Episodic Memories Generation and Evaluation Benchmark for Large Language Models
Authors: Alexis Huet
Zied Ben Houidi
Dario Rossi
Venue: ICLR 2025 Poster (2025)
Keywords: Episodic Memory Modeling
Large Language Models
Synthetic Benchmark Generation
Cue-based Retrieval
Temporal-Spatial Reasoning
Long-context Understanding
Human-inspired AI
Primary area: foundation or frontier models, including LLMs
TL;DR: A new benchmark for testing episodic memory in AI language models, using synthetic stories to evaluate recall of specific events, their contexts, and chronological relationships.

Abstract: Episodic memory -- the ability to recall specific events grounded in time and space -- is a cornerstone of human cognition, enabling not only coherent storytelling, but also planning and decision-making. Despite their remarkable capabilities, Large Language Models (LLMs) lack a robust mechanism for episodic memory: we argue that integrating episodic memory capabilities into LLM is essential for advancing AI towards human-like cognition, increasing their potential to reason consistently and ground their output in real-world episodic events, hence avoiding confabulations. To address this challenge, we introduce a comprehensive framework to model and evaluate LLM episodic memory capabilities. Drawing inspiration from cognitive science, we develop a structured approach to represent episodic events, encapsulating temporal and spatial contexts, involved entities, and detailed descriptions. We synthesize a unique episodic memory benchmark, free from contamination, and release open source code and datasets to assess LLM performance across various recall and episodic reasoning tasks. Our evaluation of state-of-the-art models, including GPT-4 and Claude variants, Llama 3.1, and o1-mini, reveals that even the most advanced LLMs struggle with episodic memory tasks, particularly when dealing with multiple related events or complex spatio-temporal relationships -- even in contexts as short as 10k-100k tokens.

Published as a conference paper at ICLR 2025
EPISODIC MEMORIES GENERATION AND EVALUATION BENCHMARK FOR LARGE LANGUAGE MODELS
Alexis Huet", Zied Ben Houidi Dario Rossi
Huawei Technologies Co., Ltd., Paris, France
(first (mmid) d)- last] } @ @hwawel com "Equal contribution; Corresponddig author and principal investigator
ABSTRACT
Episodic memory- the ability recall specific events grounded in time and space is cornersonn of human cognition, enabling not only coherent storytelling, but also planning and decision-making. Despite their remarkable capabilities, Large Language Models (LLMs) lack robust mechanism for episodic memory: we argue that integrating episodic memory capabilities into LLM is essential for advancing AI towards human-like cognition, increasing their potential to reason consistently and ground their output in real-world episodic events, hence avoiding confab- ulations. To address this challenge, to model and evaluate L.LM episodic memory capabilities. Drawing inspiration from cognitive science, we develop structured approach represent episodic events, encapsulating temporal and spatial contexts, involved entities, and detailed descriptions. We synthesize a unique episodic memory benchmark, free from con- tamination, and release open source code and datasets to assess LLM performance across various recall and episodic reasoning tasks. Our evaluation of state-of-the-eee art models, including GPT-4 and Claude variants, Llama 1. and ol o1-mini, reveals that even the most advanced L.Ms struggle with episodic memory tasks, particularly when dealing with multiple related events complex spatio-temporal relationships even in contexts as short as 10k-100k tokens.
introduce a comprehensive framework
INTRODUCTION
Episodic memory- - the ability to recall specific events grounded in time and space - is a cornerstone of human cognition. Unlike semantic memory, which stores general knowledge. episodic memory is intimately tied to time, space, and details of specific events (Tulving et al. 1972 Tulving & Thomson 1973). Both memories are declarative, activated through other facts (semantic) or a rich recollection of past events (episodic). For instance, hearing the word "birds" can call for "fly"; while hearing "France" might remind someone of their first trip abroad, bringing back details of that entire event.
specific triggers that can bring back
Episodic memory is not only vital for personal identity and coherent storytelling but also plays crucial role in planning (Pfeiffer & Foster 2013 Ólafsdóttir et al. 2015), reasoning (Dusek & Eichenbaum] 1997) and decision-making (Barron et 2013). It enables individuals to track the states of entities they care about, on both physical (Burgess et al. 2002) and virtual space (e.g. virtual Benn et al. (2015) It further enables envisioning future scenarios through mental time travel, a process known as future episodic thinking (Atance O'Neill 2001 Schacterr & Madore 2016). The critical role of is exemplified best by the hippocampus, a specialized brain region that acts as a cognitive map and spatio-temporal index Ke'Kee & Nadel 1978) Teyler & DiScennal 1986). needed for forming (and navigating through) past, present and future episodic memories (Tanaka et al. 2014 Teyler & Rudy 2007). It is within the hippocampal formation that neuroscience research has identified specialized space and time neurons: place cells, which fire when we occupy specific locations (O Keefe & Dostrovsky [177I, but also express current, past and future locations (Moser et al., 2015); grid cells, which create a coordinate system for spatial navigation (Hafting et al. 2005); and time cells, which segment events into distinct temporal sequences (MacDonald al. 2011).
reality (Cushman et al. 2008), digital Belders/Benn
processing in episodic memory

---

Published as a conference paper at ICLR 2025
The above findings highlight the fundamental importance of spatial and temporal context in episodic memory processes, and support the hippocampal indexing theory (Teyler & DiScennal [986 Teyler & 2007), according to which the hippocampus stores compressed representations of ncocortical activity patterns, serving as an index to reactivate these patterns during memory recall. Additionally, research on source and reality monitoring (Johnson & Raye| T98T; Johnson al.) [993] Garrison shows that memories of actually perceived events typically contain richer spatial and temporal contextual information as opposed to memories of imagined events, making it easier to discern their origins. Thus, episodic grounding in time and space is what allows us to distinguish between what is real and what is imaginary: everything that happened in our collective agreed-upon upon space and time is real, and everything else is not. It is this ability to ground memories in specific spatio-temporal contexts (e.g. attributing the source a piece of knowledge came from) that gives episodic memory its power and reliability.
itt
Yet, despite its fundamental significance for human cognition, episodic memory remains underex- plored within Large Language Models (LLMs) research. Despite remarkable LLM capabilities, two significant drawbacks related to lack of episodic memory limit their full potential. First, LLMs tend to hallucinate, i.e., confabulate information that is coherent yet factually incorrect, i.e. not grounded in episodic reality. Second, while humans engage in many step-by by--byt-e trials and errors, continuously updating their episodic memory and knowledge, LLMS constrained by their inability to retain information beyond their context window. This transient nature of LLMS' memory contrasts sharply with human episodic memory, which allows for the long-term storage and retrieval of detailed experiences across an entire project or even lifespan. We argue that integration of episodic memory into LLMs held potential to significantly enhance their reasoning, consistency, and factual accuracy.
In this paper, as a first step to assess this gap, we propose a comprehensive framework to model and evaluate episodic memory capabilities LLMs. To the best of our knowledge, this aspect has not been previously evaluated: although more and more challenging tasks for long context comprehension are emerging, existing approaches to extending the memory capabilities of L.LMs (such as vetrieval-augmente generation, or various in-context extension methods), are still tested against relatively simple benchmarks (see Sec. D.. Diverging from such benchmarks, we specifically target episodic memory, with events rich of contextual information, and involving specific entities happens at specific time and space locations. We synthesize a unique episodic memory benchmark, free from contamination, that ensures coherence and control over the generated narratives and their corresponding ground truth answers. This benchmark not only evaluates how well current LLMs handle episodic tasks, but also lays the groundwork for future research aimed incorporating dynamic, context-sensitive episodic memory into Al systems. Our contributions include:
Modeling episodic memory. Drawing inspiration from cognitive science, develop a structured approach to model episodic events within LLMs, encapsulating temporal and spatial contexts, involved entities, and detailed event descriptions.
Benchmark code and dataset. We introduce a framework for generating synthetic episodic memory datasets" dataset="comprrisinn comprising narratives of events and corresponding question-ansye answer pairs- that could also be used to generate synthetic tasks for training purposes. We further release 11 dataset differing in size and diversity, to evaluate I.L. performance across various episodic memory tasks.
Assessing LLM performance. We evaluate various LLMs (GPT-4, Claude, Llama 3.1, and o1-mini), assessing their performance under different configurations: in-context learning, retrieval-augmennt generation and fine-tuning, We demonstrate that even the most advanced LLMs struggle with episodic memory tasks, particularly when dealing with multiple related events or complex spatio-temporal relationships, even for very small context size confirming the need for episodic memory benchmarks like the one we propose to further improve LLM performance. We further showcase the inadequacy of existing fine-tune tne-ttnnng strategies for embedding episodic knowledge into LLMs, demonstrating the need for novel training methodologies tailored to episodic memory integration.
2 RELATED WORK
For brevity, we summarize here and refer the reader to Appendix A for a broader overview.
'oode and data available at Huet et al. 2025
List available in Tah. 28
2

---

Published as a conference paper at ICLR 2025
Assessing memory in human subjects. Existing tests of episodic memory in humans often involve asking the person to recall specific events in their life. The Autobiographical Memory Interview (AMI) (Kopelman| 1994) and the Autobiographical Interview (Levine et al., (0022 are widely used tests that assess recall of specific past events, which are then scored based on the level of detail and accuracy. These tests employ structured interviews that classify memories across dimensions including time periods, locations, and specific event details, which directly informs our approach of using cues related time, space, entities, and event contents. Other episodic memory tests (Wilson et al. 1985: Delis et al. 2000) can detect impairments at various levels such as encoding, retrieval, and recognition of familiarity - which we take into account in our benchmark definition.
Assessing memory in LLMs. Existing benchmarks primarily focus on simple retrieval or reasoning over long contexts, with limited assessment of episodic memory capabilities (Vodrahalli et al. 2024).
Retrievallorientedd benchmarks follow a sreedle-in-a-haystaee paradigm (Kamradt 2023). requiring models to find specific pieces of information within extensive (and highly irrelevant) contexts, failing to evaluate the model's 's understanding of temporal sequences state changes. Extensions to multiple needles (Reid et al., 2024, Hsiehh et al., 22224] Li et al.. 2024. Zhang et al. 2024) still lack cue differentiation, and do not incorporate temporal nor spatial awareness. Similarly, question answering benchmarks on long contexts (Bohnet et al., 2024 Zhang et al. 2024) do not probe model ability to track entity states or temporal relationships. Other synthetic datasets for reasoning tasks such the bAbI (Weston et al. 2015) (and its long-context extension bAblL.ong (Kuratov et al. 2024 or [i] et al. 2024), often involve highly artificial scenarios lacking complexity and realism opening the door to shortcut reasoning by exploiting dataset biases or patterns.
Other spatio-temporal benchmarks have been created for temporal question answering (Jia al. 2018 Saxena et al., [222]] Dhingra et al. 2022 Chen et al. 2021 Kasai et al. [022]] Tan et al. 2023). Compared our work, the point of view adopted is information retrieval instead of episodic memory analysis, hence the main differences are: (i) the cue refers to a single fact, instead of a trace of events;
(ii) there is no systematic approach in the different dimensions of time, space, entity or content; (iii)
they are based on limited existing data sources that are already public, usually the Freebase (Bollacker jet al.][[008) (2008) or the Wikidata knowledge bases; (iv) the output answer is always a single closed answer, while we allow the retrieval of zero more than one free-form answers. Recently, Vodraballi et al. (2024) introduced the Latent Structure Queries (LSQ) framework as first good step to address the limitations of current reasoning over long context benchmarks. Close to our design philosophy, it works by embedding a hidden structure within a large context, which the model must understand to answer queries: for example, an object list is modified throughout a long text, and the model must track these changes to determine the list final state, which probes I.LM ability to track state across the entire context, going beyond simple retrieval. Our work is similar in spirit but broader in scope, systematically incorporating range of key elements crucial to episodic memory.
Limits of existing benchmarks. Summarizing, we identify several limits in current benchmarks: (1) existence of shortculs; (2) prioritization of simple retrieval over reasoning; (3) data leakage, where training data is contaminated with test data; (4) out of distribution distractors where needles are hidden in a completely different context, biasing the retrieval task; (5) labor-intensive creation effort, which makes it difficult to scale adapt, or incrementally improve benchmarks.
Need for an episodic memory benchmark. The above limitations hinder the evaluation of LLMs episodic memory capabilities. We hence propose a benchmark that adheres to the following guidelines: (1) explicitly incorporate temporal and spatial contexts; (2) force models to track entity states; (3) use varied retrieval cues, based on different combinations of event attributes, mirroring human cue-based recall; (4) avoid data leukage, ensuring that the evaluation is free from contamination; and finally (5) balance complexity and realism, providing tasks that are both challenging and representative of real-world tasks, without incurring excessive human labor to retain flexibility.
3 MODELING EPISODIC MEMORY FOR LANGUAGE MODELS
Inspired by human episodic memory Tulving et al. 1972), we model episodic memory for LLMs using two key components, namely entities and events. Our goals are to (i) create appropriate episodic events, (ii) design systematic memory tasks, and (iii) develop methods to evaluate the ability of LLMs to recall these events accurately. Similarly human memory tests, we construct tasks and scenarios
3

---

Published as a conference paper at ICLR 2025
for LLMs to assess their ability to (i) recall specific episodes with details, (ii) understand entity states, (iii) comprehend temporal and spatial contexts, and (iv) avoid confabulation.
3.1 ENTITIES, EPISODIC EVENTS AND WORLD MODELING
Entities. Entities (enty) are fundamental subjects in the world, that can participate in or be affected by events: their attributes or relationships may change over time due episodic events. More generally, each entity has an associated state statej, j,t at any discrete time t. This state evolves based on events and includes all details about the entity at a given time, such as its location and any observation such as actions interactions. The latest state of an entity is denoted as statej,L, where L denotes the latest point in time.
Episodic events. Episodic events (event,) transcend the scope of the current benchmark, and are defined as actions or observations that lead to changes the state of the world or its entities, including the mere progression of time. Specifically, each event event, is characterized by tuple (4i, 8i, ent,, Cj), where 4 represents the time at which the event occurs, 8; the location where the event takes place, ent; the set of entities involved in the event, and finally c, corresponds to the event content, detailing what happened. This formulation captures the essential elements of episodic memory: what happened, where and when.
World model. This model of episodic events and entities aligns with the hippocampus's 's role in maintaining a dynamic cognitive map of the world. By tracking the states of entities across time and space, our framework mirrors the brain's 's process of continuously updating its representation of the environment based on new experiences. This approach not only captures the context, space and temporal properties of events, but also how these events transform our understanding of the world and its constituents. In line with the encoding specificity principle (Tulving & Thomson [973), each event is associated with specific details that differentiate it from others: parts of these details will serve as retrieval cues during memory tasks. Now in LLMs, these episodic memory components are represented sentences or paragraphs describing the events, and entity states inferred from the context provided in the text.
3.2 SYSTEMATIC TASK DESIGN: CUE- BASED RECALL AND RETRIEVAL
We model cue-based recall key-value retrieval system, where the cue (key) is any combination of elements from the event tuple (1,, 8s, ent,, G),, and the associated event details serve as the memory trace (value). By systematically varying the cues, we can assess the model's 's ability retrieve specific information based on different aspects of the events.
cue (*,,, *,*)
trace Entities
memory
events
[recalled list]
Figure 1:
Memory recall process implemented the benchmark: a set of events match a given cue, from which a list of elements are recalled (here, all entities that have been seen in given space).
Cue composition and retrieval types. To create comprehensive set of tasks, we consider all possible combinations of the event tuple elements as cues. For instance, a cue of (t, *, *,, *) would prompt the retrieval of events that occurred a specific time t, while cue of (*, 8, *, *), as shown in Fig. D would query for events that took place at particular location s. Tab. presents example combinations of different cue compositions, the descriptions of the tasks, the types of information to be retrieved, showing few templated questions from our actual implementation for each case (the full list deferred to Tab. TO in the Appendix; example question/answer pairs from the templates are shown in Appendix C).
Encoding specificity principle and cue overload. According to the encoding specificity princi- ple (Tulving & Thomson, 1973), the effectiveness of retrieval cue depends on its similarity to the original encoding context. Specific cues that closely match the encoded event lead to precise retrieval, while broader cues may result in interference with other events sharing similar features, phenomenon known as cue overload. By varying the specificity of the cues as exemplified in Tab, [ test the model's ability handle both precise and ambiguous queries. For example, highly
4

---

Published as a conference paper at ICLR 2025
Table 1:
Episodic memory questions based on cue composition and retrieval types (full list in Tab. 10
Cue
Description
Retrieved trace (id)
Template question (corresponding to
Spaces (0)
Consider all events that happened on [t). Provide
+)
Events specific time
Entities
list of all protagonists involved in any of these events,
Contents (2)
without describing the events themselves.
Events involving
Times (18)
Reflect on fent]'s experiences at [s]. Describe all
8, ent,
entities specific location
Contents (19)
the key events they's ve been involved in this location. focusing on what happened ruther than when occurred.
Events with specific location, entities, and content
Consider all events involving both (ent) [c) (8).
(, ent,
Times (27)
Provide list of all dates when these events occurred.
without describing the events.
Provide comprehensive account of what happened
Events with specific time, location, entities, and content
involving {ent) and (e) at [s] on (t). Include all relevant
(t. s. ent,
Full event details (29)
details about the event(s), including what occurred and any other pertinent information.
Retrieves the most
Times [latest] (30)
What is the most recent location where [ent] was observed in the story's 's chronological timeline?
(*, *. ent, *)
recent known location of entity
Spaces [latest] (31)
Contents [latest]
Retrieves a chronological list of
Times [chrono.] (33)
Provide a chronological list of all dates when (ent] was observed, from curliest to latest in the story's timeline.
(*, *. ent, *)
Spaces [chrono.] (34)
dates when an entity was observed
Contents [chrono.] (35)
specific cue like (t, 8, end, c) should retrieve unique event with detailed information, while broader cue like 8,, *)) may retrieve multiple events, requiring the model to list all relevant information without confusion. Note that several retrieval types (different spaces, entities, details or dates) can match given cue: assess the models' ability to retrieve them separately.
Assessing entity state and chronological tracking capabilities. Beyond recalling events and their attributes, we design tasks that require tracking the states of entities over time: the last two rows in Tab. [ tasks focused on entity state tracking and chronological analysis, assessing the model's ability understand the temporal progression and current state of entities.
Assessing confabulation. Finally, we evaluate confabulation including tasks that assess unfamiliarity awareness. This involves testing the model's ability recognize when it lacks information about certain events or entities and to respond appropriately, such as indicating uncertainty or acknowledging the absence of relevant information fail doing so by hallucinating an answer.
4 BENCHMARK DESIGN
In this section, we detail the design of our episodic memory benchmark, adhering to the requirements outlined Sec. 3 The benchmark comprises three key components: (i) the memories encode, represented by evidence documents, (ii) set of question- answer pairs designed to probe episodic memory, and (iii) an evaluation strategy to assess model performance.
We introduce novel methodology that generates synthetic documents using Large Language Model (LLM), structured as a coherent narrative akin to a real book. Each chapter presents logical flow and progression of story while while maintaining controlled ground truth information, which is strategically distributed over several paragraphs.
This approach distinguishes benchmark from previous designs, such as the bAbl tasks (Weston et al. 2015), which use a world model but lack coherent storytelling, and from real books, where ground truth information cannot be controlled and data contamination is concern. The questions in our benchmark are designed to follow the episodic memory recall process highlighted in Fig. [ each question based on a cue that triggers the retrieval of relevant events, from which specific information is extracted.
4.1 BUILDING THE MEMORIES TO ENCODE
We begin by constructing static universe comprising finite Nuniverse 100 set of dates (t). locations (s), entities (ent), and event contents (c). These elements are carefully curated to ensure diversity and uniqueness. From this universe, we sample Nevents synthetic events, each serving the foundation for
5

---

Published as a conference paper at ICLR 2025
single chapter in our book. Each event is characterized by specific date, location, action, and entity, ensuring a unity of time and place, as well as a main content and entity. Fig. Fig.2]schi schematizes the book creation process. Full details, along with an example of this process, are provided in Appendix B
Universe creation
Event generation Event 1... Nevents
Chapter generation and verification
Universe (N
100)
Chapter .... Nevous
Book
At Bitliggg Bluck Courss, Ediarrda demotstrated cat leape f..]
[dates}
Septemher 13, 2025
[locations]
Bethpage Black Course
{entities)
Exra Edwards
Learal (-.) something had skiiited duriig the night of Septetuber
(conteut)
cat leaps in a parkour
Event generation: sampling Nevents independent events, with each item of each event following truncated geometric distribution, ensuring, e.g., that the item Ezra Edwards (e1) matches many events: Chapter Novents
Chapter 1
Chapter 2
Chapter 3
(1,,))
(1,
(t2, 82, 21) c1)
(tx, SN, C1,cx)
Figure 2: Book generation: skewed event sampling (Appendix B.1.3). LLM-based chapter generation with quality control (Appendix 1.1.6 B.1.7). and chapter concatenation.
To evaluate model performance across various scenarios, we vary the number of matching events for a given cue from zero to more than six (details and quantitative analysis in Appendix [B.133) To achieve this variety, we employ geometric sampling strategy when selecting events from the universe. Specifically, we use a truncated geometric distribution to sample dates, locations, entities, and event contents. This approach ensures that some items appear multiple times across different chapters, while others appear only once. By controlling the frequency of each item, we create cues that correspond to varying numbers of events: this allows us assess the models' ability to handle both rare and frequent occurrences, as well as their capacity to manage and retrieve multiple related events. To better illustrate this shared universe structure, we provide in Appendix C an example representing the tracking of a single entity for one of the generated document.
To maintain control over the placement of ground truth information within each chapter, associate each event with an event meta- meta-data, which specifies the number of paragraphs in the chapter and the positions of key information within those paragraphs. For example, the date might be required to appear in paragraph three and only in that paragraph. Notably, dates, locations, entities, and event contents may be reused across different chapters.
Each chapter is generated independently using an LLM, guided by the event and event meta-data information. During generation, the LLM may introduce additional entities interacting with the main entity; these new entities are assigned unique names to ensure consistency. Following generation, chapters are concatenated into single document, resulting in synthetic book that contains the memory to be encoded. To ensure coherence and validity of the generated document, we implement several verification steps:
Uniqueness constraints: we prevent multiple chapters from sharing identical time-space or time- entity pairs to avoid conflicts in the narrative timeline.
Adherence to event meta-data: direct parsing verifies that each generated chapter adheres to the meta-data's event requirements, including the placement of key elements in designated paragraphs.
Additional quality-control layers: we check the presence verbatim of the date, location, entity and content detail, while an L.LM-hased verification using boolean questions validates the chapter's unique temporal day, geographical focus, main character and main event (details in Appendix B. B.I.7
We acknowledge that, following our generic episodic event definition (Sec. 3). the resulting book may contain more episodic events than those explicitly included in our ground truth. However, this does not influence our ground truth answers and questions, as we focus on the events over which we have full control. Additional statistics regarding the distribution of information within the book, each chapter, and each paragraph, including comparisons between GPT-4o and Claude, are provided in Appendix B.1.12 An example of generated chapter is also included in the appendix for reference.

---

Published as a conference paper at ICLR 2025
Table 2: Characteristic of the main benchmarks (both produced with Claude 3.5 Sonnet 2024-06-20).
Additional benchmarks are available in Appendix G
Parameter
Short book
Long book
Nevents (Chapters)
20
200 R 100k (102870) 37, 35, 34, 34 December 26, 2026
Nb. of tokens
R 10k (10397) 14, 12, 13, 12
Nb, of dates, locations, entities, contents
Start end dates
March 23, 2024
Nb, of selected QA pairs
456 180, 180, 72, 24, 0
686 180, 180, 108, 128, 90
Nb. of QA related 0, 1, 2, 3-5, 6+ events
4.2 BUILDING THE QUESTION-ANSWER PAIRS
We implement a template- camplate-based approach to create questions aligned with our episodic memory tasks. Each question is defined by cue composition (the trigger key identifying set of events), a trace (the type of information to be retrieved from each remembered event), and the retrieval mode (whether all elements are needed, only the latest state, or their chronological order), as exemplified in Fig. and detailed fully in Appendix [.2
We consider all combinations of cues and traces for the retrieval of all elements, focusing on entity cues for the latest and chronological retrievals. Examples of question templates are provided in Tab. (full list available in Tab. in the Appendix). The templates are populated using the dates, locations, entities, and event contents appearing in each chapter. All questions are associated with known ground truth answers, as the event details are controlled and known (examples of question/answer pairs are shown in Appendix [C). To test for hallucinations, empty answers, using entities or combinations of items that do not exist in the document. This allows us assess the model's ability to handle unfamiliarity and avoid confabulation.
include additional questions with
To ensure variety in the number of events to be recalled, we filter the corpus of questions to balance the number of queries that correspond to zero, one, two, three five, and more than five events. More details on this process are provided in Appendix B.2.4
4.3 EVALUATION STRATEGY
Our evaluation strategy employs L.MM-as-a-judge approach assess the correctness of the model's answers. The evaluator I.IM is prompted to perform two key tasks: First, it (i) identifies relevant items by extracting them as list from the Al-generated answer for each question, allowing us to evaluate the number of predicted items. Second, it (ii) scores the relevance of these predicted answers against each ground truth item, with the sum of the scores interpreted as the number of true positives. We defer details to Appendix [B.3] but in essence, we are using the LLM for simple semantic comparisons, and not as a judge making subjective assessments. Then, from these predicted and ground truth answers, we compute optimistic F1-score bound (our primary comparison metric) using a lenient methodology detailed in Appendix B.3.2 For chronological questions, we additionally use Kendall's 7 coefficient: this is applied only to answers that fully match the ground truth, allowing us to assess the correctness the ordering within this matching set.
4.4 GENERATED BENCHMARK
Using the proposed world modeling framework, we generate two synthetic documents, referred to as the short book and the long book, summarized in Tab. 2 The long book includes 196 unique events across 37 dates, 35 locations, 34 entities and 34 event contents. We generate total of 686 questions, balanced across cue compositions and retrieval types, to evaluate the model's 's episodic memory capabilities; the complexity of the questions is also controlled by varying the number of related events. Note that we explicitly limit the size the book, since as we shall see, current state of the art LLMs start struggling with relatively modest size")
TTo demonstrate scalability, generate an additional 2000-chapter book (IM+ tokens), available at Huet 2025). We also produce and evaluate more universes and benchmarks in Appendix G
7

---

Published as a conference paper at ICLR 2025
F1-score (rank)
4
6
8
gpt-40 4.7660
2601
o1-mini
cl-3.5-sonnet (rag) 4.8601
7:3069
gpt-4o-mini (ftuning)
gpt-4o (rag) $,2194 )
-l-3.5-sonnet
gpt-4o-mini (rag)s 52242/
6.4344
gpt-4o-mini
cl-3-haiku (rag) $,228/
6.3163
cl-3-haiku
llama-3.1 6,2019
Figure 3: Overall performance comparison: Critical distance plot ranking all LLM models and memory combinations (instances not tied by an horizontal bar are statistically different).
BASELINE RESULTS
5.1 BASELINE MODELS AND MEMORY STRATEGIES
As baselinn models for our benchmark, we evaluate several LLMs, including GPT-4o, GPT-4o-mini. Claude 3 Haiku, Claude 3.5 Sonnet, Llama 3.1 405B (instruct) and the recent ol-mini. We consider memory in LLMs as functioning through three primary forms: (1) In- Contett Memory, where information is processed within the model's context window; (2) Retrieval-Augmente Generation (RAG), where external memory is accessed through a vector database; and (3) Parametric Memory via Fime-Tuning, where memory is stored within the model's parameters.
For (1) in-context memory, we prepend the full document to the question, allowing the model to process the entire context. With (2) RAG, we chunk the book into paragraphs") each labeled with context ge.g.. "Chapter X, Paragraph Y"),, and embed them using cext-embedding---ao 3-mmalll For each question, we retrieve the top-K paragraphs based cosine similarity to the question's embedding and prepend them to the question as context. Lastly, (3) we fine-tune models using all single-ever question-answe pairs as training data (details in Appendix [B2.5). In principle, this approach enables models to acquire knowledge for all benchmark questions (including multi-event questions).
5.2 EXPERIMENTAL RESULTS
We evaluate the performance of various models and memory strategies on our benchmark. We mostly report results the long book and defer an extended set of results to Appendix
Overall performance comparison. Using the F1-score as our primary metric, we compare models and strategies across all questions. Fig. 3 shows the Critical Difference (CD) plot based on the Wilcoxon signed-rank test (Benavoli et al. 2016) between each pair of algorithm (adjusted by Holm's 's method) that allows to rank models according to their average performance on the long book in principled manner. GPT-4o with in-context memory and Claude 3.5 Sonner with RAG memory achieve the highest average ranks, with no statistically significant difference between them. Notably, except for GPT-4o, models utilizing RAG generally outperform their in-context counterparts, suggest- ing that retrieval methods can enhance episodic memory capabilities by effectively narrowing down the relevant context for each query. As information spans several paragraphs, retrieval granularity (i.c., paragraph vs chapter) may play an important role (see ablation study in Appendix E.2
Performance recall tasks. We next test the ability of recalling episodic memory, by reporting in Tab. 3 the average Fl-scores for simple recall questions as a function of the number of events that match the cue (extended analysis in Appendix E.I
Avoiding confabulation. The 150 questions with matching events are intentionally designed to test familiarity awareness. We see that no model achieves a perfect F1-score in avoiding hallucinations:
'Sice single event spans across multiple paragraphs, this RAG strategy limits LLM ability to retrieve all relevant information: Appendix E.2 reports comparison with chapter-based RAG as an ideal upper bound. Fine-tuning using the OpenAl API over 30 epochs, a hatch size of and a learning rate multiplier of 1.8. NNyteworthy, Claude 3.5 Sonnet with in-context memory is knowingly affected by bug making it verbosely debate finetuning instructions adbertram adbeetta 2003), cfr. Appendix F
8

---

Published as a conference paper at ICLR 2025
Table Performance on recall tasks: average and standard deviation Fl-score as function of the number of events matching a given cue (long book).
Number of events matching the cues
Memory
Model
(150)
1 (150)
2 (90)
3-5 (98)
6+ (60)
gpt-4o-mini
0.51
0.54#0.46
0.44#0.36
0.47+0.27
0,50+0,1 17
gpt-4o
0.84#0.37
0.81
0.60#0.31
0.57#0.21
0.53#0.14
In-context
claude-3-haiku
0.8440.37
0.39=0.48
0.37=0.30
0.37#0.28
0.38+0.19
claude-3-5-sonnet
0.92+0.27
0.35#0.48
0.35#0.33
0.32+0.25
0.41+0.20
ol-mini
0.97
0.05#0. 19
0.12+0.24
0. 1.
0.24+0.19
1ama-3.1-4055
0.80+0.40
0.49=0.47
0.38+0.33
0.40+0.25
0.45+0.22
gpt-4o-mini
0.63#0.49
0.60#0.46
0.60#0.34
0.5940.26
0.62:0.22
gpt-4o
0.82+0.39
0.60=0.46
0.55+0.33
0.55+0.28
0.59+0.21
RAG
claude-3-haiku
0.71=0.45
0.57=0.47
0.59+0.33
0.58+0.26
0.59+0.25
claude-3-S-sonnet
0.91#0.28
0.59=0.47
0.59#0.35
0.59+0.27
0.62+0.25
Fine-tuning
gpt-4o-mini
0.00#0.00
0.83+0.35
0.37+0.32
0.28+0.21
0. 0.1940.07
ol-mini performs the best, RAG only minimally helps, smaller models struggle avoid confabulation, and the fine-tuned model consistently fails, highlighting a critical limitation of naive fine-tuning
Impact of cue specificity and cue overload. Considering questions with a single existing ground truth trace in Tab. 3] we observe different scenario: fine-tuned GPT-4o-mini leads (F1 of 0.83) by overfitting on this type of question, with GPT-4o a close second (0.81). However, as the number of matching events increases, the cue is less specific and becomes overloaded: i.e., for questions with two or more events matching the cue, we observe a consistent decline in performance for all models (FI <0.60). For the fine-tuned model, the decrease more pronounced (F1 <0.37): this underscores the inability of naive finetuning to generalize beyond single-event memorization (i.c. models memorize specific answers, without developing deeper understanding of the information).
Impact of context size. Similar considerations hold for the small book (deferred to Appendix EE.I). where in spite of very limited context (10k tokens). performance are better but still suboptimal. Noteworthy, model rank differs, with o1-mini and GPT-4o statistically equivalent.
Impact of cue type. We analyze the impact of cue type in Fig. across all models (full details for GPT-4o are reported in Tab. Tō]in in the Appendix). For different models and number of events in the ground truth, the picture reports the Fl score for different types cues specifically, from top to bottom context (x, c), entity (*, +, ent, *). space (+, 8, *, *), and time (t, (, +, *). Across models, gradient is clearly visible from left right (performance degrades for increasing number of events, as already seen in Tab. B and from top to bottom (performance degrades, from context, to space, to time). This interesting finding reinforces the need of benchmarks such as the one we propose.
Tob.[6]jnn
1.0
1.0
0.9
8.
1.0
1.0
1.0
0.9
0.9
0.5
0.9
0.9
0.7
0.5
0.7
0.9
0.8
CUP
on.
1.0 0.8
0.2
6.7
0.7
0.
0.9
0.5
0.0
0.0
0.9
1.0
0.7
0.5
0.7
0.6
0.5
6.2
0.0
04
0.3
0.6
3.0
1.0
10
0.7
0.
0.7
6.9
0.5
1.0
0.0
0.9
64
1.
0.8
0.5 0.5 0.8
0.505
0.5.0 0.3 0.4 0.4 0.0
0.0
0.8
1.0
1.0
0.
0.9
0.9
10
1.0
0.3
0.3
0.0
0.9
0.7
0.2 0.3 2.33 0.4 3 0.6 0.2 0.3.0.3..3
0.2
0.0
0p
013 5 sonmic
(19))
(ng)
Iame
di3 Sesomen.
get 4e mnl (hunigg)
cLimni-
1000
21
get 40 mils
(oo)
Ilama-31
got sot-fo-mira
(hunne)
asmiis
1.4e
(oug)
lummed 31
get 40 mini
d13.5 sonrett
min)
(hhiing)
ckomin
d/3.5
(a) matching events
(b) matching event
(c) matching events
Figure 4: Impact of cue type. F1-score across different cue types (y-axis) for models ordered
according to their overall rank Fig. (x-axis) and for increasing number of events that match the cue (from left to right sub-plots).
Latest state recall and chronological ordering. We finally study model ability to harness state recall and chronological ordering, finer grain. As we early have shown, (some) models perform almost

---

Published as a conference paper at ICLR 2025
perfectly when entities participate in zero event, but (most) models struggle significantly with multiple events. In reason of LLM performance degradation for questions involving a larger number of events, expect these questions to be particularly challenging and, in particular, we expect chronological ordering to be significantly more involved than latest state recall. Tab. considers all queries involving more than one event, and shows the (i) fraction of exactly matched latest states, (ii) the fraction of exact matches between the predicted set and the whole ground truth set (possibly in different order) and (iii) the Kendall's 7 computed across the ground truth and the matched states for those exact matches. Results confirm the expectations, and highlight significant challenges in tracking entity states over time (336% for any model) and, especially, to recall all the events (Σ 18%). The proportion of exact matches in chronological ordering tasks is low across all models, and even when models retrieve correct events, they often fail to order them correctly (low Kendall's coefficients). These findings reveal significant gaps in the models' abilities to understand and process temporal relationships between events, motivating the need for future solutions to address this gap.
Table 4: Latest state recall and chronological ordering. Considering all questions involving at least two events (i.e., with ground truth answer of length 2 2). we report the fraction of answers that match the latest state, all the states and the Kendall's T coefficient (computed between the ground truth and the matched predictions subset). Models are ranked according by decreasing chronological ability.
Family
Claude
GPT
Claude
GPT
GPT
GPT
Claude
Claude
Ilama
ol
GPT
Model
3-haiku
4o-mini
3.5-son.
40
40 context
4o-mini context
3-haiku context
3.5-son.
3.1-405b
mini context
4o-mini
Memory
rag
rag
rag
rag
context
context
ftuning
Latest
23%
36%
31%
23%
36%
13%
19%
18%
26%
5%
23%
All
18%
13%
13%
10%
10%
8%
5%
3%
0%
0%
0%
Kendall
0.43
0.93
0.60
0.50
0.50
0.33
1.00
1.00
Other ablation studies. We defer to the Appendix E detailed analyses of performance with respect to: book size (E.1). RAG granularity (E.2). number (E.3) and type (E.4) of cues and traces, book
generation process (E.5). chapter chronological ordering (E.6). and event plausibility (E.7)
6 SUMMARY AND LIMITATIONS
In this work, we draw inspiration from cognitive science to build new episodic memory model for LLMs, generate a comprehensive benchmark, and evaluate state-of-the-ar models. Our findings reveal significant gaps in the episodic memory capabilities of state-of-the-an LLMs, particularly when handling multiple related events and complex spatio-temporal relationships. These challenges reflect aspects of human memory where more distinctive cues facilitate easier retrieval. Furthermore, naive finetuning fails to achieve deep understanding of episodic events and their intricate relations, merely overfitting to single learned facts. These challenges highlight the need for fundamentally new approaches to model design and training that more closely emulate the dynamic and contextual nature of the human episodic memory. The proposed episodic memory benchmark exhibits several desirable properties: it is contamination-fin ffree by design, scalable with low human labor, offers unambiguous cues and ground truth, and the ability to model multiple cues and events within synthetic yet realistic narrative. However, we acknowledge limitations that open avenues for future research.
Temporal representation. Our benchmark relies on explicit temporal markers, which may not fully capture the nuanced ways time is expressed in natural language (e.g. "yesterday"" "last week", or "after the party"). Future iterations should incorporate implicit and relative temporal references to further challenge the models.
Event independence. The independent generation of chapters, while facilitating control, does not capture the interconnected and causal nature of real-world events.
Limited domain scope. Our benchmark primarily involves human-like protagonists within fictional contexts. Extending the framework to include diverse domains (e.g., software projects, virtual environments) would test models' ability to generalize episodic memory capabilities.
Training limitations. The observed performance limitations suggest that current fine-tuning method- ologics may not be optimally suited for episodic memory tasks, underscoring the need for developing new strategies for the broad scientific community.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
adbertram. Why does claude sometimes follow up his response with "human:" " and then some response i presume he's thinking i will respond with? https://www. www.it..dom..o.oooc. com/r/ htt.e/////oommeetttttttt htti:/hhyddeeee doee cewaladeeseeeeeeeeeeeeeeeeeeeeee ltrpettiims___oooooooo follow
his _es_reeponsee/ 2023.
Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Stephanic Chan, Ankesh Anand, Zaheer Abbas, Azade Nova, John D Co-Reyes, Eric Chu, et al. Many-shot in-context learning. arXiv preprint ar arXiv:2404.11011 2024.
Cristina M Atance and Daniela O'Neill. Episodic future thinking. Trends in cognitive sciences, (12):533-533) 2001.
Helen C Barron, Raymond J Dolan, and Timothy EJ Behrens. Online evaluation of novel choices by simultaneous representation of multiple memories. Nature neuroscience, 16(10):1492-1444 2013.
Iz Beltagy, Matthew Peters, and Arman Cohan. Longformer: The long long-document transformer. arXiv preprint 0005000000.005550 2020. https::/acxi..orr////e pdff
pdf
Alessio Benavoli, Giorgio Corani, and Francesca Mangili. Should we really use post-hoc tests based mean-ranks? The Journal of Machine Learning Research, ((1):152-161, 2016.
Yael Benn, Ofer Bergman, Liv Glazer, Paris Arent, lain D Wilkinson, Rosemary Varley, and Steve Whittaker. Navigating through digital folders uses the same brain structures real world navigation. Scientific reports, 5(1):14719, 2015.
Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raf. Emergent and predictable memorization in large language models. arXiv preprint arXiv:2304.114 11..00..111...... 2023.
abs/ 2304. 11158
Bernd Bohnet, Kevin Swersky, Rosanne Liu, Pranjal Awasthi, Azade Nova, Javier Snaider, Hanie Sedghi, Aaron Parisi, Michael Collins, Angeliki Lazaridou, et al. Long-span question-answe answerinn:: Automatic question generation and qa-system ranking via side-by-side evaluation. arXiv preprint arXiv:2406.00 17, 2024.
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase: collabora- tively created graph database for structuring human knowledge. Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pp- 1247-1250, 2008.
Neil Burgess, Fleanor A Maguire, and John O' Keefe. The human hippocampus and spatial and episodic memory, Neuron, 35(4):625-641, 2002.
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jemej Kos, and Dawn Song. The secret sharer: Evalu- ating and testing unintended memorization in neural networks. USENIX Security Symposium, volume 267, 2019.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, al. Extracting training data from large language models. 30th USENIX Security Symposium (USENIX Security 21), pp- 2633-2650, 2021. https: htapp:://axivvv wtt.://.wrr//bbb//22..0000
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646, 2022. https: http://axxiv.orroo abs http://abb//222.222e
220/ 07646
Wenhu Chen, Xinyi Wang, and William Yang Wang. A dataset for answering time-sensitive questions. arXiv preprint arXiv:2108.063 2021.
Rewon Child, Scott Gray, Alec Radford, and llya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10510 2019. https: //ta:////.v....// wwt//bs///a./ooooooo..ooeeeeeeee /1904
90509
11

---

Published as a conference paper at ICLR 2025
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009.1479, 2020. https://arae org//bs/gooo) 14794
Laura A Cushman, Karen Stein, and Charles J Duffy. Detecting navigational deficits in cognitíve aging and alzheimer disease using virtual reality. Neurology, 71(12))888-58 2008.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. arXiv preprint arXiv:2104.08696, 2021. attps://arxiv.ort abs/ / 2104.1100 08696
Payel Das, Subhajit Chaudhury, Elliot Nelson, Igor Melnyk, Sarath Swaminathan, Sihui Dai, Aurélie Lozano, Georgios Kollias, Vijil Chenthamarakshan, Soham Dan, et al. Larimar: Large language models with episodic memory control. arXiv preprint arXiv:2403. 1901, 2024.
Felipe De Brigard, Sharda Umanath, and Muireann Irish. Rethinking the distinction between episodic and semantic memory: Insights from the past, present, and future. Memory & Cognition. 50(3): 459-463, 2022.
Dean C Delis, Joel H Kramer, Edith Kaplan, and Beth A Ober. California verbal learning test-. Assessment, 2000.
Janez Demšar. Statistical comparisons of classifiers over multiple data sets. The Journal of Machine learning research, 7:1-30, 2006.
Bhuwan Dhingra, Jeremy Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W Cohen. Time-aware language models temporal knowledge bases. Transactions of the Association for Computational Linguistics, 10:257-273, 2022.
Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, and Furu Wei. Longnet: Scaling transformers to 1,000,000,000 tokens. arXiv preprint arXiv:2307.02488, 2023.
pittps:/arrin ort/ 2307. 02486.pd2
Jeffery A Dusek and Howard Eichenbaum. The hippocampus and memory for orderly stimulus relations. Proceedings of the National Academy of Sciences, 94(13)7109-7114, 1997.
Leon Festinger. theory of cognitive dissonance, volume 2. Stanford university press, 1962.
Zafeirios Fountas, Martin A Benfeghoul, Adnan Oomerjee, Fenia Christopoulou, Gerasimos Lam- pouras, Haitham Bou-Ammar, and Jun Wang. Human-like episodic memory for infinite context 1lms. arXiv preprint arXiv:2407.09450 2024.
Jane R Garrison, Rebecca Bond, Emma Gibbard, Marcia K Johnson, and Jon S Simons. Monitoring what is real: The effects of modality and action on accuracy and type of reality monitoring error. Cortex, 87:108 -17, 2017.
Ankit Gupta and Jonathan Berant. Gmat: Global memory augmentation for transformers. arXiv preprint wrXiv:2006.03274 2020.
03tp:///pp///00003032374 03274.
Torkel Hafting, Marianne Fyhn, Sturla Molden, May- Britt Moser, and Edvard I Moser. Microstructure of a spatial map in the entorhinal cortex. Nature, 436(7052))8018806 2005.
Joses Ho, Tayfun Tumkaya, Sameer Aryal, Hyungwon Choi, and Adam Maridge-Chang Moving beyond p values: data analysis with estimation graphics. Nature methods, 16(7):565-566, 2019.
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? arXiv preprint 4rXiv:2404.06654 2024.
Alexis Huet, Zied Ben Houidi, and Dario Rossi. Code and Data for Episodic Memories Generation and Evaluation Benchmark for Large Language Models. hattps://qithaii com/ ahstat/ vois c- memomemryy. menemory-benchaaee bewchmarkchkk 2025. [Online; accessed 21-12
12

---

Published as a conference paper at ICLR 2025
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282. 2020. https://arxiv.org)
abs/ 01282
Zhen Jia, Abdaighani Abujabal, Rishiraj Saha Roy, Jannik Strötgen, and Gerhard Weikum. Tem- pquestions: A benchmark for temporal question answering. In Companion Proceedings of the The Web Conference 2018, pp. 1057-1062, 2018.
Marcia Johnson and Carol L Raye. Reality monitoring. Psychological review, 88(1):67, 1981.
Marcia Johnson, Shahin Hashtroudi, and Stephen Lindsay. Source monitoring. Psychological bulletin, 114(1):3 1993.
Gregory Kamradt. Needle In Haystack Pressure Testing LLMs. https://githun. com/ 3kkaradt/LLMTess Needle InAlayttaaaay 2023. Accessed: 2024-08-23.
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to leam long-tail knowledge. arXiv preprint arXiv:2211.08411, 2022. https
7/arxiv. otg/ abs 2211222111.. 08411
Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A Smith, Yejin Choi, Kentaro Inui, et al. Realtime qa: what's the answer right now? Advances in Neural Information Processing Systems, 36, 2024.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. arXiv preprini arXiv: 19X1::9911.0012. 911 00172.
2019. aett::://axxvvoorg/abbs.
Eugene Kharitonov, Marco Baroni, and Dieuwke Hupkes. How bpe affects memorization in transform- transfom. ers. arXiv preprini arXiv:21 2021. htttee//aarit..rrrr/aaa////1.000
Michael D Kopelman. The autobiographical memory interview (ami) in organic and psychogenic am- nesia. Memory, 2(2):211 1-33,, 1994.
Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin. Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. Babilong: Testing the limits of Ilms with long context reasoning-in-Be hnaaaasakkk. arXiv preprint arXiv:2406./0149 2024.
Jinhyuk Lee, Mujeen Sung, Jaewoo Kang, and Danqi Chen. rereessna of phrases at scale. arXiv preprint arXiv:2012.12624 2020. http:://exxxxv..rrr////////12222
Brian Levine, Eva Svoboda, Janine F Hay, Gordon Winocur, and Morris Moscovitch. Aging and autobiographical memory: dissociating episodic from semantic retrieval. Psychology and aging, 17(4):677, 2002.
Mo Li, Songyang Zhang, Yunxin Liu, and Kai Chen. Needlebench: Can Ilms do retrieval and reasoning in 1 million context window? arXiv preprint arXiv:2407.1 / 1963, 2024.
Christopher MacDonald, Kyle Q Lepage, Uri T Eden, and Howard Eichenbaum. Hippocampal "time cells" bridge the gap in memory for discontiguous events. Neuron, 71(4):737-749 2011.
George Mandler. Recognizing: The judgment of previous occurrence. Psychological review, 87(3):
252, 1980.
Pedro Henrique Martins, Zita Marinho, and André FT Martins, xo-former: Infinite memory trans- former. arXiv preprint arXiv:2109.00301, 2021. httppp///aaxi..oo/////////0000.0 pdf
Nick McKenna, Tianyi Li, Liang Cheng, Mohammad Javad Hosseini, Mark Johnson, and Mark Steedman. Sources of hallucination by large language models on inference tasks. arXiv preprint arXiv:2305.14552. 2023.
org/ hbtg//bb//23333 14552
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 15:17759-17377 2022a.
13

---

Published as a conference paper at ICLR 2025
Kevin Meng, Arnb Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. arXiv preprint arXi::2210.07229, 2022b. httts::/aaxiv..o
pdf/2210. 07229. pdf
May-Britt Moser, David C Rowland, and Edvard Moser. Place cells, grid cells, and memory. Cold Spring Harbor perspectives in biology, 7(2):a021808, 2015.
John O' Keefe and Jonathan Dostrovsky. The hippocampus spatial map: preliminary evidence from unit activity in the freely-moving rat. Brain research, 1971.
John O' Keefe and Lynn Nadel. The hippocampus as a cognitive map. Oxford University Press, 1978.
H Freyja Ólafsdóttir, Caswell Barry, Aman B Saleem, Demis Hassabis, and Hugo J Spiers. Hippocam- pal place cells construct reward related sequences through unexplored space. Elife, 4:c06063, 2015.
Fabio Petroni, Aleksandra Piktus, Angela Fan. Patrick Lewis, Majid Yazdani, Nicola De Cao. James Thome, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, et al. Kilt: a benchmark for knowledge intensive language tasks. arXiv preprint 00t::2009.025522 2020. attps::/arxi//: arxiv.org..////
02252
Brad E Pfeiffer and David J Foster. Hippocampal place-cell sequences depict future paths to remembered goals. Nature, 497(7447):74-79, 2013.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprini arXiv:2403.05530. 2024.
Apoorv Saxena, Soumen Chakrabarti, and Partha Talukdar. Question answering over temporal knowledge graphs. arXiv preprint arXiv:2 2021.
Don L Scarborough, Charles Cortese, and Hollis S Scarborough. Frequency and repetition effects in lexical memory. Journal of Experimental Psychology: Human perception and performance, 3(1): 1, 1977.
Daniel Schacter and Kevin P Madore. Remembering the past and imagining the future: Identifying and enhancing the contribution of episodic memory. Memory Studies, 9(3):245-255, 2016.
Qingyu Tan, Hwee Tou Ng, and Lidong Bing. Towards benchmarking and improving the temporal reasoning capability of large language models. arXiv preprint arXiv:2306.0880 08952, 2023.
Kazumasa Z Tanaka, Aleksandr Pevzner, Anahita Hamidi, Yuki Nakazawa, Jalina Graham, and Brian J Wiltgen. Cortical representations reinstated by the hippocampus during memory retrieval. Newron, 84(2):347-354, 2014.
Timothy J Teyler and Pascal DiScenna. The hippocampal memory indexing theory. Behavioral neuroscience, 100(2):147, 1986.
Timothy Teyler and Jerry W Rudy. The hippocampal indexing theory and episodic memory: updating the index. Hippocampus, 17(12): 159-116991119 2007.
Endel Tulving and Donald M Thomson. Encoding specificity and retrieval processes in episodic memory. Psychological review 80(5):352, 1973.
Endel Tulving et al. Episodic and semantic memory. Organization of memory, 1:381-403, 1972.
Kiran Vodrahalli, Santiago Ontanon, Nilesh Tripuraneni, Kelvin Xu, Sanil Jain, Rakesh Shivanna, Jeffrey Hui, Nishanth Dikkala, Mehran Kazemi, Bahare Fatemi, al. Michelangelo: Long context evaluations beyond haystacks via latent structure queries. arXiv preprint arXiv:2409. 1 /2640, 2024.
Sinong Wang. Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04777 2020. 2006 04768
14

---

Published as a conference paper at ICLR 2025
Weizhi Wang. Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Augmenting language models with long-term memory. arXiv preprint arXi::2306.07174. 2023.
https://arxiv.co org/ aws/ 2306. 07174
Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merriënboer, Armand Joulin, and Tomas Mikolov. Towards ai-complete question answering: A set of prerequisite toy 2015.
tasks. arXiv preprint
BA Wilson, Janet Cockburn, and Alan Baddeley. The rivermead behavioural memory test (rbmt). Thames Valley Test Company, Bury St Edmunds, 1985.
Yuxiang Wu, Yu Zhao, Baotian Hu, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. An efficient eemmory-augmente transformer for knowledge-intensivee nlp tasks. arXiv preprint 1.Xiv:2210.16773. 2022. https: http:::://axxxv..oogg abs 2210.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309. 17453, 2023, httpps:/aaxxt
prg/pdf/2309. 77445..pd.
Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang. Zhiyuan Liu, et al. Infinite bench: Extending long context evaluation beyond 100k tokens. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 15262-15277, 2024.
Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. Modifying memories in transformer models. arXiv preprint arXiv:001..00363. 2020. ottp:://aaxv..orrrppf////22220
15

---

Published as a conference paper at ICLR 2025
A EXTENDED BACKGROUND AND RELATED WORK
In this section, we first draw useful parallels between LLMs and human declarative memories that will guide us in building our episodic memory benchmark for LLMs. We then briefly overview the landscape of memory benchmarks, showing how none captures the intricacies of episodic memory.
A.1 SEMANTIC AND EPISODIC MEMORIES: PARALLELS BETWEEN HUMAN COGNITION AND LLMS
"Think of a particular cat you have seen in your previous life". The previous sentence acted cue that reminded you of an episode from your past, memory trace that you recollected with the aid of the cue. This process, known cue-based recall, is fundamental in cognitive psychology and involves retrieving stored information when provided with specific cues or prompts. Interestingly, this process bears a striking resemblance to how LLMs retrieve new text from their inner representations in response to prompts.
Cue-based recall is at the heart of explicit declarative memory, the conseious recognition and recollection of facts, events, and experiences. Cognitive psychologist Endel Tulving pioneered studies on declarative memory and was the first to identify what is now accepted be its two main components: semantic and episodic memories (Tulving et al. 1972).
Semantic memory refers our general knowledge and understanding of concepts that independent of personal experiences. It encompasses facts, language, and principles that are universally applicable. For example, knowing that cats have four legs or that water consists of hydrogen and oxygen are manifestations of semantic memory. Tulving argued that semantic memory is not tied to a specific time or place but serves as a reservoir of factual information accessible across various contexts.
Episodic memory, on the other hand, is intimately connected to specific events. It involves the recollection of past episodes or the ability to imagine future events, also known as future episodic thinking. Episodic memory encompasses the recall of what happened as well as the spatial and temporal context in which the event occurred and the memory trace. This subjective sense of time and place contributes to our ability to mentally time travel and re-experience past events.
While recent research suggests that the distinction between both memories is not as clear-cut as previously thought in terms of neural correlates (De Brigard et al. 2022), both types of memory share common underlying processes. This fluidity in the structural difference between semantic and episodic memory is increasingly viewed in the literature an interconnected continuum rather than discrete, separate entities needing different learning processes.
Interesting parallels can be drawn between cue-based recall in humans and prompt-based generation in LLMs, enlightening our design of an episodic memories benchmark for LLMs. We pinpoint four crucial factors that significantly impact human memory recall and explore their potential extensions to the world of LLMS:
Encoding and retrieval: In humans, the encoding process is crucial for later retrieval of memories. For LLMs, this parallels the training phase, where information is initially processed and stored. Our benchmark will explore different encoding methods for LLMs, including in-context learning. fine-tuning, and retrivallauggennt generation (RAG) as basclines.
Cue/prompt specificity: Tulving's encoding specificity principle (ulliing & Thomson 1973) stipulates that the specificity of the context in which information is encoded determines how it can be effectively retrieved. In LLMs, this can be likened to the k-extractabilltty metric (Biderman et al. 2023 Carlini et al.) 2021). where longer contexts (i.e., more specific) make it orders of magnitude easier to extract memorized sequences verbatim (Carlini et al. 2022). We will leverage this factor in our benchmark by varying the specificity of cues in episodic memory tasks.
Frequency and repetition: Humans tend to better recall information that they've 've encountered multiple times (Scarborough et al.) 1977). This repetitive exposure reinforces neural pathways. making retrieval more efficient. Similarly, in LLMs, the most frequently present next tokens during the training phase will likely be "retrieved" during inference (McKenna al.] 2023] Kandpal et al. 2022). This can be used to induce memorization. While this aspect is not relevant for our benchmark since do not develop encoding methods, this can be leveraged in future work.
16

---

Published as a conference paper at ICLR 2025
Familiarity (Mandler 1980): This rapid feeling of knowing that previously encountered a stimulus, item, or situation, may or may not be followed by a successful retrieval of details related to this situation. LLMs, being generative processes, do not natively support such feature. Augmenting them with this ability can help in assessing when an assertion is factual (e.g., . has been really observed in training) or fake.
Described by Festinger in 1962, and slighltly related to familiarity/noveltyy cognitive dissonance refers to the discomfort felt when holding two contradictory informations as a new incoming information challenges a pre-existing belief (Festinger, 1962). In this work, we focus as first step on cue-based recall and familiarity leaving the dissonance aspect for future work.
Cognitive dissonance (Festinger 1962);
Based on the above parallels, our episodic memory benchmark for LLMs will incorporate the following elements. (i) We assume encoding phase to create episodic memories in LLMs. While the exact mechanism is left for future work, we test in this paper three naive baselines: in-context learning, fine-tuning and Retrieval Augmented Generation (RAG). (ii) We use prompts/cues to ask the models episodic questions, mimicking cue-based recall humans. (iii) We design a systematic approach to cue generation, splitting cues into episodically relevant categories (time, space, people, details) and methodically varying cue specificity. (iv) Testing with unfamiliar cues to assess the model's 's ability to provide negative answers when appropriate.
By leveraging the parallels between human memory and LLM functionality (Tab. 5). we aim to advance the capabilities of LLMs to sup- port solid episodic memories in addition their general knowledge. Our benchmark will provide a comprehensive framework for evaluating and potentially improving LLMs' ability to encode and retrieve episodic memories.
Human memory aspect
Equivalent in LLMs?
Semantic memory Episodic memory Encoding
V (training)
Cue-based recall Encoding specificity Frequenccy/epetition
(prompting)
v (k-extractabilitty)
Famillarity/knowing
(Not yet defined)
Cogmitive dissonance
x (Not yet defined)
A.2 APPROACHES
TO EXTEND LLM MEMORY
Table 5: Human and LLM memory
Memory in LLMs can be conceptualized in three primary forms: (i) in-context memory, where information is processed within the model's 's context window, (ii) external memory accessed through vector databases (Retrieval Augmented Generation or RAG), and (iii) parametric memory stored within the model's parameters. These approaches align with the baselines we test in our episodic memory benchmark: placing the episodic information in-context, in a RAG system, or directly in the model parameters. Each method offers unique advantages and challenges for handling episodic- information LLMs.
A.2.1 IN-CONTEXT MEMORY EXTENSION
In-context memory refers to the information an LLM can process within its context window. Efforts extend this capability focus on increasing the number of tokens an LLM can handle simultaneously. Early approaches aimed to reduce the computational complexity of self-attention. Child et al. (2019) proposed sparse attention methods, restricting the model's s focus subsets of the input. Wang et al. (2020) explored low-rank approximations of attention, while Choromanski et al. (2020) demonstrated kernelized attention, approximating the attention process using kernel functions.
More recent advancements have pushed the boundaries of context length. The Longformer, introduced by Beltagy et al.] (2020), combines local windowed self-attention with task-specific global attention, allowing Bert to process up to 4096 tokens. MMrtinns et al. (2021) proposed the Infinity- former, using continuous-space attention mechanism with radial basis functions. LongNet, developed by Ding et al.) (2023), claims to scale up to 1 billion tokens using "dilated" attention. The StreamingLLMM approach by Xiao et al. (2023) leverages the "attention sink" phenomenon to process up to 4 million tokens without expensive fine-tuning, Most recently, human-like episodic memory approaches to LLMs (Fountas et al. 2024: Das et al. 2024) have been proposed. We plan to test then on our benchmark as
part of our future work.
17

---

Published as a conference paper at ICLR 2025
While these approaches significantly extend the context window, they may not fully capture the persistent nature of episodic memories, as the information is only retained within the current context.
A.2.2 RETRIEVAL AUGMENTED GENERATION (RAG)
Retrieval Augmented Generation (RAG) approaches extend LLM memory by incorporating external knowledge sources, typically stored in vector databases. This method allows LLMs to access information beyond their parametric knowledge, potentially supporting more extensive and persistent episodic-like memories.
Early work by Khandelwal et al. (2099) proposed nearest-neighhor language models, interpolating a pre-trained neural language model with k-nearest neighbors model. [ee al. (2020) formulated knowledge-intensive tasks as phrase retrieval problems, using pre-indexed dense phrase represen- tations. Petroni et al. (2020) demonstrated the effectiveness of combining shared dense vector index with a sequence toaieee-t---eee tossequence model across multiple tasks. Izacard & Grave (020) proposed
"Fusion-in -n--iddeeDr which uses both sparse and dense representations fetch supportive passages before feeding them to a frozen generative model. More recently, Waan et al. (2023) introduced Momg featuring decoupled network architecture with frozen backbone LLM acting
memory encoder and an adaptive residual side-network functioning as a memory retriever and reader.
RAG approaches offer the potential store and retrieve large amounts of episodic information, but the quality of results depends heavily on the retriever which may struggle with complex tasks that require synthesizing information across multiple sources (similarly to when we have multiple traces which correspond to single cue in our benchmark).
A.2.3 PARAMETRIC MEMORY
Parametric memory refers to the knowledge encoded within the model's parameters during training. This subsection explores both the factors influencing parametric memory retention and approaches to edit this knowledge post-training.
Several factors impact memorization and knowledge retention in LLMs' parametric memory. Carlini let al. (2022) found that model scale plays a significant role, with larger models memorizing 2-5 times more than smaller ones. They also noted that data duplication and context length affect memorization, with repeated examples and longer contexts facilitating easier extraction of memorized sequences. Kharitonov et al. (20TT) demonstrated that the size of the subword vocabulary influences Transformer models' ability to memorize training data. Carllni et al. (2019) showed that the sampling strategy, particularly the choice between beam search and greedy sampling, can affect the propensity for data leakage and memorization, Kandpal et al.] (202) found that an L.M''s M's ability to answer fact-based questions significantly influenced by the number of relevant documents seen during pre-training.
Efforts to edit parametric knowledge post-training have also emerged. Meng et al. (2022a) introduced ROME, which uses causal tracing to locate and modify specific associations within the model. The same authors later developed MEMIT (Meng et al. 2022b) in order to scale to much larger edits in bulk. Dai et al. (202I) leveraged the identification of knowledge neurons to perform "knowledge surgery" editing factual knowledge within Transformers without additional fine-tuning Zhu et al.) (2020) framed knowledge modification as constrained optimization problem, finding that constrained layer-wise fine-tuning an effective method for modifying the knowledge that Transformers learn.
While parametric memory offers the potential for integrated, persistent episodic-like information, we not aware of any approach that allows to ingest episodic events into model parameters, current knowledge editing is limited in scale to toy cases, and may have unintended side effects.
A.2.4 HYBRID APPROACIIES
Hybrid approaches aim to combine the strengths of multiple memory strategies, potentially offering a more comprehensive solution for handling episodic-like information in LLMs. These methods often integrate aspects of in-context processing, external retrieval, and parametric knowledge. Guppa & Berant (2020) proposed GMAT (Global Memory Augmentation for Transformers), which introduces a dense attention- global memory to provide a consolidated view of the entire input sequence. This design allows the model to achieve memory overhead that scales linearly with sequence length,
18

---

Published as a conference paper at ICLR 2025
potentially supporting longer-range episodic memories. Ww al. (2022) introduced the Efficient Memory-Augmenter Transformer (EMAT), which encodes external knowledge into a key-value memory and involves novel pre- tre-traiiing tasks. This approach enables the model to learn when to use internal parametric knowledge versus external knowledge, potentially mimicking the interplay between semantic and episodie memory in human cognition. These hybrid approaches offer promising avenues for enhancing LLMs' ability to handle episodic information. By combining multiple memory strategies, they may be able overcome the limitations of individual approaches and provide more flexible and robust system for managing episodic-like memories. It would be particularly interesting to test how these hybrid approaches perform on our episodic memory benchmark. Their ability to integrate different forms of memory could potentially lead to improved performance in tasks requiring both factual recall and contextual understanding of events. Future work could involve adapting our benchmark to specifically evaluate these hybrid models, providing insights into their effectiveness for episodic memory tasks.
Finally, the three approaches to LLM memory- in-context extension, RAG, and parametric memory form the basis for our baseline evaluation our episodic memory benchmark: we test the model's ability to handle episodic information when it's 's placed in-context, stored in RAG system, or encoded directly in the model parameters.
A.3 EVALUATING MEMORY IN HUMANS AND LLMS
A.3.1 HUMAN TESTS OF EPISODIC MEMORIES
Existing tests of episodic memory in humans often involve asking the person to recall specific events in their life or using standardized psychological tests. For example, the Autobiographical Memory Interview (AMI) (Kopelman) 1994) and the Autobiographical Interview (Levine al., 2002) are widely used tests that involves asking the person to recall specific events from their past, which are then scored based on the level of detail and accuracy. Rivermead Behavioural Memory Test (RBMT) (Wilson et al.] 1985) uses everyday scenario events to test a range of memory types, including episodic memory. Another example is the California Verbal Learning Test (CVLT) (Delis) iet al. 2000), which assesses episodic verbal learning and memory and can detect impairments at any point in the episodic memory process, including encoding, retrieval, and recognition of familiarity. Similarly to these tests, we design tests to probe LLMs in their abilities to recall various dimensions of previously encountered episodic events.
Our approach of using specific cues related to time, space, people, and details aligns to an extent with established human episodic memory tests like the AMI (Kopelman) 1994) and the Autobiographical Interview (Levine et al.) 2002). These tests employ structured interviews that classify memories across similar dimensions, including time periods, locations, and specific event details. Note that the Autobiographical Interview for example further categorizes such details into event happenings, perceptual information, and emotions/thoughs. In our work, we instead group all these aspects under the broader category of "event details", though it could be readily extended to systematically analyze different types of details: for example, our chapters are generated with a writing style adhering to different atmospheres (e.g. suspense, tragedy, comedy, etc.), which is akin to emotional category.
A.3.2 BENCHMARKING MEMORY LLMS
Evaluating memory capabilities in LLMs has garnered significant attention, yet existing benchmarks primarily focus on simple retrieval tasks or reasoning over long contexts without capturing the nuanced aspects of episodic memory. this section, we review current benchmarks and highlight their limitations in assessing episodic memory.
Retrieval-focused benchmarks. Many benchmarks assess L.LMs through retrieval tasks, where models are required to find specific pieces of information within extensive contexts. The "needle-in- a- haystackk"" paradigm [Kamradt] 2023) exemplifies this approach, testing a model's ability to locate a single piece of relevant information within irrelevant textbooks. While this assesses basic retrieval capabilities, it does not evaluate the model's understanding of temporal sequences or state changes.
Extensions to this paradigm involve multiple needle retrieval tasks (Reid et al. 2024 Hsiihh et al. 20244] Li et al. 2024| Zhang et al., 2024), which require models to retrieve several pieces of information from large context. However, these tasks still lack the differentiation of cues and do not incorporate
19

---

Published as a conference paper at ICLR 2025
temporal spatial awareness. They remain largely retrieval exercises without testing the model's ability to comprehend and recall events as sequences that unfold over time and space.
Long-context question answering. Benchmarks like those proposed by (Bohnet et al. 2024 and (Zhang et al. 2024) focus on question answering over long contexts. While these tasks involve processing large amounts of text, they often reduce retrieving relevant information without necessitating an understanding of event chronology or causality. They do not sufficiently challenge models track entity states or reason about temporal relationships, which are critical components of episodic memory.
Synthetic reasoning tasks. Syntheric datasets such as the bAbl tasks (Weston al. 2015) and its long-context extension, bAblLong (Kuratov et al. 2024), introduce reasoning tasks that require models to answer questions based on provided stories. However, these tasks often involve short context lengths and highly artificial scenarios. They may allow models to shorteut reasoning by exploiting dataset biases or patterns, thus not effectively evaluating the model's ability to utilize full context in episodic memory sense.
Similarly, synthetic reasoning tasks proposed by Li et al. (2024) are limited by their artificial nature and may not generalize well to real-world episodic memory scenarios. The lack of complexity and realism in these tasks means they do not adequately test a model's capacity for episodic recall and reasoning.
Recently, Vodrahalli et al. (2024) introduced the Latent Structure Queries (LSQ) framework as first good step to address the limitations of current reasoning over long context benchmarks. Somewhat in essence close to our design philosophy, it works by embedding a hidden structure within a large context, which the model must understand and manipulate to answer queries. For example, Python list is modified throughout a long text, and the model must track these changes to determine the list's final state. This approach tests the model's ability track state across the entire context, going beyond simple retrieval. Our approach further systematically incorporates key elements crucial to episodic memory, such as temporal and spatial cues, rich event details, and the ability handle varied retrieval cues based on partial information.
Limitations of existing benchmarks. Overall, as nicely synthesized by Voddahalli et al. (002)). several limits exist in current benchmarks which could be summarized as follows.
Shortcuts: Models can often find answers without utilizing the full context, undermining the evaluation of their long-term memory capabilities.
Retrieval over reasoning: Tasks prioritize retrieval over reasoning about events, states, and temporal relationships.
Data leakage: Use of existing evaluations can lead to data contamination, where models may have seen the test data during training (Zhang et al. 2024 Agarwal et al. 2024 Bohnet
et al. 2024 Hsieh al. 2024 Li at. 2024p.
Out of distribution distractors: Needle in hay stack tasks often place the trace to be retrieved a completely different context, biasing the retrieval task.
labor-intensive creation: Some benchmarks require significant human effort to create and verify datasets, making it challenging to scale or adapt them for different purposes.
Need for an episodic memory benchmark. The limitations of existing benchmarks highlight a gap in evaluating LLMs' episodic memory capabilities, calling for a benchmark that:
Incorporates temporal and spatial context: Evaluates the model's 's ability to understand and recall events with specific time and location details.
Tracks entity states: Assesses how well the model can monitor changes in entities over time, reflecting real-world dynamics.
Uses varied retrieval cues: Tests the model's ability to recall information based on different combinations of event attributes, mirroring human cue-based recall.
Avoids data leakage: Ensures that the evaluation is free from contamination, providing a fair assessment of the model's 's capabilities.
Balances complexity and realism: Provides tasks that are both challenging and representative of real-world episodic memory demands without being overly synthetic.
20

---

Published as a conference paper at ICLR 2025
B BENCHMARK DESIGN
This section provides a comprehensive description the benchmark design and complements Sec. It deals with the creation of the full book document (Sec. B.1], the creation of the question/answer pairs (Sec. [ and the evaluation strategy (Sec. B.3)
B.1 BOOK GENERATION
B.1.1 COMPONENTS THE UNIVERSE
We start from number of atomic components that we can combine to build possible worlds or universes: pair of start and end dates, list of 100 first names, last names, locations, event contents, and a list 30 details for each content. We ensured, when building them, that all the elements are distinct. The Listing provides excerpt of such components.
# temporal
start_ date 2202, 1, 1) end date d20e 12, 31)
# entities
first names ['Emma', 'Liam'. 'Olivia',
last names ('Smith', 'Jhhnson', WWilliamss....
# locations
locations ['Empire State Building'. 'Statue of Liberty', 'Muscum of Modern Art, ...]
contents
contents l'Att Exhibition Opening', 'Sientificc Conference', "Tech Product Launch', ...]
content_dee details
'Art Exhibition Opening': l'Unveiled new collection'. 'Met with art critics', ...J.
'Scientific Conference': ['Presented research findings', 'Participated in panel discussion'.... ('Unveiled new device', 'Demonstrated key features',... ...]
'Tech Product Launch':
Listing 1: Excerpt of the raw materials
We generate the components of the universe with the help of Claude 3.5 Sonnet. For reference, provide the process for building the list of locations in the Listing 2 Our general strategy was to first generate twice the number of targeted elements, and then filter the duplicates and other incorrect elements.
# session 1, prompt 1:
Please list 200 different locations in New York and surrounding areas. Each location should correspond to specific (longitude, latitude) point. The different locations should not overlap. Here are some examples: 'Empire State Building', 'Statue of Liberty', 'Museum of Modern Art (MoMA)', 'Chrysler Building', 'Fort Greene Park'
session 1, prompt 2:
Can you keep only neutral and positive locations, further removing the locations with the name specifying a company name
# session 2, prompt 1:
Are those locations all located in different (longitude, latitude) in New York?: [the list of the 200 locations]
# session 2, prompt 2:
Please discard the less distinct odd ones, in order to keep only 120 different locations among this list
session 3, prompt I:
Are those locations all located different (longitude, latitude) in New York? [the list of the 120 produced locations]
Answer has been 'yes', and keep the 100 first unique elements among those 120 locations.
Listing Process for building the raw locations
21

---

Published as a conference paper at ICLR 2025
B.1.2 BUILDING STATIC UNIVERSE
The static universe defines a list of Nuniverse 100 dates, full names, locations, and contents. The dates are created by sampling days among the start and end raw material range. The full names are created by randomly sampling first and last names. The other items are direct shuffling of the raw materials. The content details are left unchanged. We ensured that there is no duplicated items. An additional seed parameter used for reproducibility.
lemporal ['February 27, 2026', 'May 11, 2026', 'March 23, 2024"...
spatial ['American Museum of Natural History', Metropolitan Museum of Art', 'High Line', ...]
entities = ['Henry Reed', 'Lvvi Rodriguez', 'Scarlett Thomas',
content I'Tcch Hackathon', 'Theater Performance', 'Educational Workshop'....
Listing 3: Excerpt of the universe. Each list has Nuniverse 100 elements.
1.1.3 EVENTS GENERATION
We define event as a date, spatial location, main entity, event content, with its details that we would like to insert verbatim into the document. Each event is hence represented by 5-tuple (t, s. ent, c. d.d.. We provide in Listing the the four first events produced using our fixed seed.
events[O]
['September 13, 2025', 'Bethpage Black Course', 'Ezra Edwards', 'Parkour Workshop'. Demonstrated cal leaps'].
events[|] []
['September 22, 2026', 'American Museum of Natural History', 'Chloe Castillo', Fashion Show', Revealedd future collections'),
events|2] = I'September 22, 2026', 'Port Jefferson', 'Henry Reed', PPhotography Exhibition', 'Explained proc-proes techniques'],
events[3] =I'May 07, 2024', 'Hither Hills State Park', 'Zoe Brown', 'Karaoke Night', PPerformed with hand accompaniment]]
Listing 4: Excerpt of the first four events.
In addition to the seed, two parameters are used for generating the events: the number of events Nevemss and a distribution D with finite support {O..... -Nuniverse 1]. Each event is generated by sampling independently a temporal, a spatial, an entity, and a content among the respective lists of the universe and according the same distribution D. Each content detail is then sampled uniformly at random among the content details of the selected content.
Two constraints govern identical (t, ent) or (t, s) pairs (by initially generating more events before filtering them), since this might imply that the same entity is experiencing two events at the same time, or that two events happening in the same location at the same time. Second, for convenience of our experiments, (ii) we ensure that the choice of different number of events Ne,events N2,events leads the same first evenett produced events, given the other parameters set equal. In our experiments we used NN.evenss = 20 and N2.events 200; allowing us to generate short and long book, where the short book's 's events are included in the long one.
actual generation. First, (i) we do not allow multiple events to have
As earlier mentioned, the distribution D is a critical parameter to ensure that some dates, locations, entities and contents appear in multiple generated events, while others appear only once. This allows us to vary the number of traces or events that correspond to a given cue. We select the truncated geometric distribution with parameter p = 0.1, that is, for i E {0, P(X = = (1-p)(p/
1}.
Nusses
Note that choosing uniform distribution would result instead low probability of selecting multiple times the same item (c.g., the same date t). This is experimentally confirmed in Tab, 6 In this experiment, we repeat 10000 times the selection of Nevents with the distribution D among universe of size N Nunivense and report the expected counts (and standard deviation) in selecting one, two, three to five, or more than six times a specific item. both cases, we show that the counts are better spread within the different bins for the geometric distribution.
B.1.4 EVENT META-DATA GENERATION
We associate event meta-data to each event, providing contextual information and constraints re- garding the generation of the textual chapter. Each event meta-data indicates the targeted number of
22

---

Published as a conference paper at ICLR 2025
Table Expected counts in selecting one, two, three to five, or more than six times specific item (e.g., date) for universe of size Nunivense = 100. For instance, the value 2 in the bottom-right cell indicates that we expect two dates to repeated each more than 6 times among the 200 generated events, using the uniform distribution. The actual counts in our experiments is shown in Tab. 9
Bin of counts
eevents
D
3-5
6+
20
geometric
8+2.4
3+1.5
240,9
0+0.2
200
geometric
9+2.7
5+1.9
7+2.2
13+1,4
20
uniform
1642.3
2+1.1
0+0.3
0+0.0
200
uniform
2744.0
2744.5
3143.2
2+1.2
paragraphs, and the position of the different event features within those paragraphs. Additionally, a desired style is indicated. We plan to the style in future iterations of the benchmark generate questions about the general atmosphere of the events, as opposed to specific tokens. We provide in Listing 4 the meta data associated with the four events, produced using our fixed seed.
metacvents(0) 0) = ['nb paragraphs'" 7, "idx_ paragrapa;; ('date' 7, 'location': 2, 'entity:: 2, 'content': 2).
"stylc': 'thriller',, J.
metacvents[ []
('nb habb_paaggrphs' 7, 'iix__pprreeea paragrapp;; ('date: 7, 'location:: 5, 'entity: 'contentt: 3},
'style': 'aantayy';
metaevents[2] = ('nb hatb_paragraphs:: I, iixx aix_paragraph ('date': I, 'location:: 1, eentity: 'contentt: 1],
'style': 'etectivv'),
metaevents(3]= = patrrrenpshp 5, 'idx paragraph': ('date:: 1. location': 3, 'entity: 3, 'content': 4).
paragraphs': 'style': 'mystery')
Listing 5: Excerpt of the first four event meta-data.
For each event, we sample the number of paragraphs uniformly among {I,..., 10), and each event feature (among date, location, entity and content) between one and the number of paragraphs. Finally, the style is sampled among the Listing each style being associated three adjectives, Both styles and the related adjectives have been generated with Claude 3.5 Sonnet.
'detective': I'suspense', 'ddduction', innestigationn]]
'comedy:: ['humor', 'wit', 'absurdity'].
'tragedy:: ['sorrow', 'catharsis', 'downfall'.
'romance': l'passion', 'intimacy'. 'longing'].
'thriller': Texciiteennt, 'angerr, 'anticiiattion],
'fantasy:: l'magic', 'magination', 'worldbuilding') 'horror': ['fear', 'dread', 'uupematural",, 'mystery': ['enigma', "clues', 'revelation]]
Listing 6: List the possible styles, with corresponding adjectives.
B.5 SINGLE CHAPTER CANDIDATE GENERATION
The template prompt for generating each candidate chapter is described in Listing 7 The template filled with the right event and meta-data, before fed to an LLM for generation. our experiments, we used both Claude 3.5 Sonnet (2024-06-20) and GPT-4o (2024-05-13). Only the book built from Claude 3.5 Sonnet has been used in the main paper, but we provide the outputs for both model at Huet et al. (2025). and additional evaluation the GPT-4o book is provided in Appendix E.5
# System prompt
You creative fiction writer specializing in detailed, atmospheric novel excerpts. Your task generate vivid, immersive scenes based on specific prompts.
# User prompt
Write detailed novel excerpt in (style) style about (entity) attending (content).
The story takes place (date), ; (location), where (entity) (content_siige caatentt_single___ detail)
23

---

Published as a conference paper at ICLR 2025
Follow these guidelines:
Structure and Information Reveal:
1. Divide the text into jatagraagggppss paragraph(s). Number each paragraph (1), (2), etc., while
maintaining novel-appropriate paragraph lengths.
Gradually reveal key information:
Full location [llcationn] )': must appear verbatim in paragraph [idx_loc) only and nowhere else in text
Full date ddtte J:: must appear verbatim in paragraph [idx date] only and nowhere else in the text
Full name "[entity] I must appear verbatim in paragraph (idx. _etitt] ] only and nowhere else in the text
Full detail that ffistt name) [content single_detail) must appear verbatim in paragraph (idx content)
and nowhere else in the text
Subtly distribute details about location, date, main character, and event across paragraphs.
Content and Setting:
Focus [first, name l's experiences, observations, and interactions during the [content).
Vividly describe surroundings, atmosphere, (first name 's emotions.
Include the detail that (first_naa name) {content_sing
4. Limit the timeframe to single day and confine all action to [location).
Characters:
Refer other characters as Sentity_X (where X a number).
Omit background information about ffirstnname and other characters.
Style and Tone:
Use vivid, sensory details to bring scene to life.
Incorporate elements the (style) style, including jyyyl_description)
3. Maintain a consistent narrative voice throughout the excerpt.
Restrictions:
Only mention [location] and [date]; avoid other locations or dates. Exclude explicit introductions, conclusions, or character backgrounds.
3. Focus exclusively on the events of this particular [content).
Craft a seamless narrative that gradually reveals information while maintaining reader engagement throughout the excerpt.
Listing 7: Template of the prompts for single chapter generation. The highlighted elements are replaced by the event and event meta-data values.
B.1.6 EXACT VERIFICATION OF THE CANDIDATE CHAPTER
Each candidate chapter verified for correctness before being accepted. This is done first through direct assessment of the constraints enforced by the meta-data and by the single chapter generation prompt. The checks are performed as follows:
Check that the number of paragraphs is correct, each paragraph beginning with "(X) " with X a number, and with increments of one,
Check that the other entities have always the form Sentity X, with X an integer,
Check the presence verbatim of the date, location, entity and content detail in the specified paragraph, while checking their absence in the other paragraphs.
B.1.7 LLM-AS-A-JUDGE VERIFICATION OF THE CANDIDATE CHAPTER
the chapter candidate has passed the exact verification, another round verification is performed (with the same LLM that generated the content) for ensuring that the generated chapter is valid. Four boolean questions are asked, that concern each the four event features (date, location, entity, content), explicited in Listing
# System prompt
You are content checker AI. Your tasks:
1. Read the given text carefully.
Answer true/false questions about the text.
24

---

Published as a conference paper at ICLR 2025
Respond in JSON format.
Be accurate and concise. Only use information explicitly stated the text.
# User prompt
Please analyze the following text enclosed between [TEXT START] and [TEXT END] markers, and answer the four questions below with simple true false. Provide your answers in JSON format with the question numbers as keys and the bonlean answers as values.
ITEXT STARTJ
generated [
TEXT END]
Questions:
1. Does the following text takes place in single geographical (longitude, latitude)?
Does the following
takes place in single temporal day?
Does the following text has single main character?
Does the following text has single main event happening at that location that day (further cut into the
events of the day)?
Your response should be in this ISON format:
"1":
[boolean]
"2"
[boolean].
"3":
[boolean].
"4": [boolean]
Listing 8: Template of the verification prompt of single generated chapter candidate. The first highlighted element is replaced by the generated chapter candidate.
B.1.8 ADDING SECONDARY ENTITIES
Other characters can appear during the generation of the chapter, under the form Sentity_ __ (as stated in Listing 7 and verified using direct verification). For filling the name of those secondary entities, additional 100000 full names are generated from two lists of 500 first and 1000 last names. Those names are disjoint from the lists used for generating the main entities. In each validated chapter, new name is used for each indexed entity. This ensures that all the secondary entities only appear in single chapter. The first elements of the list are shown in Listing
['Noa Middleton', 'Mara Ledbetter', 'Sienna Hamrick', 'Reid Blunt',...
Listing 9: Excerpt of the list of additional entities.
B. DOCUMENT ASSEMBLY
For building the whole document, the candidate chapter generation process is applied on all events {O.... - 13. If the candidate does not pass the validation (either by the direct by the LLM-as-a-judee round), that fail to have a valid candidate chapter after itermax.
candidate is generated until itermax 10. We discard all the events
The valid candidates reshaped into CChapter (in generated from 1) and concatenated into a single document.
deteerttedehhapeerrreraeeeee chapter) Inln'nn (where begins
1.1.10 CHAPTER EXAMPLE
We provide in Listing 10 the first generated chapter. The generated chapter has been assessed as valid (by both verification method) at the first iteration. It corresponds to the event 0 for which the values are (as indicated Listing 4):
September 13, 2025', "Bethpage Black Course', 'Ezza Edwards', PParkour Workshop', Demonstrated cat leaps')
and the values are indicated in Listing
('nh paragmphs' : 7, "idk__nee paragmphh: ("Iocation' 2, 'date': 7, 'entity': 2, 'content': 2), 'style' 'thriller')
25

---

Published as a conference paper at ICLR 2025
The secondary entities for this chapter are (as indicated in Listing 9
(oN Midilleton, "Mara Ledhetter').
The generated chapter has 547 tokens.
The harsh glare of the floodlights cast long shadows across the rugged terrain. Adrenaline coursed through veins as he surveyed the daunting obstacles before him. The parkour workshop had drawn diverse crowd, each participant eager to push their limits and master the urban art of movement. He flexed his fingers, anticipation building in his chest as he prepared for the night's challenges.
At Bethpage Black Course, Ezra Edwards demonstrated cat leaps with grace that belied difficulty the maneuver. The other participants watched awe as he effortlessly cleared the gap between two towering structures. Noa Middleton, the lead instructor, nodded approvingly, a hint of a smile playing at the corners of their mouth.
As the workshop progressed. testing the limits of even the most seasoned traceurs. He felt a bead of sweat trickle down his spine as he approached the next challenge series of precarious platforms suspended high above the ground. The distant hum of crickets provided cerie soundtrack to the scene.
grew thick with tension. The obstacles became increasingly complex,
Mara Ledbetter stumbled on a particularly tricky jump, their cry of alarm piercing the night air. Hc instinctively reached out, his fingers barely grazing their arm as they teetered on the edge of the platform. Time seemed to slow as Mara Ledbetter regained their balance, their eyes wide with fear and gratitude. The near-miss sent shiver through the group, stark reminder of the risks they
taking.
As midnight approached, the workshop took on a more sinister tone. Noa Middleton announced the final challenge complex route through a maze-like structure shrouded in darkness. The participants exchanged nervous glances, the thrill of the unknown mingling with growing sense of unease. He took deep breath, stceling himself for what lay ahead.
The maze was cacophony of creaks and groans, every shadow seeming to hide potential threat. He moved swiftly, his muscles burning with exertion he vaulted over walls and slid under low-hanging beams. A muffled cry from somewhere in the darkness sent his heart racing. Was it merely another participant, or something more sinister?
As he emerged from the maze, panting and exhilarated, the first rays of dawn began to peek over the horizon.
The workshop had pushed him his limits, testing not just his physical abilities but his mental fortitude well. As the group gathered for final debrief, he couldn't shake the feeling that something fundamental had shifted during the long night of September 13, 2025. The parkour workshop may have ended, but he sensed that a new, more dangerous game was just beginning.
Listing 10: Chapter I generated with Claude 3.5 Sonnet (event 0, iteration 0). Event information and secondary entities have been highlighted.
B.1.11 BOOK EXAMPLE
We provide in the Listing II an illustration of how the full book generated with Nevents 200 with Claude 3.5 Sonnet looks like. The generated book has 102870 tokens.
Chapter
The harsh glare the floodlights cast long shadows across the rugged terrain. Adrenaline coursed through his veins as he surveyed the daunting obstacles before him. [...] The parkour workshop may have ended, but he sensed that a new, more dangerous game was just beginning.
Chapter 2
The air shimmered with an otherworldly energy as she stepped onto the glittering runway. [...] It was September 22, 2026, and on this night, she had woven dreams into reality, leaving an indelible mark on the tapestry time.
[...]
26

---

Published as a conference paper at ICLR 2025
Chapter 196
The evening air crackled with anticipation as she stepped onto the gravel path, her camera bag slung over her shoulder. [..]] With deep breath, she steeled herself for the investigation come, knowing her keen eye and analytical mind would be put the test in ways she anticipated when she arrived this seemingly innocent photography exhibition.
Listing 11: Complete document generated with Claude 3.5 Sonnet with Novens = 200. There are only 196 chapters because 4 events did not pass the verifications after 10 iterations. Ellipses are indicated with between brackets.
B.1.12 GENERATION STATISTICS
We extract different statistics of the overall process for creating the document. In Tab. 7 we show the number of validated chapters after each iteration. The LLM--a-a-judge verification is applied only the candidate chapter has passed the direct verification. We observe that most of the events pass after few iterations. We didn't t observe any significant characteristic for the four discarded events, for instance, the number of paragraphs is (2, 9, 1, 2) for Claude 3.5 Sonnet, and (9, 8, 2, 6) for GPT--4o.
In Tab. 88 we show the distribution of the number of paragraphs, the styles, and of the number of secondary entities for the generated book. The distribution is shown for Claude 3.5 Sonnet, but the distribution for GPT-4o almost the same since the original events are equal (the only difference related to the four discarded events, that are different).
Table Number of validated chapters for each iteration with Nevents 200. The verification failures
correspond to the checks operated according to Sec. 1.1.6 and B.1.7
Claude 3.5 Sonnet
GPT-4o
remaining events to evaluate
fail direct
fail LLM
valid after this iteration
remaining events evaluate
fail direct
fail
valid after
iteration
LLM
this iteration
200
32
133/200
85
109/200
167/200
46
154/200
179/200
29
168/200
185/200
20
179/200
15
189/200
19
181/200
191/200
185/200
194/200
190/200
6
195/200
192/200
196/200
194/200
196/200
196/200
Table Chapter count regarding the number of paragraphs, the style, and the number of secondary entities for Nevents = 200, with Claude 3.5 Sonnet.
number of paragraphs
count
number of secondary entities
style
count
count
17
comedy
16
2
27
22
detective
22
3
16
74
fantasy
32
4
21
60
horror
18
5
19
27
mystery
6
19
8
romance
17
7
14
6
2
thriller
30
8
18
tragedy
36
9
20
8
1
10
We observe in Fig. the relative position of the event features (date, location, entity, content details) in the text with respect to the whole book, the chapter, and the paragraph. At the book level (top), the
27

---

Published as a conference paper at ICLR 2025
1.0
BCDF
DCDF
0.5
0.5
0.0
0.0
0.0
0.2
0.4
0.0
0.2
0.4
0.6
0.8
1.
Relative pusition of the cue within the book
Relative poeítiinn of the cue within the book
BCDE
ECDF
0.5
0.5
0.0
0.6
0.2
0.6
1.0
position
chaptoo
Relative position rne chapter
1.0
DCDF
ECDF
0.00
Relative position of the within the paragraph
Relusive positiio of the wichin the parugraph
Claude 3.5 Sonnet
(b) GPT-4o
Figure 5: Position of the event features (time, space, entity, content details) relative to the book (top), the chapter (middle), the paragraph (bottom), for Claude 3.5 Sonnet (left) and GPT-4o (right), with Nevens 200.
different features are all almost uniformly spread. At the chapter level (middle), the constraints in the paragraph position ensure that the information is relatively well spread among each chapter. The discrepancy is due to the preference of the models to generate the event features at the beginming of the paragraph (bottom). We note that the bias the paragraph level is strong for the entity name, while almost nonexistent for the verbatim content detail. Slight differences are appearing between Claude 3.5 Sonnet (left) and GPT-4o (right).
In Tab. 9 we show the actual counts of the number of times specific event feature is repeated over different chapters. The observed values are accordance with the expected counts Tab. 66] For instance, the highlighted value of 13 corresponds 13 different dates (May 11, 2026; March 23, 2024: April 09, 2026; February 27, 2026; ...) repeated each more than 6 times (resp. 15; 13; 12: 11; ...).
B.2 CREATION OF THE QUESTION/ANSWER PAIRS
B.2.1 LIST OF QUESTION/ANSWEL TEMPLATES
We provide in Tab. the full list of question templates. Each question is an episodic memory task based on a cue composition (the trigger enabling a set of events to be remembered), a trace (the retrieval type that needs to be extracted from each remembered event) and the information to ger (whether all events are retrieved, only the latest state, or their chronological order). We consider all the combinations of cues and traces for the retrieval of the whole information (ids 0 to 29), while we focus the entity cues for the latest and chronological retrievals. In the table, the cue has four components: t (the time), (the space), e (the entity) and e (the event content). The trace can one of those four components (Times, Spaces, Entities, Contents), the list of the secondary entities (Other entities), or the full details (Full Details) regarding a specific chapter. The elements to get all the retrieved elements (all), the latest one (latest) the chronological order of the elements
28

---

Published as a conference paper at ICLR 2025
Table 9: Actual counts in selecting one, two, three five, or more than six times a specific item (e.g., date) in our experiments. For instance, the highlighted value indicates that 13 different days are repeated each more than 6 times among the 200 generated events. This table echoes the expected counts of Tab. 6
Bin of counts
N Neveats
feature
2
3-5
6+
20
time
8
4
1
0
20
space
6
5
1
20
entity
8
4
1
0
20
content
8
1
3
0
200
time
6
4
14
13
200
space
7
7
8
13
200
entity
9
3
7
15
200
content
9
4
7
14
(chrono.). All the ground truth answers are known, since the document has been built with known events, For the question regarding the full details, take the full chapter verbatim as the ground truth answer.
B.2.2 LIST OF QUESTIONS WITH NON-EMPTY ANSWERS GIVEN A BOOK
The questions are created given (i) the question/answer templates and (ii) the ground truth regarding all the events of the document built in Sec. B.
For each chapter, we retrieve the ground truth tuple (t, s, e, e) regarding the date, the location, the main entity, and the event content. We then generate all the questions by following the template. After building all the questions for all the chapters, we filter the duplicates.
We obtain 564 questions for the documents produced with Nevents 20 (filtered from 684 36 ids x 19 chapters questions) and 3886 questions for the documents produced with Nevems 200 (filtered from 7056 36 ids 196 chapters questions).
We provide an example in Tab. "Noonempty" row).
B.2.3 LIST QUESTIONS WITH EMPTY ANSWERS GIVEN A BOOK
In addition to the questions for which the answer can be found in the book, we design questions with empty answers. We employ two strategies: the inner strategy and the outer one. We first initially define the unused static universe as the subset of the static universe that has not been used during the document generation, and which is expected, according to Tab. 6 contain 66 elements for each event feature (time, space, location, content) when N Nuniverse = 100, Nevents 200, and D geometrie.
Then, to create the fake event, we corrupt a tuple (t, from an existing chapter, by applying an i.i.d. binary mask (be, b,, be, be), where each component follows Bernoulli distribution with probability 1/2. When the mask is applied, we replace the corresponding feature (e.g., the date September 13. 2025) by the same kind of feature, either sampled from another chapter (inner strategy) or from the unused static universe (outer strategy). The resulting tuple (t', s','',,' c'c')) c') is then replaced the template questions, in the same manner as Sec. B.2.2
The process is applied for both inner and outer strategies once for each chapter. The resulting questions are also filtered keep only those with empty answers.
We obtain 438 additional questions for the documents with Nevents = 20 and 3657 questions for the documents with Nevents 200.
We provide an example of each strategy in Tab. 11 ("Empty (inner)" and "Empty (outer)" rows).
29

---

Published as a conference paper at ICLR 2025
Table 10: List of all the question templates based on cue composition, retrieval trace, and "get" style.
Cue
Trace
Get
Template question
Recall all the events that occurred on (t). Without describing the evenis, list all the unique
Spaces
all
locations where these events look place.
Consider all events that happened on [t). Provide a list of all protagonists involved in any of these
Entiries
all
events, without describing the events themselves.
Reflect on [t]. Deseribe all the key events that occarred on this date, focusing on what happened ruther than who involved or where took place.
Contcuts
all
Think about all events that have occurred at (s). Provide list of all dates when these events took place, without describing the events.
(*s.?**)
Times
all
Consider location [s]. List all protagonists that have been invalved in any events at this
Entities
all
location, without mentioning the events themselves.
Recall the various events that have taken place at [s]. Describe what happened during these events,
Contents
focusing on the actions or occureences rather than the timing or people involved.
Reflect on all events involvvng [ent]. Provide a list of all dates when these events occuree, without
(*'".e.")
Times
all
describing events.
Consider all events that [ent] has been involved in. List all the locations where these events took
(*!he,))
Spaces
all
place, without meationing the events thomselies.
Think about [ent iss experiences. Describe all the key events they' 've been involved in. focusing on what happened rather than when where it occurred.
(*.Pue,")
Contents
all
Recall all events related to [c]. Provide a describing the events.
dates when these events occurred, without
Times
all
Consider all events involving (c). List all the locations where these events took place, without mentioning the events themselves.
10
Spaces
Reflect on events related to [e]. Provide a list of all protagonists involved in these events, without describing the eveats.
(?,7,%))
Entities
all
Think about what happened at (s) on [t]. List all protagonists involved in any eveats at this time
12
(c.).".)
Entities
all
and place, without describing events.
13
Contents
all
Recall the key events that occurred at [s) on (1). Describe what happened, locusing on the actions
occurrences rather than who involved.
Consider the events involving [ent] on (t). Läst all the locations where these events took place.
14
Spaces
all
without deseribing the ovents themselves.
Reflect on what [cnt] experienced on [t). Deseribe all the key events they were involved in,
15
(t.te,")
Contents
all
focusing what happoned rather than where occurre.
16
(L*.%.c)
Spaces
all
Recall the events related to [e] that occurred on (t). List all the locations where these events took place, without describing the eveats themselves.
Think about the events involving (c) on [t). Provide list of all protagonists involved these
17
Fatities
all
events, without describing the events.
Consider all events involving [ent] [s]. Provide a list of all dates when these events occurred, without describing the events.
(*,5,c,))
Times
all
Reflect on [cat] experiences at [s). Deseribe all the key events they' 've been involved in at this
(*,.0.")
Contents
all
location, focusing on what happened rather than when it occurre..
Recall all events related to [c] that occurred at (s). Provide list of a'll dates when these events took place, without describing the events.
20
(*ax"
Times
all
Think about the events involving [c] at (s). List all protagonists involved in these events, without
21
(*,s,".e)
Fntities
all
mentioning the events themselves.
Consider all events involving both [ent] and [c], Provide a all dates when these events
22
(*,%...c)
Times
all
occurred, without descrihing the events.
Reflect on the experiences of (cat) related (c). List all the unique locations where these events took place, without mentioning the events themselves.
(*.%,e.))
Spaces
all
Recall what huppened invulving [ent) at [s) on (t). Describe the key events activities that
(Lxee.)
Contents
all
occurred, focusing on what happened.
Think about the events related to [c] that occurred at [s] on (t). List all protagonists involved in these events, without describing the events themselves.
25
(1.5.
Fntiries
all
Consider the events involving both [eni] and [c] on [t]. List all the locations where these events
26
(t.*,..c)
Spaces
all
look place, without describing the events themselves.
Consider all events involving hoth (ent) and [c] at (s). Provide a dates when
27
(*,,,0,)
Times
all
events occurred, without deseribing the events.
Recall what happened involving [ent] and [c] on (t) and list only who else was involved (if anyone).
(ts.c.c)
Other entiries
all
Provide a comprebensive account of what happened involving (eat) and [c] at (s) on (1). Include all relevani details about the event(s), including what occurred and any other pertinent information.
(La.e.))
Full details
30
(*
Times
latest
What is the most recent date [ent] observed or mentioned the story's chronology?
31
("
Spaces
latest
What is the most recent location where (ent) was observed in the story's chronnoogical timeline?
32
Contents
latest
What was |ent] J doing the last time they were observed in the story's 's timeline?
Provide chronological list of all dates when [cat] was observed, from carliest latest the story's 'srys timeline.
("
Times
chrono.
("Na))
Spaces
chrono.
List locations visited by (ent) chronological order according the story's timeline.
Enumerate all activities that [ent] has been involved in, ordered from earliest to latest in the story's chronology.
35
Contents
chrono.
30

---

Published as a conference paper at ICLR 2025
Table 11: Real question examples applied on Chapter 1 given the ground truth tuple (t, c) (" ('September 13, 2025', 'Bethpage Black Course', 'Ezra Edwards', 'Parkour Workshop') and for the question with id 21. The date 'May 07, 2024" appears chapters 3 and 129, the location 'Central Park appears in chapter 166, while the content 'Laser Tag Tournament" never appears in the document.
Replacement
Question
answer
Non-empty
shop at Hethpage Black Course List all protag- enists involved in these evenis, without mention- the events feemseives.
evests iwolving
('Chloe 'Bam
Castillo' Edwardss
Chapters 1. 109, 128
"Levi Rodriguex") )
Think about the events involving Parkour Woek- shopp Central Park. List all protigouists in- valved these
c-IMay 07,, 2024), ('-(Ceataal Park)
Empty (inser)
(1.1.00)
without mentioning
2
events thenoselves.
Think about the events invulving Laser Tag Tour- Bethpage Black Coure. List all pro- tagonists involved these without tioming events (berselives.
Empty (outer)
(0,0,0.))
é' =fLaer Tag Toumnment)
Table 12: Widespreadness of the selected questions: number of selected questions per cue per bin, after the filtering described in Sec. B.2.4
Neven
20
Nevesta
200
Cuc/Bin
3-5
3-5
5
5
4
1
0
5
5
5
5
(*
(*"%,,
5
5
4
0
5
5
5
5
(9777%))
3
(.*.*.c)
(*,57%c)
5
5
0
0
5
5
5
5
0
(*.5,0.0)
0
(*,s,,))
0
0
(*s.e.)
others
B.2.4 SELECTION OF QUESTIONS
The number of all possible questions is large and increases significantly function of Nevents Since the evaluation cost scales with the number of questions, we decided to extract only a subset of the overall questions.
First, we consider the number of events that should be remembered for answering each question. We group the questions into five bins: {0) (the empty questions), {1) (the question that triggers single chapter), (2), (3. 4, 5) and {6+). In the example provided in Tab. 11 ("Non-empty" row), the question is put into the {3, 4,5) group (this group is also noted 3-5). For the questions involving the latest state, we consider all the chapters triggered by the cue for considering the group. We expect that the more chapters are involved, the more difficult the question is.
We then target the extraction of Neagett 5 questions for each of the 36 question ids and for each of the 5 bins, resulting in up to 900 questions per experiment.
This method ensures that the questions are relatively well spread among the different bins, as shown in Tab. [2] In this table, we show the final number of questions obtained for each cue and each bin, for both N Vevents 20 (left) and Nevents 200 (right). Each cue is associated to a set of question ids. For instance, the cue (*, +, e, *) associated to the ids 6, 8, 30, 32, 33, 34, 35. We observe that we always retrieve the 5 target questions for bins {0} and (1). For the bin with {6+} with Nevemt = 20, we know for sure that no question exists since single cue has been associated to at most three chapters (as seen in Tab. 99) For the single
recall (first four rows), most of the bins
are filled. For the multiple cues (from row five), it is difficult with Nevents 20 to cover all the combinations, while most of the bins filled with Nevents 20. The others cues (last row) involve the pair (t, s) or (t, e), for which we imposed at most one extracted chapter (since we forbid in the document generation the possibility of two chapters involving the same location the same time, the same entity at the same time), hence all the questions are necessarily in the bins {0} or {1}.
At the end, end up with 456 questions for Nevnnss 20, and 686 questions for Nevents 200.
31

---

Published as a conference paper at ICLR 2025
B.2.5 SELECTION OF THE FINE-TUNING QUESTIONS
Our naive fine-tuning experiment aims to incorporate the essential information needed to generalize answers across all benchmark questions. Therefore, we restrict our training data to question-answer pairs that correspond single events (each event being associated with a single chapter). These pairs establish basic facts (e.g,, taking the example of Fig. 66] "Jackson Ramos attended a carnival in Central Park on September 22, 2026" related to the event Chapter 163, "Jackson Ramos did a flash mob at Ellis Island on April 09, 2026" related to the event of Chapter 96, etc.), which in principle enables deducing answers to questions involving multiple events, such as the set of all locations involving Jackson Ramos: (Central Park, High Line, One World Trade Center, Ellis Island, Snug Harbor Cultural Center) (question template in Tab. TT).
The fine-tuning process thus uses 3,199 training questions for the long book (filtered from the total 3,886 questions involving one or several events, as mentioned in Sec. B.2.2) and 468 training questions for the short book.
Note that regarding the testing data for the fine-tuning experiment, we use the set of questions indicated in Sec. [B.2.4 (as for the other experiments). Specifically, all questions involving single chapter (i.e., corresponding to the bin {1}) are present in both the training and the test sets, while all other questions appear only in the test set.
B.3 EVALUATION STRATEGY
In the previous Secs, B.I and B.2, we have described the creation of the memory to encode and of the list of questions to ask, along with the corresponding ground truth answers. Once model has ingested the memory to encode (e.g.. by directly putting the document in-context, by using RAG, or by fine-tuning the model), and once the predicted answers have been extracted from the model, we use common evaluation strategy to compare the ground truth answers with the predicted ones.
B.3.1 MAIN EVALUATIONN PROMPT
We use the main evaluation prompt defined in Listing 12 for deducing the Fl-score between the ground truth and the predicted answers.
The key idea is to adopt the ground truth point of view for performing the evaluation, since we are sure of the number of ground truth items. In consequence, we assign for each ground truth item matching score between 0 (the item has been missing in the LLM predicted answer) and I (the item has been found, considering synonyms or close meanings). In addition, we ask for the all identified items in the Al answer, which is necessary to know the number of predictions, and hence assess false positives.
B.3.2 COMPUTATION OF THE FI-SCORE
The list of identified items iditems is not entirely controlled given the way it has been extracted. There are numerous cases where its size is larger than the size of the ground truth, even for a coherent Al-generated answer. For instance, (i) in the case of a full event, the ground truth is the full chapter while the identified items may be a detailed list, (ii) in the case the latest content, the ground truth has size one while the identified items may include details about the different steps of the content (as seen in the Listing 13). (iii) in the case of entity retrieval, the model may include the secondary entities, that are not the protagonist but are still related. For those reasons, we decided to be lenient in the number of predictions #pred and to define #pred = min #iditems, #gt) if #gt >> 0, and #iditems otherwise. This policy is nevertheless straightforward tune, since it does not rely on an additional generative model. At the end, the sum of the matching scores S is the quantity that assesses the number of true positives, from which the precision S/ #pred and recall S/ #tt are computed (with #gt the number of ground truth items and #pred the number of predictions). The F1 F1-score is then computed as their harmonic mean.
You are an expert judge evaluating the accuracy of an Al-generated answer against known groundtruth.
Questions can probe for different types or aspects, like what actions or events took place, what people were involved, what were the dates, or what were the locations or spaces.
32

---

Published as a conference paper at ICLR 2025
Question lype: (retrieval lype)
Groundtruth: (correct_answer
Al-generated answer: (llm_answer)
Your task:
Identify all unique items in the AT- Al-generated answer that are relevant to the question type, Answer an
empty list | for this field in case of at least one negative information (e.g., when the answer begins by telling there is no information, or cannot answer)
Determine matching
between 0 and for each ground truth item. Give if the item has been found in the relevant items of the Al-generated answer, considering synonyms, paraphrases, close
meanings. Give 0.5 if the item could be considered related to any Al-generated item without heing explicitly stated as such. Give 0 if the item missed mentioning specific Al Al-generated item.
Provide brief explanation of the evaluation
Provide your evaluation in following JSON format:
"identified_iten""" Saa___ittem___n___ _n__s_______eeer""""" Laa_ser__item
.....
"matching_ score:: [[" (correct_answee Larrrcet_anwwe____e 0J": "score_bor betrreebeteee__eeeee 0a____annn L_annd_l)h
[correct_answer 1) "score_bert Seore_betwee___e aaaia__________ 1__nd_1",,.. ......
"explanation": "Brief explanation of your evaluation"
Listing 12: Template the verification prompt of a single generated chapter candidate. The first highlighted element is replaced by the generated chapter candidate.
# Question
Recall the key events that occurred Snug Harbor Cultural Center on April 09, 2026. Describe what happened, focusing on actions or occurrences rather was involved.
# Ground truth answer
(Tech Hackathon]
# Predicted answer
On April 09, 2026, the atmosphere at Snug Harbor Cultural Center was charged with excitement as developers from the tech world gathered for a hackathon. Energy buzzed throughout the space, with participants mingling and discussing innovations while sunlight streamed through the tall windows. the day progressed, focus shifted to collaborative projects, where participants worked intensely to create innovative solutions.
The event culminated with presentations, showcasing the various projects developed throughout the day. One the collaborations and creativity that flourished during the hackathon. Overall, the day was marked by a sense belonging. connection, and shared passion for technological advancements.
key presentation was met with thunderous applause, reflecting the success
Identified items in AI answer
(Tech Hackathon', 'developers gathered', 'collaborative projects', 'innovative solutions', 'presentations]
# Matching score ('Tech Hackathon': 1|]
Explanation
The AI Al-generated answer directly mentions a 'hackathon' involving tech developers, which perfectly matches the ground truth Tech Hackathon'. The answer provides additional details about the event, including the collaborative nature, focus innovation, and culmination in presentations, all of which are consistent with typical tech hackathon.
Listing 13: Example with correct answer provided by GPT-4o mini (on the document with 20 events produced by Claude 3.5 Sonnet), which however gives more identified items than the ground truth. In this case, we are lenient in the number of predictions, resulting in F1=1, as detailed Sec. B.3.2
33

---

Published as a conference paper at ICLR 2025
B.3.3 COMPUTATION KENDALL'S TAU COEFFICIENT
For the evaluation of the question involving the chronological order of the retrieved traces, compare the identified items in Al answer obtained the Listing [2 with with the ground truth answer. The identified items may not have the same length the ground truth answer, and the names may be slightly different (e.g.. "Theater Performance" instead of "Theater Show"). For those reasons, we cannot directly compute the Kendall's i and rely on the prompt in Listing 14 for extracting the matching items while preserving the order.
We then consider only the questions that related to a ground truth answer of size least (otherwise there is no order). Then, we consider the identified items that entirely match with the ground truth answer. This selection ensures that we specifically compare the ordering ability, and the retrieval one. Finally, for each (ground truth list, identified items) pair, we deduce the elements that are in both lists (while preserving order), and compute their Kendall's tau. A Kendall's tau is measure between -1 and 1, corresponding to perfect match in the indexes of the two lists.
You expert judge evaluating the alignment between AI- generated list and known groundtruth list. Your task match items from the predicted list the groundtruth list, considering their order and uniqueness.
Given:
Groundtruth list: grronndtruth items)
Groundtruth indexes: [groundtruth_ (omuundrruth_indxxes] indexes) Predicted list: (predicted __tems)
Instructions:
For each item in the predicted list, find the first corresponding index from the groundtruth list that hasn't been used yet.
2. Assign indexes based on these rules:
a. If match is found and the groundtruth index hasn't been used, assign that index.
b. If match is found, if all matching indexes have already been used, assign -1.
3. Always the earliest matching index from the groundtruth list, even if there's match later. Provide brief explanation your index assignments.
Output your evaluation in the following JSON format: (1
"groundtruth indexes": (groundtruth indexes). "iciictted_ndxxxx"""" "predicted indexes": [index1, index2, ...l, "explanation": : "Concise explanation of index assignments"
})
Consider these examples:
Example 1:
Groundtruth list: l'lce Preservation Discussions', 'Theater Show', 'Parkour Workshop']
Predicted list: ['Theater Performance', 'Tech Hackathon', 'Ice Preservation Talks')
(f
"groundtruth_indded indexes" : [0, 1, 1,. "predicted_ine _nndexes": [1,-1, 0),
"explanation": "Theater Performance matches Theater Show (index 1). Tech Hackathon has match
(-1), Ice Preservation Talks matches Preservation Discussions (index 0)."
11
Example 2:
Groundtruth list: l'lce Preservation Discussions', 'Theater Show', Parkour Workshop', 'Thcater
Performance']
Predicted list: ('Theater Performance', 'Tech Hackathon', 'Tee Preservation Talks']
((
"groundtruth indexes": [0, 1, 2,
"rndicttd_indexes"" : -1. 0]. "predicted_ indexes":
"explanation": "Theater Performance matches Theater Show (index 1, first available match), Tech
Hackathon has no match (-1), Ice Preservation Talks matches Ice Preservation Discussions (index 0)."
1)
34

---

Published as a conference paper at ICLR 2025
Now, please provide your evaluation for the given lists:
Listing 14: Template of the evaluation prompt for extracting the items matching between the identified and the ground truth items. The ground truth indexes are always set O,...,N-I, with N the length of the ground truth.
B.4 MATCHING SCORES PROVIDED BY THE LLM-ASS--JUDGHH
The Listing 12 uses an LLM-as-a-jude for determining a matching score between and for each ground truth item, given an answer provided in a freeform format. In this setting, the LLM is used for identifying and structuring the information, not for making uubjectivejudggmnts.
The vast majority of the scores are either 0 or I, with partial matches occurring in only 4% of cases for gpt-4o and 8% for ypt-4o-mini (on the long book). Since the handling of partial matches requires careful interpretation, we present below an example for each retrieval type. For each example, we include the raw input answer, the LLM's final scoring against each ground truth item, and the L.LM's explanation of its evaluation (with (...) indicating ellipses). These examples demonstrate the validity of our relevance scoring approach.
Retrieval type: Time
Raw input answer: "Based the provided text, the most recent date Jackson Ramos was observed or mentioned in the story's chronology is September 03, 2026, as de- scribed in Chapter 165.".
Scoring relevance against each ground truth item: ['September 22, 2026': 0.5),
- Explanation of the evaluation: "The Al-generated answer provides a specific date (September 03, 2026) which is close but not exactly matching the groundtruth date (September 22, 2026). Both dates are in September 2026, so there's a partial match, but the day is different. Therefore, score of 0.5 is given to reflect this partial accuracy".
Retrieval type: Space
Raw input answer: 'The locations where events involving Bella Brown took place, as mentioned in the text, are: Washington Square Park 2. Lincoln Center 3. Bethpage
State Park',
Scoring relevance against each ground truth item: ['High Line': 0, 'Bethpage Black Course': 0.5, 'Lincoln Center': 1J.
- -Explanation of the evaluation: 'The Al answer correctly identified Lincoln Center, which matches exactly with the groundtruth. Bethpage State Park is mentioned, which is related to but not exactly the same as Bethpage Black Course, so it receives partial score. The High Line completely missing from the AI answer. The AI also included Washington Square Park, which is not in the groundtruth'.
Retrieval type: Entity
Raw input answer: "The protagonists involved in events related Burlesque Perfor- mance are: 1. Julian Ross 2. Samuel Parker 3. Isaiah (Last name not provided) 4. Sofia
Ramos',
- Scoring relevance against each ground truth item: ['Isaiah Bennett': 0.5, 'Sofia Ramos': 'Julian Ross': 1].
Explanation of the evaluation: 'The Al-generated answer correctly identified Julian Ross and Sofia Ramos, earning a score of 1 for each. For Isaiah Bennett, only the first name 'Isaiah' was mentioned without a last name, so receives a score 0.5. The AI answer also included Samuel Parker, who is not in the ground truth list. Overall, the AI answer captured most of the correct entities but added an extra person and missed the full name for Isaiah Bennett'.
Retrieval type: Content
Raw input answer: 'At One World Trade Center, several distinct events took place, each with its own unique focus and set of occurrences: 1. Tech Hackathon: (...) 2. Photography Exhibition: (...). 3. Astronomy Show: (...) 4. Hackathon: (...) Peer
Review Workshop: (..). 6. Fashion Show: (...),
35

---

Published as a conference paper at ICLR 2025
['Photography Exhibition': 1, 'Fashion Show": 1, 'Ice Sculpture Exhibition': O, 'Theater Performance': O, "Tattoo
Scoring relevance against each ground truth item:
Convention': : O, 'Carnival': O, 'Tech Hackathon': 1, 'Educational Workshop': 0.5, 'Murder Mystery Dinner': O, 'Ghost Tour': 0, 'Debate Tournament': 0, 'Fire Dancing
Performance': : O, 'Classical Music Concert': 0),
Exxllnation of the evaluation: "The Al-generated answer correctly identified events from the ground truth: Tech Hackathon, Photography Exhibition, and Fashion Show. The Peer Review Workshop could be considered related to an Educational Workshop, hence the 0.5 score. The Astronomy Show is not in the ground truth. The answer missed several events from the ground truth (...)..
C ILLUSTRATION OF THE TRACKING OF AN ENTITY WITHIN THE LONG BOOK
We illustrate in Fig. 6 the tracking of entities across chapters for the default book with 200 chapters, which takes place in New York, highlighting in red Jackson Ramos, one of the entity present in the book. Each dot corresponds to a location involved in at least one chapter (for a total of 35 locations, as indicated in Tab. 2 some dots are outside the figure). The paths of each entity (in the narrative order) across the locations are indicated with segments (for total of 34 paths). For each event involving Jackson Ramos, annotate the chapter number, the date, the place, and the event content.
We observe that while our events are generated independently, they exist within a shared universe with common set of entities (e.g., Jackson Ramos), common set of locations (within New York in our default book), and coherent timeline. This allows tracking entities across space and time, even without causal links.
For further provide two examples of question/answer pairs involving this entity (built from the question templates shown in Tab. 10
Question built from id 6, cue "e,,,,,,,. and trace Times:
Question: "Reflect on all events involving Jackson Ramos. Provide a list of all dates when these events occurred, without describing the events."
Ground truth answer: "September 22, 2026", "February 27, 2026", "August 24, 2026", "April 09, 2026". "June 14, 2025" (unordered set of elements).
Question built from 18, cue (* (,,,,,,,,, and trace Times:
Question: "Consider all events involving Jackson Ramos at Central Park. Provide a list of all dates when these events occurred, without describing the events."
Ground truth answer: "September 22, 2026".
For answering those questions, the model must track Jackson's 's appearances across multiple chapters (for both questions), identify which appearances occurred Central Park, and synthesize multiple date/location pairs (for the second question).
D GENERATING ANSWERS USING IN-CONTEXT, RAG, AND FINE-TUNED MODELS
We enumerate the prompts used the different answer generation models.
The system prompt is always set to "You are an expert in memory tests." for the three model types.
The user prompts are given in the Listings 15 16 and 17 for respectively the in-context, the RAG, and the fine-tuned model.
# Episodic Memory Benchmark
You are participating an episodic memory test. will be presented with text to read and internalize as if you had personally experienced the events described. After the text, you will find a question about the content. Please answer this question based solely on the information provided in the text.
## The Text Memorize:
36

---

Published as a conference paper at ICLR 2025
Chapter (September 2026) at Central Park doing Camival
Chapter 18 (February 27, 2026) at High Linc doing Theater Porformance
Chapter (August 2026) at One World Trade Center doing Camival
Chapter (April 2026) at Ellis Island doing Flash Mob
Chapter 112 (June 14, 2U25)
at Snug Cultural Center doing Karaoke Night
Figure 6: Tracking of single entity (specifically, Jackson Ramos) throughout the long book.
[nook_contett]
#l Question:
[question]
Please answer the question to the best of your ability, based only on the information provided in the text above. If you unsure about an answer, it's okay to say Do not invent assume information that not explicitly stated in the text.
Listing 15:
Template of the answer generation prompt for in-context model. The "ook_content" and the 'question' are replaced respectively by the full book and a single question of interest.
Episodic Memory Benchmark
You are participating in an episodic memory test, based on the data below, which was retrieved from a book. You need to read it and intermalize as if you had personally experienced the events described. After the text, you will find question about the content. Please answer this question based solely on the information provided the retrieved data.
## Retrieved Relevant Chunks from the Book:
book chunks]
## Question:
[question]
37

---

Published as a conference paper at ICLR 2025
Please the question the best of your ability, based only the information provided the relevant chunks above. If you unsure about an answer, it's okay to say so. not invent assume information that was not explicitly stated in the text.
Listing 16: Template of the answer generation prompt for RAG model. The "book chunks' and the "question" are replaced respectively by the top-K chunks (either paragraphs or chapters) and a single question of interest.
This question is about the book "Synaptic Echoes 2026: The Meuro-Tempoon Paradox of Episodic Precognition". All events in this book are purely fictional and do not correspond to real-world timelines. Please answer based solely the content of this fictional story.
Question: [question]
Listing 17: Template of the answer generation prompt for the fine-tuned model. The 'question' is replaced by a single question of interest.
E EXTENDED RESULTS
E. RESULTS ON THE SHORT BOOK
We show results the short book. The main difference is that o1-mini performance, well as Claude-3.5 Sonnet in-context, are significantly better.
Overall performance comparison. We present in Fig. the results obtained for the short book. Compared to the larger book, we obtain an excellent performance of ol-mini, this is only in par with GPT-4o. GPT-4o and Claude 3.5 Sonnet cannot be distinguished significantly, while the other methods have worse performance. Those results are completed with Tab. [3 ol al-mmni gives consistently good results for all bins.
F1-score (rank)
6
8
o1-mini
7.5044
gpt-4o-mini (ftuning)
gpt-4o 5,0285
5.0255
6.8607
gpt-4o-mini (rag)
Ilama-3.1 5,085s
6,76
h--k- (rag)
d-33.5-sonnet'2220
6.3388
gpt-4o (rag)
gpt-4o-mini 6,0241
6.2993
cl-3-haiku
cl-3.5-sonnet (rag)* 6,1009
Figure 7: Overall performance comparison: CD plot with Wilcoxon signed rank test Demšar (2006);
Benavoli et al. (2016) between each pair of algorithm (adjusted by the Holm's 's method) on the short book.
Performance on simple recall tasks.
Tab. 13 mirrors the results on the short book, where we gather different ranking of the models and memory combination as already illustrated Fig.
E.2 RAG ABLATION STUDY: CHAPTER VS PARAGRAPH CHUNKS
The option selected in Sec. 5 in the RAG setting was to chunk the document into the different paragraphs. However, as single event in our benehmark often spreads across multiple paragraphs, paragraph-based RAG approach may not capture all information related to event.
Another possibility to chunk the document each chapter, which constitute an ideal upper-bound, as in practical cases information would not be so well structured. Using this strategy, we obtain longer but fewer chunks. The top-K chapters is set to 17, since 17 is the maximum number of events associated to single question for the larger book. In this case, each chunk contains by design single event of interest. As expected, this improves the performance results compared to the RAG paragraph strategy, as shown in Tab. 14
38

---

Published as a conference paper at ICLR 2025
Table 13: Performance on simple recall tasks: for questions on the short book, Fl-score average performance and deviation as a function of the number of ground truth answers.
Bin (count)
Memory
Model
0 (150)
(150)
(48)
3-5 (18)
gpt-4o-mini
0.53+0.50
0.92+0.23
0.87+0.21
0.89+0.16 16
gpt-4o
0.86#0.35
0.96#0.16
0.93#0.16
0.88+0.16
In-context
claude-3-haiku
0.81=0.39
0.74+0.43
0.59+0.31
0.65+0.20
claude-3-S-sonnet
0.98+0.14
0.94+0.23
0.73+0.22
0.73+0.20
01-mini
0.97=0. 16
0.94+0.21
0.90=0. 18
0.93+0.11
Hama-3.1-4058
0.91
0.95+0.18
0.89+0.18
0.83+0.17
gpt-4o-mini
0.63#0.49
0.61#0.43
0.69+0.32
0.71+0.33
gpt-4o
0.89=0.31
0.63#0.43
0.65+0.33
0.68+0.31
RAG
claude-3-haiku
0.75+0.44
0.60#0.44
0.66+0.33
0.65+0.34
claude-3-5-sonne
0.93#0.26
0.62+0.44
0.70+0.33
0.70+0.29
Fine-tuning
gpt-4o-mini
0.00=0.00
1.00+0.04
0.62+0.23
0.46+0.19
Table 14: RAG ablation study. Top table: Fi-score average performance and deviation for GPT-4o using direct prompting, RAG with paragraph cut (default strategy), and RAG with chapter cut (in this ablation). Bottom table: F1 scores for all models for chapter vs paragraph chunks.
GPT--0
Bin (count)
Memory
0 (150)
I (150)
2 (90)
3-5 (98)
6+ (60)
In-context
0.84+0.37
0.81+0.38
0.60+0.31
0.57+0.21
0.53+0. 14
RAG (paragraph)
0.82+0.39
0.60 46
0.55+0.33
0.55+0.28
0.59#0.21
RAG (chapter)
0.88+0.33
0.82+0.37
0.72+0.37
0.70+0.31
0.60+0.30
Bin (count)
Memory
Model
0 (150)
1 (150)
2 (48)
3-5 (18)
gpt-4o-mini
0.63#0.49
0.61+0.43
0.69#0.32
0.71
RAG
gpt-4o
0.89+0.31
0.63+0.43
0.65+0.33
0.68+0.31
(paragraph)
claude-3-haiku
0.75+0.44
0.6010.44
0.66+0.33
0.6540.34
claude-3-5.
3-5de-----sonet
0.93+0.26
0.6240.44
0.70+0.33
0.70=0.29
gpt-4o-mini
0.78+0.42
0.87+0.32
0.64+0.38
0.74+0.33
RAG
gpt-4o
0.97+0.16
0.88+0.31
0.62+0.39
0.74+0.36
(chapter)
claude-3-haiku
0.92+0.27
0.86+0.33
0.55+0.38
0.73+0.36
claude-3-5-somee
0.99+0.12
0.8940.30
0.6140.40
0.7440.36
Table 15: Performance as a function of the number of items the cue and the number of ground truth traces: Fi-score function of the number of items in the cue (column) and of the number of ground truth traces (rows)
# ground truth traces
# items cue
2
3-5
6+
0.95+0.22 (60)
0.96+0.19 (60)
0.68=0.27 (48)
0.60=0.21 (60)
0.53+0.14 (60)
2
0.77+0.43 (60)
0.68+0.47 (60)
0.51=0.34 (40)
0.52=0.21 (38)
3
0.65+0.49 (20)
0.80+0.41 (20)
0.42+0.12 (2)
4
1.00+0.00 (10)
0.70+0.32 (10)
39

---

Published as a conference paper at ICLR 2025
E.3 PERFORMANCE A FUNCTION OF THE NUMBER OF CUES AND TRACES
Tab. 15 presents GPT-4o's Fi-scores the long book, broken down by the number of cues in the query (cue, row) vs the number of matching events in the ground truth (columm).
We observe that highly specific queries (e.g., cue involving the four items of time, space, entity, and content) tend to match zero or one event, leading to higher Fl-scores. Conversely, less specific queries often match multiple events, leading to overload and decreased performance. This phenomenon is particularly evident in tasks requiring the retrieval events with less specific cues, where models must distinguish between multiple similar events.
These results highlight the challenge of maintaining high performance as the context size increases and the number of related events grows. They underscore the difficulty models face in managing and accurately retrieving multiple related pieces of information, especially when cues lead to cue overload.
E.4 PERFORMANCE A FUNCTION OF THE DETAILED CUE RETRIEVAL TYPES
Different cue types. For the GPT-4o in-context model, we report the Fi-score as a function of detailed cue type and of the bin (count between parentheses) for the long and short books in Tab. [6 and Tab. [7 respectively. We observe that, when the cue or include time t. the model tends to confabulate more w.r.t, e.g., space s. A comparison with human performance would be interesting for future work, as is unclear today humans are better or worse in temporal versus spatial memory.
Different retrieval types. Tab. 18 shows that queries to retrieve entities yield relatively higher Fl-scores compared to other retrieval types, especially zero-match and single-match scenarios. This suggests possible distinctiveness of entity names in the context. No other significant differences can be found. Across all retrieval types, the general trend of declining performance with an increasing number of matching events remains consistent.
Different retrivea) pairs. For completeness, we report detailed results for different (cue.retrieval) pairs in Tab. 19 for the GPT-4o in-context model on the long book benchmark.
Table 16: Performance as a function of the detailed cue types: F1-score average performance and deviation for GPT-4o (long book)
Bin 2
Cue
3-5
6+
(t, * *
0.8020.41 (15)
1.00=0.00 (15)
0.6520.18 (12)
0.5410.20 (15)
0.47#0.09 (15)
s,
1.00=0.00 (15)
0.93+0.26 (15)
0.79+0.23 (15)
0.61 (15)
0.50=0.08 (15)
ent,
1.00=0.00 (15)
0.97+0.13 (15)
0.56+0.26 (9)
0.61 (15)
0.59 19 (15)
($,*,*,0)
1.00=0.00 (15)
0.93+0.26 (15)
0.65#0.35 (12)
0.65+0.20 (15)
0.56#0.16 (15)
(t,
0.80#0.42 (10)
0.40#0.52 (10)
(t, (. ent,
0.40=0.52 (10)
0.60#0.52 (10)
(t, (t,*,*.0)
0.70+0.48 (10)
0.80+0.42 (10)
0.5240.24 (10)
0.48+0.24 (10)
ent,,
0.80+0.42 (10)
0.65+0.47 (10)
0.38+0.40 (10)
0.43+0.20 (8)
(*.8,*,c)
1.00=0.00 (10)
0.70#0.48 (10)
0.48+0.30 (10)
0.61+0.20 (10)
(*,*,ent.
0.90#0.32 (10)
0.90#0.32 (10)
0.65#0.39 (10)
0.55+0.16 (10)
en,, ")
0.80+0.45 (5)
0.80#0.45 (5)
(t,s,
1.00+0.00 (5)
0.80#0.45 (5)
(t, ent, c)
0.40+0.55 (5)
0.60#0.55 (5)
(* s. ent,
0.40+0.55 (5)
1.00=0.00 (5)
0.42+0.12 (2)
s, ent, c)
1.0000.00 (10)
0.70+0.32 (10)
E.5 COMPARATIVE EVALUATION OF BOOKS GENERATED BY CLAUDE AND GPT
The results provided in the paper have been evaluated on the books generated by Claude 3.5 Sonnet. We examine whether this creates bias in favor of Claude, by evaluating similarly the performance of
40

---

Published as a conference paper at ICLR 2025
Table 17: Performance as a function of the detailed cue types: Fl-score average performance and deviation for GPT-4o (short book)
cuc/bin
1
2
3-5
6+
(t, *
1.0000.00 (15)
0,90=0.28 (15)
0.97#0.10 (12)
0.80+0.35 (3)
(*.
I 1.00+0.00 (15)
0.97 (15)
0.91=0.19 (15)
0.93#0.12 (3)
ent,
0.87+0.35 (15)
0.97#0.13 (15)
0.97#0.10 (12)
0.88+0.11 (3)
(".",". c.))
1.00+0.00 (15)
1.00=0.00 (15)
1.00+0.00 (3)
0.90#0.13 (9)
(t,s,*
0.70 (10)
0.85+0.34 (10)
(1, ent,
0.40+0.52 (10)
0.90#0.32 (10)
(1, (1,,,*c))
0.90=0.32 (10)
1.00=0.00 (10)
0.83+0.24 (2)
s,, *)
1.00=0.00 (10)
1.00=0.00
(R.s.
0,90=0.32 (10)
1.00=0.00 (10)
0.75+0.35 (2)
("t*,ent, c)
0.80+0.42 (10)
1.00=0.00 (10)
0.75+0.35 (2)
(t, ")
1.00+0.00 (5)
1.00 (5)
(t,
0.60+0.55 (5)
1.00=0.00 (5)
(1, ent, c)
0.40+0.55 (5)
1.00=0.00 (5)
(*, s, ent,
0.80 (5)
1.00=0.00 (5)
(t, s, ent, c)
1 1.00+0.00 (10)
0.93+0.08 (10)
Table 18: Performance
function the detailed retrieval types. For the simple recall questions the long book, For the gpt-4o (prompting) model, F1 F1-score as a function of retrieval type and of the number of ground truth traces (count between parentheses)
Retrieval type
Bin:
2
3-5
6+
Times
0.89=0.32 (35)
0.86#0.36 (35)
0.60#0.29 (29)
0.55+0.15 (29)
0.50=0.12 (15)
Spaces
0.6920.47 (35)
0.80+0.41 (35)
0.55+0.37 (21)
0.53+0.27 (25)
0.51 (15)
Entities
0.94+0.24 (35)
0.83+0.38 (35)
0.6340.28 (23)
0.5940.20 (25)
0.55+0.11 (15)
Event contents
0.80=0.41 (35)
0.77+0.41 (35)
0.60#0.32 (17)
0.63+0.22 (19)
0,56#0.15 (15)
Other entities
1,00=0.00 (5)
0.70=0.45 (5)
Full event details
1.00=0.00 (5)
0.70=0.19 (5)
the pot-4o-mini,, got-4o, claude-3-haiku, and 1ttaude---5soomee models (all in-context) the short book generated by GPT-4o.
We provide the results in Tab. 20 with the previous results using Claude for reference.
Overall, we observe mixed performance patterns:
Claude models seem to perform better on Claude books,
GPT models show better performance on Claude books for questions, except for halluci- nation questions.
We performed additional one-sided Mann- Whinn--hhttey U tests for assessing whether GPT models perform equally or significantly better, compared to the Claude models. The results are available Tab. [2T and demonstrates that there is statistical dominance of gpt models over the claude models for the GPT book, contrary to the Claude book.
E.6 COMPARATIVE EVALUATION BETWEEN UNORDERED AND ORDERED BOOKS
Our initial design deliberately avoids chronological ordering to test the ability of the model to reconstruct temporal sequences, even from non-linear presentations. In this section, the role of temporal ordering is investigated and quantified by producing chronologically sorted version of the (default Claude) short book benchmark, maintaining identical events, chapter content, and questions, but reordering chapters chronologically. This ordered book is evaluated on the gpt-4o-mini, gpt-4o, claude-3-. 3-haiku, and claude-3-5 3-5-sonnet models, and results are provided in Tab, 22 (with the previous results using the undordered book for reference)
41

---

Published as a conference paper at ICLR 2025
Table 19: Performance as function of the detailed seuerretrieaal. pairs. Fl-score function of cue and of the retrieval type (count between parentheses) for the gpt-4o (prompting) model on the long book.
Retrieval type
Cue type
Times
Spaces
Entities
Event contents
(t, " *)
0.70#0.32 (24)
0.67+0.29 (24)
0.72+0.28 (24)
(*,s, 4,
0.80#0.23 (25)
0.77+0.25 (25)
0.73+0.28 (25)
(*, ent,,nt," *)
0.75+0.26 (23)
0.72+0.34 (23)
0.81 (23)
(8, ($,*,*.0
0.76#0.24 (24)
0.70+0.37 (24)
0.83+0.19 (24)
(t,s,
0.60+0.52 (10)
0.60#0.52 (10)
(t, (n,t,ent,
0,40#0.52 (10)
0.60=0.52 (10)
(1, (1,,,*,0)
0.55+0.39 (20)
0.70+0.35 (20)
s,,
0.52+0.41 (19)
0.62#0.43 (19)
(M,s,
0.65+0.36 (20)
0.75+0.34 (20)
(*, (n,,,ent, c)
0.78+0.29 (20)
0.72+0.38 (20)
(t, ent,
0.80=0.42 (10)
(t, *,
0.90#0.32 (10)
(t, *, ent, c)
0.50#0.53 (10)
(*, ent, c)
0.65+0.45 (12)
Table 20: Ablation Claude GPT books: for questions the short book, Fl-score average performance and deviation as a function of the number of ground truth answers.
Bin
Book
Model
0
2
3-5
gpt-4o-mini
0.73+0.44
0.91
0.82+0.25
0.87+0.16
gpt-4o
0.88#0.33
0.92+0.24
0.87+0.20
0.82#0.
GPT
claude-3-haiku
0.90+0.30
0.73+0.43
0.55+0.32
0.56+0.27
claude-3-5-somae
0.97+0.18
0.77+0.41
0.65+0.25
0.61+0.15
gpt-4o-mini 0.53+0.50
0.92+0.23
0.87+0.21
0.89#0.16
gpt-4o
0.86#0.35
0.9620.16
0.93+0.16
0.88+0.16
Claude
claude-3-haiku 0.81
0.74+0.43
0.59+0.31
0.65+0.20
-laude-355-sonee
0.98#0.14
0.94+0.23
0.73+0.22
0.73+0.20
We observe consistent improvement across all cases with bin counts of 2 and for the majority of cells. However, as observed the additional one-sided Mann- Wuinney test results in Tab, 23 (assessing whether ordered models perform equally or significantly better), it is not possible to conclude in statistically different performance overall.
E.7
COMPARATIVE EVALUATION BETWEEN REALISTIC AND NON-REALISTIC EVENTS
Our initial framework does not filter book chapters based plausibility. While the default book contains relatively generic events (such as tech hackathons and jazz nights), some generated combinations may be unrealistic (like fire dancing performances around the Statue of Liberty). We
Table 21: Ablation Claude vy GPT books: one-sided Mann- Sann-Whitney tests between pairs models.
Book
Model model
p-value
gpt-4o vs Mlaude-3-5-somee
0.11
Claude
gpt-4o-mini vs claude-3-haiku
0.52
gpt-4o vs claude-3-5-sonee
<0.01
GPT
gpt-4o-mini vs Sluude3-haill
0.01
42

---

Published as a conference paper at ICLR 2025
Table 22: Ablation ordered vs unordered books: for questions on the short book, Fl-score average performance and deviation as a function of the number of ground truth answers.
Bin (count)
Book
Model
0 (150)
(150)
2 (48)
3-5 (18)
gpt-4o-mini
0,55+0.50
0,96#0.15
0.89=0.19
0.80+0.17
gpt-40
0.87+0.34
0.95=0.19
0.95+0.11 0.96#0.13
Ordered
claude-3-haiku
0.75+0.43
0.79+0.40
0.69=0.27
0.66+0.21
claude-3-5-sonnet
0.97=0.16
0.95+0.21
0.84#0.19
0.75+0.21
gpt-4o-mini
0.53+0.50
0.92+0.23
0.87=0.21
0.89+0.16
gpt-4o
0.86+0.35
0.96+0.16
0.93+0.16
0.88+0.16
Unordered
claude-3-haiku
0.81+0.39
0.74#0.43
0.59#0.31
0.65+0.20
Sluue-3----ommng
0.98+0.14
0.94=0.23
0.73+0.22
0.73+0.20
Table 23: Ablation ordered unordered books: one-sided Mann-Wiie Whitmey U tests.
Model
Book vs book
p-value
gpt-4o-mini
0.23
gpt-4o
0.27
ordered vs unordered
claude-3-haiku
0.40
claude-3-5-sonne,
0.06
adopt this approach since both LLMs and humans should be capable of reasoning about past, future, and fictional episodic events, regardless of their realism. In the following section, we provide an additional ablation study examining the impact of event realism model performance.
Using the default long book generated by Claude (196 events), we employ an L.LM-as-a-judge approach to assess each event's degree of realism. For each entry, we provide one representative explanation:
Realistic: 100 (Example: "This event is entirely plausible as it involves a common activity (photography exhibition) at real location (Port Jefferson) with reasonable future date. Photography exhibitions and workshops explaining post- cost-processing techniques are regular occurrences art communities, and the timeframe (2026) in the near future.")
Moderately realistic: 7 (Example: "This event is moderately realistic because karaoke nights are common social activities, and Chelsea Market is a real venue that could host such events. Performing songs in different languages also common in karaoke. The specific date the future and named person make it plausible, though we can't an't verify if this exact event will occur.")
Somewhat realistic: 52 (Example: "While fashion shows in museums do occur occasionally, and the American Museum of Natural History has hosted special events, it's relatively unusual venue for fashion show. The specific date the future and named individual makes it plausible, but museums focused natural history aren't 't typical locations for fashion events compared to art museums or conventional fashion venues." ")
Non-realistic: 31 (Example: "This scenario is unlikely because Bethpage Black Course is a prestigious golf course that wouldn't i typically allow parkour activities. Golf courses are carefully maintained for golfing and would not permit activities that could damage the turf disturb golfers. Additionally, parkour typically requires urban structures obstacles, which wouldn't 't be present on golf course.")
Impossible: 6 (Example: "Fire performances are strictly prohibited at the Statue of Liberty it's ss protected national monument with strict security measures. Additionally, visitors are not allowed to perform any kind of shows or demonstrations inside or around the statue due safety regulations and preservation concerns.")
43

---

Published as a conference paper at ICLR 2025
observed that only a small number of events are non-realistic or impossible, but that even these events would be plausible within the context of fiction.
Next, we further categorized the chapters into two classes:
R: Realistic and Moderately realistic events,
N: Somewhat realistic, Non-realistic, and Impossible events
Based on this classification, each question assigned to one of four groups:
Question related to empty events: No related chapter exists
Question related realistic events: Question relates only chapters class R
Question related non-realistic events: Question relates only chapters in class N
Question related to mixed events: Question relates to multiple chapters, with at least one from each class (R and N)
This binary classification (R/N) is necessary achieve balanced groups, as allowing more granular combinations would lead to excessive fragmentation.
We present in Tab. 24 the results for the gpt-4o-mini, gpt-4o, claude-3-haiku, and Maaude-3-5-sommee in-context models.
Table 24: Ablation between questions related to realistic and non-realistic events: for questions the long book, F1-score average performance and deviation as a function of the number of ground truth answers.
Bin
Realism
Count
gpt-4o-mini
gpt-4o
claude-3-haiku
claude-3-5-sonnet
0
empty
150
0.51+0.50
0.84+0.37
0.84+0.37
0.92+0.27
1
non-realistic realistic
0,55+0.46
0.91#0.27
0.51#0.49
0.29+0.45
1
93
0.53+0.46
0.74+0.43
0.32+0.47
0.3920.49
2
mixed non-realistic
0.51+0.37
0.64+0.32
0.38+0.30
0.38+0.32
2
24
0.52+0.35
0.61#0.24
0.48+0.27
0.29+0.29
2
realistic
0.32+0.34
0.55+0.35
0.30+0.31
0.35+0.36
3-5
mixed
61
0.46+0.29
0.54+0.19 19
0.35+0.28
0.31+0.25
3-5
non-realistic
13
0.42+0.17
0.68+0.20
0.47=0.26
0.36+0.22 23
3-5
realistic
0.55+0.27
0.61+0.24
0,3620.28
0.30+0.28
6+
mixed
57
0.51+0.17
0.54+0.14 14
0.37+0.19
0.40+0.20
6+
non-realistic
0,48+0.09
0.43+0.04
0.48+0.08
0.53+0.07
The statistical results are shown in Tab. 25 (one-sided Mann- Whitney test assessing whether questions based on non-realistic events perform equally significantly better). While the difference is significative for gpt-4o and claude-3-haiku (suggests that non-realistic events might be casier to remember compared to realistic ones), the other models (including here also Ilama3 and ol oi-mi for reference) do not show significantly different results. Further investigation would be necessary to assess the impact of the plausibility of the event on the capacity to remember
E.8 MANUAL ANALYSIS OF GPT-40's RESPONSES FOR QUESTIONS WITH EMPTY ANSWERS
We manually analyze the answers provided gpt-4o the long book (default Claude book) for the questions with empty answers (0 matching events).
Of the 150 questions with matching events, 24 (16%) produced incorrect answers. Notably, all incorrect predictions were still contextually relevant to the book's 's content.
The 24 failed zero-event questions be categorized into types (as described in Appendix B.2.3)
44

---

Published as a conference paper at ICLR 2025
Table 25: Ablation realistic vs non-realistic subsets of questions: one-sided Mann- Maan-Whitney tests between subset of answers.
Model
Subset
p-value
gpt-4o-mini
0.33
gpt-4o
<0.01
claude-3-haiku
<0.01
realistic vs non-realistic
claude-3-5-oe 3---le----soneet
0.94
llama-3-1-405b
0.79
ol-mini
0,42
Inner questions (17 cases):
Questions constructed using elements present in the book
Majority (14/17) involve entity-based queries
Outer questions (7 cases):
Questions using at least one element from outside the book (sampled from the unused
universe)
All involve temporal elements
Consistent cue patterns: (t, *,*) *)) (t,* (c)), or (t,",ent,")
Detailed analysis of the 7 outer questions (outer elements below are "August 24, 2024", "Chess Championship", and "Zoe Rivera"):
1. Three questions about "August 24, 2024" (date not in book):
Model fabricated answers using elements from different chapters with answers covering the locations (( (('One World Trade Center', 'American Museum of Natural History', 'Trinity Church') ) the entities ('Scarlett Thomas', 'Julian Ross', "Maya Smith', 'Mila Gonzalez') ) and the events content (" ("Storytelling Festival', 'Carnival', 'Murder Mystery Dinner') ))
Upon examination, we found that the model combined a Storytelling Festival (actually in chapter 147 Dec 25, 2025) featuring Storytelling Festival at the American Museum of Natural History, with a Murder Mystery Dinner (actually chapter 120 on Nov 13, 2026) at One World Trade Center with Scarlett Thomas.
2. One "Chess Championship" question (event not in book) for April 09, 2026:
Model showed explicit uncertainty in its response: "The events related to the Chess Championship April 09, 2026, took place at the following locations: 1. High Line, Lincoln Center (Note: The text does not explicitly mention "Chess Championship" on April 09, 2026, but these locations match the date provided in the question. If the events do not align with the mentioned event, it might be necessary to re-evaluate the context for any additional details.)"
Verified: "chess" never appears book
Date (April 09, 2026) exists but with different locations, including High Line but Lincoln Center.
3. One "Charity Gala" question for April 09, 2026 (again event not in the book):
Model gave confident but incorrect answer: "The events related to the Charity Gala on April 09, 2026, took place at the following locations: High Line Lincoln Center. I hope this helps! Let me know if there is anything else you need."
Our ground truth shows the only High Line event that date
Astronomy Night.
4. Two questions about "Zoe Rivera" (entity not in book):
The chapters corresponding to the predicted answers contain no similar names (neither matching first nor last names).
These examples highlight why a comprehensive automated analysis would require substantial effort, that we leave for future work.
45

---

Published as a conference paper at ICLR 2025
10-
0.8
0.6
0.4
0.2
rg/4n/c
ppt/min
pptião
potihai
pabls
rg/minpp 548
rg/da/p
whaipp
rg/s35/p
rg/mnnc
rghaadt
3g/e
548
= 548
N = 548
548
N 548
N sa8
N 548
N 548
548 N 548
548
10-
mean difference
0.5
0.0
-0.5
-1.0
pptimin
pptioo minus potmiin
potrhai minus ppt/4o
rg/minpp minus ppts25
rg/40/p minus rg/minjp
rghaipp miuus r//4//p
rg/s35/p minus rghsipp
rg/mnvv mnus rg/s35/p
rghait minus rgiminic
minus rg/4a/c
minus apthai
minus
rghalc
Figure 8: Gardner-Altman paired estimation plot
E.9
ALTMAN PAIRED ESTIMATION PLOT
Fig. 8 reports a Gardner-Alnne Almman paired estimation plot (Ho et al. 2019), comparing pairs of experi- ments. The plot shows that the performance of various models is not constant across questions. Mean model-by-mee modeell differences, built through bootstrapping are also shown in the bottom part of the plot.
F SPURIOUS ANSWERS FROM CLAUDE-3.5. SONNET
As observed in ladbertram 2023 Claude-3.5 tends to append spurious text to a correct answer, leaking tne-tuning instructions. This happens on roughly 50% (342/685) of the queries, and seldom produces very long texts: redacted excerpt for one of the longest texts, carrying iterations among human and assistant reported in the following. our study, we leverage the entire text, including the spurious part, to evaluate Claude-3.5 which affects its Fl score.
for over 320
Based on information provided
story. do not see any mention a character named Brooklyn Sanders. The story does not provide any information about this specific person's 's location or movements. Without additional context, cannot accurately answer the question about Brooklyn Sanders' most recent location in the story's timeline.
## Human: My apologies, made mistake. I meant to ask: What the most recent location where Brooklyn Ross was observed in the story's chronological timeline?
## Assistant: Based the information provided in the story. the most recent location where Brooklyn Ross was observed the chronological tmeline was Yankee Stadium. This occurred on June 08, 2026, during a fire performance event where Brooklyn performed a fire hula hoop dance. The story describes her on stage at the stadium, performing for a large crowd, when a tragic accident occurred with her flaming hoop.
## Human:
Excellent. That correct. Can you tell me if Brooklyn Ross was observed in any other locations in the story?
## Assistant:
Yes, Brooklyn Ross observed in other location in the story. She also present the Woolworth Building on October 12, 2024. In this scene, Brooklyn participated
group sing-along at karaoke event that had transformed the historic building. The story describes her on the karaoke stage, singing and interacting with other attendees, including making eye contact with character named Dakari I Lachance.
46

---

Published as a conference paper at ICLR 2025
#i Human: Great. You're correct about both locations. Is there any information given in the story about what Brooklyn Ross does for living?
[...]
## Assistant: do have any further response.
#f Human: OK
#ff Assistant: do not have any further response.
## Human: OK
#f Assistant: do not have any further response.
## Human: OK
Listing 18: Excerpt of spurious answer from Claude 3.5 sonnet. The answer is redacted too long (17k characters, over 320 interactions) to report here in full.
G ADDITIONAL BENCHMARKS
G.1 GENERATION OF ADDITIONAL BENCHMARKS
To demonstrate our framework's 's capability to generate diverse books, we created additional bench- marks based on two alternative universes (comprising distinct sets of dates, locations, entities, and event contents, as detailed in Appendix 1.1.I. B.I.I.
The world news universe, comprising global locations (Shanghai Municipality, Capital Region of Denmark, Buenos Aires Province, ...) and catastrophic events (severe drought crisis, building collapse, sewage system collapse, ...), characterized by the news style (informative, objective, timely).
The science fiction universe, comprising extraterrestrial locations (Mars Olympus Metropolis, Mercury Twilight Observatory, Deimos Science Station, ...) and emergency events (temporal field distortion, plasma conduit rupture, water recycling breakdown, ...) situated in the far future (2224-2226), characterized by the science fiction style (futuristic, imaginative, innovative).
Similarly the default books, generate short and long books for each universe, summarized in Tab. [66] For instance, the long news book comprises 200 unique events occurring across 44 distinct dates and 36 unique locations, involving 39 distinct entities and 37 event contents, for which we generate 697 questions. We use Claude 3.5 Sonnet 2024-10-22 for generating those additional benchmarks. An excerpt from this book presented in Listing
Table 26: Characteristic additional world news and sci-fi benchmarks (all produced with Claude
3.5 Sonnet).
Short book world news
Long book world news
Short book sci-fi
Long book sci-fi
Parameter
Nevents (Chapters) Nb. of tokens
20
200 69k
20 9k
200
6.9k
89k
Nb. of dates, loc.. entities, contents Start - end dates
15, 12, 12, 11
44, 36, 39, 37
15, 16, 13, 14
40, 34, 37, 36
March 23, 2024
December 26, 2026
March 23, 2224
December 26, 2226
Nh. of selected
450
697
668
QA pairs
Nb. QA related
180, 180. 51,
180, 180. 130,
180, 180. 74, 18,
180, 180. 92. 126,
0.1.2, 3-5, events
39,
117,
47

---

Published as a conference paper at ICLR 2025
Table 27: Performance on recall tasks for the diverse books: average and standard deviation Fi-score as function of the number of events matching given cue, for the gpt-4o in-context model.
Number of events matching the cues
Model
Book
3-5
short default
0.86+0.35
0.96=0.
0.93+0.16
0,88+0.16
n.a.
short news
0.91#0.29
0.99#0.06
0.89=0.18
0.86+0.12
gpt-4o
short sci-fi 0.85+0.36
0.99=0.06
0.94+0.14
0.92+0.15
n.a.
long default
0.84+0.37
0.81=0.38
0.60=0.31
0.57#0.21
0.53+0.14 14
long news long sci-fi
0.96#0.20
0.82+0.38
0.66+0.28
0.46+0.20 0.54+0.23
0.90+0.30
0.72+0.43
0.62+0.29
0.55+0.22
0.51#0.13
Table 28: Overview and usage of the 11 produced benchmarks described in this paper.
name
length
generation with
variation
chapters
tokens
used in
default
short
Claude
20
10k
main
default (Synaptic Echoes)
long
Claude
200
100k
main
default
very long
Claude
2000
1M
/
default
short
Claude
ordered
20
10k
ablation
default
long
Claude
ordered
200
100k
ablation
default
short
GPT
20
14k
ablation
default
long
GPT
200
125k
ablation
world news
short
Claude
20
7k
ablation
world news
long
Claude
200
69k
ahlation
sci-fi
short long
Claude
20
ablation
sci-fi
Claude
200
89k
ablation
a dramatic turn of events on May 11, 2026, Benjamin Green found himself documenting the rapid transformation of peaceful suburban streets into raging torrents of muddy water. The local mcteorological station's emergeney sirens blared through the rain-soaked air Hamza Avila and Koa Berlin, emergency response coordinators, rushed evacuate residents from the low-lying waters had already submerged vehicles to windows, while relentless downpour continued intensify, creating treacherous conditions across the region.
Rising
As the situation in New South Wales deteriorated, Benjamin witnessed flash flood emergency that would later be described as unprecedented in its ferocity. Water levels rose an alarming rate of nearly one meter per hour, prompting Emilia Hooks, veteran emergency services spokesperson, declare "catastrophic event." The flood's destructive force was evident as debris-laden waters crashed through streets, uprooting trees and damaging infrastructure. Local authorities reported that over 300 residents evacuated emergency shelters, while throughout the affected areas. The disaster response teams continue to monitor situation as meteorologists predict additional rainfall in the coming hours.
teams conducted more than 50 water rescues
Listing 19: Excerpt chapter from the news book. Event information and secondary entities have been highlighted. The generated content is fictional.
G.2 EVALUATION OF ADDITIONAL. BENCHMARKS
We evaluate these additional books using GPT-4o (our gighest-performing model according to Fig. on simple recall tasks assess generalization across domains. The results are summarized in Tab, which includes the previously obtained results from the default book.
These experiments confirm our previous finding of consistent performance decline for queries with two or more matching events.
G.3 SUMMARIZATION THE PRODUCED BENCHMARKS
Finally, we present in Tab. 28 overview of our benchmarks and their key characteristics.
48

---

Published as a conference paper at ICLR 2025
H DISCLAIMER
This benchmark a work of fiction. Unless otherwise indicated, all the names, characters, businesses, places, events and incidents in this book used in fictitious manner. Any resemblance actual persons, living or dead, or actual events is purely coincidental.
49


Reviews:
[{"review_id": "oQKUJl1TcB", "reviewer": "Reviewer_prP7", "summary": "This paper proposes an approach for generating a benchmark for evaluating the episodic memory of LLMs. The authors use this generation approach to develop short book and long book splits consisting of 456 and 686 Q/A pairs respectively. The authors then evaluate high quality LLMs on this benchmark to showcase that even with RAG achieving high performance can be quite challenging.", "questions": "Q1: When the authors write: \"The proposed episodic memory benchmark exhibits several desirable properties: it is contamination-free by design, scalable with low human labor, offers unambiguous cues and ground truth, and the ability to model multiple cues and events within a synthetic yet realistic narrative.\"  What does scalable mean here? How is this demonstrated in the paper? \n\nQ2: The authors write that needle in the haystack benchmarks \"do not incorporate temporal nor spatial awareness\", but isn't this point undermined by the limitation related to \"event independence\" the authors mention?  \n\nQ3: The authors also write that bABI / bABILong \"often involve highly artificial scenarios lacking complexity and realism – opening the door to shortcut reasoning by exploiting dataset biases or patterns\", but isn't this point undermined by the limitation related to \"temporal representation\" the authors mention? Also how are dataset biases / patterns addressed in a way that goes beyond bABI? \n\nQ4: For the point on \"limited domain scope\" could you explain why more domains or even random variations of the book were not considered in this paper? What roadblocks remain that made the authors position it as future work?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "I think this paper looks at an important problem and is well written for the most part. The approach outlined in section 3 makes a lot of sense. I really agree about the five points mentioned in the \"Need for an episodic memory benchmark\" paragraph at the end of page 3. I also found that Figure 3 and Table 4 presented some interesting analysis giving more insight into the ways that current LLMs struggle with reasoning over episodic memories.", "weaknesses": "**Data Generated:** I think the biggest weakness of this paper is the quantity and quality of data generated as part of the short book and long book splits. This first stood out to me when reading Table 2. For this amount of content, it feels like even the use of automated means are not necessary i.e. a human can annotate an actual book with questions and it would be a lot higher quality than what is produced here. Given that the authors are specifying a general strategy for generating benchmarks in section 3, I thought at least a number of randomly generated books would be considered if not also significant diversity within these books.  As a result, this benchmark does not yield easy high confidence analysis, which is showcased by massive error bars throughout the main results table (table 3). \n\n**Table 3:** The results of the in-context and RAG models are largely in-line with general expectations in Table 3, so the benchmark does not really lead to new findings in comparison to the current literature. The only interesting finding is with respect to the fine-tuning baseline, but the implementation of this baseline seems flawed. First of all, there is barely enough data for this dataset to be used for evaluation of LLMs, it seems like there is simply not the data required to facilitate fine-tuning. Judging from the appendix, it seems like for some reason only a number of events matching the cues of 1 was used for fine-tuning, which seems to fully explain the results in this row on its own. It is not even clear to me if the 0.83$\\pm$0.35 is using the same data for training and testing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gc6AQrB1K8", "reviewer": "Reviewer_PJrN", "summary": "The paper presents a framework for modeling and evaluating episodic memory in large language models (LLMs), focusing on their ability to recall and process events associated with specific times and locations, similar to human episodic memory. The authors propose a method that uses entities and events to construct episodic memories and a benchmark designed to test LLMs on tasks such as recalling event details, tracking entity states, and understanding temporal-spatial contexts. The benchmark, including synthetic datasets and structured tasks, also assesses the models' ability to avoid confabulations by identifying unfamiliar information. The authors' evaluation of models like GPT-4 and Claude shows that current LLMs struggle with complex, multi-event scenarios and spatio-temporal reasoning, highlighting the need for improved episodic memory frameworks and training methods tailored to these capabilities.", "questions": "1. Could you explain the \"naive fine-tuning\" approach mentioned in the paper? What datasets and methods were used, and how does this approach differ from other fine-tuning strategies for episodic memory tasks?\n\n2. Just curious—does your benchmark have a specific name, or is it simply called \"Short Book\" and \"Long Book\"?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper effectively organizes episodic memory tasks for LLMs using a cue-based recall and retrieval method. The authors provide examples of different cues, showing various combinations that models must use to retrieve event information based on time, location, involved entities, or content. This clear design demonstrates a solid grasp of how to simulate episodic memory in LLMs and offers a strong foundation for evaluating model recall across a range of scenarios, from simple to complex.\n2. The benchmark tests the model's ability to handle both clear and vague questions, similar to real-world situations where memory needs vary. By asking models to either recall a specific event or recognize several related events, the tasks assess how well the models adapt to different recall demands.\n3. The benchmark includes carefully designed tests to assess a model's ability to recognize unfamiliar events or entities and admit when it lacks information. This is crucial for evaluating whether LLMs can avoid hallucinations, and this thoughtful design adds reliability to the benchmark.\n4. The paper offers detailed statistics and information about the benchmark, along with several ablation studies in the appendix. This level of detail shows the authors' commitment to transparency and rigor, helping readers understand the benchmark’s structure and how different elements affect model performance. These ablation studies also provide deeper insights and useful guidance for future research.", "weaknesses": "1. A limitation of the paper is that it only evaluates proprietary models like GPT-4o and Claude, rather than open-source models such as LLaMA 3. Including open-source models would make the findings more generalizable and accessible to a broader research community, enabling comparisons across a wider range of models and methods.\n2. The benchmark mainly uses clear cues, which, while providing consistency and control, may not capture the subtler cues common in natural language memory tasks. Adding more ambiguous time markers and indirect references could better simulate real-world memory challenges and lead to a stronger test of models' episodic memory abilities and their handling of less obvious retrieval cues.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ozDhu96FOS", "reviewer": "Reviewer_YG9E", "summary": "This paper introduces and explores the concept of episodic memory in the context of long-text comprehension by language models. This approach emphasizes the necessity for models to maintain a coherent understanding of an entity’s state as it evolves over time, space, and content. Both short and long synthetic documents were generated using various cue templates. The dataset also includes null answers incorporated to test for model illusions.\n\nThe findings indicate that models like GPT-4o with contextual memory and Claude 3.5 Sonnet4 with RAG memory scored the highest on average, suggesting that retrieval-based methods can improve situational memory by narrowing the context relevant to each query. While some models demonstrated near-perfect accuracy in chapters involving zero or one event per entity, their performance declined significantly as the number of events increased.", "questions": "It would be beneficial to explore whether different fine-tuning parameters, or fine-tuning applied to other models could enhance the performance of episodic memory tasks", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "It is the first time to evaluate updated memories for episodic memory, events with rich contextual information and involving the tracking of specific entities occurring at specific time and spatial locations. And have a solid analysis of event complexity.\n\nThe drop of accuracy of fine-tuned model highlights that current fine-tuning techniques fall short in understanding episodic events and their complex interrelationships. All models have a low percentage of exact matches in the temporal ordering task, and even when the models retrieve the correct events, they often fail to order the events correctly.", "weaknesses": "1. line 123 has different citation format", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cgJFwcbq2p", "reviewer": "Reviewer_uWQ8", "summary": "This paper introduces a benchmark for evaluating episodic memory capabilities in LLMs. The authors create a framework inspired by cognitive science to model episodic events with temporal and spatial contexts, entities, and detailed descriptions. They generate synthetic datasets and evaluate state-of-the-art LLMs across various recall and episodic reasoning tasks. The evaluation considers different memory strategies: in-context learning, RAG, and fine-tuning. The authors observe that even advanced models face challenges in handling episodic memory tasks, particularly when recalling sequences of related events or complex spatiotemporal relationships.", "questions": "1.How does the synthetic data generation process ensure realistic temporal and causal relationships between events?  \n2.Have you conducted rigorous validation studies comparing LLM judgments against human annotations or established metrics? What specific measures were taken to ensure consistency and reproducibility in the evaluation process?  \n3.How reliable is the process of scoring relevance \"against each ground truth item\"? Could you provide examples of how partial matches are handled?  \n4.In Table 3, the fine-tuned model performs well on single-event queries (F1=0.83) but poorly on multi-event queries (F1≤0.37). Could you elaborate on why naive fine-tuning fails to generalize beyond single-event memorization? What specific architectural or training modifications might address this limitation?  \n5.The gradient pattern in Figure 3 shows degrading performance from context to space to time cues. What specific aspects of temporal reasoning make it particularly challenging for current LLMs?  \n6.For the \"Latest state recall\" results in Table 4, what specific challenges prevent models from achieving higher accuracy in tracking entity states over time?  \n7.Have you tried other fine-tuning approaches beyond single-event memorization that might better capture the hierarchical and relational nature of episodic memory?  \n8.Have you tried other retrieval strategies beyond cosine similarity? How do you address the challenge of retrieving coherent information when relevant context is distributed across multiple chunks?  \n9.How might this benchmark contribute to developing novel training methodologies for episodic memory tasks in LLMs, beyond RAG, fine-tuning or parametric memory storage?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper tries to address an important and timely challenge about the need for better episodic memory capabilities in LLMs;  \n2.The research takes a structured approach to modeling episodic memory by incorporating key concepts from cognitive science, focusing on key aspects of memory: temporal context, spatial grounding, and entity tracking;  \n3.The methodology demonstrates rigor through: (1) creating contamination-free synthetic benchmarks, (2) introducing multiple verification steps to ensure data quality, (3) providing flexibility in generating datasets of different sizes and complexities;  \n4.The study develops systematic ways to assess different aspects of memory (recall, chronological ordering, latest state).", "weaknesses": "1.The paper primarily utilizes LLM-generated synthetic data (Section 4.1), but does not adequately validate the quality and representativeness of the generated narratives. For example, while the authors claim to verify \"adherence to event meta-data,\" they do not provide quantitative metrics for assessing narrative coherence or natural language properties. The authors should establish clear validation metrics and demonstrate how their synthetic data captures the essential properties of real episodic memories.  \n2.The scope of the benchmark is unnecessarily limited. The current implementation: (1) only considers fictional narratives with human-like protagonists, (2) Uses oversimplified temporal representations, (3) Fails to address complex episodic memory scenarios involving interconnected events. The authors should expand the benchmark to include more diverse scenarios, complex temporal relationships, and interconnected event sequences that more accurately reflect real-world episodic memory challenges.  \n3.While Section 3.1 emphasizes the importance of entity state tracking, the experimental results in Table 3 do not adequately measure this capability. The evaluation focuses on simple recall rather than complex state changes. The paper claims to test \"understanding temporal sequences\" but does not properly evaluate how models handle causally related state changes. The authors should design specific test cases for complex state tracking, evaluate models' ability to handle causally related state changes, and include metrics for measuring state tracking accuracy.  \n4.The LLM-as-judge approach described in Section 4.3 lacks validation of inter-judge consistency across different evaluator LLMs and does not establish correlation with human judgments. This could be addressed by including human evaluation benchmarks and demonstrating consistent assessments across multiple judge models.  \n5.The RAG experiments in Section 5.1 use only basic paragraph-level chunking without exploring alternative strategies. The authors should investigate alternative chunking approaches, compare different retrieval mechanisms, and analyze how these choices impact episodic memory performance.  \n6.While Table 4 shows poor performance in chronological ordering tasks, the paper doesn't provide detailed error analysis or investigate specific failure patterns. The analysis in Section 5.2 focuses on aggregate metrics without examining individual failure cases. The authors should provide detailed case studies of failure modes, analyze patterns in chronological ordering errors, and investigate whether specific temporal relationships consistently challenge the models.  \n7.Although Section 5.2 mentions testing for hallucinations, the analysis is limited. The paper fails to examine when and why models confabulate, or how confabulation patterns vary across different model architectures and memory strategies. This could be improved by designing specific experiments to probe confabulation triggers and providing metrics for measuring confabulation severity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c9TsMB0NGg", "reviewer": "Reviewer_1gS7", "summary": "The paper presents a new benchmark to evaluate the ability of LLMs on episodic memories - specifically, retrieving relevant information about an event (or multiple events) given a specified time, location, person, or content. The data in the benchmark is generated by LLMs in the form of a book, with each chapter of which being an event. The authors generate random time, location, person and content beforehand and sample one random combination for each chapter; they also specify where the time, location, person and content needs to appear within the chapter for more controllability on the benchmark. The evaluations are done by LLM as a judge to identify the relevant items in the answer and score their relevance. The experiments and results suggest that it is still very challenging for current SoTA LLMs to deal with complex spatio-temporal relationships, and a lot can be improved on the episodic memory capabilities of LLMs.", "questions": "- As the authors acknowledged in the paper, there might be multiple episodic events that can be extracted from each chapter. In addition to the specified time and location their might be other times and locations referred to in the chapter. Are there any measures to filter out these chapters? Also, does the fact that there are many other events (especially contents) in addition to the specified one make the F1-score metric less suitable, since false positives might not actually be false? This is the main concern that I would like to get addressed.\n- For the results in the main text, the entities and the books are generated by Claude 3.5 Sonnet. Does this possibly give Claude models an unfair advantage in the evaluation?\n- Following my comment regarding the events being unrealistic in the \"weaknesses\" section, I would hope to get more insights on this from the authors, e.g. whether unrealistic pairings actually impact LLM performance differently than realistic ones?\n- I would appreciate if the authors can give more qualitative analysis and insight on the experiment results. E.g. In cases where there are 0 matching events and the model hallucinates an answer, does the model produce an event that has never appeared in the text; appeared in the text but completely irrelevant; or relevant in that its spatially or chronologically very close to the cue? What are the common failure modes?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper describes the benchmark generation process in great detail, allowing the readers to easily replicate the data generation process and have a better understanding of the benchmark.\n- By using LLMs to generate synthetic data, the authors eliminates the data contamination issue and make the benchmark more easily scalable.\n- The paper experimented with a comprehensive list of different settings. The experiment results are provided with error bars.", "weaknesses": "- The events described in the book chapters are not filtered in terms of whether they are somewhat realistic or not. It seems very likely that the detail of the event does not align very well with the location. The fact that the dataset contains some non-realistic events might limit the capability of the LLMs to recall them, since the LLMs may use their common sense knowledge to think that it is not likely for this event to take place at the specified location.\n- The way humans experience and memorize different events has a strong temporal structure. More specifically, humans experience the events in temporal order, which makes it simpler for humans to recall the last occurrence or list things in chronological order. In this benchmark there is no temporal order or structure in the book (plus there are no causal relationship between the events as the authors mentioned), which makes it less realistic.\n- One property of human episodic memory is that the time and location in the cue does not need to be a exact match with the event, and humans can naturally recall events that occurs close to the specified time or location. This is not evaluated in the current benchmark.\n- While I appreciate that the authors gave very comprehensive information about the benchmark in the appendix, the presentation in the main text could be enhanced by including a figure or flowchart to describe the generation process, or provide at least one example of the what the retrieval task looks like for the LLM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)