Episodic Memories Generation and Evaluation Benchmark for Large Language Models

Authors: Alexis Huet, Zied Ben Houidi, Dario Rossi
Primary area: foundation or frontier models, including LLMs
Keywords: Episodic Memory Modeling, Large Language Models, Synthetic Benchmark Generation, Cue-based Retrieval, Temporal-Spatial Reasoning, Long-context Understanding, Human-inspired AI

Abstract:
Episodic memory -- the ability to recall specific events grounded in time and space -- is a cornerstone of human cognition, enabling not only coherent storytelling, but also planning and decision-making. Despite their remarkable capabilities, Large Language Models (LLMs) lack a robust mechanism for episodic memory: we argue that integrating episodic memory capabilities into LLM is essential for advancing AI towards human-like cognition, increasing their potential to reason consistently and ground their output in real-world episodic events, hence avoiding confabulations. To address this challenge, we introduce a comprehensive framework to model and evaluate LLM episodic memory capabilities. Drawing inspiration from cognitive science, we develop a structured approach to represent episodic events, encapsulating temporal and spatial contexts, involved entities, and detailed descriptions. We synthesize a unique episodic memory benchmark, free from contamination, and release open source code and datasets to assess LLM performance across various recall and episodic reasoning tasks. Our evaluation of state-of-the-art models, including GPT-4 and Claude variants, Llama 3.1, and o1-mini, reveals that even the most advanced LLMs struggle with episodic memory tasks, particularly when dealing with multiple related events or complex spatio-temporal relationships -- even in contexts as short as 10k-100k tokens.

Decision: accept (poster)
Reviewer scores: 8, 8, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1729189703519

thread_type:
official_review

<thread_object>
creator: reviewer_1gs7

<content>
summary:
The paper presents a new benchmark to evaluate the ability of LLMs on episodic memories - specifically, retrieving relevant information about an event (or multiple events) given a specified time, location, person, or content. The data in the benchmark is generated by LLMs in the form of a book, with each chapter of which being an event. The authors generate random time, location, person and content beforehand and sample one random combination for each chapter; they also specify where the time, location, person and content needs to appear within the chapter for more controllability on the benchmark. The evaluations are done by LLM as a judge to identify the relevant items in the answer and score their relevance. The experiments and results suggest that it is still very challenging for current SoTA LLMs to deal with complex spatio-temporal relationships, and a lot can be improved on the episodic memory capabilities of LLMs.

strengths:
- The paper describes the benchmark generation process in great detail, allowing the readers to easily replicate the data generation process and have a better understanding of the benchmark.
- By using LLMs to generate synthetic data, the authors eliminates the data contamination issue and make the benchmark more easily scalable.
- The paper experimented with a comprehensive list of different settings. The experiment results are provided with error bars.

weaknesses:
- The events described in the book chapters are not filtered in terms of whether they are somewhat realistic or not. It seems very likely that the detail of the event does not align very well with the location. The fact that the dataset contains some non-realistic events might limit the capability of the LLMs to recall them, since the LLMs may use their common sense knowledge to think that it is not likely for this event to take place at the specified location.
- The way humans experience and memorize different events has a strong temporal structure. More specifically, humans experience the events in temporal order, which makes it simpler for humans to recall the last occurrence or list things in chronological order. In this benchmark there is no temporal order or structure in the book (plus there are no causal relationship between the events as the authors mentioned), which makes it less realistic.
- One property of human episodic memory is that the time and location in the cue does not need to be a exact match with the event, and humans can naturally recall events that occurs close to the specified time or location. This is not evaluated in the current benchmark.
- While I appreciate that the authors gave very comprehensive information about the benchmark in the appendix, the presentation in the main text could be enhanced by including a figure or flowchart to describe the generation process, or provide at least one example of the what the retrieval task looks like for the LLM.

questions:
- As the authors acknowledged in the paper, there might be multiple episodic events that can be extracted from each chapter. In addition to the specified time and location their might be other times and locations referred to in the chapter. Are there any measures to filter out these chapters? Also, does the fact that there are many other events (especially contents) in addition to the specified one make the F1-score metric less suitable, since false positives might not actually be false? This is the main concern that I would like to get addressed.
- For the results in the main text, the entities and the books are generated by Claude 3.5 Sonnet. Does this possibly give Claude models an unfair advantage in the evaluation?
- Following my comment regarding the events being unrealistic in the "weaknesses" section, I would hope to get more insights on this from the authors, e.g. whether unrealistic pairings actually impact LLM performance differently than realistic ones?
- I would appreciate if the authors can give more qualitative analysis and insight on the experiment results. E.g. In cases where there are 0 matching events and the model hallucinates an answer, does the model produce an event that has never appeared in the text; appeared in the text but completely irrelevant; or relevant in that its spatially or chronologically very close to the cue? What are the common failure modes?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730127761307

thread_type:
official_review

<thread_object>
creator: reviewer_uwq8

<content>
summary:
This paper introduces a benchmark for evaluating episodic memory capabilities in LLMs. The authors create a framework inspired by cognitive science to model episodic events with temporal and spatial contexts, entities, and detailed descriptions. They generate synthetic datasets and evaluate state-of-the-art LLMs across various recall and episodic reasoning tasks. The evaluation considers different memory strategies: in-context learning, RAG, and fine-tuning. The authors observe that even advanced models face challenges in handling episodic memory tasks, particularly when recalling sequences of related events or complex spatiotemporal relationships.

strengths:
1.The paper tries to address an important and timely challenge about the need for better episodic memory capabilities in LLMs;  
2.The research takes a structured approach to modeling episodic memory by incorporating key concepts from cognitive science, focusing on key aspects of memory: temporal context, spatial grounding, and entity tracking;  
3.The methodology demonstrates rigor through: (1) creating contamination-free synthetic benchmarks, (2) introducing multiple verification steps to ensure data quality, (3) providing flexibility in generating datasets of different sizes and complexities;  
4.The study develops systematic ways to assess different aspects of memory (recall, chronological ordering, latest state).

weaknesses:
1.The paper primarily utilizes LLM-generated synthetic data (Section 4.1), but does not adequately validate the quality and representativeness of the generated narratives. For example, while the authors claim to verify "adherence to event meta-data," they do not provide quantitative metrics for assessing narrative coherence or natural language properties. The authors should establish clear validation metrics and demonstrate how their synthetic data captures the essential properties of real episodic memories.  
2.The scope of the benchmark is unnecessarily limited. The current implementation: (1) only considers fictional narratives with human-like protagonists, (2) Uses oversimplified temporal representations, (3) Fails to address complex episodic memory scenarios involving interconnected events. The authors should expand the benchmark to include more diverse scenarios, complex temporal relationships, and interconnected event sequences that more accurately reflect real-world episodic memory challenges.  
3.While Section 3.1 emphasizes the importance of entity state tracking, the experimental results in Table 3 do not adequately measure this capability. The evaluation focuses on simple recall rather than complex state changes. The paper claims to test "understanding temporal sequences" but does not properly evaluate how models handle causally related state changes. The authors should design specific test cases for complex state tracking, evaluate models' ability to handle causally related state changes, and include metrics for measuring state tracking accuracy.  
4.The LLM-as-judge approach described in Section 4.3 lacks validation of inter-judge consistency across different evaluator LLMs and does not establish correlation with human judgments. This could be addressed by including human evaluation benchmarks and demonstrating consistent assessments across multiple judge models.  
5.The RAG experiments in Section 5.1 use only basic paragraph-level chunking without exploring alternative strategies. The authors should investigate alternative chunking approaches, compare different retrieval mechanisms, and analyze how these choices impact episodic memory performance.  
6.While Table 4 shows poor performance in chronological ordering tasks, the paper doesn't provide detailed error analysis or investigate specific failure patterns. The analysis in Section 5.2 focuses on aggregate metrics without examining individual failure cases. The authors should provide detailed case studies of failure modes, analyze patterns in chronological ordering errors, and investigate whether specific temporal relationships consistently challenge the models.  
7.Although Section 5.2 mentions testing for hallucinations, the analysis is limited. The paper fails to examine when and why models confabulate, or how confabulation patterns vary across different model architectures and memory strategies. This could be improved by designing specific experiments to probe confabulation triggers and providing metrics for measuring confabulation severity.

questions:
1.How does the synthetic data generation process ensure realistic temporal and causal relationships between events?  
2.Have you conducted rigorous validation studies comparing LLM judgments against human annotations or established metrics? What specific measures were taken to ensure consistency and reproducibility in the evaluation process?  
3.How reliable is the process of scoring relevance "against each ground truth item"? Could you provide examples of how partial matches are handled?  
4.In Table 3, the fine-tuned model performs well on single-event queries (F1=0.83) but poorly on multi-event queries (F1≤0.37). Could you elaborate on why naive fine-tuning fails to generalize beyond single-event memorization? What specific architectural or training modifications might address this limitation?  
5.The gradient pattern in Figure 3 shows degrading performance from context to space to time cues. What specific aspects of temporal reasoning make it particularly challenging for current LLMs?  
6.For the "Latest state recall" results in Table 4, what specific challenges prevent models from achieving higher accuracy in tracking entity states over time?  
7.Have you tried other fine-tuning approaches beyond single-event memorization that might better capture the hierarchical and relational nature of episodic memory?  
8.Have you tried other retrieval strategies beyond cosine similarity? How do you address the challenge of retrieving coherent information when relevant context is distributed across multiple chunks?  
9.How might this benchmark contribute to developing novel training methodologies for episodic memory tasks in LLMs, beyond RAG, fine-tuning or parametric memory storage?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730723195684

thread_type:
official_review

<thread_object>
creator: reviewer_yg9e

<content>
summary:
This paper introduces and explores the concept of episodic memory in the context of long-text comprehension by language models. This approach emphasizes the necessity for models to maintain a coherent understanding of an entity’s state as it evolves over time, space, and content. Both short and long synthetic documents were generated using various cue templates. The dataset also includes null answers incorporated to test for model illusions.

The findings indicate that models like GPT-4o with contextual memory and Claude 3.5 Sonnet4 with RAG memory scored the highest on average, suggesting that retrieval-based methods can improve situational memory by narrowing the context relevant to each query. While some models demonstrated near-perfect accuracy in chapters involving zero or one event per entity, their performance declined significantly as the number of events increased.

strengths:
It is the first time to evaluate updated memories for episodic memory, events with rich contextual information and involving the tracking of specific entities occurring at specific time and spatial locations. And have a solid analysis of event complexity.

The drop of accuracy of fine-tuned model highlights that current fine-tuning techniques fall short in understanding episodic events and their complex interrelationships. All models have a low percentage of exact matches in the temporal ordering task, and even when the models retrieve the correct events, they often fail to order the events correctly.

weaknesses:
1. line 123 has different citation format

questions:
It would be beneficial to explore whether different fine-tuning parameters, or fine-tuning applied to other models could enhance the performance of episodic memory tasks
</content>
</thread_object>

</thread>
<thread>
creation date:
1730728623229

thread_type:
official_review

<thread_object>
creator: reviewer_pjrn

<content>
summary:
The paper presents a framework for modeling and evaluating episodic memory in large language models (LLMs), focusing on their ability to recall and process events associated with specific times and locations, similar to human episodic memory. The authors propose a method that uses entities and events to construct episodic memories and a benchmark designed to test LLMs on tasks such as recalling event details, tracking entity states, and understanding temporal-spatial contexts. The benchmark, including synthetic datasets and structured tasks, also assesses the models' ability to avoid confabulations by identifying unfamiliar information. The authors' evaluation of models like GPT-4 and Claude shows that current LLMs struggle with complex, multi-event scenarios and spatio-temporal reasoning, highlighting the need for improved episodic memory frameworks and training methods tailored to these capabilities.

strengths:
1. The paper effectively organizes episodic memory tasks for LLMs using a cue-based recall and retrieval method. The authors provide examples of different cues, showing various combinations that models must use to retrieve event information based on time, location, involved entities, or content. This clear design demonstrates a solid grasp of how to simulate episodic memory in LLMs and offers a strong foundation for evaluating model recall across a range of scenarios, from simple to complex.
2. The benchmark tests the model's ability to handle both clear and vague questions, similar to real-world situations where memory needs vary. By asking models to either recall a specific event or recognize several related events, the tasks assess how well the models adapt to different recall demands.
3. The benchmark includes carefully designed tests to assess a model's ability to recognize unfamiliar events or entities and admit when it lacks information. This is crucial for evaluating whether LLMs can avoid hallucinations, and this thoughtful design adds reliability to the benchmark.
4. The paper offers detailed statistics and information about the benchmark, along with several ablation studies in the appendix. This level of detail shows the authors' commitment to transparency and rigor, helping readers understand the benchmark’s structure and how different elements affect model performance. These ablation studies also provide deeper insights and useful guidance for future research.

weaknesses:
1. A limitation of the paper is that it only evaluates proprietary models like GPT-4o and Claude, rather than open-source models such as LLaMA 3. Including open-source models would make the findings more generalizable and accessible to a broader research community, enabling comparisons across a wider range of models and methods.
2. The benchmark mainly uses clear cues, which, while providing consistency and control, may not capture the subtler cues common in natural language memory tasks. Adding more ambiguous time markers and indirect references could better simulate real-world memory challenges and lead to a stronger test of models' episodic memory abilities and their handling of less obvious retrieval cues.

questions:
1. Could you explain the "naive fine-tuning" approach mentioned in the paper? What datasets and methods were used, and how does this approach differ from other fine-tuning strategies for episodic memory tasks?

2. Just curious—does your benchmark have a specific name, or is it simply called "Short Book" and "Long Book"?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730753676435

thread_type:
official_review

<thread_object>
creator: reviewer_prp7

<content>
summary:
This paper proposes an approach for generating a benchmark for evaluating the episodic memory of LLMs. The authors use this generation approach to develop short book and long book splits consisting of 456 and 686 Q/A pairs respectively. The authors then evaluate high quality LLMs on this benchmark to showcase that even with RAG achieving high performance can be quite challenging.

strengths:
I think this paper looks at an important problem and is well written for the most part. The approach outlined in section 3 makes a lot of sense. I really agree about the five points mentioned in the "Need for an episodic memory benchmark" paragraph at the end of page 3. I also found that Figure 3 and Table 4 presented some interesting analysis giving more insight into the ways that current LLMs struggle with reasoning over episodic memories.

weaknesses:
**Data Generated:** I think the biggest weakness of this paper is the quantity and quality of data generated as part of the short book and long book splits. This first stood out to me when reading Table 2. For this amount of content, it feels like even the use of automated means are not necessary i.e. a human can annotate an actual book with questions and it would be a lot higher quality than what is produced here. Given that the authors are specifying a general strategy for generating benchmarks in section 3, I thought at least a number of randomly generated books would be considered if not also significant diversity within these books.  As a result, this benchmark does not yield easy high confidence analysis, which is showcased by massive error bars throughout the main results table (table 3). 

**Table 3:** The results of the in-context and RAG models are largely in-line with general expectations in Table 3, so the benchmark does not really lead to new findings in comparison to the current literature. The only interesting finding is with respect to the fine-tuning baseline, but the implementation of this baseline seems flawed. First of all, there is barely enough data for this dataset to be used for evaluation of LLMs, it seems like there is simply not the data required to facilitate fine-tuning. Judging from the appendix, it seems like for some reason only a number of events matching the cues of 1 was used for fine-tuning, which seems to fully explain the results in this row on its own. It is not even clear to me if the 0.83$\pm$0.35 is using the same data for training and testing.

questions:
Q1: When the authors write: "The proposed episodic memory benchmark exhibits several desirable properties: it is contamination-free by design, scalable with low human labor, offers unambiguous cues and ground truth, and the ability to model multiple cues and events within a synthetic yet realistic narrative."  What does scalable mean here? How is this demonstrated in the paper? 

Q2: The authors write that needle in the haystack benchmarks "do not incorporate temporal nor spatial awareness", but isn't this point undermined by the limitation related to "event independence" the authors mention?  

Q3: The authors also write that bABI / bABILong "often involve highly artificial scenarios lacking complexity and realism – opening the door to shortcut reasoning by exploiting dataset biases or patterns", but isn't this point undermined by the limitation related to "temporal representation" the authors mention? Also how are dataset biases / patterns addressed in a way that goes beyond bABI? 

Q4: For the point on "limited domain scope" could you explain why more domains or even random variations of the book were not considered in this paper? What roadblocks remain that made the authors position it as future work?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732192725368

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global answer summary

comment:
Dear reviewers, 

We greatly appreciate your detailed reviews and insightful feedback, which are helping us to significantly improve our work.
We answer in detail to each reviewer in the separate answers, while we provide below a summary of the new experiments performed along with the additional material created.

Sincerely, 

The authors

## Generated books and question/answer pairs

- For assessing the impact of the temporal order (highlighted by reviewer 1gS7), we created the chronologically ordered version of the default short and long book,
- For demonstrating the capability in generating variations of the book (concern express by reviewer prP7), we added the more diverse 'world news' and 'science fiction' books, with short (20 chapters) and long (200 chapters) variation for each (one excerpt available below),
- For demonstrating the scalability of our approach, we generated the default book with 2000 chapters, for a total of 1M+ tokens.

In the following table, we show the existing and additional books (together with the related question/answer pairs) that have been generated. The additionally generated benchmarks are indicated with an asterisk *.

| chapters            | 20 | 200 | 2000 |
|---------------------|----|------|-----|
| Claude default         | ✔  |  ✔  | ✔*  |
| Claude default ordered | ✔* |  ✔* | ✘   |
| GPT default            | ✔  |  ✔  | ✘   |
| Claude world news      | ✔* |  ✔* | ✘   |
| Claude scifi           | ✔* |  ✔* | ✘   |

## Additional experiments for providing the answers

- For assessing whether evaluating only the book produced with Claude gives an unfair advantage in the evaluation (highlighted by reviewer 1gS7), we evaluated the short GPT default book on the four (gpt-4o-mini, gpt-4o, claude-3-haiku, claude-3-5-sonnet) models,
- We assessed the impact of the temporal order by evaluating the ordered version of the default short book on the four (gpt-4o-mini, gpt-4o, claude-3-haiku, claude-3-5-sonnet) models,
- We have evaluated the Claude world news and Claude scifi variations with gpt-4o on the short default book.
- ~~We tried to evaluate the benchmark on llama3, but we are facing some issues detailed to reviewer PJrN. We hope that those issues will be solved in order to add this model in the comparison~~ We have evaluated llama-3.1-405b and llama-3.2-3b on the short default book.

In the following tables, we show the additional experiments performed (indicated with an asterisk *).

- Ablations:
|book|gpt-4o-mini|gpt-4o|claude-3-haiku|claude-3-5-sonnet|
|---|---|---|---|---|
| Short Claude default ordered|✔*|✔*|✔*| ✔* |
| Short GPT default|✔*|✔*|✔*|✔*|
| Short Claude world news|✘|✔*|✘|✘|
| Short Claude scifi|✘|✔*|✘|✘|

- New model:
|book|llama-3.1-405b|llama-3.2-3b|
|---|---|---|
|Short Claude default|✔*|✔*|

## Additional experiments

- We evaluated the degree of realism of each produced event (concern express by reviewer 1gS7), and evaluated the difference of performance between the realistic and the non-realistic events,
- We analyzed manually the hallucinations observed in the gpt4o answers when there is 0 matching events

## Visual aids and examples

Please find at the following anonymous addresses:
- The [global flowchart of our generation process](https://figshare.com/s/863956f3e6592d3dad34?file=50683452)
- [An example of the journey of of a single entity within the default long book](https://figshare.com/s/863956f3e6592d3dad34?file=50682921) This example shows the tracking of a single entity (here Jackson Ramos with red segments) over the chapters (other entities are indicated with gray segments)
- [Detailed fine tuning explanation for building the training data in this setting](https://figshare.com/s/863956f3e6592d3dad34?file=50683632)

## For reproducibility

We provide the supplementary [generated benchmark data at this address](https://figshare.com/s/7b634effbf6a71ca722c), while the [following notebooks for reproducing the additional experiments are there](https://figshare.com/s/863956f3e6592d3dad34?file=50784417)

- rebuttal_ablation_on_news_and_scifi_books.ipynb (evaluating the world news and the scifi short books with gpt-4o)
- rebuttal_ablation_with_gpt_book.ipynb (applying the experiment on the GPT generated book)
- rebuttal_generating_book_variations.ipynb (building the world news, the scifi, and the very long default books)
- rebuttal_hallucinations_0_matching_events.ipynb (manual analysis of the hallucinations observed in the gpt-4o answers when there is 0 matching events)
- rebuttal_llama3.ipynb (evaluating the short default book with llama 3.1 405b and llama 3.2 3b)
- rebuttal_map.ipynb (illustration provided)
- rebuttal_ordered_books.ipynb (building and evaluation of the ordered book)
- rebuttal_realistic_partition_and_evaluation.ipynb (assess the degree of realism of each event and evaluation in the difference of performance between the realistic and the non-realistic events),

Edit: adding anonymous links and solving issue for llama3 model
</content>
</thread_object>

</thread>
<thread>
creation date:
1732909779783

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Changes integrated to the manuscript

comment:
Dear reviewers, 

We sincerely thank you all for your thorough feedback throughout the review process. Thanks to your suggestions, we have substantially strengthened our work through methodology clarifications, dataset diversification, and extended evaluations and ablations, enriching our findings and significantly increasing the confidence in our earlier results.

Our improvements encompass three main areas:

## Additional benchmark datasets illustrating our framework's scalability

We are now releasing 11 benchmark datasets (Table 28), including:
- Short (10k tokens), long (100k tokens) and very long (1M+ tokens) Claude-generated books;
- Time-ordered versions of the short and long Claude-generated books; 
- Short and long GPT-generated books; 
- Two additional diverse universes, world news and sci-fi, that we use to generate additional books (characteristics in Table 26, examples of universe elements and excerpts of a chapter in Appendix G.1).

## Extended evaluation and ablations
We have conducted comprehensive evaluations leading to both novel findings, and increased confidence in our earlier results:
- Applied on both short and long books:
-
  + Evaluation on Llama 3.1 405B instruct, applied on the long book (Figures 3 and 4 and Tables 3 and 4 in the main section), and on the short book (Figure 7 and Table 13 in the appendix). Results show that on the short book, llama-3.1 performance is equivalent to gpt-4o and cl-3.5-sonnet, while on the long book, gpt-4o is still statistically better than both cl-3.5-sonnet and llama-3.1 (that are statistically equivalent). Summarizing table below.

|Model|Book|0|1|2|3-5|6+|
|-|-|-|-|-|-|-|
|llama-3.1|short default|0.91±0.28|0.95±0.18|0.89±0.18|0.83±0.17|n.a.|
|llama-3.1|long default|0.80±0.40|0.49±0.47|0.38±0.33|0.40±0.25|0.45±0.20|

-
  + Evaluation on the world news and the sci-fi books for the gpt-4o model (in Appendix G.2, table also reported below). The figures are inline with our previous universe, confirming a consistent performance decline for queries with two or more matching events.

|Model|Book|0|1|2|3-5|6+|
|-|-|-|-|-|-|-|
|gpt-4o|short default|0.86±0.35|0.96±0.16|0.93±0.16|0.88±0.16|n.a.|
|gpt-4o|short news|0.91±0.29|0.99±0.06|0.89±0.18|0.86±0.12|n.a.|
|gpt-4o|short sci-fi|0.85±0.36|0.99±0.06|0.94±0.14|0.92±0.15|n.a.|
|gpt-4o|long default|0.84±0.37|0.81±0.38|0.60±0.31|0.57±0.21|0.53±0.14|
|gpt-4o|long news|0.96±0.20|0.82±0.38|0.66±0.28|0.54±0.23|0.46±0.20|
|gpt-4o|long sci-fi|0.90±0.30|0.72±0.43|0.62±0.29|0.55±0.22|0.51±0.13|

- Applied on the short book only:
  + Comparative evaluation of Claude- vs GPT-generated short books on four models (gpt-4o-mini, gpt-4o, claude-3-haiku, claude-3-5-sonnet) in Appendix E.5.
  + Comparative evaluation of unordered vs ordered events, in the short books, on four models (gpt-4o-mini, gpt-4o, claude-3-haiku, claude-3-5-sonnet) in Appendix E.6.
  + Comparative evaluation on questions related to a set of realistic vs non-realistic events reported in Tab. 24 for four models (gpt-4o-mini, gpt-4o, claude-3-haiku, claude-3-5-sonnet) in Appendix E.7.

## Methodology clarifications
We have enhanced the paper's clarity through:
- [Flowchart of the book generation process (Fig. 2 in the main section), with an explicit illustration that an item can match many events](https://figshare.com/s/863956f3e6592d3dad34?file=50683452).
- [Illustration of the shared universe structure, with examples of entity tracking and question/answer pairs (Appendix C, and Figure 6)](https://figshare.com/s/863956f3e6592d3dad34?file=50682921).
- Clarification of book generation quality-control layers: (i) exact-match parsing for event requirements and (ii) LLM-based verification for geographical focus, temporal day, main character and main event (Section 4.1, Appendix B.1.6 and B.1.7).
- Evaluation clarifications on LLM-as-a-judge usage and F1-score methodology (Appendix B.3.2, B.4).
- Fine-tuning methodology details in Appendix B.2.5.
- Assessment of event realism (Appendix E.7) and analysis of GPT-4o's empty-answer responses (Appendix E.8).

These improvements significantly strengthen our empirical validation while ensuring full reproducibility for the community. Again, we appreciate the reviewers' guidance in helping us achieve these meaningful enhancements.

Sincerely, 

The authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work introduces a novel, cognitively inspired episodic memory benchmark for LLMs. Episodic memory is a crucial ability for decision making. Even though there have been benchmarks evaluating the decision making abilities of LLMs, there has not been a benchmark that explicitly the episodic memory capacity of LLMs. The extensive evaluation of recent LLMs reveals the limited capacity of current LLMs on episodic memory tasks. 

There were several concerns in the initial reviews: the quality of the generated data compared to human annotations, the limited scope of the benchmark, implementation of the finetuning and RAG baselines, the confidence of the results, lack of evaluation of open sourced LLMs, the lack of ambiguous time markers and indirect references, and the lack of detailed analysis of errors and model hallucination. The authors' responses adequately addressed the concerns. 

Overall, the benchmark is a good addition to the existing evaluation suite of LLMs, highlighting an important but less explored aspect of LLMs. It could be particularly useful for diagnosing LLMs' limitations in decision making and storytelling. I do want to suggest a human baseline for the benchmark. The benchmark design is inspired by human cognition. Thus, it would be informative to know humans' performance on these tasks to indicate the human-model performance gap.