LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory

## Abstract

Abstract Recent large language model (LLM)-driven chat assistant systems have integrated memory components to track user-assistant chat histories, enabling more accurate and personalized responses. However, their long-term memory capabilities in sustained interactions remain underexplored. This paper introduces LongMemEval , a comprehensive benchmark designed to evaluate five core long-term memory abilities of chat assistants: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. With 500 meticulously curated questions embedded within freely scalable user-assistant chat histories, LongMemEval presents a significant challenge to existing long-term memory systems, with commercial chat assistants and long-context LLMs showing 30% accuracy drop on memorizing information across sustained interactions. We then present a unified framework that breaks down the long-term memory design into four design choices across the indexing, retrieval, and reading stages. Built upon key experimental insights, we propose several memory designs including session decomposition for optimizing value granularity, fact-augmented key expansion for enhancing the index structure, and time-aware query expansion for refining the search scope. Experiment results show that these optimizations greatly improve both memory recall and downstream question answering on LongMemEval . Overall, our study provides valuable resources and guidance for advancing the long-term memory capabilities of LLM-based chat assistants, paving the way toward more personalized and reliable conversational AI. To facilitate future research, we publicly release the benchmark and our experiment code at https://github.com/xiaowu0162/LongMemEval .

## 1 Introduction

Large language models (LLMs) have exhibited impressive capabilities in solving diverse tasks through natural language, leading to numerous successful chat assistant applications (OpenAI, ; Microsoft, ). Nevertheless, LLMs face limitations on tasks relying heavily on personal knowledge accumulated through long-horizon user-AI interactions, such as psychological counseling or secretarial duties (Zhong et al., ). Failing to incorporate user background and preferences into responses can diminish the response’s accuracy as well as user satisfaction. To personalize LLM-based assistants, long-term memory, the ability to memorize, recall, and reason with the long-term interaction history, is indispensable. Recently, several commercial (OpenAI, ; Coze, ) and open-sourced memory-augmented assistant systems (Zhong et al., ; Zhang et al., ) have been introduced. These systems leverage techniques like compressing, indexing, and retrieving from chat histories to generate more accurate and personalized responses.

Despite these advances in memory-augmented chat systems, there has been limited progress in holistically evaluating their memory capability in long-term interactions. While several benchmarks evaluate LLMs on understanding long chat histories (Xu et al., ; ; Zhong et al., ; Maharana et al., ; Du et al., ; Kim et al., ), they have two major shortcomings. First, they do not accurately reflect the nature of long-term user-AI interactions. Many datasets focus solely on human-human conversations (Xu et al., ; Maharana et al., ; Kim et al., ), while others omit task-oriented dialogues, which represent a significant portion of chat assistant usage. Their interactive histories also typically have short and non-configurable length, spanning only a few thousand tokens, limiting the challenge as current systems continue to improve. Second, current benchmarks’ questions only offer a limited coverage of the memory abilities required to leverage the dynamic, ever-changing, and accumulative user information in long-term interactions. For instance, MemoryBank (Zhong et al., ) and PerLTQA (Du et al., ) under-evaluate the ability to synthesize information across numerous sessions or reasoning with temporal metadata or time references. All benchmarks including recent ones such as LoCoMo (Maharana et al., ) also fail to evaluate recall of assistant-side information or reasoning with updated user information.

We introduce LongMemEval, a comprehensive, challenging, and scalable benchmark designed to assess the long-term memory capabilities of chat assistants. LongMemEval consists of 500 human-curated, high-quality questions to test five core
memory abilities: information extraction, cross-session reasoning, temporal reasoning, knowledge updates, and abstention. Each question requires recalling information hidden within one or more multi-turn task-oriented user-AI dialogues that are LLM-simulated and human-edited. Inspired by the “needle-in-a-haystack” test (Kamradt, ), we design an attribute-controlled pipeline to compile a coherent, extensible, and timestamped chat history for each question. A chat system, then, is required to parse the dynamic interactions online for memorization, and answer the question after all the interaction sessions. While the length of the history is freely configurable, we provide two standard settings for consistent comparison: LongMemEvalS with approximately 115k tokens per problem’s history and LongMemEvalM with 500 sessions (around 1.5 million tokens). Preliminary evaluations highlight the difficulty of LongMemEval, as long-context LLMs show a 30%$\sim$60% performance drop on LongMemEvalS, and manual evaluations reveal that state-of-the-art commercial systems (such as GPT-4o) only achieve 30%$\sim$70% accuracy in a setting much simpler than LongMemEvalS (section 3.5).

Finally, we present a unified view for memory-augmented chat assistants. Leveraging LongMemEval, we comprehensively analyze memory design choices across three key execution stages—indexing, retrieval, and reading—and four control points: value, key, query, and reading strategy. Experimental insights identify several effective memory designs:

- •
(Section 5.2) Instead of sessions, round is the best granularity for storing and utilizing the interactive history. While further compression into individual user facts harms overall performance due to information loss, it improves the multi-session reasoning performance.
- •
(Section 5.3) While using a flat index with the memory values themselves as the keys is a strong baseline, expanding the keys with extracted user facts greatly facilitates both memory recall (4% higher recall@$k$) and downstream question answering (5% higher accuracy).
- •
(Section 5.4) Simplistic memory designs perform poorly on temporal reasoning questions. We propose a simple time-aware indexing and query expansion strategy to narrow down the search range, which improves the memory recall for the temporal reasoning by 7%$\sim$11%.
- •
(Section 5.5) Even with perfect memory recall, accurately reading the retrieved items is still non-trivial. Applying Chain-of-Note (Yu et al., ) and structured prompt format (Yin et al., ) improves the reading accuracy by as much as 10 absolute points across three LLMs.

## 2 Related Work

**Table 1: A comparison between LongMemEval and existing long-term memory benchmarks. We use different colors to denote human-human dialogue and human-AI dialogue. #Sess and #Q denote the total number of sessions and questions. Context depth is calculated as the number of tokens in the history. Finally, we the coverage of five core abilities: information extraction (IE), multi-session reasoning (MR), knowledge update (KU), temporal reasoning (TR), and abstaining on unanswerable questions (ABS). ^∗Not reported in the paper, based on our approximation. ^∗∗at most 2 sessions.**
| Benchmark | Domain | #Sess | #Q | Context Depth | Core Memory Abilities |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| IE | MR | KU | TR | ABS |  |  |  |  |  |
| MSC (Xu et al., ) | Open-Domain | 5k | - | 1k | ✗ | ✗ | ✗ | ✗ | ✗ |
| DuLeMon (Xu et al., ) | Open-Domain | 30k | - | 1k | ✗ | ✗ | ✗ | ✗ | ✗ |
| MemoryBank (Zhong et al., ) | Personal | 300 | 194 | 5k | ✓ | ✗ | ✗ | ✓ | ✗ |
| PerLTQA (Du et al., ) | Personal | 4k | 8593 | 1M^∗ | ✓ | ✗ | ✗ | ✗ | ✓ |
| LoCoMo (Maharana et al., ) | Personal | 1k | 7512 | 10k | ✓ | ✓ | ✗ | ✓ | ✓ |
| DialSim (Kim et al., ) | TV Shows | 1k–2k | 1M | 350k | ✓ | ✓^∗∗ | ✗ | ✓ | ✓ |
| LongMemEval (this work) | Personal | 50k | 500 | 115k, 1.5M | ✓ | ✓ | ✓ | ✓ | ✓ |

#### Long-Term Dialogue Benchmarks

Early benchmarks focused on language modeling evaluation, requiring models to generate personalized responses by referencing human-human (Xu et al., ) or human-AI (Xu et al., ) conversation histories. To more precisely evaluate how accurately models recall past information, subsequent benchmarks shifted toward question answering (QA). For example, MemoryBank (Zhong et al., ) features multi-day chat histories from 15 users with 194 human-written probing questions. LoCoMo (Maharana et al., ) includes 50 long-term chat histories and questions testing single-hop, multi-hop, temporal, commonsense, world knowledge, and adversarial reasoning. LoCoMo also introduces two new tasks – event graph summarization and multimodal dialogue generation. PerLTQA (Du et al., ) scales the evaluation to 3,409 dialogues and 8,593 questions, covering world knowledge, personal profiles, social relationships, events, and dialogue history. DialSim (Kim et al., ) evaluates models’ memory ability by role-playing TV show characters and introduces a time constraint that penalizes slow system responses. Despite these advancements, existing QA-based benchmarks overlook several memory capabilities critical to long-term user-assistant interactions: synthesizing information across numerous sessions, recalling assistant-side information, and reasoning about updated user details or complex temporal references. Additionally, the chat histories are often too brief and do not reflect the nature of task-oriented user-assistant interactions. Table 1 compares between LongMemEval and previous works, highlighting its advantages in both (1) featuring a long and freely extensible iterative history and (2) holistically covering critical memory abilities in a uniquely challenging way (further examples in Figure 1).

#### Long-Term Memory Methods

To equip chat assistants with long-term memory capabilities, three major techniques are commonly explored. The first approach involves directly adapting LLMs to process extensive history information as long-context inputs (Beltagy et al., ; Kitaev et al., ; Fu et al., ; An et al., ). While this method avoids the need for complex architectures, it is inefficient and susceptible to the “lost-in-the-middle” phenomenon, where the ability of LLMs to utilize contextual information weakens as the input length grows (Shi et al., ; Liu et al., ). A second line of research integrates differentiable memory modules into language models, proposing specialized architectural designs and training strategies to enhance memory capabilities (Weston et al., ; Wu et al., ; Zhong et al., ; Wang et al., ). Lastly, several studies approach long-term memory from the perspective of context compression, developing techniques to condense lengthy histories into compact representations, whether in the form of LLM internal representations (Mu et al., ; Chevalier et al., ), discrete tokens (Jiang et al., ; Xu et al., ), or retrievable text segments via retrieval-augmented generation (Shi et al., ; Wang et al., ; Sarthi et al., ; Chen et al., ; Gutiérrez et al., ). Although LongMemEval can evaluate all types of memory systems, we will take an online context compression perspective, where each history interaction session is sequentially processed, stored, and accessed on-demand through indexing and retrieval mechanisms (Section 4). This formulation aligns with current literature (Zhong et al., ; Gutiérrez et al., ) and commercial memory systems (OpenAI, ; Coze, ). Its plug-and-play nature also facilitates the integration into other existing chat assistant systems.

Figure: Figure 1: Examples of the seven diverse question types in LongMemEval. For each example, we show the associated evidence statements on the left and the question with the answer on the right.
Refer to caption: x1.png

## 3 LongMemEval

### 3.1 Problem Formulation

The evaluation of LongMemEval requires an instance of 4-tuple $(\mathbf{S},q,t_{q},a)$. $\mathbf{S}\equiv[(t_{1},S_{1}),(t_{2},S_{2}),...,(t_{N},S_{N})]$ is a sequence of $N$ history chat sessions ordered from the earliest to the latest, where $S_{i}$ is a multi-turn interaction between the user and a chat assistant and $t_{i}$ is the date and time of the session. During test time, the tested system observes $\mathbf{S}$ sequentially. $q$ and $t_{q}>t_{N}$ represent the question from the user and its date. $a$ is a short phrase indicating the answer, or a natural language rubric describing the preferred answer in the case where $q$ is open-ended.

### 3.2 LongMemEval : Benchmark Curation

We argue that the main challenge in building a reliable personalized assistant is performing online recording, recalling, updating, and reasoning on the dynamically evolving user information. To faithfully reflect the challenge, LongMemEval formulates five core long-term memory abilities:

- •
Information Extraction (IE): Ability to recall specific information from extensive interactive histories, including the details mentioned by either the user or the assistant.
- •
Multi-Session Reasoning (MR): Ability to synthesize the information across multiple history sessions to answer complex questions that involve aggregation and comparison.
- •
Knowledge Updates (KU): Ability to recognize the changes in the user’s personal information and update the knowledge of the user dynamically over time.
- •
Temporal Reasoning (TR): Awareness of the temporal aspects of user information, including both explicit time mentions and timestamp metadata in the interactions.
- •
Abstention (ABS): Ability to refrain from answering questions that involve unknown information, i.e., information not mentioned in the interaction history.

As shown in Table 1, LongMemEval represents a more comprehensive ability coverage compared to prior long-term memory benchmarks like MemoryBank and PerLTQA. To thoroughly assess these abilities, LongMemEval features seven question types. Single-session-user and single-session-assistant test memorizing the information mentioned by user or assistant within a single session. Single-session-preference tests whether the model can utilize the user information to generate a personalized response. The other types of questions are multi-session (MR), knowledge-update (KU), and temporal-reasoning (TR). Finally, we draw 30 questions from the previous question types and modify them “false premise” questions, testing whether the model can correctly abstain from answering (ABS). Figure 1 presents an example for each question type.

#### Question Curation Pipeline

We design an attribute-controlled pipeline for question curation, as depicted in Figure 2. We first define an ontology of 164 user attributes in five categories: lifestyle, belongings, life events, situations context, and demographic information. For each attribute, we leverage an LLM(^1^11Unless otherwise mentioned, Llama 3 70B Instruct (Dubey et al., ) is used as the LLM in the pipeline.) to generate attribute-focused user background paragraphs, each of which includes detailed discussion of the user’s life experience. To create a question, we randomly sample a paragraph and use an LLM to propose several seed (question, answer) pairs. However, we observe that LLM-proposed seed questions often lack depth and diversity. We thus manually rewrite all the questions to achieve the desired difficulty and diversity. Then, based the background, we manually decompose the answer into one or more evidence statements with optional timestamps. Each evidence statement is then separately embedded into a task-oriented evidence session created by self-chatting (Xu et al., ). The user LLM is instructed to convey the evidence statement indirectly. For instance, to mention “the user bought a new car last month”, the user LLM may begin with questions regarding car insurance, and reveal the fact incidentally within the dialogue. This approach enhances the benchmark’s difficulty by requiring systems to recognize and memorize key details not directly emphasized in conversations. We present the full details in Section A.1.

To ensure high-quality test data, all the evidence sessions are manually adjusted to (1) verify evidence inclusion, (2) distribute evidence across different conversation positions, and (3) rephrase statements into more natural, colloquial language, especially for time mentions, which LLMs often express too formally. We also meticulously annotate the position of the evidence statement within each evidence session. For questions involving temporal information, we manually add timestamps to both the sessions and the questions. Figure 3 presents the basic statistics of LongMemEval, revealing that most questions require evidence from multiple sessions (up to six) and that evidence statements are positioned diversely within sessions, increasing the challenge to the memory design.

Figure: Figure 2: Data creation pipeline of LongMemEval. The question construction process is performed by human experts (a), and the evidence sessions are LLM-simulated and human-edited (b). The history construction process (c) is performed at test time and is freely configurable.
Refer to caption: x2.png

Figure: (a) Distribution of question types in LongMemEval.
Refer to caption: x3.png

### 3.3 History Compilation

For each question, LongMemEval compiles a coherent user-AI chat history that contains the associated evidence sessions (Figure 2c). Specifically, we sample a number of unrelated user-AI chat sessions, randomly insert the evidence sessions in the middle, and assign a plausible timestamp to each session. We draw the irrelevant sessions from two sources: (1) self-chat sessions simulated using facts extracted from the user backgrounds with non-conflicting attributes and (2) publicly released user-AI style chat data including ShareGPT (Zheng et al., ) and UltraChat (Ding et al., ). This design allows us to simulate realistic and extensible user-AI chat histories with minimal conflicts. We document the full details of history session sampling and timestamp specification in Section A.2. Although our pipeline is able to output histories of arbitrary length, we provide two standard benchmark settings for consistent comparison: LongMemEvalS with approximately 115k tokens per question and LongMemEvalM with 500 sessions (around 1.5 million tokens).

It is worth noting that LongMemEval’s testing strategy is inspired by the needle-in-a-haystack test for long-context understanding (Kamradt, ), which asks a model to retrieve certain short information (the “needle”) embedded in a long document (the “haystack”). However, LongMemEval presents a more challenging and realistic scenario, as it involves identifying and synthesizing information from multiple extended evidence sessions. Additionally, the “needles” in LongMemEval are contextually integrated within the chat history, creating a coherent haystack.

### 3.4 Evaluation Metric

#### Question Answering

As the correct answers in LongMemEval can take flexible forms, using an exact matching strategy as in previous works can result in high error rates. To address this, we employ a prompt-engineered large language model to assess response quality (Liu et al., ). Specifically, we use the gpt-4o-2024-08-06 model via the OpenAI API. Our meta-evaluation study demonstrates that the evaluator achieves more than 97% agreement with human experts. We include the human meta-evaluation details and the evaluation prompt in Section A.3.

#### Memory Recall

As LongMemEval contains human-annotated answer location labels, intermediate retrieval metrics can be reported if the chat system exposes its retrieval results. In this paper, we report Recall@$k$ and NDCG@$k$, where $k$ is the number of top items retrieved by the system.

### 3.5 LongMemEval represents a significant challenge

Using LongMemEval, we conduct a pilot study on commercial systems and long-context LLMs.

#### Commercial systems

We evaluate two commercial systems that maintain a set of memorized user facts as the user chats with the assistant: ChatGPT (OpenAI, ) and Coze (Coze, ). Since these systems only support memory features via their web interfaces, we randomly selected 97 questions and created a short chat history of 3-6 sessions (approximately 10x shorter than LongMemEvalS). Human annotators interacted with the chat assistants session-by-session, ending with a new session where the question was posed(^2^22All evaluations were conducted in the first two weeks of August 2024.). In Figure 4(a), we compare this online memory evaluation with offline reading, where GPT-4o is prompted to answer with the complete history provided as context. Both ChatGPT and Coze exhibited significant performance drops compared to offline reading, underscoring the challenging nature of LongMemEval and highlighting the substantial gap between recalling isolated user facts and demonstrating a comprehensive memory ability. We find ChatGPT tends to overwrite crucial information as the number of interaction sessions increases, while Coze often fails to record user information presented indirectly. We provide further human study details and analyses in Appendix B.

#### Long-Context LLMs

While LongMemEval poses a significant challenge to online memory systems, is the benchmark easily tackled with offline reading over the entire history? In Figure 4(b), we evaluated four advanced long-context LLMs on LongMemEvalS (with a history length of approximately 115k tokens): GPT-4o, Llama 3.1 Instruct (Dubey et al., ), and Phi-3-Medium (Abdin et al., ). Compared to the oracle retrieval setting (answering with only the evidence sessions as the context), these LLMs showed a 30% to 60% performance decline when tasked with reading the entire LongMemEvalS history, regardless of whether the chain-of-note technique (Yu et al., ) was applied (discussed further in Section 4.2). It’s important to note that the histories in LongMemEvalS is still short ($\sim$50 sessions), and the performance is likely to further degrade as the interaction history expands. In summary, even the most capable long-context LLMs currently require an effective memory mechanism to manage an ever-growing interaction history.

Figure: (a) Commercial memory-augmented LLMs exhibit weak performance on LongMemEval. The accuracy on 97 questions degrades by a large amount compared to directly reading the context (“Offline Reading”). Specifically, ChatGPT and Coze instantiated with GPT-4o exhibits 37% and 64% performance drop, respectively.

## 4 A Unified View of Long-Term Memory Assistants

In this section, we formulate a three-stage long-term memory model for chat assistants. Despite its simplicity, this model provides a unified view of existing long-term memory assistant works and enables us to investigate crucial control points for each stage’s design.

### 4.1 Long-Term Memory System: Formulation

We formulate long-term memory as a massive key-value datastore $[(k_{1},v_{1}),(k_{2},v_{2}),...]$, where the keys $k_{i}$ can be heterogeneous and the values $v_{i}$ might repeat. As shown in Figure 5, we formulate three stages for a memory-augmented assistant: (1) indexing, converting each history session $(t_{i},S_{i})$ into one or more key-value items, (2) retrieval, formulating a retrieval query and collecting $k$ most relevant items, and (3) reading, an LLM $\mathcal{M}$ reads the retrieval result and generates a response. In Appendix C, we further revisit nine memory-augmented chat assistant systems with this framework.

### 4.2 Long-Term Memory System: Design Choices

Based on this formulation, we identify four crucial control points for long-term memory of chat assistants (illustrated in Figure 5). We analyze design choices from existing works and their limitations, and propose our optimizations. Due to space constraints, we present high-level descriptions here, with detailed designs in Section 5 and Appendix D.

#### CP 1: Value

The value represents the format and granularity of each session stored in memory. Given that user-AI chat sessions are often lengthy and cover multiple topics, storing each session as a single item can hinder effective retrieval and reading. Conversely, compressing sessions into summaries or user-specific facts (as seen in prior work such as Zhong et al. () and Du et al. ()) can lead to information loss, harming the performance of the system to answer detailed questions. In Section 5.2, we compare three value representation strategies: storing entire sessions, decomposing sessions into individual rounds, and further applying summary/fact extraction.

#### CP 2: Key

Even when sessions are decomposed and compressed, each item still contains substantial information, with only a fraction relevant to the user’s query. Therefore, using the value itself as the key, a common practice in prior works (Zhong et al., ; Maharana et al., ), may be suboptimal. We introduce a key expansion approach in Section 5.3, where summaries, keyphrases, user facts, and timestamped events are extracted from the values to augment the index. This optimization highlights the key information and enables effective retrieval with multiple pathways.

#### CP 3: Query

For straightforward user queries, the aforementioned key-value optimizations may yield high retrieval accuracy. However, when queries involve temporal references (e.g., “Which restaurant did you recommend last weekend?”), naive similarity search proves insufficient. We address this with a time-aware indexing and query expansion strategy, where values are indexed with timestamped events, and retrieval is restricted to items within the relevant time range (Section 5.4).

#### CP 4: Reading Strategy

Answering complex queries may require recalling numerous memory items. Although the retrieval accuracy can be enhanced through the preceding designs, it does not guarantee that the LLM can effectively reason over the extensive context (Shi et al., ; Liu et al., ). In Section 5.5, we explore reading strategies and demonstrate that optimizations such as extracting key information before answering (Chain-of-Note, (Yu et al., )) and using structured format prompting (Yin et al., ) are crucial for achieving high reading performance.

Figure: Figure 5: A unified view of a chat assistant with long-term memory in operation. We formulate three stages and four control points (CP). We provide further examples in Appendix C.
Refer to caption: x6.png

## 5 Experiment Results

### 5.1 Experimental Setup

We mainly study three LLMs: GPT-4o, Llama 3.1 70B Instruct, and Llama 3.1 8B Instruct. For the retriever, we choose the 1.5B Stella V5 embedding (Zhang, ), given its high performance on MTEB (Muennighoff et al., ). Extensive comparisons between Stella V5 and alternative retrievers are provided in Section E.1. For the indexing stage, we employ Llama 3.1 8B Instruct to extract summaries, keyphrases, user facts, and timestamped events. All tasks use in-context learning, except for summaries and keyphrases, as detailed in Appendix D. When sessions or rounds are used as the key, we only keep the user-side utterances. In the reading stage, the retrieved items are always sorted by their timestamp to help the reader model maintain temporal consistency. Throughout Section 5.2 to Section 5.4, we apply the two optimizations discussed in Section 5.5 by default. All experiments were conducted on a local server with 8 NVIDIA A6000 GPUs.

Figure: Figure 6: QA performance on LongMemEvalM with different value designs. Decomposing sessions into rounds improves the QA performance. For the multi-session reasoning questions, further representing the values with the extracted facts improves the QA accuracy.
Refer to caption: x7.png

### 5.2 Value: Decomposition improves RAG performance

Using LongMemEvalM, we compare different value choices in a budget-aware manner. As shown in Figure 6, decomposing sessions into rounds significantly enhances reading performance with GPT-4o as the reader, while performing similarly to non-decomposed sessions when using Llama 3.1 8B Instruct as the reader. However, despite their efficiency in token usage, replacing sessions or rounds with extracted summaries or facts negatively impacts QA performance due to information loss. The only exception is with multi-session reasoning questions, where fact decomposition consistently improves performance. We hypothesize this is because fact decomposition extracts the same type of information across all sessions in a more uniform and simplified format, aiding retrieval and reading (Chen et al., ). Finally, we observe that the optimal token budget varies by the reader’s capability: while Llama 3.1 8B Instruct’s performance drops sharply beyond 3k retrieved tokens, GPT-4o continues to improve even with over 20k retrieved tokens.

**Table 2: Retrieval and end-to-end QA performance on LongMemEvalM with different key designs for indexing. L3.1 = Llama 3.1 Instruct. We find applying document expansion with the extracted user facts (row K = V + fact) greatly improves both retrieval and the downstream QA.**
| Key Design | Retrieval | End-to-End QA |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metrics@5 | Metrics@10 | GPT-4o | L3.1 70B | L3.1 8B |  |  |  |  |  |  |
| Recall | NDCG | Recall | NDCG | Top-5 | Top-10 | Top-5 | Top-10 | Top-5 | Top-10 |  |
| Value = Round |  |  |  |  |  |  |  |  |  |  |
| K = V | 0.582 | 0.481 | 0.692 | 0.512 | 0.615 | 0.670 | 0.600 | 0.624 | 0.518 | 0.534 |
| \hdashlineK = fact | 0.530 | 0.411 | 0.654 | 0.449 | 0.588 | 0.664 | 0.564 | 0.610 | 0.510 | 0.534 |
| K = keyphrase | 0.282 | 0.159 | 0.392 | 0.303 | 0.425 | 0.489 | 0.404 | 0.450 | 0.378 | 0.432 |
| \hdashlineK = V + fact | 0.644 | 0.498 | 0.784 | 0.536 | 0.657 | 0.720 | 0.638 | 0.682 | 0.566 | 0.572 |
| K = V + keyphrase | 0.478 | 0.359 | 0.636 | 0.410 | 0.541 | 0.652 | 0.538 | 0.620 | 0.472 | 0.524 |
| Value = Session |  |  |  |  |  |  |  |  |  |  |
| K = V | 0.706 | 0.617 | 0.783 | 0.638 | 0.670 | 0.676 | 0.592 | 0.570 | 0.524 | 0.464 |
| \hdashlineK = summary | 0.572 | 0.448 | 0.648 | 0.468 | 0.554 | 0.252 | 0.498 | 0.512 | 0.444 | 0.216 |
| K = fact | 0.642 | 0.524 | 0.814 | 0.571 | 0.644 | 0.512 | 0.544 | 0.550 | 0.470 | 0.404 |
| K = keyphrase | 0.482 | 0.375 | 0.576 | 0.401 | 0.618 | 0.498 | 0.440 | 0.450 | 0.388 | 0.414 |
| \hdashlineK = V + summary | 0.689 | 0.608 | 0.749 | 0.624 | 0.658 | 0.666 | 0.568 | 0.560 | 0.518 | 0.494 |
| K = V + fact | 0.732 | 0.620 | 0.862 | 0.652 | 0.714 | 0.700 | 0.588 | 0.584 | 0.530 | 0.490 |
| K = V + keyphrase | 0.710 | 0.587 | 0.768 | 0.602 | 0.665 | 0.672 | 0.590 | 0.566 | 0.526 | 0.508 |

### 5.3 Key: Multi-key indexing improves retrieval and RAG

In Table 2, we first explore whether summaries, keyphrases, or user facts condensed from the value can serve as better keys than the value itself. Interestingly, despite their more focused semantics, using these condensed forms alone does not enhance the memory recall performance. We hypothesize that this is due to the retriever’s ability to already effectively handle long-text semantics.

To leverage both the highlighted information from compression and the completeness of the original value, we applied a simple document expansion technique (Tao et al., ; Efron et al., ), where the compressed information is concatenated with the original value to form the key during indexing(^3^33We also experimented with merging at the retrieval stage by combining the ranks from different pathways, but it underperformed compared to indexing-stage merging. See Section E.2 for details.). This approach, particularly when using user facts, yielded an average improvement of 4% in retrieval metrics and 5% in final accuracy across all models. In Section E.1, we further analyze different different retrievers and find with alternative retrievers, key expansion with summary and keyphrases could improve Recall@5 when session is used as the value granularity. These results suggest that multi-pathway retrieval can significantly enhance memory retrieval performance. In the following section, we will investigate the time constraint as another pathway to optimize.

### 5.4 Query: Time-aware query expansion improves temporal reasoning

A key challenge highlighted by LongMemEval in building real-world assistant systems is the need to utilize temporal information present in both metadata and user utterances to correctly answer time-sensitive queries. To address this need, we introduce a simple yet effective time-aware indexing and query expansion scheme. Specifically, values are additionally indexed by the dates of the events they contain. During retrieval, an LLM $\mathcal{M}_{T}$ extracts a time range for time-sensitive queries, which is used to filter out a large number of irrelevant values.

As shown in Table 3, this simple design improves recall by an average of 11.4% when using rounds as the value and by 6.7% when using sessions as the value. This improvement remains consistent when key expansion is applied during indexing. We also find that the effectiveness of this method depends on using a strong LLM for $\mathcal{M}_{T}$ to accurately infer time ranges from queries. Llama 8B, on the other hand, struggles to generate accurate time ranges, often hallucinating or missing temporal cues even with numerous in-context examples. Further analysis is provided in Section E.3.

**Table 3: Retrieval performance on the temporal reasoning subset of LongMemEvalM. Time-aware query expansion significantly facilitates retrieval by narrowing down the retrieval scope.**
| Key Setting | Value = Session | Value = Round |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metric@5 | Metric@10 | Metric@5 | Metric@10 |  |  |  |  |  |
| Recall | NDCG | Recall | NDCG | Recall | NDCG | Recall | NDCG |  |
| (1) K = V | 0.639 | 0.630 | 0.651 | 0.707 | 0.421 | 0.462 | 0.499 | 0.511 |
| (1) w/ Query Expansion ($\mathcal{M}_{T}$ = GPT-4o) | 0.654 | 0.660 | 0.707 | 0.679 | 0.451 | 0.565 | 0.495 | 0.538 |
| (1) w/ Query Expansion ($\mathcal{M}_{T}$ = Llama 3.1 8B Instruct) | 0.624 | 0.627 | 0.647 | 0.692 | 0.384 | 0.448 | 0.489 | 0.488 |
| \hdashline(2) K = V + fact | 0.684 | 0.688 | 0.721 | 0.782 | 0.489 | 0.500 | 0.550 | 0.598 |
| (2) w/ Query Expansion ($\mathcal{M}_{T}$ = GPT-4o) | 0.722 | 0.732 | 0.797 | 0.758 | 0.526 | 0.548 | 0.722 | 0.669 |
| (2) w/ Query Expansion ($\mathcal{M}_{T}$ = Llama 3.1 8B Instruct) | 0.677 | 0.688 | 0.711 | 0.744 | 0.481 | 0.532 | 0.570 | 0.647 |

Figure: Figure 7: Question answering performance under the oracle retrieval setting. CoN with JSON format outperforms the other three parameter combinations by a large margin.
Refer to caption: x8.png

### 5.5 Improving reading with chain-of-note and structured format

Since LongMemEval requires synthesizing information from multiple sessions in the interaction history, even an optimal memory retrieval mechanism might need to return several items to capture all relevant details. This makes correctly extracting and reasoning with retrieved information challenging. To enhance the model’s ability to handle long contexts, we applied two key optimizations. First, we present retrieved items in a structured JSON format (Yin et al., ), which helps the model clearly recognize memory items as structured data for processing. Additionally, we adopt the Chain-of-Note (CoN) reading approach (Yu et al., ), instructing the LLM to first extract information from each memory item and then reason based on these notes. This effectively decomposes the long-context reading task into two simpler subtasks: extracting important details through copying, and reasoning with the concise notes. As shown in Figure 7, even under the oracle retrieval setting, a suboptimal reading strategy can result in up to a 10-point absolute performance drop compared to the recommended approach. Notably, when CoN is not applied, the JSON format does not consistently outperform the natural language format. However, with CoN, the JSON format consistently benefits reader LLMs of various sizes and capabilities. In Section E.4, we provide an analysis of the error patterns of different LLMs with the enhanced reading strategy.

## 6 Conclusion

In this study, we introduced LongMemEval, a comprehensive and challenging benchmark designed to evaluate the long-term memory abilities of chat assistants across five core memory tasks: information extraction, cross-session reasoning, temporal reasoning, knowledge updates, and abstention. Through extensive experiments with both commercial systems and long-context LLMs, we demonstrated the significant challenges posed by LongMemEval, with current systems exhibiting substantial performance drops. By analyzing key design choices across indexing, retrieval, and reading stages, we proposed effective strategies such as session decomposition, fact-augmented key expansion, and time-aware query expansion, which collectively improved both memory recall and the question answering performance. Our findings highlight the need for more sophisticated memory mechanisms to achieve personalized and reliable conversational AI, and LongMemEval offers a valuable benchmark to drive future advancements in long-term memory capabilities for chat assistants.

## Reproducibility Statement

In the paper writing and the subsequent code release, we are committed to enabling other researchers to easily leverage our resources, replicate our results, and build upon our findings.
We have documented the benchmark construction process in detail in Section 3.2 and Section A.1, including all the attributes and instructions used to prompt LLMs. We have created and will publicly release the two fixed evaluation datasets, LongMemEvalS and LongMemEvalM. In addition, we will also release the algorithm and source mixture used to create these two datasets, so that future studies could build upon them to create chat histories of any length. Finally, we have meticulously documented all the implementation details of our memory optimization in Appendix D, and our code will be released along with the benchmark as well. We believe that these efforts of transparency can help advance the field and foster future research endeavors.

## Appendix A Supplemental Details for LongMemEval

### A.1 Dataset Construction

In this section, we discuss the details of our benchmark construction process.

#### Attribute Ontology

In Table 4, we provide the full attribute ontology used in LongMemEval. This attribute ontology is constructed manually to reflect commonly mentioned topics in user-assistant chats. Five major categories are included: demographic information, lifestyle, situational context, life events, and belongings.

**Table 4: Human-constructed user attribute ontology. Each attribute represents a unique dimension of human experience along which a user biography could be constructed. For LongMemEval, we sample user backgrounds based on each of the attributes and construct questions on top of them.**
| Attribute 1: Demographic Information |
| --- |
| age, gender, ethnicity, nationality, language, education level, occupation |
| Attribute 2: Lifestyle |
| 2.1: Shopping: online shopping frequency, favorite stores, loyalty program, sales events, coupons, gift purchasing habits, eco-friendly product preferences, luxury vs budget shopping, technology gadget purchasing, fashion and apparel, grocery shopping, shopping for others |
| 2.2: Media Consumption: book, movie, tv show, music, podcast, video game, streaming service, theater, magazine and newspaper, youtube, educational content, audiobook and e-book |
| 2.3: Social Media Engagement: posting, commenting, followers, groups, hashtags, campaigns, messaging, live streaming, social media breaks |
| 2.4: Daily Routines: wake-up time, bedtime, work or school start time, meal time, exercise routines, coffee or tea break, commuting, evening activities, weekend routines, cleaning schedules, time spent with family or friends |
| 2.5: Travel: frequency, destination, road trips, travel agencies, outdoor adventures, airlines, hotel, travel with family vs solo travel, packing habits |
| 2.6: Recreation: reading, painting, musical instruments, dancing, watching sports, participating in sports, gardening, bird watching, fishing or hunting, board games, video games, fitness classes, yoga, sculpting, photography, stand-up comedy, writing, collecting, model building, aquarium keeping |
| 2.7: Eating and Cooking: home cooking, food delivery, vegetarian or vegan, favorite cuisines, snacking habits, barbecue, baking, cocktail mixing, cooking classes |
| 2.8: Event Participation: concerts, theater, galleries and museums, sports games, film festivals, religious services, book readings, charity events, trade shows, lectures or workshops, theme parks, local markets, networking events, sports, auto racing, workshops, museum tours |
| Attribute 3: Situational Context |
| 3.1: Home: living room, kitchen, bathroom, room style, room lighting, furniture, technology, plants |
| 3.2: Social Context: alone, family, friends, interactions with strangers |
| 3.3: Time Context: time of day, day of week, seasonal |
| Attribute 4: Life Events |
| graduations, academic achievements, study abroad, significant academic projects, job promotions, starting a business, births and adoptions, marriages, family reunions, illness or surgeries, mental health journeys, purchasing a home, trips, movement, living abroad, refugee or immigration, loss of loved ones, name change, belief, milestone |
| Attribute 5: Belongings |
| cars, bikes, vehicles, computer, phone, pet, farm animal, animal care items, home, land, art, antiques, collectible, rare items, clothing, jewelry, shoes, bag, sports gear, musical instruments, health related devices, crafting, photography |

#### Background Sampling

Based on each of the attributes, we prompt Llama 3 70B Instruct to generate a background paragraph outlining the user memory and experience. In our preliminary studies, we find that the following zero-shot prompt in Figure 8 can already guide the model to generate a long and focused user background with sufficient details, which suffices for the next step of question creation. We thus use the same prompt for the final version of LongMemEval.

Figure: Figure 8: The prompt for constructing user backgrounds based on an attribute.

#### Question Construction

As discussed in Section 3.2, based on the generated user backgrounds, Llama 3 70B Instruct is used to propose question and answers for each question type. Additionally, for the question type temporal-reasoning, multi-session reasoning, and single-session-preference, we use GPT-4o to propose several questions. Nevertheless, we find most of the questions to be unsatisfactory and manually filter and edit most of the questions. In total, approximately 1000 questions were generated for each question type, and the final yield rate is about 5%. For each question, we then manually decompose the answer into the evidence statements. If the question or the evidence statements involve time mentions, we assign a timestamp to the question and the evidence statements at this stage. Note that if timestamps are specified for the evidence statements at this stage, these timestamps will always be used for corresponding evidence sessions. Otherwise, the timestamps will be randomly assigned at the history construction stage with all the other sessions.

Figure: Figure 9: The prompt for instruction an LLM to act as a user and initiates a task-oriented dialogue with another LLM. Both the background and the evidence statement is provided. This prompt is used for the question type single-session. For the other question types, the prompt components are slightly different but the prompt overall follows the same style.

#### Evidence Session Construction

Using the question and the decomposed evidence statements, we use Llama 3 70B Instruct to simulate one user-AI chat history per evidence statement via self-chat. In Figure 9, we present an example of the chat simulation prompt, where we ask the user LLM to indirectly mention the evidence statement while avoiding to talk about other evidence statements for the question, if there are any. We include two crucial instructions in the prompt to make sure (1) the evidence statement is provided in an indirect way and (2) the generated messages are concise and thus mimic the style of user messages. On the assistant side, we directly provide the input generated by the user LLM without any prompt engineering. We simulate the chat for 10 round at maximum and stopped prematurely when either side of the LLMs generates an empty sequence indicating the end of the conversation.

Subsequently, expert annotators manually inspect and edit each of the generated sessions to ensure that (1) the required evidence statements are present in the conversation, (2) no other evidence statements are leaked into the conversation, (3) the evidence statements are provided in a colloquial style, especially for the data and time mentions, and (4) the conversation ends gracefully. In total, roughly 70% of the sessions are human edited. We note that in a few rare instances, the user LLM fails by assuming the assistant role instead. When these failures are identified, we discard the instance if the conversation cannot be fixed.

### A.2 History Construction

In order to construct a coherent and freely extensible chat history, we design a three-staged pipeline that include session pool construction, session sampling, and timestamp resolution.

#### Session pool construction

For each question, we draw the history sessions from three sources: ShareGPT (Zheng et al., ), UltraChat (Ding et al., ), and the simulated sessions corresponding to other attributes using the same pipeline mentioned in the previous section. This pool ensures that the non-evidence history sessions have similar topic or format as the evidence sessions, while avoiding providing conflicting information that would invalidate the question.

#### Session sampling

To sample a history containing $x$ sessions, we randomly sample from the aforementioned three sources and shuffle the sessions together with the question’s evidence sessions. For LongMemEval, we always use the following mixture: 25% ShareGPT, 25% UltraChat, and 50% simulated sessions. If the evidence sessions need to follow a specific order, we swap their orders accordingly after shuffling.

#### Timestamp resolution

Finally, we randomly assign timestamps to the session following their order of the history. If the evidence sessions are associated with pre-defined timestamps, we use them as anchors to determine the range of timestamp of the non-evidence sessions preceding or following them. Other wise, we randomly assign tiemstamps in May 2023.

### A.3 Evaluation Metric Building

To accurately evaluate the diverse responses of LLMs, we use an expert-written prompt to instruct GPT-4o as the correctness judge. We present the full prompt in Figure 10. To enable the model to handle detailed edge cases as how expert evaluators would do, we design separate prompts for a number of tasks. To ensure the prompt has a high agreement with expert judge, we sample 30 questions per problem type, collect the long-context generation results from GPT-4o and Llama 3.1 8B Instruct, and report the judgment correctness by category.

As shown in Table 5, the prompt-engineered GPT-4o judge achieves reliable performance in evaluating both GPT-4o and Llama-3.1-8B-Instruct as the generation model. The evaluator’s judgment slightly deviates from human experts for the single-session-preference and abstention problems due to the open-ended nature of the response. Nevertheless, our evaluation prompt still achieves 90% or higher accuracy under all settings. We will include this prompt in the benchmark package that we will release to enable consistent comparisons for future work.

Figure: Figure 10: Evaluation instructions for the GPT-4o judge. We provide the question, answer, and the model’s hypothesis after the instruction and ask GPT-4o to directly generate “yes” or “no”.

**Table 5: Meta-evaluation results of prompt-engineered GPT-4o judge. We observe a high evaluation accuracy across all the problem types in LongMemEval.**
| Question Type | Answer Model |  |
| --- | --- | --- |
| GPT-4o | Llama-3.1-8B-instruct |  |
| single-session-user | 1.00 (30/30) | 0.97 (29/30) |
| single-session-assistant | 1.00 (30/30) | 1.00 (30/30) |
| single-session-preference | 0.90 (27/30) | 0.97 (29/30) |
| multi-session | 1.00 (30/30) | 1.00 (30/30) |
| knowledge-update | 1.00 (30/30) | 1.00 (30/30) |
| temporal-reasoning | 1.00 (30/30) | 0.97 (29/30) |
| abstention | 0.97 (29/30) | 0.90 (27/30) |
| Average | 0.98 | 0.97 |

## Appendix B A Human Study on Commercial Memory Chatbots

We evaluate two commercial memory-augmented chatbots: ChatGPT (OpenAI, ) and Coze (Coze, ). We randomly selected 97 questions and created a short chat history of 3-6 sessions by sampling according to a mixture of 50% ShareGPT and 50% simulated sessions. We skip two type of questions that the assistants cannot answer: (1) a subset of temporal-reasoning, since our manual analysis cannot afford interacting with the (potentially evolving) systems across multiple months, and (2) single-session-assistant, since the systems do not remember any information given by the assistant. We also did not evaluate the abstention ability of these two systems because an early version of the dataset without any abstention questions was used for this analysis.

Since these systems only support memory features via their web interfaces, human annotators manually interacted with the chat assistants session-by-session, ending with a new session where the question was posed. After collecting the model’s response, the annotator manually evaluates the answer’s correctness. Finally, to start evaluating the next instance from a fresh state, the annotator manually clears the model’s memory through the web interface. We distribute the questions across five annotators. A discussion is performed among the annotators whenever there is a concern with the model’s response or with the evidence sessions. All evaluations were conducted in the first two weeks of August 2024.

In Table 6, we present the detailed human evaluation results by problem types. We observe that when the task is memorizing the information from a single session (column IE), both systems can answer a considerable number of problems correctly. However, for the other question types where aggregation across multiple sessions is generally required, both systems exhibit significant performance drops. Compared to ChatGPT, we find most of Coze’s errors are due to failing to record information from some session. On the other hand, ChatGPT generally records the evidence statements immediately after it has been presented in the evidence session. However, as the interaction proceeds, ChatGPT often modify this information when it compresses the history, resulting in information loss. This highlights the potential trade-off between reliable personalization and efficiency.

**Table 6: Human evaluation results of two systems categorized by evaluated ability types. We use the questions from single-session-user for the IE column. For the temporal reasoning column (TR), we use the questions that do not require reasoning with metadata.**
| System | Model | Memory Ability |  |  |  |
| --- | --- | --- | --- | --- | --- |
| IE | MR | KU | TR |  |  |
| ChatGPT | GPT-4o-mini | 1.000 | 0.647 | 0.667 | 0.652 |
| GPT-4o | 0.688 | 0.441 | 0.833 | 0.435 |  |
| Coze | GPT-3.5-turbo | 0.625 | 0.118 | 0.375 | 0.043 |
| GPT-4o | 0.813 | 0.147 | 0.208 | 0.391 |  |

## Appendix C Existing Memory Systems from the Unified View

In Section 4, we have presented a unified view of the memory systems with three stages and four control points. In this section, we provide a closer look at nine memory-augmented chat systems and view them as specific instances of the unified framework. Specifically, we consider in-context RAG (Shi et al., ), Memorybank (Zhong et al., ), LD-Agent (Li et al., ), CoN (Yu et al., ), ChatGPT, Coze, RAPTOR (Sarthi et al., ), MemWalker (Chen et al., ), and HippoRAG (Gutiérrez et al., ). Table 7 provides an overall comparison between the systems, among which we also situate the recommended design identified through the paper’s empirical study.

#### Indexing

For the value representation, most of the surveyed systems either directly use the sessions themselves or use a mixture of them with the extracted facts or summaries. Only three systems (LD-Agent, ChatGPT, and Coze) use only the compressed values to replace the original ones, which may incur an information loss. When it comes to the indexing key, we observe three major types of decisions. First, a number of systems (in-context RAG, Memorybank, LD-Agent, and CoN) simply use the values as the keys, which wins in terms of simplicity. By comparsion, HippoRAG builds an entity-centric index, and RAPTOR/Memwalker build a hierarchical index using recursive summarization. It is noteworthy that while a more complex memory indexing structure can potentially benefit certain types of queries, they also increase the cost of creating and maintaining the index in online interactions. Specifically, for HippoRAG, RAPTOR, and Memwalker, some level of re-indexing is required when a new session is added to the memory, increasing the computational overhead of these sytems.

#### Retrieval

For most of the systems, we find that the question is generally used as the query, which is intuitive since the questions are generally short. LD-Agent and HippoRAG extract additional information and leverage them to better pinpoint entity-centric knowledge within the index. In terms of the query-key matching process, most systems use a flat retrieval setup, where a similarity search is directly conducted between the query and the keys. HippoRAG uses Personalized PageRank (PPR) to leverage its entity graph structure to match passages that feature entities close to the seed entities extracted from the query. Different from these approaches, MemWalker bridges the retrieval and inference through an interactive reading mechanism where the reader LLM self-navigates the indexing structure to find the answer. This method brings the additional robustness in case the retriever fails, but also incurs additional latency costs due to more LLM inference.

#### Generation

Most of the systems apply a direct reading mechanism where the reader model is provided with the retrieved memory items and directly asked to generate the response. However, as we show in the main results, adapting an extract-before-read strategy is important to a high performance. On the other hand, the interactive reading mechanism such as that used in MemWalker allows the reader to backtrack and search again in case the recalled memory is inadequate.

**Table 7: A comparison of nine memory-augmented frameworks through the lens of the proposed unified framework. For ChatGPT and Coze, we skip several designs as they are unknown. We also remark that they may have other value representations that are not directly exposed to the user.**
| Method | Value | Key | Query | Retrieval | Time-aware | Reading |
| --- | --- | --- | --- | --- | --- | --- |
| In-context RAG | round/session | K = V | question | flat | No | direct |
| MemoryBank | summary + round | K = V | question | flat | Yes | direct |
| LD-Agent | summary + fact | K = V | keyphrase | flat | Yes | direct |
| CoN | round/session | K = V | question | flat | No | CoN |
| ChatGPT | fact | - | - | - | No | - |
| Coze | fact | - | - | - | No | - |
| RAPTOR | round/session | node summary | question | flat/interactive | No | - |
| MemWalker | round/session | node summary | question | interactive | No | interactive |
| HippoRAG | round/session | entity | entity | PPR | No | direct |
| Our Design | round | K = V + fact | question + time | flat | Yes | CoN |

## Appendix D Memory Optimizations: Implementation Details

In this section, we describe our implementations of the memory optimizations in detail.

#### Value Decomposition

We design an LLM-based method for compressing the value, which are either sessions or rounds, into three different formats: summaries, keyphrases, or user facts. In Figure 11, we present the corresponding zero-shot and few-shot prompts. Note that few-shot learning is only applied for user fact extraction, and we find it to be unnecessary for the rest two tasks. For in-context examples, we randomly sample ten examples, five from the simulated user sessions and five from ShareGPT. The expected responses are generated by GPT-4o and modified by human annotators. We manually tune the prompts by observing several examples. Llama 3.1 8B Instruct is used for all the extraction experiments. Note that for all the experiments, we only provide the user-side messages for the extraction, skipping the assistant’s responses.

#### Key Expansion

To expand the value documents with summaries, keyphrases, or user facts, we follow the same pipeline in the previous section to first extract these pieces of information from each value. Then, we directly prepend the extracted information to the corresponding key.

#### Time-Aware Indexing and Query Expansion

At the indexing stage, we instruct Llama 3.1 8B Instruct to extract the event mentions in the text whose timestamp could be inferred. At the retrieval stage, we instruct an LLM (we explore GPT-4o and Llama 3.1 8B Instruct) to extract the a time range for retrieval for the problems that specifically focus on a range of time. We present the corresponding prompts in Figure 12. Ten human-written in-context examples are additionally provided.

#### Reading Strategy

We present the prompt for reading in Figure 13, which is a small variation of Chain-of-Note (CoN). Overall, the prompt shares the same idea with CoN, that the model is asked to traverse the documents and extract the evidence before generating the answer to the question. We use greedy search for all the experiments, and set the maximum generation length to 800 tokens.

Figure: Figure 11: Zero-shot prompts for extracting information from the value items for indexing. For user fact extraction, we additionally provide ten in-context examples.

Figure: Figure 12: Zero-shot prompt for extracting timestamped events from the value items for indexing and the prompt for extracting time range from the user question. For both settings, we additionally provide ten in-context examples.

Figure: Figure 13: The prompt for reading with recalled memory with and without Chain-of-Note.

## Appendix E Extended Analyses

We provide additional analyses on LongMemEval, including retriever selection, memory optimization, and error analyses of end-to-end retrieval-augmented generation.

### E.1 Ablations on Retriever Selection

In Table 8, we compare the performance of Stella V5 with two alternative popular retrievers: BM25 (Robertson & Zaragoza, ) and Contriever (Izacard et al., ). We do not report the results of retrievers that use larger embedding models, as their latency is generally unrealistic for real applications. The retrievers are compared under two settings: (1) vanilla key design where the values themselves are used as the key and (2) the key expansion strategies introduced in the main text. We list the key observations below:

- •
Both dense retrieval embeddings have significantly higher performance than the BM25 sparse retrieval method. The trend is the same across most settings.
- •
The recommended technique of performing key expansion with user fact consistently improves the performance over directly using the value as the key.
- •
Key expansion with summary and keyphrases could improve the performance in some settings. For example, both summary and keyphrase improve the Recall@5 of Contriever when session is used as the value granularity. However, expanding the key with facts still gives the greatest performance gain.

**Table 8: A comparison between three retrievers under different key-value indexing settings.**
| Key Design | Retriever | Value = session | Value = round |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metric@5 | Metric@10 | Metric@5 | Metric@10 |  |  |  |  |  |  |
| Recall | NDCG | Recall | NDCG | Recall | NDCG | Recall | NDCG |  |  |
| K = V | BM25 | 0.634 | 0.516 | 0.710 | 0.540 | 0.472 | 0.352 | 0.538 | 0.372 |
| Contriever | 0.723 | 0.634 | 0.823 | 0.663 | 0.589 | 0.454 | 0.747 | 0.495 |  |
| Stella V5 1.5B | 0.720 | 0.594 | 0.794 | 0.615 | 0.660 | 0.498 | 0.784 | 0.528 |  |
| K = V + summary | BM25 | 0.626 | 0.544 | 0.694 | 0.565 | - | - | - | - |
| Contriever | 0.732 | 0.632 | 0.823 | 0.657 | - | - | - | - |  |
| Stella V5 1.5B | 0.689 | 0.608 | 0.749 | 0.624 | - | - | - | - |  |
| K = V + fact | BM25 | 0.683 | 0.560 | 0.757 | 0.582 | 0.554 | 0.454 | 0.608 | 0.473 |
| Contriever | 0.762 | 0.632 | 0.862 | 0.658 | 0.612 | 0.489 | 0.756 | 0.530 |  |
| Stella V5 1.5B | 0.732 | 0.520 | 0.862 | 0.552 | 0.644 | 0.498 | 0.784 | 0.536 |  |
| K = V + keyphrases | BM25 | 0.632 | 0.546 | 0.711 | 0.569 | 0.453 | 0.391 | 0.538 | 0.416 |
| Contriever | 0.740 | 0.638 | 0.849 | 0.668 | 0.546 | 0.450 | 0.672 | 0.489 |  |
| Stella V5 1.5B | 0.710 | 0.587 | 0.768 | 0.602 | 0.478 | 0.359 | 0.636 | 0.410 |  |

### E.2 Post-retrieval rank merging for index expansion

In Section 5.3, we have introduced and evaluated the key expansion strategy for multi-path retrieval, where important information is extracted from the value items and then used to augment the key. However, an alternative strategy is to directly create a separate key with the retrieved information and place it in parallel as the original keys. At the retrieval stage, the query could be used to retrieve from pools defined multiple types of keys and the values from different sequences could be merged subsequently according to their rank. Effectively, this implements the true “multi-path retrieval. We call this strategy “rank merging”. In Table 9, we thoroughly evaluate rank merging verse key merging, the strategy we recommended in the main text. We find that rank merging has much lower performance than key merging. One potential reason is that rank merging increases the index size by $m+1$ times, where $m$ is the number of information pieces extracted from each (key, value) pair. By comparison, key merging highlights the important information while avoiding explording the size of the index.

**Table 9: Retrieval and end-to-end QA performance on LongMemEvalM with different key designs for indexing. L3.1 = Llama 3.1 Instruct. RM = Rank Merging and KM = Key Merging.**
| Key Design | Retrieval | End-to-End QA |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metrics@5 | Metrics@10 | GPT-4o | L3.1 70B | L3.1 8B |  |  |  |  |  |  |
| Recall | NDCG | Recall | NDCG | Top-5 | Top-10 | Top-5 | Top-10 | Top-5 | Top-10 |  |
| Value = Round |  |  |  |  |  |  |  |  |  |  |
| K = V | 0.582 | 0.481 | 0.692 | 0.512 | 0.615 | 0.670 | 0.600 | 0.624 | 0.518 | 0.534 |
| \hdashlineK = V + fact (RM) | 0.478 | 0.372 | 0.568 | 0.403 | 0.558 | 0.596 | 0.536 | 0.586 | 0.458 | 0.482 |
| K = V + keyphrase (RM) | 0.386 | 0.376 | 0.536 | 0.329 | 0.485 | 0.590 | 0.460 | 0.558 | 0.426 | 0.466 |
| \hdashlineK = V + fact (KM) | 0.644 | 0.498 | 0.784 | 0.536 | 0.657 | 0.720 | 0.638 | 0.682 | 0.566 | 0.572 |
| K = V + keyphrase (KM) | 0.478 | 0.359 | 0.636 | 0.410 | 0.541 | 0.652 | 0.538 | 0.620 | 0.472 | 0.524 |
| Value = Session |  |  |  |  |  |  |  |  |  |  |
| K = V | 0.706 | 0.617 | 0.783 | 0.638 | 0.670 | 0.676 | 0.592 | 0.570 | 0.524 | 0.464 |
| \hdashlineK = V + summary (RM) | 0.608 | 0.488 | 0.684 | 0.511 | 0.600 | 0.620 | 0.510 | 0.544 | 0.468 | 0.466 |
| K = V + fact (RM) | 0.618 | 0.511 | 0.754 | 0.548 | 0.622 | 0.688 | 0.574 | 0.574 | 0.468 | 0.466 |
| K = V + keyphrase (RM) | 0.550 | 0.435 | 0.650 | 0.466 | 0.560 | 0.620 | 0.454 | 0.506 | 0.492 | 0.482 |
| \hdashlineK = V + summary (KM) | 0.689 | 0.608 | 0.749 | 0.624 | 0.658 | 0.666 | 0.568 | 0.560 | 0.518 | 0.494 |
| K = V + fact (KM) | 0.732 | 0.620 | 0.862 | 0.652 | 0.714 | 0.700 | 0.588 | 0.584 | 0.530 | 0.490 |
| K = V + keyphrase (KM) | 0.710 | 0.587 | 0.768 | 0.602 | 0.665 | 0.672 | 0.590 | 0.566 | 0.526 | 0.508 |

### E.3 Strong and weak LLMs for extracting time ranges from queries

In Section 5.4, we concluded that a strong model is required to accurately extract the time ranges from the questions, even if in-context examples (with balanced labels) are provided. In this section, we further illustrate this finding with examples in Table 10. We find that the Llama 3.1 8B Instruct model can generate a correct temporal range when the time references are clearly defined (example 2). However, in many of the cases where the question does not have any temporal reference (example 1, 2, 4), the model often mistakenly extracts a time range, which erroneously prunes out the search space, leading to a low memory recall. By contrast, GPT-4o is able to refuse to generate a time range when the question does not have a time reference.

**Table 10: Example of time span extraction outputs from GPT-4o and Llama 3.1 8B Instruct.**
| Example 1 |
| --- |
| Question Date: 2023/05/28 |
| Question: How long had I been taking guitar lessons when I bought the new guitar amp? |
| Predicted time range (GPT-4o): No date extracted [correct] |
| Predicted time range (Llama 3.1 8B Instruct): 2023/05/01$\sim$2023/05/28 [false positive] |
| Example 2 |
| Question Date: 2023/04/27 |
| Question: Which airline did I fly with the most in March and April? |
| Predicted time range (GPT-4o): 2023/03/01$\sim$2023/04/30 [correct] |
| Predicted time range (Llama 3.1 8B Instruct): 2023/03/01$\sim$2023/04/30 [correct] |
| Example 3 |
| Question Date: 2023/06/28 |
| Question: How many days before the ’Rack Fest’ did I participate in the ’Turbocharged Tuesdays’ event? |
| Predicted time range (GPT-4o): No date extracted [correct] |
| Predicted time range (Llama 3.1 8B Instruct): 2023/06/21$\sim$2023/06/28 [false positive] |
| Example 4 |
| Question Date: 2023/03/10 |
| Question: Which seeds were started first, the tomatoes or the marigolds? |
| Predicted time range (GPT-4o): No date extracted [correct] |
| Predicted time range (Llama 3.1 8B Instruct): 2023/03/01$\sim$2023/03/10 [false positive] |

### E.4 error analysis

In this section, we analyze the source of error of various LLMs with our best memory design. Specifically, we use round as the value granularity, expand the key with extracted user facts, and apply CoN as the reading strategy. Stella v5 1.5B is used as the retriever, and top-10 items are provided to the model with a JSON structured prompt. In Figure 14, we analyze the distribution of the correct and error cases for three different reader LLMs. First, we observe that a substantial proportion of errors corresponds to correct retrieval yet wrong generation (15%$\sim$19% of all instances, and 40%$\sim$50% among the error instances). The proportion is higher when a weaker reader LLM is used. This indicates that the reading strategy still has a large room of improvement. In addition, we observe that for the reader LLM to generate a correct answer, performing correct retrieval is necessary in $\sim$90% of the time. We observe that the rest 10% instances are mostly of the question type knowledge-update and the the retriever was only able to identify the updated knowledge but failed to retrieved the previous information before update. For these cases, our strict retrieval evaluation criteria will deem that the retrieval has failed. Overall, this result also highlights the high quality of LongMemEval, as the reader LLM cannot take any shortcut to answer the question correctly without a correct memory recall.

Figure: Figure 14: An analysis of the error distribution of different reader models. We use R and G to indicate correct Recall@10 and correct answer based on the top-10 retrieved results.
Refer to caption: x9.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. according to (c) in figure 3, it seems there are no answer located in 7 8 9 10 round.
2. in figure6, what motivate you to list multi-session subset independently? and why not analyze this in later experiemnts?
3. why some experiments use both llama 3.1 70b and 8b instruction model, and some only use 8b model?

other typos

1. the answer in (a) of figure 2 is wrong?
2. best performance of llama 3.1 70b when value = session on table 2 is wrong?
3. typo in line 468
soundness: 3
strengths: 1. the proposed benchmark is valuable, being diverse, scalable and practical.
2. the experimental results are solid and comprehensive and the analysis is insightful such as the choices of key, value and query expansion.
3. the presentation and organization is relatively clear and easy-to-understand.
summary: the paper proposes LONGMEMEVAL, a comprehensive benchmark designed to evaluate five core long-term memory abilities of chat assistants: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. Based on that, the present a unified framework that breaks down the long-term memory design into four design choices across the indexing, retrieval, and reading stages. Built upon key experimental insights, they propose several memory designs including session decomposition for optimizing value granularity, fact-augmented key expansion for enhancing the index structure, and time-aware query expansion for refining the search scope. Generally, i think this is a good paper and demonstrates some insightful observations via comprehensive experiments.
weaknesses: 1. lots of human intervention is required to ensure the quality of dataset, such as line 248
2. most of experiments are conducted using gpt4o and llama3.1 8b instruct models. there is no comprehensive analysis about exsiting models.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 2
questions: Please give feedback to Weaknesses -> No.1 and No.2.
soundness: 2
strengths: 1. It divides the memory workflow into three phases: indexing, retrieval, and reading. This segmentation not only helps to conceptualize the entire process but also lays the groundwork for systematically improving each stage. Then this paper further enhances this workflow. These modifications represent a thoughtful approach to refining how information is indexed, retrieved, and read from long-term memory. The process is sound and convincing.

2. This paper did a comprehensive evaluation across different existing models and provided detailed insights into how various models perform differently depending on the task—some models excel at reading (i.e., interpreting memory), while others are better at understanding or processing the retrieved information.
summary: This paper evaluates long-term memory abilities in dialogue tasks from extraction, multi-session reasoning, temporal ability, knowledge management, and abstention. They found that it's a challenge for existing LLMs to perform sustained interaction with those chat histories. To tackle this issue, this paper presents a unified framework to redefine the process when treating long-term memory. Beyond that, extensive experiments show optimizing in the new proposed framework can improve memory recall and QA ability in their benchmark. The key point of this work is to divide the pipeline workflow into single parts when facing long-term memory dialogue situations. They create larger data to make the long-term memory better represented, then make the model perform better.
weaknesses: 1. This work was generated by human-AI talk. However they did not mention how they control the hallucination and other issues when talking to AI chatbots (i.e., repeating problems, proportion of effective dialogue turns, etc.)

2. The proposed five aspects to evaluate long-term memory should be discussed more (Which aspect contributes more? Do different aspects have overlaps? What if one aspect performs better but another one fails?).

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Why is the proposed framework called a "unified" framework? It is very pipeline, not unified. 
2. Since the timestamp is randomly assigned, how to guarantee the evaluation of temporal reasoning?
soundness: 3
strengths: 1. Long-term dialogue is an important problem in the era of LLMs, which also lacks of comprehensive problem formulations. This work formulates several core memory capabilities in this problem.
2. Construct a new challenging benchmark for studying this problem.
3. Propose a unified framework for improving the long-term dialogue capabilities in LLM-based conversational systems.
summary: This paper focuses on the problem of long-term dialogues. To study this problem, this work constructs LongMemEval, a new benchmark for evaluating five long-term memory capabilities of chat assistants, including information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention.  Experimental results on this new benchmark indicate significant challenges in existing LLMs on long-term dialogues. Furthermore, the authors propose a new framework, consisting of several designs, such as session decomposition, fact-augmented key expansion, and time-aware query expansion. Further results show that these designs can improve both memory recall and downstream question answering.
weaknesses: 1. The construction of history sessions is not well motivated. The constructed history could be very noisy and the temporal order is just random. Not sure what is the motivation behind this messy historical session construction.
2. The proposed framework is actually a combination of several existing design. 
3. There are too many terms to remember in this work, which hampers the readability of this paper.
4. It would be better to include some long-term memory models to make a comparison for better presenting the challenge of the proposed benchmark and the effectiveness of the proposed framework.

---

confidence: 4
contribution: 3
initial_rating: 3
presentation: 1
questions: Please address the concerns/questions in the above presentation (P1-P3), soundness (S1-S5), contribution (C1-C2), and the following other questions.

Other:

1. The Ethics Statement is missing. 

* Since the LongMemEval dataset and the pilot study require human annotators, the ethics statement *should* be included, even though they are not as strongly recommended as the Reproducibility Statement. Moreover, given that this paper studies a three-stage framework for improving a chatbot's long-term memory, could this be exploited by malicious users to produce harmful results? I cannot find it in the appendix, however. 

2. Do the authors include the Limitations section? If not, then is this work thoroughly studied? What are the future work? 

3. What is the temperature used in this paper? What is the version of GPT-3.5 turbo tested in Figure 4-a?

4. In Figure 4-a, do the author have insights regarding why gpt-4o-mini is better than gpt-4o?

5. Do the authors touch upon the NDCG metric in the discussion section?

6. In Lines 419-420, are the timestamps sorted by ascending or descending order? Do the authors evaluate the performance when the timestamps are not sorted in the preliminary experiments?

7. In Table 2, why K = summary and K = V + summary experiments are not included in this paper? Is it because of the budget-aware manner? Moreover, why is the numeric value (0.590) in QA performance using Llama-3 (70B) in bold? It is smaller than the corresponding value when K = V (0.592).

8. The rounding is confusing.

* In the third contribution (Line 99), the authors report improvement is 7% ~ 11%, but it's 6.7% ~ 11.4% in Lines 480-481. Moreover, the result is for GPT-4o LLM, and this improvement does not hold when Llama-3.1 (8B) is used, as stated in Lines 482 (*We also find that the effectiveness of this method depends on using a strong LLM*). Nevertheless, this strong prerequisite is not mentioned in Line 99, which could be a bit misleading regarding the "7% ~ 11%" improvement.

* Can the author provide how to compute the numerical values in Line 467 (*This approach, particularly when using user facts, yielded an average improvement of 4% in retrieval metrics and 5% in final accuracy across all models.*)? Specifically, how to get the result of a 5% improvement in Table 2? Is it $1/3 * [(0.714-0.670)/0.670 + (0.584-0.570)/0.570 + (0.490-0.464)/0.464] = 0.04875$, then 4.9% rounds to 5%? If so, why **only** use Top-5 in GPT-4o and Top-10 in Llama-3.1 (70B and 8B)? Because I can get the 4% result in retrieval metrics if I average all the improvement in "Recall@5, NDCG@5, Recall@10, and NDCG@10": $1/4 * [(0.732-0.706)/0.706 + (0.620-0.617)/0.617 + (0.862-0.783)/0.783 + (0.652-0.638)/0.638] = 0.0411$ (4.1%); however, when I try to average all the top-5 and top-10 improvement in these LLMs (6 entries), it does not match 5%.

* Similar to the previous question, can the authors list the steps when computing the average metric in Lines 480-481? How can we get the 11.4% and 6.7% results from Table 3? Is there any additional rounding method involved after the computation?

9. Are Figure 4-b and Figure 7 related in any sense? If so, please explain the following questions.
* There is a discrepancy between Figure 4-b and Figure 7. Specifically, the oracle of GPT-4o is 0.924 and Llama-3.1 (*70*B, I guess) is 0.848 in Figure 4-b, and they can be found in Figure 7 (JSON + CoN). However, the oracle of Llama-3.1 (8B) is 0.710 in Figure 4-b, which is the natural language with CoN (NL + CoN). Since the JSON input format is not mentioned in Section 3.5, the authors should report when the input format is NL but not JSON. 
* In this vein, if I understand correctly, how can we derive the oracle of those LLMs w/o CoN from Figure 7? These values are not identical (I thought it could be done by looking at NL + Direct Prediction).
* Is it a coincidence that Llama-3.1 (8B) has no improvement when CoN is used?

10. Other typos and suggestions:
* To improve the clarity of this paper, including a simple, concrete example in Figure 5 (or mentioned in Sections 4 and 5) would be beneficial. Otherwise, readers must refer to Appendix D (in Figures 11 and 12) to know what keyphrases/user facts/query expansion is exactly.
* The term "round" is not precisely defined in this paper (see Line 377: decomposing sessions into individual rounds). I only find the session is defined in Section 3.1. For instance, can a session be decomposed into 2 (i.e., user and chat assistant) rounds? If more rounds ($>$ 2) are possible, how do you handle this process in this paper?
* cross-session $\rightarrow$ multi-session (for consistency). Note that other words need to be revised for consistency, yet it is too cumbersome to list all of them.
* In Line 65 (verb missing): ... Finally, we the coverage of five core ...
* In Line 138, the em dash is inconsistent.
* In Line 161, RAG is not mentioned in retrieval-augmented generation but is later used in Sections 5.2 and 5.3.
* In Lines 375-376, should use \citep.
soundness: 2
strengths: 1. This paper aims to tackle a problem of LLM from the time perspective, which may be a useful method.
2. Apart from the single-session, the knowledge update and the temporal reasoning are known issues in contemporary LLMs.
3. They study these phenomena in the long-context, long-term scenario.
summary: This paper proposes a 500 LongMemEval dataset focusing on evaluating five crucial long-term capabilities of LLMs: (single-session) information extraction, multi-session reasoning, knowledge update, temporal reasoning, and abstention. 
The authors define each conversation as a key-value pair, where the key is the "time index" and the value is the corresponding chat.
They propose a three-stage framework for indexing, retrieval, and reading.
The framework has four control points: key (in the indexing stage), value (in the indexing stage), query (in the retrieval stage), and reading strategy (in the reading stage).
They conduct experiments to test their framework using LongMemEval on GPT-4o and Llama-3 (70B and 8B).
weaknesses: This paper has several drawbacks regarding presentation, soundness, and contribution. 

TL;DR, the major weaknesses are as follows (see **Details** below for further elaboration):

**Weakness 1.** [Presentation and Soundness] **This paper lacks careful proofreading, as it contains an unusually high number of typos and inconsistent writing. The numerical rounding is also confusing.** For example, in the third contribution, the authors report improvement is 7% ~ 11% in Line 99, but it's 6.7% ~ 11.4% in Lines 480-481 (should report the exact value or *at least* add the term "around" in Line 99). **Additionally, the flow in each section is not strongly connected, requiring significant revision to reach a publishable standard. Many tables and figures need substantial revisions and adjustments. The authors also need to be crisp in their writing.** For one thing, in the main content, the "result and discussion" of their proposed framework only spans two pages (Sections 5.2 to 5.5), including two tables and one figure. Without these, the text is roughly a page only in the main paper, which is insufficient for the ICLR community. 

**Weakness 2.** [Presentation and Soundness] The LongMemEval dataset should be tested on other papers' existing baselines (with external systems like RAG). However, no "baseline section" is mentioned in this paper, and I cannot find the baseline mentioned in the "experimental setup" in Section 5.1 as well as in Tables 2 and 3 and their titles (e.g., "K = V is the baseline from xxx paper"). Hence, it is challenging to understand the prior works' baselines based on each individual's assumption. If non-familiar readers try to search for the "baseline" keyword, they will find it only appears once in the second contribution (in Line 95). After jumping to Section 5.3, they cannot find the recent papers conducting LLM with RAG baselines within this section either. If no comparable baselines are suitable for this task, see S2 below for suggestions.

**Weakness 3.** [Presentation and Soundness] **The authors claim this paper presents a *unified* framework, as shown in Figure 5, but without a formal mathematical definition, it is difficult to justify whether this is indeed true.** For example, how does the proposed framework encompass the existing RAG frameworks? Moreover, it is also hard for other future researchers to follow/revise/expand their ongoing work, particularly the four control points (CP) in Figure 5. Take the conducted CoN experiment (Section 5.5) as an example: The connection between the case when CoN is applied and Figure 5 is unclear. This figure is not illuminating, and the imprecise definition will let the readers interpret this paper on their own. Note that each separated component can be visualized and formalized in a better way (see Figure 1 in [0] and [1] for visualization; see also Section 3.1 in [1] for formalization). In addition, this paper only displays general terms without an explicit example to showcase their framework in Figure 5, whereas it is common for various RAG papers (see [2] and [3]). 

**Weakness 4.** [Contribution] **As for the dataset comparison, LongMemEval needs justification to significantly distinguish it from other related works, such as task-oriented dialogues (as stated in Line 50) and other QA-based datasets (see C2 below).** Specifically, several question types are highly similar to (that is, can be leveraged by) those existing datasets, especially the single-session scenarios. Nonetheless, they are missing in the references. Hence, further analysis is necessary regarding why not to leverage their datasets.

[0] Tree of Thoughts: Deliberate Problem Solving with Large Language Models, Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, Karthik Narasimhan, NIPS 2023

[1] Memory-Based Model Editing at Scale, Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning, Chelsea Finn, ICML 2022

[2] Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection, Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, Hannaneh Hajishirzi, ICLR 2024

[3] REPLUG: Retrieval-Augmented Black-Box Language Models, Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Richard James, Mike Lewis, Luke Zettlemoyer, Wen-tau Yih, NAACL 2024


# Details

## Presentation (P)

**P1. The contexts need to be reorganized to provide a clear flow.**

* As shown in Figure 2, the dataset construction pipeline is (a) Question Construction $\rightarrow$ (b) Evidence Session Construction $\rightarrow$ (c) History Construction. However, the (a) and (b) steps are merged into a single "paragraph" in Line 241 (within Section 3.2: LongMemEval: Benchmark Curation), while the (c) step has its own single "subsection" (Section 3.3: History Compilation).

* In the main paper, the results and discussion of their proposed framework only have two pages (Sections 5.2 to 5.5), including the two tables and a figure. More analysis should be moved to the main content. For instance, I find Table 7 in Appendix C informative, so perhaps the authors could consider moving this to the main content.

* There is a discrepancy between Section 4.2 and Figure 5. In Section 4.2, the authors use "CP 1: Value" and "CP 2: Key" as the paragraph title. However, it is "CP 1: Key" and "CP 2: Value" in Figure 5. 

* What is NL in Figure 7? I think it is natural language (as opposed to JSON format), but "NL" is not a common abbreviation and is not mentioned in the main content either. Moreover, this figure reports the result on LongMemEval_s, yet it is not mentioned in Figure 7 and Section 5.5. 

**P2. The figures/tables do not follow the order or ICLR format. Many of them require adjustment in format and location in this paper for better reference.**

* Should follow the ICLR format and place the table title on top (see Tables 1, 2, and 3). 
* Tables and Figures are placed on page "i" but first mentioned on page "i+1." For example, Table 1 is first mentioned on page 3 (Line 146) but placed on page 2. For another example, Figure 2 is first mentioned on page 5 (Line 242) but placed on page 4.
* As shown in Figures 2, 3, and 4, it really confuses me to place (a), (b), and (c) at the end. Take Figure 3 as an example: 
```
LongMemEval challenges chat assistants through its diverse question types (a), emphases on multi-session reasoning (b), 
and diverse evidence locations within sessions (c).
```
It should be:
```
LongMemEval challenges chat assistants through its (a) diverse question types, (b) emphases on multi-session reasoning, 
and (c) diverse evidence locations within sessions.
```
* The column separator line is inconsistent (Tables 1 and 3). 
* In Table 1, can the authors explain why you do not use "# data" but "# Q"?
* In Line 83, the term session has a different meaning compared to Table 1 (session means the total number of LongMemEval dataset in "with 500 sessions"). 
* The text in Figure 2 is small. Moreover, the "Answer: 5" is wrong in Question Construction (should be 50).
* In Figure 3-c, the round index 7 ~ 10 is "invisible," compared to the "barely visible" number of sessions = 6 in Figure 3-b. If they do have "height," perhaps insert the numeric values above these bars.
* In Figure 4-b, two "Llama-3.1 8B Instruct" are presented in both w/o CoN and w/ CoN. Which one is 70B? 
* Moreover, no explicit evaluation metric (accuracy) is found in Figure 4-b and its caption. While the authors mention "... compared to answering the questions based on only ...," this makes readers need to first observe it is the question answering task, then refer to Section 3.3 to conclude that this table is indeed using accuracy (but not Recall@k and NDCG@k).
* In Figure 5, presenting "key 4," "value chunk 2," etc. without a simple example is hard to grasp the idea. This should be done by showing different color blocks in "legend" (just like Figure 3) and providing a concrete example (possibly re-use the examples in the previous figures). Additionally, Figure 5 does not visualize a "smooth" pipeline; there is a break between the"(1) indexing" process and the "(2) retrieval and (3) reading" process. Specifically, there is no connection between the "key 4 and key 2" and the data storage system.

**P3. Too much emphasis on many terms and two (or more) adjectives to describe certain words.** Specifically, the overuse of "\textit" and compound words (excluding common words in NLP like long-term). While I could understand that the authors want to emphasize the *scenario* they aim to test and *why* not conduct more experiments etc., it further adds the difficulty to grasp the main point in this paper. The texts in italics are already overwhelming even without these. Below are several sentences that may distract the readers:

* In Lines 287-288: This design allows us to simulate *realistic* and *extensible* user-AI chat histories with *minimal conflicts*.
* In Lines 319-320: ... we compare this *online* memory evaluation with *offline reading*, where GPT-4o is prompted to answer with the complete history provided as context. (Note: online is already in italics in Lines 80 and 183)
* In Line 426: Using LongMemEval_m, we compare different value choices in a *budget-aware* manner.
* In Line 430: The only exception is with *multi-session reasoning* questions, where fact decomposition consistently improves performance. (Note: multi-session reasoning is already mentioned and abbreviated as MR before)
* In Lines 476-477: To address this need, we introduce a simple yet effective *time-aware indexing and query expansion* scheme. (Note: "time-aware indexing and query expansion" is already in italics in Line 400; moreover, "time-aware indexing and query expansion" first appears in Line 98, but only time-aware is in italics)
* The authors spend 6 lines mentioning that this paper is inspired by the needle-in-the-haystack test (in Lines 292-297); however, it is already stated in the Introduction Section (in Lines 77-78). In my opinion, the authors should remove this paragraph and elaborate more on either (1) the framework formulation or (2) other experiments that the authors found worth noting in the main content (for instance, Table 7).
* As for the over-use of adjectives/compound words, please refer to Lines 72-79 as an example (others are scattered in the main content, particularly in the Introduction):
```
We introduce LongMemEval, a comprehensive, challenging, and scalable benchmark designed to assess the long-term memory capabilities of chat assistants. LongMemEval consists of 500 human-curated, high-quality questions to test five core memory abilities: (...). Each question requires recalling information hidden within one or more multi-turn task-oriented user-AI dialogues that are LLM-simulated and human-edited. Inspired by the “needle-in-a-haystack” test (Kamradt, 2023), we design an attribute-controlled pipeline to compile a coherent, extensible, and timestamped chat history for each question.
```


## Soundness (S)

**S1. The proposed unified framework is not formalized.** See Weakness 3 above.

**S2. The authors do not compare their proposed framework with other similar baselines such as LLM with RAG or external systems.** While the authors do report 97 data using proprietary LLMs (ChatGPT and Coze) in the pilot study, this subset of data is 5x smaller than the original size of the data (500), not to mention the conversation sessions are extremely short, which is mentioned in Lines 317-318: approximately 10x shorter than LongMemEval_s. What's more, the distribution does not even really match the original LongMemEval_s dataset (see Lines 1030-1034 in Appendix B). Is there at least an implicit baseline comparison between your and other works in this paper in some way (see Weakness 2 above)? Are the works mentioned in Line 393 the baselines?

On the other hand, *if the nature of this dataset is that no prior baselines are suitable for comparison, then the authors should test their framework using more LLMs to benchmark the LongMemEval dataset for future researchers and demonstrate its effectiveness across various LLMs.* 
In this scenario, to strengthen the method, it would be necessary to (a) run the experiments multiple times (see [4]) or (b) enlarge the existing LongMemEval dataset. This paper only has 500 data and experiments on two LLMs in its current state: GPT-4o and Llama 3 (70B and 8B). If the budget is an issue, testing other small LLMs (e.g., below 7B) is also a welcome contribution.

**S3. Regarding the soundness in Figure 4-a, the authors only test the closed-source LLMs with a memory system once in a small (97) data.** While the authors already mention (multiple times) that they are interested in testing "*online* interactions with chatbot", this topic further narrows down to a very specific setting. Moreover, as these LLM-based memory systems do not have a snapshot (nor do they release a technical report demonstrating the robustness of their memory systems on various reasoning tasks), including this potentially immature result in the main content could be a problem for future reference (despite the evaluation time mentioned in the footnote), because these systems would be constantly improved over time without any notification, increasing the difficulty in terms of reproducibility.

**S4. The dataset contains only 500, which is rather small. Moreover, many questions focus on *single-turn* sessions (31%; see Figure 3-a).** As these settings are constantly tested in previous long-term/short-term datasets (see Contribution C2 below), the authors should create a dataset with more "multi-session, knowledge update, and temporal reasoning" for current LLMs.

**S5. In Figure 1, the definition of those question types is unclear.** Specifically, in Figure 1, why not treat the "knowledge-update" as a "multi-session" example, and vice versa? Moreover, after adding the time and date in each session, they can also be generalized to the "temporal-reasoning." The authors do not define these types of questions clearly and only use "The other types of questions are multi-session (MR), knowledge-update (KU), and temporal-reasoning (TR)" in Lines 236-237, and yet the distribution of question types is distinctly shown in the pie chart in Figure 3-a.

## Contribution (C)

**C1. Could the authors explain why it is necessary to differentiate "human-AI" and "human-human" conversations?** Am I missing something in this paper? If so, could you kindly explain why you are interested in the user-AI setting and further differentiate this in Table 1? For instance, is there a significant difference between them when training an LLM or in some tasks that prefer user-AI datasets over human-human ones? 

**C2. Prior works of task-oriented dialogues (TOD) are missing, such as the MultiWoZ dataset [5].** On the other hand, the AirDialogue dataset [6] has a more narrower scope. **As a result, there may be an issue in Lines 48-50**: *Many datasets focus solely on human-human conversations (Xu et al., 2022a; Maharana et al., 2024; Kim et al., 2024), while others omit task-oriented dialogues, which represent a significant portion of chat assistant usage.* **As for the Related Work, there is another issue in Lines 142-144** (*Despite these advancements, existing QA-based benchmarks overlook several memory capabilities critical to long-term user-assistant interactions: synthesizing information across numerous sessions, recalling assistant-side information, ...*) **because the CoQA dataset [7] encompasses these two issues.** The model needs to refer back to the conversation history. Lastly, **the SituatedQA dataset [8] $-$ related to temporal context and retrieval $-$ is missing in the Related Work.**

[4] Self-consistency improves chain of thought reasoning in language models, Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou, ICLR 2023

[5] MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselines, Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, Jindong Chen, NLP4ConvAI 2020

[6] AirDialogue: An Environment for Goal-Oriented Dialogue Research, Wei Wei, Quoc Le, Andrew Dai, Jia Li, EMNLP 2018

[7] CoQA: A Conversational Question Answering Challenge, Siva Reddy, Danqi Chen, Christopher D. Manning, TACL 2019

[8] SituatedQA: Incorporating Extra-Linguistic Contexts into QA, Michael J.Q. Zhang, Eunsol Choi, EMNLP 2021

Meta-review: Summary: 

Long-term memory capabilities of LLM-based chat assistants in sustained interactions remain underexplored. This paper introduces LongMemEval, a new benchmark designed to evaluate five core long-term memory abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. LongMemEval consists of 500 meticulously curated questions embedded within freely scalable user-assistant chat histories, and presents a significant challenge to existing long-term memory systems.  Commercial chat assistants and long-context LLMs show a 30% accuracy drop on memorizing information across sustained interactions. 

The paper also presents a unified framework that breaks down the long-term memory design into four design choices across the indexing, retrieval, and reading stages. Experiment results show that these designs greatly improve both memory recall and downstream question answering on LongMemEval. 

Strengths:

1. The proposed benchmark is a new and challenging benchmark, which is valuable for long-term memory evaluation.

2. The paper identifies several core memory capabilities and formulates a unified framework.

3. The paper is promising to provide valuable resources and guidance for advancing the long-term memory capabilities of LLM-based chat assistants.   

Weaknesses:

Most concerns or weaknesses raised in original reviews are addressed during the rebuttal. Here are some outstanding issues, solving which could further enhance the paper: 

1. Deeper insights about the five aspects (information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention) for evaluating long-term memory should be discussed. (Reviewer eitt). The authors did provide a response that partly addresses this issue. 

2. There seem to be many writing issues in the original draft and still some in the revised version (Reviewer LhuP).

Decision: Accept (Poster)