LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory
Authors: Di Wu
Hongwei Wang
Wenhao Yu
Yuwei Zhang
Kai-Wei Chang
Dong Yu
Venue: ICLR 2025 Poster (2025)
Keywords: long-term memory
retrieval-augmented generation
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Recent large language model (LLM)-driven chat assistant systems have integrated memory components to track user-assistant chat histories, enabling more accurate and personalized responses. However, their long-term memory capabilities in sustained interactions remain underexplored. We introduce LongMemEval, a comprehensive benchmark designed to evaluate five core long-term memory abilities of chat assistants: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. With 500 meticulously curated questions embedded within freely scalable user-assistant chat histories, LongMemEval presents a significant challenge to existing long-term memory systems, with commercial chat assistants and long-context LLMs showing a 30% accuracy drop on memorizing information across sustained interactions. We then present a unified framework that breaks down the long-term memory design into three stages: indexing, retrieval, and reading. Built upon key experimental insights, we propose several memory design optimizations including session decomposition for value granularity, fact-augmented key expansion for indexing, and time-aware query expansion for refining the search scope. Extensive experiments show that these optimizations greatly improve both memory recall and downstream question answering on LongMemEval. Overall, our study provides valuable resources and guidance for advancing the long-term memory capabilities of LLM-based chat assistants, paving the way toward more personalized and reliable conversational AI. Our benchmark and code are publicly available at https://github.com/xiaowu0162/LongMemEval.

Published as a conference paper at ICLR 2025
LONGMEMEVAL: BENCHMARKING CHAT ASSIST- ANTS ON LONG-TERM INTERACTIVE MEMORY
Di Wu': Hongwei Wang", Wenhao Yu², Yuwei Zhang", Kai-Wei Chang', Dong Yu? UCLA,, Tencent Lab Seattle, PCC San Diego [diwu, kwchang) ucla.edu
ABSTRACT
Recent large language model (LiM)-driven chat assistant systems have integrated memory components to track user-assistant chat histories, enabling more accurate and personalized responses. However, their long-term memory capabilities in sustained interactions remain underexplored. We introduce LONGMEMEVAL. comprehensive benchmark designed to evaluate five core long-term memory abilities of chat assistants: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. With 500 meticulously curated questions embedded within freely scalable user-assistant chat histories, LoNGMEMEVal presents significant challenge to existing long-term memory systems, with commercial chat assistants and long-context LLMs showing 30% accuracy drop on memorizing information across sustained interactions, We then present unified framework that breaks down the long-term memory design into three stages: indexing, retrieval, and reading. Built upon key experimental insights, propose several memory design optimizations including session decomposition for value granularity, aact-augmented key expansion for indexing. and time-aware query expansion for refining the search scope. Extensive exper- iments show that these optimizations greatly improve both memory recall and downstream question answering on LONGMEMEVAL Overall, study provides valuable resources and guidance for advancing the long-term memory capabilities of LLM-based chat assistants, paving the way toward more personalized and reliable conversational AI. Our benchmark and code are publicly available at https: htt///gitttt// com/ htmm/xiawwwll6e0m 62 xlaovvoll.//oogggeevvallll LongMenEval.
INTRODUCTION
Large language models (LLMs) have exhibited impressive capabilities in solving diverse tasks through natural language, leading numerous successful chat assistant applications (OpenAI, 2022; Microsoft, 2023). Nevertheless, LLMs face limitations on tasks relying heavily on personal knowledge accumulated through long-term user-Al interactions, such psychological counseling or secretarial duties (Zhong et al., 2024). Failing to incorporate user background and preferences into responses can diminish the response's accuracy as well user satisfaction. To personalize LLM--based assistants, long-tern term memory, the ability to memorize, recall, and reason with a long interaction history, is indispensable. Recently, several commercial (OpenAI, 2024; Coze, 2024) and open-source assistant systems with memory (Zhong et al., 2024; Zhang et al., 2024) have been introduced. These systems leverage techniques like compressing, indexing, and retrieving from chat histories to generate more accurate and personalized responses.
Despite these advances, there has been limited progress in holistically evaluating the memory capability in long-term interactions. While several benchmarks evaluate LLMs understanding long chat histories (Xu et al., 2022a;b; Zhong et al., 2024; Maharana et al., 2024; Du et al., 2024; Kim al., 2024), they have major shorteomings. First, they do not accurately reflect user-Al interactions: many focus solely on human-human conversations (Xu et al., 2022a; Maharana et al., 2024; Kim et al., 2024), while others omit task-oriented dialogues, which represent significant
"work done during internship Tencent AI Lab, mentored by Hongwei and Wenhao.

---

Published as a conference paper at ICLR 2025
portion of chat assistant usage and challenge memorization with the long-context inputs and long- form responses. Their interactive histories also typically have a non-confi con-configeeable length spanning only few thousand tokens, limiting the difficulty as current systems continue to improve. Second, current benchmarks' questions only offer a limited coverage of the memory abilities required in dynamic long-term interactions. For instance, MemoryBank (Zhong et al., 2024) and PerL.TQA (Du et al., 2024) insufficiently evaluate the ability to synthesize information across numerous sessions or to reason with temporal metadata or time references. All long-term memory benchmarks including recent ones such as LoCoMo (Maharana et al., 2024) also fail to evaluate recall of information provided by the assistant or reasoning with updated user information.
We introduce LONGMEMEVAL, a comprehensive benchmark for assessing the long-term memory capabilities of chat assistants. LONGMEMEVAL. consists of 500 manually created questions to test five core memory abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. Each question requires recalling information hidden within one or more task-oriented dialogues between a user and an assistant. Inspired by the "needle- in-a-haystack" test (Kamradt, 2023), we design a pipeline to compile a coherent and length- configurable chat history for each question. A chat system, then, is required to parse the dynamic interactions online for memorization, and answer the question after all the interaction sessions. While the length of the history is freely extensible, we provide two standard settings for consistent comparison: LONGMEMEVALs with approximately 115k tokens per problem and LONGMEMEVALM with 500 sessions (around 1.5 million tokens). Preliminary evaluations highlight the difficulty of LONGMEMEVAL, as long-context LLMs show a 30%~~60% performance drop on LONGMEMEVALS, and manual evaluations reveal that state-of-the-ar commercial systems only achieved 30%~~00% accuracy in a setting much simpler than LONGMEMEVALS ($3.4).
Finally, we present a unified view for nemory-augmentt MEMÉVAL, we comprehensively analyze memory design choices across three key execution stages-indexing retrieval, and reading-and four control points: value, key, query, and reading strategy. Experimental insights identify several effective memory designs:
chat assistants. Leveraging LONG-
($5.2) Instead of sessions, round is the more optimal granularity for storing and utilizing the interactive history. While further compression into individual user facts harms overall performance due to information loss, improves the multi-session reasoning accuracy.
($5.3) While using a flat index with the memory values themselves as the keys is a strong baseline, further expanding the keys with extracted user facts improves both memory recall (9.4% higher recall@k) @k) and downstream question answering (5.4% higher accuracy).
($5.4) Naive agnostic memory designs perform poorly on temporal reasoning ques- tions. We propose simple indexing and query expansion strategy to explicitly associate timestamps with facts and narrow down the search range, improving the memory recall for temporal reasoning by 6.8%~11 1.3% when a strong LLM is employed for query expansion.
($5.5) Even with perfect memory recall, accurately utilizing retrieved items is non-trivial. Applying Chain-of-Note (Yu al., 2023) and structured data format (Yin et al., 2023) improves question answering accuracy by as much as 10 absolute points across three LLMs.
2 RELATED WORK
Long-Term Dialogue Benchmarks As the ability of dialogue systems improve, research starts to focus on long-term dialogue understanding beyond traditional dialogue modeling benchmarks (Budzianowski et al., 2018; Wei al., 2018). Early works focused on language modeling evaluation on generating personalized responses from Human-human (Xu et al., 2022a) or human-Al (Xu et al., 2022b) chat histories. To more precisely evaluate memory accuracy, subsequent benchmarks shifted toward question answering (QA, Reddy et al. (2019); Zhang & Choi (2021)). For example, MemoryBan Bank (Zhong et al., 2024) features multi-day chat histories from 15 users with 194 human- written probing questions. LoCoMo (Maharana et al., 2024) includes 50 long-term chat histories and questions testing single-hop. multi-hop, temporal, commonsense, world knowledge, and adversarial reasoning. PerLTQA (Du et al., 2024) scales the evaluation to 3,409 dialogues and 8.593 questions, covering world knowledge, personal profiles, social relationships, events, and dialogue history. DialSim (Kim et al., 2024) evaluates models' memory ability by roleplaying TV show characters
2

---

Published as a conference paper at ICLR 2025
Table 1: A comparison between LONGMEMEVAI. and existing long-term memory benchmarks. We use different colors to denote human-human and human-Al dialogue. #Sess and #Q denote the total number of sessions and questions. Context depth is defined as the number of tokens in the history. Finally, we compare the coverage of five core abilities: information extraction (IE), reasoning (MR), knowledge update (KU), temporal reasoning (TR), and abstaining on unanswerable questions (ABS). " "Not reported in the paper, based on our approximation. ** most 2 sessions.
Core Memory Abilities
Benchmark
Domain
#Sess
#Q
Context Depth
IF
MR
KU
TR
ABS
MSC (Xu et al., 2022a)
Open-Domaid
5k
1k
X
X
X
X
X
DuLeMon (Xu et al., 20226)
Open-Domain
30k
1k
x
X
X
X
X
MemoryBank (Zhong al., 2024)
Personal
300
194
5k
V
X
PerL.TQA (Du al., 2024)
Personal
8593
IM"
<
X
v
LoCoMo (Maharana al., 2024)
Personal
1k
7512
10k
V
DialSim (Kim et al., 2024)
TV Shows
Ik-2k
1M
350k
X
V
LONGMEMEVAL (this work)
Personal
50k
500
115k. 1.5M
<
V
single-session-user
single-session cesssiion-ssee an-siiitant
Ivee been Ustening bocks during daily communo, which takes minutes each
work?
Which one would you say best for romarttc dinner?
you ofehe romantio nestiunant flome you recommended for dinner?
min tes each vay
would reco momee Resctall, It has cory and intimats comosphiee here with soft lighting and sopellant servic....
single-session-prefrree
Rosoieii
Can you recommend some camara flashh options compotible with my Sony IV?
Can you suggest eccessor les that complerentt photography setup?
temporal-reassonng
What's the best wayto clean my Sony 24- 70mm f/2.8 lons?
1/22
tswnt to beeind--th--seeeee tour of the Science Museum todayy with friend who's
6/25
The user would prefer suggestions of Bury-soomparille sccessories.
How many montha have passed since my last museum vtart with friend?
Sorr
been thirking
chemistry professor.
about upgrading my camera bog something more comfortable and durabto.
5/11
attendeed guided Natural History Museum yearu they with my ded.
knowtedge update
I'm
trip.. was Hawaii grean snorkeling togecher.
4/20
just lase red lot in lecture at Hisstory Museum about encient civiizations this morth.
montth
thinking about
on my most recont family tri??
mutti-session
I'm actually thinkkng of iiing Paris, we just weit there wa femily list swek ind amazing...
mmlloking to find a plane techniclan to service my Korg B1.
Parts.
How mony musical instrumants del curminlly pwnn?
I'ws bean playing my black Fander Stratocaster electric guttar a lot lately...
abstention
lupgraded 10 10-galion which my Bubilee.
there 3D- gallor
I've had my acoustic guttar, Yamaha FO600, for about 8 years.
I boced decorations to the 20- gation tank to create more Ni plazes for the fish.
You did not mention that you have a 3D gallon tark..
I'm thinking selling my old orum set, piece Pearl Export.
Figure 1: Examples of the seven question types in LONGMEMEVAL. For each example, we show the associated evidence statements on the left and the question with the answer on the right.
and introduces a time constraint that penalizes slow system responses. Despite these advancements, existing QA-based benchmarks overlook several memory capabilities critical long-term user- assistant interactions: synthesizing information across numerous sessions, recalling assistant side information, and reasoning about updated user details or complex temporal references. Additionally, the chat histories are often too brief and do not reflect the nature of task-oriented interactions. Table compares between LONGMEMEVAL. and previous works, highlighting its advantages in both (1) featuring long and freely extensible iterative history and (2) holistically covering critical memory abilitics in a uniquely challenging way (further examples in Figure l).
Long-Tere Term Memory Methods To equip chat assistants with long-term memory capabilities. three major techniques are commonly explored. The first approach involves directly adapting LLMs to process extensive history information as long-context inputs (Beltagy et al., 2020; Kitaev et al., 2020; Fu et al., 2024; An et al., 2024). While this method avoids the need for complex architectures, is inefficient and susceptible to the "'ost-in-the-midlle" phenomenon, where the ability of LLMS to utilize contextual information weakens as the input length grows (Shi et al., 2023; Liu et al., 2024). A second line of research integrates differentiable memory modules into language models, proposing specialized architectural designs and training strategies to enhance memory capabilities (Weston et al., 2014; Wu et al., 2022; Zhong et al., 2022; Wang et al., 2023). Lastly, several studies approach long-term memory from the perspective of context compression, developing techniques
3

---

Published as a conference paper at ICLR 2025
to condense lengthy histories into compact representations. whether in the form of LLM internal representations (Mu et al., 2023; Chevalier et al., 2023), discrete tokens (Jiang et al., 2023; Xu et al., 2024), retrievable text segments via retrieval-augmenned generation (RAG, Shi et al. (2024); Wang et al. (2023); Sarthi et al. (2024); Chen et al. (2023a); Gutiérrez. et al. (2024)). Although LoNGMEMEVAL. can evaluate any memory system, will take online context compression perspective, where each history interaction session is sequentially processed, stored, and accessed on-demand through indexing and retrieval mechanisms ($4). This formulation aligns with current literature (Zhong et al., 2024; Gutiérrez et al., 2024) and commercial systems (OpenAI, 2024; Coze, 2024). Its plug-and and-play nature also facilitates the integration into existing chat assistant systems.
LONGMEMEVAL
3.1 PROBLEM FORMULATION
The evaluation of LONGMEMEVAL requires an instance of 4-tuple (g, S = [(t,. (t2, S2). Sy]] is a sequence of N history chat sessions ordered from the carliest to the latest, where S, is a multi-turn interaction between the user and chat assistant and t; is the session's 's timestamp. Each session can be further decomposed into rounds: one user message followed by one assistant response. During test time, S is provided to the system one by one. g and t > tN represent the question from the user and its date. a is a short phrase indicating the answer, or natural language rubric describing the preferred answer in the case where q is open-ended.
3.2 LONGMEMEVAL. BENCHMARK CURATION
One major challenge in building reliable personalized assistant is performing online recording, calling, updating. and reasoning on the dynamically evolving user information. To comprehensively reflect the challenge, LONGMEMEVAL formulates five core long-term memory abilities:
Information Extraction (IE): Ability recall specific information from extensive interactive histories, including the details mentioned by either the user or the assistant.
Multi-Session Reasoning (MR): Ability to synthesize the information across multiple history sessions to answer complex questions that involve aggregation and comparison.
Knowledge Updates (KU): Ability to recognize the changes in the user's personal information and update the knowledge of the user dynamically over time.
Temporal Reasoning (TR): Awareness of the temporal aspects of user information, including both explicit time mentions and timestamp metadata in the interactions.
Abstention (ABS): Ability to identify questions seeking unknown information, i.e.. information not mentioned by the user in the interaction history, and answer "I don't know".
As shown in Table 1. this formulation represents a more comprehensive ability coverage compared to prior long-term memory benchmarks like MemoryBank and PerLTQA. To thoroughly assess these abilities, LONGMEMEVAL features seven question types. Single-session-sser and single-session- assistant test memorizing the information mentioned by user assistant within single session. pesall-session-ppefeeeee tests whether the model can utilize the user information to generate a personalized response. multi-session (MR) tests aggregating user information across two more sessions. knowledge-update (KU) focuses on the ability to recognize changes in the user's life states and update the memory accordingly. temporal-reasoning (TR) tests reasoning with both the timestamp in metadata and explicit time references. Finally, we draw 30 questions from the previous question types and modify them into "false premise" questions, testing whether the model can correctly abstain from answering (ABS). Figure I presents an example for each question type.
Question Curation Figure 2 depicts the question curation pipeline. We define an ontology of 164 user attributes in five categories: lifestyle, belongings, life events, situations context, and demographic information. For each attribute, we leverage an L.LM to generate attribute-oocusel user background paragraphs, each of which includes detailed discussion of the user's 's life experience.
Unless otherwise mentioned, Ilama 70B Instruct (Dubey al., 2024) used the LLM in the pipeline.
4

---

Published as a conference paper at ICLR 2025
(a) Question Construction
(b) Evidence Session Construction
Attribute
occupption
User LLM
Dur nu paak campaigh as maiscrs my work
Assistart LLM
hours por week incicoses by 10 hours.
Bockground Sampling
Imm planning 1o stort looking now job noxt few months, wis wcnce ing you holo uodate my resume and Linked proffie...
Background
I've been wo King a Markening Specio let on mid sized tech firm sprty named TechSewwy Inc. Myj titte is officialyy "Cigial Har ruting Specials" and report direclly to Ha koding Har rager Rachol, My primary responsiiiiioes includo... ... in past 6 mentis, I've been workking on average 40 hours per weok, with wooks going up 6o hours during pask campaign sessoms
happy to help ger stw ted, could you shorm your curnent resume?
I've wttechedn cummntt nenum..... I'd lsee to highight mym accommence mansgghn poats, motr ca. During poak campaign sessons, increase my work hours by 10 hours
sociel media prosencc, creating and scchduling posts, and anaagaargg angggeement
weekly to accommodate date the additional workload. (want cmppesske my atality to wrtept to chen ging workinods and pricetton freie affercively
Question Construction
Thank staannn pripoovd you with some feodbacck and suggestions....
Evidence Statements
Fwork everage of 40 hours werw
2024/5/1
(c) History Construction
During paak campaigg houuse hours
work
2024/5/3
Publicly Released
<:3007>
Chat Sessions
Question wi/ Timestamps
Evidence Sessions
History Sessions
How mony hours look in typical during compaign
2024/5/20
Non-Conftin Confeiitiit Simulated Chat
Araiwer: 50
Figure 2: Data creation pipeline of LONGMEMEVAL (a) Human experts construct all the questions and evidence statements. (b) Then, the evidence sessions are LLM-simulated and human-edited. (c) The full user-Al Al chat history is constructed at test time, whose length is freely configurable.
To create question, we randomly sample paragraph and use an LLM to propose several seed (question, answer) pairs. As these LLM LLM-propssed questions often lack depth and diversity, human experts manually filter and rewrite all the questions to achieve the desired difficulty. Then, we manually decompose the answer into one or more evidence statements with optional timestamps.
Evidence Session Construction Each evidence statement is then separately embedded into a task- oriented evidence session created by self-chatting (Xu et al., 2023). The user LLM is instructed to convey the evidence statement indirectly, e.g., instead of stating "I bought a new car last month," it might instead ask for help about car insurance and reveal the information incidentally. This approach enhances the benchmark's 's difficulty by requiring systems to recogmize and memorize user details not explicitly emphasized conversations. We present the full details in Appendix A.
To ensure the data quality, all the evidence sessions are then manually screened and edited to (1) verify evidence inclusion, (2) distribute the evidence statement across different conversation positions, and (3) rephrase statements into more natural, colloquial language, especially for time mentions, which LLMs often express too formally. We also meticulously annotate the position of the evidence statement within each evidence session. For questions involving temporal information, we then manually add timestamps to both the evidence sessions and the questions. Most questions require evidence from multiple sessions (up to six) with evidence statements positioned diversely within sessions. Appendix A.3 presents further statistics of the final constructed data.
History Compilation For each question, LONGMEMEVAL compiles a coherent user-Al chat history (Figure 2c). Our approach is analogous to the needle-in-a-haysten test (Kamradt, 2023), which asks a model to retrieve brief information (the "needle") embedded in a long document (the "haystack")). In comparison, LONGMEMEVAL. is more challenging and realistic as it involves retrieving and synthesizing information from multiple extended evidence sessions. Specifically, we sample a number of unrelated user-Al chat sessions, randomly insert the evidence sessions in the middle, and assign plausible timestamp to all sessions. We draw the irrelevant sessions from two sources: (I) self-chat sessions simulated based on other non-confiittin attributes and (2) publicly released user-Al style chat data including ShareGPT (Zheng et al., 2023) and UltraChat (Ding et al., 2023), This design creates extensible realistic chat histories with minimal conflicts. While the pipeline allows us to compile chat histories of arbitrary length, we provide two standard settings: LONGMEMEVALS (~~15 tokens/question) and LONGMEMEVALy (500 sessions, L.5M tokens).
3.3 EVALUATION METRIC
Question Answering As the correct answers can take flexible forms, an exact matching strategy as in previous works can result in inaccurate evaluations. To address this, EONG.
5

---

Published as a conference paper at ICLR 2025
Model
Size Oracle
S
% Drop
No Chain-of-Note
System
LLM
Accuracy
GPT-4o
0.870
0.606
30.3%
Offline Reading
GPT-40
0.9184
Llama 3.1 Instruct
70B
0.744
0.334
55.1%)
Llama 3.1 Instruct
8B
0.710
0.454
36.1%%
GPT-4o
0.5773
Phi-3 128k Instruct
14B
0.702
0.380
45.9%
ChatGPT
GPT-4o-mini
0.7113
Phi-3.5 Mini Instruct
4B
0.660
0.342
18.1%
With Chuin-of-Note
GPT-40
0.3299
Coze
GPT-3.5-turbo
0.2474
GPT-40
0.924
0.640
30.7%.l
Llama 3.1 Instruct
70B
0.848
0,286
66.3%%
(a) Commercial
chat assistants exhibit weak performance LoNGMFMEYAL. The accuracy of ChatGPT and Coze degrades by large amount compared to directly reading the context ("Offline Reading") with the same LLM. Specifically, ChatGPT and Coze instantiated with GPT-4o exhibits 37% and 64% performance drop, respectively.
Llama 3.1 Instruct
8B
0.710
0.420
40.8%
Phi-3 128k Instruct
14B
0.722
0.344
52.4%.
Phi-3.5 Mini Instruct
4B
0.652
0.324
50.3%
(b) Long-context LLMs exhibit large QA performance drops on LONGMEMEVALs (column "S"). compared to the accuracy of answering the questions based on only the evidence sessions (column "Oracle").
Figure 3: Pilot study of (a) commercial systems and (b) long-context LLMs on LONGMEMEVAL.
employs LLM assess response quality (Liu et al., 2023). Specifically, we prompt-engineer the gpt-40-22200 2024 06-06 model via the OpenAl API. Our meta-evaluation study demonstrates that the evaluator achieves more than 97% agreement with human experts. The prompts for each problem type as well as the human meta-evaluation details are presented in Appendix A.4.
As LONGMEMEVAL contains human-annotated answer location labels, interme- diate retrieval metrics can be easily calculated if the chat system exposes its retrieval results. We report @ecall@k k and NDCG@k, k. where k is the number of top items retrieved by the system.
Memory Recall
3.4 LONGMEMEVAL REPRESENTS A SIGNIFICANT CHALLENGE
Using LONGMEMEVAL, we conduct pilot study on commercial systems and long-context LLMs.
Commercial systems We evaluate two commercial systems that maintain a set of memorized user facts as the user chats with the assistant: ChatGPT (OpenAI, 2024) and Coze (Coze, 2024). Since these systems only support memory features via their web interfaces, we randomly selected 97 questions and created a short chat history of 3-6 sessions (approximately 10x shorter than LONGMEMEVAL.s) Human annotators interacted with the chat assistants tum-by-turn, and finally ask the question in a new session". In Figure 3a, we compare this online memory evaluation with offline reading, where GPT-4o is prompted to answer with the complete history provided as context. Both ChatGPT and Coze exhibited significant performance drops compared to offline reading, underscoring the challenging nature of LONGMEMEVan ChatGPT tended to overwrite crucial information as the chat continues, while Coze often failed to record indirectly provided user information. We provide analyses in Appendix B. Overall, this result highlights the gap between building a seemingly personalized chat assistant by recalling isolated facts and demonstrating genuinely strong memory ability.
and
We found
Long-Context LLMs While LONGMEMĒVAL poses a significant challenge to online memory systems, is the benchmark easily tackled with offline reading over the entire history? In Figure 3b, evaluated four advanced long-context LLMs on LENGMEMEVALS (with history length of approximately 115k tokens): GPT-4o, Llama 3.1 Instruct (Dubey et al., 2024), and Phi-3 (Abdin al., 2024a). Compared to the oracle retrieval setting (answering with only the evidence sessions as the context), these LLMs showed a 30% to 60% performance decline when tasked with reading the entire LONGMEMEVALS history, regardless of whether the chain-of-note technique (Yu et al., 2023) was applied (discussed further in $4.2). As the histories in LONGMEMEVALS is still short (~50 sessions). the performance is likely to further degrade as the interaction history expands. Overall, these findings suggest that even the most capable current long-context LLMs struggle to manage an ever-growing interaction history without an effective memory mechanism.
All evaluations were conducted in the first weeks of August 2024.

---

Published as a conference paper at ICLR 2025
OP2: Key
OPT: Value
CP3: Query
CF4: Reading
MeadingSttrnngg Stratogy Ry
Key
Value Chrunk
Key'2
Key.
Chtunk
Key
Duery
Key
Viluam Chunn
Chunk
Question
Question
K
Value Chink
(1) Indexing
(2) Retrieval
(3) Reading
Figure 4: A unified view of chat assistant with long- -erm memory in operation. We formulate three stages and four control points (CP). We provide further examples in Table 2 and Appendix C.
A UNIFIED VIEW OF LONG-TERM MEMORY ASSISTANTS
In this section, we formulate a three-stage long-term memory model for chat assistants. Despite its simplicity, this model provides unified view of existing long-tern memory assistant works. Along each its stages, we then investigate crucial control points and propose our optimizations.
4.1 LONG-TERS MEMORY SYSTEM: FORMULATION
We formulate long-term memory as massive key-value datastore [(k) 01), (k2, v2), The keys k; can be heterogeneous, and could be discrete continuous. In the discrete be a sentence, paragraph, a fact, or an entity, etc. In the continuous case, the key could be c.g., the model's internal representation under some inputs. The values U, might repeat. As shown in augmentdd assistant: (1) indexing, converting
the key could
Figure 4, we formulate three stages for more key-value items, (2) retrieval, formulating retrieval
each history session (t,, St)) into query and collecting A most relevant items, and (3) reading, an LLM M reads the retrieval result and show how nine eeamory-augmented chat assistant systems can
generates a response. In Table be viewed as instantiations of this framework. An alternative mathematical formulation is presented in Appendix C. For its conciseness, the rest this paper follows this section's 's formulation.
4.2 LONG-TERM MEMORY SYSTEM: DESIGN CHOICES
We identify four crucial control points for long-term memory of chat assistants, illustrated in Figure 4. We analyze design choices from existing works and their limitations, and propose our optimizations. Due space constraints, we present these designs high level here, with detailed designs further described in $5 and Appendix D.
CP 1: Value The value represents the format and granularity of each session stored in memory. Given that user-Al chat sessions are often lengthy and cover multiple topics, storing each session as a single item can hinder effective retrieval and reading. Conversely, compressing sessions into summaries user-specific facts, seen in prior work such as Zhong et al. (2024) and Du et al. (2024), can lead to information loss, harming the performance of the system to answer detailed questions. In $5.2, compare three value representation strategies: storing entire sessions, decomposing sessions into individual rounds, and further applying summary/fact extraction.
CP Key Even when sessions are decomposed and compressed, each item still contains substantial information, with only fraction relevant the user's query. Therefore, using the value itself as the key, a common practice in prior works (Zhong et al., 2024; Maharana et al., 2024), may be suboptimal. We introduce a key expansion approach in $5.3, where summaries. keyphrases, user facts, and timestamped events are extracted from the values to augment the index. This optimization highlights the key information and enables effective retrieval with multiple pathways.
CP 3: Query For straightforward user queries, the aforementioned key-value optimizations may yield high retrieval accuracy. However, when queries involve temporal references (e.g., "Which restaurant did you recommend last weekend?"), naive similarity search proves insufficient. We address this with a time-aware indexing and query expansion strategy, where values are indexed with timestamped events, and retrieval is restricted to items within the relevant time range ($5.4).

---

Published as a conference paper at ICLR 2025
Table 2: A comparison of nine memory-augmmnnted frameworks through the lens of the proposed unified framework. We provide detailed references and discussions of each work in Appendix C. For ChatGPT and Coze, we skip several designs as they are unknown.
Method
Value roundisession summary round summary + fact
Key
Query
Retrieval
Time-aware
Reading
In-context RAG (Shi et al., 2024)
question
flat
No
direct
MemoryBank Bank (Zhong
20244
question
flat
Yes
direct
LD LD-Agent (Li et al., 2024)
K
V
keyphrase
flat
Yes
direct
CoN (Yu al., 2023)
roundkession fact fact round/session round/session roundkession
question
CoN
ChatGPT
No
Coze
No
RAPTOR (Sarthi al., 2024)
node summary node summury entity
question question entity
flat/ineractive interactive
No
Mem Wellel (Chen et al., 202.3a
No
interactive
HippoR AG (Guttérrez et ul,, 20224)
PPR
No
direct
Our Design
round
K + fact
question time
flat
Yes
CoN
Reader LLM GPT-4o
Reader LLM Liama 3.1 8B Instruct
Full
Multi-Session
Multi-Session Subset
0.6
Accuracy
P----0
0.4
10000
200000
10000 Number of Tokens
20000
10000 Number of Tokens
20000
10000 Number of Tokens
20000
Number of Tokens
Session
Round
. Session Summary
- Sessian Facts -A- Round Facts
QA performance on LONGMEMEVALM with different value designs. Decomposing sessions into rounds improves the QA performance. For the multi-session reasoning questions, further representing the values with the extracted facts improves the QA accuracy.
Figure 5:
Reading Strategy Answering complex queries may require recalling numerous memory items. Although the retrieval accuracy can be enhanced through the preceding designs, it does not guarantee that the LLM can effectively reason over the extensive context (Shi al., 2023; Liu al., 2024). In $5.5, we explore reading strategies and demonstrate that optimizations such as extracting key information before answering (Chain-of-Note, (Yu et al., 2023)) and using structured format prompting (Yin et al., 2023) are crucial for achieving high reading performance.
CP 4:
5 EXPERIMENT RESULTS
5.1 EXPERIMENTAL SETUP
We mainly study three LLMs: GPT-4o, Llama 3.1 70B Instruct, and Ilama 3.1 Instruct?. For the retriever, we choose dense retrieval with the 1.5B Stella V5 model (Zhang, 2023), given its high performance on MTEB (Muennighoff et al., 2023). Extensive comparisons between Stella V5 and alternative retrievers are provided in Appendix E.2. For the indexing stage, we employ Llama 3.1 8B Instruct to extract summaries, keyphrases, user facts, and timestamped events. When sessions or rounds are used as the key, only keep the user-side utterances. In the reading stage, the retrieved items are always sorted by their timestamp to help the reader model maintain temporal consistency. Throughout 85.2 to $5.4, apply Chain-of-Note and json format (discussed in $5.5) by default.
5.2 VALUE: DECOMPOSITION IMPROVES RAG PERFORMANCE
Using LONGMEMEVALM. we compare different value choices in a budget-aware manner. As shown in Figure 5, decomposing sessions into rounds significantly enhances reading performance with GPT-40 as the reader, while performing similarly to non-decomposed sessions when using Llama 3.1 8B Instruct as the reader. However, despite their efficiency in token usage, replacing sessions or rounds with extracted summaries facts negatively impacts QA performance due to information loss. The only exception is with multi-session reasoning questions, where fact decomposition consistently improves performance. We hypothesize this is because fact decomposition extracts
Mooe results five additional I.IMs reported in Appendix E.1.
8

---

Published as a conference paper at ICLR 2025
Table Retrieval and end-to-end QA performance on LONGMEMEVALy, with different key designs for indexing. L331 = Llama 3.1 Instruct. We find applying document expansion with the extracted user facts (row V + fact) greatly improves both retrieval and the downstream QA
Retrieval
End-to-End QA
Key Design
@etrics@5
Metrics@ 10
GPT-40
L3.1 70B
L3.1 8B
Recall NDCG
Recall NDCG
Top-5
Top-10
Top-5
Top-10
Top-5
Top-10
Value
Round
K == V
0,582
0.481
0.692
0.512
0.615
0.670
0.600
0.624
0,518
0.534
K fact
0.530
0.41 i 0.654
0.449
0.588
0.664
0.564
0.610
0.510
0.534
K = keyphrase
0.282
0.159
0.392
0.303
0.425
0.489
0.404
0.450
0.378
0.432
fact
0.644
0.498
0.784
0.536
0.657
0.720
0.638
0.682
0.566
0.572
K = V keyphrase
0.478
0.359
0.636
0.410
0.541
0.652
0.538
0.620
0.472
0.524
Value
Session
K V
0.706
0.617
0.783
0.638
0.670
0.676
0.592
0.570
0.524
0.464
K = summary
0,572
0.448
0.648
0.468
0.554
0.252
0.498
0.512
0.444
0.216
K = fact
0.642
0.524
0.814
0.571
0.644
0.512
0.544
0.550
0.470
0.404
= keyphrase
0.482
0,375
0.576
0.401
0.618
0.498
0.440
0.450
0.388
0.414
K summary
0.689
0.608
0.749
0.624
0.658
0.666
0.568
0.560
0.518
"0.494
K = V fact
0.732
0.620
0.862
0.652
0.714
0.700
0.588
0.584
0.530
0.490
V keyphrase
0.710
0.587
0.768
0.602
0.665
0.672
0.590
0.566
0.526
0.508
the same type of information across all sessions in more uniform and simplified format, aiding retrieval and reading (Chen et al., 2023b). Finally, we observe that the optimal token budget varies by the reader's capability: while Llama 3.1 8B Instruct's 's performance drops sharply beyond 3k retrieved tokens, GPT-4o continues to improve even with over 20k retrieved tokens.
5.3 KEY: MULTI-KEY INDEXING IMPROVES RETRIEVAL AND RAG
In Table 3, we explore whether summaries, keyphrases, or user facts condensed from the value can serve better keys than the value itself. Interestingly, despite their more focused semanties, using these condensed forms alone does not enhance the memory recall performance. We hypothesize that this is due the retriever's ability already effectively handle long-text semanties.
To leverage both the highlighted information from compression and the completeness of the original value, we applied simple document expansion technique (Tao et al., 2006; Efron al., 2012), where the compressed information is concatenated with the original value to form the key during indexing". This approach. particularly when using user facts, yielded an average improvement of 9.4% in recall @ l and 5.4% in final accuracy across all models. In Appendix E.2, we further analyze different retrievers and find with alternative retrievers, key expansion with summary and keyphrases could improve Recall @5 when session is used the value granularity. These results suggest that mulli-pathway retrieval can significantly enhance memory recall performance. In the following section, we will investigate the time constraint another pathway to leverage.
5.4 QUERY: TIME-AWARE QUERY EXPANSION IMPROVES TEMPORAL REASONING
A key challenge highlighted by LONGMEMEVAL. in building real-world assistant systems is the need utilize temporal information present in both metadata and user utterances to correctly answer time-sensitive queries. To address this need, we introduce simple yet effective time-aware indexing and query expansion scheme. Specifically, values are additionally indexed by the dates the events they contain. During retrieval, an LLM My extracts time range for timessensitvee queries, which is used to filter out a large number of irrelevant values.
As shown in Table 4, this simple design improves recall by average of 11.3% when using rounds as the value and by 6.8% when using sessions as the value. This improvement remains consistent when key expansion is applied during indexing. We also find that the effectiveness of this method depends on using a strong LLM for Mr to accurately infer time ranges from queries. Llama 8B, on
We also experimented with merging at the retrieval stage by combining the ranks from different pathways, but it underperformed compared indexing-stage merging. See Appendix E.3 for details.

---

Published as a conference paper at ICLR 2025
Table 4: Retrieval performance on the temporal reasoning subset of LONGMEMEVALM. Time-
aware query expansion significantly facilitates retrieval by narrowing down the retrieval scope.
Value
Session
Value
Round
Key Setting
Metric@5
Metric 10
Metric
Metric@ 10
Recall
NDCG
Recall NDCG
Recall
NDCG
Recall NDCG
(1) K
0.639
0.630
0.651
0.707
0.421
0.462
0.499
0,511
(1) w/ Query Expansion (Mr GPT:40)
0.654
0.660
0.707
0.451
0.565
0.495
0.538
(1) w/ Query Expansion (My Llama 3.1 8B Instruct)
0.624
0.627
0.647
0.692
0.384
0,448
0.489
0.488
(2) K V fact
0.684
0.688
0.721
0.782
0.489
0.500
0.550
0.598
(2) w/ Query Expansion (My =GPT-4)
0.722
0.732
0.797
0.758
0.526
0.548
0.722
0.669
(2) w/ Query Expansion (My Llama 3.1 8B Instruct)
0.677
0.688
0.711
0.744
0.481
0.532
0.570
0.647
0.95
0.910
0.848
0.022
0,800
0.80
0.75
0.762
0.744
0.746
0.756
Direct Prediction
JSON Direct Prediction
Chain-of-Note
ISON Chain
0.60
GPT-4c
0B
Figure 6: Question answering performance under the oracle retrieval setting. CoN with JSON format outperforms the other three parameter combinations by a large margin. NL Natural Language.
the other hand, struggles to generate accurate time ranges, often hallucinating or missing temporal cues even with numerous in-context examples. Further analysis is provided in Appendix E.4.
5.5 IMPROVING READING WITH CHAIN-OF-NOTE AND STRUCTURED FORMAT
As LONGMEMEVAL requires syntheses across multiple sessions, even optimal memory retrieval mechanism needs to return a long context to capture all relevant information. To enhance the model's ability to handle long retrieved contexts, items in a structured JSON format (Yin et al., 2023), which helps the model clearly recognize memory items as the data for reading. Additionally, we apply the Chain-of-Note (CoN) reading approach (Yu et al., 2023). instructing the LLM to first extract information from each memory item and then reason based on these notes. This effectively decomposes long-context reading into two simpler subtasks: copying important details and reasoning with more concise notes.
apply two key optimizations. First, present retrieved
In Figure 6, we evaluate the reading designs under the oracle retrieval setting, where only evidence sessions are provided. Surprisingly, even with perfect retrieval, suboptimal reading strategy results in up to a 10-point absolute performance drop compared to the best approach for GPT-4o. Notably, when CoN is not applied, JSON format does not consistently outperform the natural language format. However, with CoN, JSON format consistently benefits reader LLMs of various capabilities. Appendix E.5 further analyzes error patterns of LLMs with the enhanced reading strategy.
6 CONCLUSION
In this paper, we introduced LONGMEMEVAL. comprehensive and challenging benchmark designed to evaluate the long-term memory abilities of chat assistants across five core memory tasks: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. Through extensive experiments with both commercial systems and long-context LLMss, we demonstrated the significant challenges posed by LONGMEMEVAL, with current systems exhibiting substantial performance drops. By analyzing key design choices across indexing, retrieval, and reading stages, we proposed effective strategies such as session decomposition, fact- augmented key expansion, and time-aware query expansion, which collectively improved both memory recall and the question answering performance. Our findings highlight the need for more sophisticated memory mechanisms to achieve personalized and reliable conversational AI, and LongMemEval offers a valuable benchmark to drive future advancements in long- memory capabilities for chat assistants.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
In the paper writing and the subsequent code release, we are committed to enabling other researchers to easily leverage our resources, replicate our results, and build upon our findings. We have documented the benchmark construction process in detail in $3.2 and Appendix A.I, including all the attributes and instructions used to prompt LLMs. We have created and will publicly release the two fixed evaluation datasets, LONGMEMEVALS and LONGMEMEVALN. In addition, we will also release the algorithm and source mixture used to create these two datasets, so that future studies could build upon them to create chat histories of any length. Finally, we have meticulously documented all the implementation details of our memory optimization in Appendix D, and our code will be released along with the benchmark as well. We believe that these efforts for transparency can help advance the field and foster future research endeavors.
ETHICS STATEMENT
The major artifact released by this work is the LONGMEMEVAL evaluation dataset. To construct the chat history, we utilize ShareGPT which has an Apache 2.0 license, and UltraChat', which has an MIT license. the data release, we plan to use an MIT license. Since the questions and the evidence sessions are newly created, we conducted rigorous human screen process to ensure that the new dataset does not contain personally identifialle information or offensive content. This paper involves human annotators in two places: (1) the dataset construction and filtering ($3.2) and (2) the manual analysis of commercial systems ($3.4). The process was mainly conducted by three expert annotators, who are in-house NLP researchers that have more than three years of NLP research experience. All the annotators are properly briefed with the annotation objective, and a discussion among them is conducted to resolve uncertain cases. In total, approximately 400 human hours are spent on the dataset construction and 150 hours on the study of commercial systems. The annotators are paid biweekly or monthly, and their salaries include the working hours dedicated to annotation. The data collection protocol has been determined exempt by the IRB of the author's institution. Finally, LONGMEMEVAL and the studied memory system could induce several societal impacts. Storing and recalling user information could cause personal information leakage, and the lack of a memory "deletion" operator could harm the system's 's trustworthiness. It is also possible that the memory mechanism could be leveraged by malicious parties to toxic contents into the datastore, which may cause a jailbreak behavior during inference. To mitigate such behaviors, it is essential to implement moderation mechanisms that monitor the read/write data flow of the memory. We encourage producers of memory-augmentee augmented assistant systems to be aware of these potential harmful effects and apply thorough testing and mitigation.
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat S. Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan, Dan Iter, Amit Garg, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Mahoud Khademi, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Chen Liang, Weishung Liu, Eric Lin, Zeqi Lin, Piyush Madan, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilae, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Xia Song, Masahiro Tanaka, Xin Wang, Rachel Ward, Guanhua Wang, Philipp Witte, Michael Wyatt, Can Xu, Jiahang Xu, Sonali Yadav, Fan Yang. Ziyi Yang, Donghan Yu, Chengruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi-3 technical report: A highly capable language model locally
Accessed via http::///sttbb...cm/////////tttt
Accessed via
11

---

Published as a conference paper at ICLR 2025
your phone. CoRR, abs/2404.14219, 2024a. doi: 10.48500ARRXV.2444.11222 URL
48550/ 240...22..........
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat S. Behl, Alon Benhaim, Misha Bilenko, Johan Bjorek, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan, Dan Iter, Amit Garg, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Mahoud Khademi, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Chen Liang. Weishung Liu, Eric Lin, Zeqi Lin, Piyush Madan, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Xia Song, Masahiro Tanaka, Xin Wang, Rachel Ward, Guanhua Wang, Philipp Witte, Michael Wyatt, Can Xu, Jiahang Xu, Sonali Yadav, Fan Yang, Ziyi Yang. Donghan Yu, Chengruidong Zhang. Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi-3 technical report: highly capable language model locally doi: 10.4550/ARXI.22404.1444
on your phone. CoRR, abs/2404. 14219, 2024b.
URL
https: http://doo..gggoo
48550/555000 arrvvv.ivvvvv.////v 2404 1.44..422 14219.
Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, and Jian-Guang Lou. Make your LLM fully utilize the context. CoRR, abs/2404.16811, 2024. doi: 1044550/AARX1V24444.6811 URI.
https://doi. http:///oooorg//..//// wtt.oo/rgg.......... 198550/axxiv/ 2404.16814.111
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer.
arXiv:2004.05/500, 2020.
Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, lñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasić. MultWWOZ a large-scale multi-domain Wizard-of-Or dataset for task-oriented dialogue modelling. In Ellen Riloff. David Chiang, Julia Hockenmaier, and Jun't ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 5016-5026, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653v1/D18-- 1547. aclanthology. org/D18 -14.7777.111.
URL https:/
Howard Chen, Ramakanth Pasumuru, Jason Weston, and Asli Celikyilmaz. Walking down the memory maze: Beyond context limit through interactive reading. CORR, abs/2310.05029, 2023a. 000.0008500/.........
doi: 10.4855/ARXIV.231000555 05029.
URL https: / /oi.
Tong Chen, Hongwei Wang, Sihao Chen. Wenhao Yu, Kaixin Ma, Xinran Zhao, Hongming Zhang, and Dong Yu. Dense X retrieval: What retrieval granularity should we use? CoRR, abs/2312.066448, 2023b. doi: 10-4850/ARXIV23112.666 48550/arxiv 2312205644
URL https: /ooi http://oo.o.oo...//////o
Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 3829-3846, Association for Computational Linguistics, 2023. doi: 10.18653//1/2023.MMMB WWW.EEENNLLPPPPPPP URL https: /ooii (tt.oooorgg11...83/1lllll.
Coze. Memory overview guide. https://wwer coze com/ htt/:/occc/////// overview?_ 2024. Accessed: September 15, 2024.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.142333 2023.
Yiming Du, Hongru Wang. Zhengyi Zhao, Bin Liang Baojun Wang, Wanjun Zhong, Zezhong Wang, and Kam-Fai Wong. Perl.TQA: personal long-term memory dataset for memory
12

---

Published as a conference paper at ICLR 2025
classification, retrieval, and fusion in question answering. In Kam-Fai Wong, Min Zhang, Ruifeng Xu, Jing Li, Zhongyu Wei, Lin Gui, Bin Liang, and Runcong Zhao (eds.), Proceedings of the 10th SICHAN Workshop Chinese Language Processing (SIGHAN-1O), pp. 152-164, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https: / //aclanthoology....ooooo .rg 2024. 1..
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.2178: 2024.
Miles Efron, Peter Organisciak, and Katrina Fenlon. Improving retrieval of short texts through document expansion. In William Hersh, Jamie Callan, Yoelle Maarek, and Mark Sanderson (eds.), The 35th International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 12, Portland, OR, USA, August 12-16, 2012, pp- 911-920, ACM, 2012. doi: 111145/2248283.24488000 URL https: /tt:///oiiiii wrg. 1ttp://...1155/244333 2348405.
Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannanch Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https: wwwp///eeee.eeeeeeeeeeeee
Bernal Jiménez Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. Hipporag: Neurobiologically inspired long-term memory for large language models. CoRR, abs/2405. 2024. doi: 00.48550/ARXIV.2405.00 14831. URL https: 2405 14831.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning- Trans. Mach. Learn. Res., 2022, 2022. URL https: /ttp:s/////peeeeeee.... net cotcfforrt forum? id=
jKN1pXi7b0.
Huiqiang Jiang, Qianhui Wu, Chw Lin, Yuqing Yang, and Lili Qiu. LImlingua: Compressing prompts for accelerated inference of large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Pruceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. EMNLP 2023, Singapore, December 6-10, 2023, pp. Association for Computational Linguistics, 2023, doi: (0.18653/V1/22233EMM //tp:i::///o.io/ /doi orgg 10. 186536/vvll w2022222222222 emn emnll p-mma in. 825.
825.
URL httpt:ttp///o.....t.
Gregory Kamradt. Needle in a haystack pressure testing llms. GitHub, 2023. URL https: //gi thub. com/ gtt.///t////////////// http:/MMMMSetteeeee -el_NeeeeeeAnaasttett
Jiho Kim, Woosog Chay, Hyeonji Hwang, Daeun Kyung, Hyunseung Chung, Eunbyeol Cho, Yohan Jo, and Edward Choi. Dialsim: A real-time simulator for evaluating long-term dialogue understanding of conversational agents, 2024. URL https: wwwiaaaivvvo.................... ab4//406.. 13144.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.0445,, 2020.
Hao Li, Chenghao Yang, An Zhang, Yang Deng, Xiang Wang, and Tat-Seng Chua. Hello again! Ilm-powered personalized agent for long-term dialogue. CoRR, abs/2406.05925, 2024, doi: 2.550AARRIVV224000555555. URL htwptttt:/ //t::://oo..... wwwddoo.oooooo http:///oo.oooo/./../
Nelson F, Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Trans. Assoc. Comput. Linguistics, 12:157-173. 2024. doi: 10.1162/TACL) 10. 1 00638.
URL https: //oi org/
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G- eval: NLG evaluation using gpt-4 with better human alignment. Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods
13

---

Published as a conference paper at ICLR 2025
in Natural Language Processing, pp. 2511-2522, Singapore, December 2023. Association
for Computational Linguistics. doi: 10. 10.18653/112023.emm cetp223.emllmmmat main. 153. URL https://
aclanthology org/2023. eew.emmannmpmmm..e 153.
Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. Evaluating very long-term conversational memory of LLM agents. Lun-Wet Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting the Association for Computational Linguistics VVlume I: Long Papers), pp. 13881-13878 Bangkok, Thailand, August 2024. Association for Computational Linguisties. URL https: http:///aaaaaatttaaaoee org/202 acl-long 747.
Microsoft.
Announcing
microsoft copilot, your everyday companion, URL https://blogg microsoft www.blcg////cccccccco.......oocc http://lllgo//a//eeeeeeeea
2023.
www.coooooo..ooooo.oooooootttt.e com
Accessed:
September 15, 2024.
Mistral AI Team. Mistral nemo: Our new best small model. Mistral AI, July 2024. URL https: / //mmsttral.... ai htt///ew/////// www.aaaeeeewsstteeee wttps//mmstaaa-neeeee
Jesse Mu, Xiang Li, and Noah D. Goodman. Learning to compress prompts with gist tokens. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http: httpppapprssseeam n www.:iisssscppaaeee wwwpeeeffflleepppppeeeeee Abs-:ezattaabbbatttt---- Cow.raactt-oofeennceee.. html.
Niklas Muennighoff, Nouamane Tazi. Loic Magne, and Nils Reimers. MTEB: Massive text embedding benchmark. In Andreas Vlachos and Isabelle Augenstein (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 2014-2037, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. doi: 10.18653/v 2023. eacl sasel-maing main. 148.
URL https:/ http///aaaaaaaltaaaeeeee org/
OpenAI. Chatgpt, 2022. URL httph:tt////ttt openai chat. Accessed: September 15, 2024.
OpenAL. Memory and new controls for chatgpt. https: com/ indx forool--fo-cchaappt 2024. Accessed: September 15, 2024.
Siva Reddy, Danqi Chen, and Christopher D. Manning. CoQA: conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266, 2019. doi: 10. a a_00266. URL https: http:/::/aaaaaathhaaacece wtt.llogy.orgg9999011/0
Stephen E. Robertson and Hugo Zaragoza. The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr. 3(4):333-33) 389, 2009. doi: 10.1561/1500000019 URL https: //doi. org.orrg1100
Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D. Manning. RAPTOR: recursive abstractive processing for tree-organized retrieval. The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL /tttp:://ppenreview. net hott//foummiic GN921JHCRW.
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 31210-31227. PMLR, 2023. URL https: /ptppppeecinngsssee mlr., press/ v202 /h//shi22 23a html.
14

---

Published as a conference paper at ICLR 2025
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Richard James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. REPLUG: Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 8371-8384, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: ht pattps:// / www.annnhhlooo.o..... org/ 2024. nwacn- long..lon... 463.
black-box language models. In
URL
Tao Tao, Xuanhui Wang, Qiaozhu Mei, and ChengXiang Zhai. Language model information retrieval with document expansion. In Robert C. Moore. Jeff Bilmes, Jennifer Chu-Carroll, and Mark Sanderson (eds.). Proceedings of the Human Language Technology Conference of the NAACI, Main Conference, pp. 407-414, New York City, USA, June 2006. Association for Computational Linguistics. URL ht: /aclanthooooyyy...oooo.. org/ 6006-1022
Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https: / /ww.nm.. github. it/bllg/bllooo qwen2.
Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao. and Furu Wei. Augmenting language models with long-term memory. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA. USA, December 10 16, 2023, 2023. URL http::/paapeee ebd82705 FSFFF447736665 (933666ade556699000 -tt.AAaadaaOo---atataatttee www.taaattooooeeeeeeaaaeaaeeeeee Calel--oooofreee
www..ci.pspsaeeeee cc/paper htcp/paperfflls/////eeeeesssssse www.ffllsppeee https//pppe///////eeeeeeeee
html.
Wei Wei, Quoe Le, Andrew Dai, and Jia Li. AirDialogue: An environment for goal-oriented dialogue research. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun' ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3844-3854, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v 1419. URL https: http::://aclaathologggo org/D1e 1419.
Jason Weston, Sumit Chopra, and Antoine Bordes. UrXiv:1410.3911 2014.
Memory networks. arXiv preprint
Yuhuai Wu, Markus N Rabe, Delee Hutchins, and Christian Szegedy. Memorizing transformers. arXiv preprint arXiv:2203.089 08913, 2022.
Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. Baize: An open-source chat model with parameter-efffiient tuning on self-chat data. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2268-6278, Singapore, December 2023. Association for Computational Linguisties. doi: 1..11655/1//2223.rr mer.emmlllmminn35... URI. https: /atps///aaaaatthho/eooooo ogy org/ 2023. emn in..nin...... 385.
Fangyuan Xu, Weijia Shi, and Eunsol Choi. RECOMP: improving -uureetdl-augmented Ims with context compression and selective augmentation. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
URL https: wpppeeveeeeeeeeeeeeeeeeeeee net wooomriidniiiii wtw.:mmmdd-mmmLLiiigg.
Jing Xu, Arthur Szlam, and Jason Weston. Beyond goldfish memory: Long-term open-domain conversation. In Smaranda Muresan, Preslay Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5180-5197. Dublin, Ireland, May 2022a. Association for Computational Linguisties. doi: 10. (0.18653/v1/2000 / URL https: http://aaaaathoooooo org/ 356.
Xinchao Xu, Zhibin Gou, Wenquan Wu, Zheng-Yu Yu Niu, Hua Wu, Haifeng Wang, and Shihang Wang. Long time no see! open-domain conversation with long-term persona memory. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguistics: ACL 2022, pp. 2639-2650. Dublin, Ireland, May 2022b. Association for Computational Linguistics. doi: 10.18653/w1/0022.ff http://iii//.....nnnnnaattta acl antho www.g/oooooo.o.ooo findings-eal 207.
URL https://
15

---

Published as a conference paper at ICLR 2025
Fan Yin, Jesse Vig, Philippe Laban, Shafiq Joty, Caiming Xiong, and Chien- Shien-Sheng Wu. Did you read the instructions? rethinking the effectiveness of task definitions instruction learning. In Anna Rogers, Jordan Boye-Graber. and Naoaki Okazaki (eds.), Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), Pp. 3063- 3079, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 18.5331//v1 2023.ac1-long.177 URL ht tps::p//// https///aaaaatlog.....e. org/ ael.:.cc-ooongg 177.
Wenhao Yu, Hongming Zhang, Xiaoman Pan, Kaixin Ma, Hongwei Wang. and Dong Yu. Chain-of-note: Enhancing robustness in reurental-uugmented language models. arXiv preprint vrXiv:2311.092/1 2023.
Dun Zhang. STELLA EN 1.5B v5, https: http:////i...a...ooo///////////a
1. 1.5B_ _5, 2023. Accessed: September 15, 2024.
Hongming Zhang, Xiaoman Pan, Hongwei Wang, Kaixin Ma. Wenhao Yu, and Dong Yu. Cognitive kernel: An open-source agent system towards generalist autopilots, 2024. URL https://
wexiv.orgio abs 2409 10277.
Michael Zhang and Eunsol Choi. SituatedQA: Incorporating extra-linguistic contexts into QA. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7371-7387, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/112021. 1t133//1/222.emmlpp www..leeeeannn..... aclanthology.coo orgoorg// 2021 we0.0.ee.m11g...mmmmmmmmmmm
URL https://
586.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. Memorybank: Enhancing large language models with long-term memory. Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan (eds.), Thirtt-Eigtth AAAI Conference Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium Educational Advances in Artificial Intelligence, EAA12014 February 20-27, 2024, Vancouver, Canada, pp. 19724-19731. AAAI Press, 2024. doi: 10. 11099/AAAL.338117.229999999 URL https:: /ttp:://doo..oo 10. 1609 aaaiaaaaa.........aaaa.aaa v38117.1200 299466.
Artificial Inteligence, AAAI 2024, Thirty-
Zexuan Zhong, Tao Lei, and Danqi Chen. Training language models with memory augmentation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 5657-5673, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 00.18633/v1/022 emnlp-main.382 URL https : / accannhologyy....o org/ emnlp main. 382.
16

---

Published as a conference paper at ICLR 2025
A SUPPLEMENTAL DETAILS FOR LONGMEMEVAL
A. DATASET CONSTRUCTION
In this section, we discuss the details of our benchmark construction process.
Attribute Ontology In Table 5, we provide the full attribute ontology used in LONGMEMEVAL. This attribute ontology constructed manually to reflect commonly mentioned topics user- assistant chats. Five major categories are included: demographic information, lifestyle, situational context, life events, and belongings.
Table 5: Human-constrrutedd user attribute ontology. Each attribute represents a unique dimension of human experience along which user biography could be constructed. For LoNG sample user backgrounds based on each of the attributes and construct questions on top of them.
Attribute I:
Demographic Information
age, gender, ethnicity, nationality, language, education level, occupation
Attribute 2: Lifestyle
2.1: Shopping: online shopping frequency, favorite stores, loyalty program, sales events, coupons, gift purchasing habits, eco-friendly product preferences, luxury vs budget shopping, technology gadget purchasing, fashion and apparel, grocery shopping, shopping for others
2.2: Media Consumption: book, movie, tv show, music, podcast, video game, streaming service, theater, magazine and newspaper, youtube, educational content, audiobook and e-book
2.3: Social Media Engagement: posting, commenting, followers, groups, hashtags, campaigns, messaging. live streaming, social media breaks
2.4: Daily Routines: wake-up time, bedtime, work or school start time, meal time, exercise routines, coffee or tea break. commuting, evening activities, weekend routines, cleaning schedules, time spent with family or friends
2.5: Travel: frequency, destination, road trips, travel agencies, outdoor adventures, airlines, hotel, travel with family solo travel, packing habits 2.6: Recreation:
reading, painting. musical instruments, dancing, watching sports, participating in sports, gardening. bird watching. fishing or hunting. board games, video games, fitness classes, yoga, sculpting, photography, stand-up comedy, writing, collecting, model building, aquarium keeping 2.7: Eating and Cooking: home cooking, food delivery, vegetarian or vegan, favorite cuisines, snacking habits, barbecue, baking, cocktail mixing, cooking classes Event Participation:
2.8:
concerts, theater, galleries and museums, sports games, film festivals, religious services, book readings, charity events, trade shows, lectures or workshops, theme parks, local markets, networking events, sports, auto racing, workshops, museum tours
Attribute 3: Situational Context
3.1: Home: living room, kitchen, bathroom, room style, room lighting, furniture, technology, plants 3.2: Social Context:
alone, family, friends, interactions with strangers time of day, day of week, seasonal
3.3: Time Context:
Attribute 4: Life Events
graduations, academic achievements, study abroad, significant academic projects, job promotions, starting business, births and adoptions, marriages, family reunions, illness or surgeries, mental health journeys, purchasing a home, trips, movement, living abroad, refugee or immigration, loss of loved ones, name change, belief, milestone
Attribute S: Belongings
cars, bikes, vehicles, computer, phone, pet, farm animal, animal care items, home, land, art, antiques, collectible, rare items, clothing, jewelry, shoes, bag, sports gear, musical instruments, health related devices, crafting, photography
Background Sampling Based on each of the attributes, we prompt Llama 3 70B Instruct to generate a background paragraph outlining the user memory and experience. our preliminary studies, find that the following zero-shot prompt in Figure 7 can already guide the model to generate a long and focused user background with sufficient details, which suffices for the next step of question creation. We thus use the same prompt for the final version of LONGMEMEYAL.
17

---

Published as a conference paper at ICLR 2025
I will give you a topic. Please imagine you user that wants recall and record recent personal facts along the topic. Generate a long text describing these personal facts. Use your imagination and generate the personal facts, Make it long and involve several recent facts or recent events spanning many days, weeks, or monthes. You may state the facts in plain language and no need make story-like.
Topic: {attribute)
Recent Personal Facts related to {attribute}:
Figure 7: The prompt for constructing user backgrounds based on an attribute.
I will give you a past memory. Use the memory to act as a normal user to chat with chat assistant. In the chat, you may ask it to assist you various tasks or ask it about various information. However, make sure that your convey the following fact about you: "(evidence "(evidence.ssatemmenn) statement]""" In addition, make sure your message is concise (1-2 simple sentences), since the real users often do not bother write a long message. I will provide you with the chat history and the response from the assistant. Directly generate the next response from the user's perspective. You must simulate the tone neutral user and do not be overly enthusiastic, verbose, formal, or polite. For conciseness, DO NOT react the assistant's message with c.g., "thanks" or "I will do that". Instead, directly state the follow-up questions or new questions.
Memory: (background}
Chat History:
assistant: Hi! How can assist you today?
(more rounds as the conversation continues)
Figure 8: The prompt for instruction an LLM to act as a user and initiates a task-oriented dialogue with another L.LM. Both the background and the evidence statement is provided. This prompt used for the question type single-session. For the other question types, the prompt components are slightly different but the prompt overall follows the same style.
Question Construction As discussed in $3.2, based on the generated user backgrounds, Llama 3 70B Instruct is used to propose question and answers for each question type. Additionally, for the question type temporal-reasoning. multi-session reasoning, and fingle-session-preeeree we use GPT-4o to propose several questions. Nevertheless, we find most of the questions to be unsatisfactory and manually filter and edit most of the questions. questions were generated for each question type, and the final yield rate is about 5%. For each question, we then manually decompose the answer into the evidence statements. the evidence statements involve time mentions, evidence statements at this stage. Note that timestamps are specified for the evidence statements this stage, these timestamps will always be used for corresponding evidence sessions. Otherwise, the timestamps will be randomly assigned at the history construction stage with all the other sessions.
total, approximately 1000
the question assign timestamp the question and the
Evidence Session Construction Using the question and the decomposed evidence statements, we use Llama 70B Instruct to simulate one user-AI chat history per evidence statement via self-chat. In Figure 8, we present an example of the chat simulation prompt, where we ask the user LLM to indirectly mention the evidence statement while avoiding talk about other evidence statements for the question, if there are any. We include two crucial instructions in the prompt to make sure (1) the evidence statement is provided in indirect way and (2) the generated messages are concise and
18

---

Published as a conference paper at ICLR 2025
thus mimic the style of user messages. On the assistant side, directly provide the input generated by the user LLM without any prompt engineering. We simulate the chat for 10 round at maximum and stopped prematurely when either side of the LLMs generates empty sequence indicating the end of the conversation.
Subsequently, expert annotators manually inspect and edit each of the generated sessions to ensure that (1) the required evidence statements are present in the conversation, (2) no other evidence statements are leaked into the conversation, (3) the evidence statements are provided in a colloquial style, especially for the data and time mentions, and (4) the conversation ends gracefully. In total, roughly 70% of the sessions are human edited. We note that in few rare instances, the user LLM fails by assuming the assistant role instead. When these failures are identified, we discard the instance the conversation cannot be fixed.
A.2 HISTORY CONSTRUCTION
In order construct a coherent and freely extensible chat history, design a three-staged pipeline that include session pool construction, session sampling, and timestamp resolution.
Session pool construction For each question, we draw the history sessions from three sources: ShareGPT (Zheng et al., 2023), Ultrahaat (Ding et al., 2023), and the simulated sessions corresponding to other attributes using the same pipeline mentioned in the previous section. This pool ensures that the non-evidence history sessions have similar topic or format as the evidence sessions, while avoiding providing conflicting information that would invalidate the question.
Session sampling To sample history containing x sessions, we randomly sample from the aforementioned three sources and shuffle the sessions together with the question's evidence sessions. For LONGMEMEVAL. we always use the following mixture: 25% ShareGPT, 25% UltraChat, and 50% simulated sessions. If the evidence sessions need follow specific order, we swap their orders accordingly after shuffling.
Timestamp resolution Finally, we randomly assign timestamps the session following their order of the history. If the evidence sessions are associated with pre-defined timestamps, we use them as anchors to determine the range of timestamp of the non-evidence sessions preceding or following them. Other wise, we randomly assign tiemstamps in May 2023.
A.3 BASIC STATISTICS
In Figure 9, we present the basic statistics of LONGMEMEVAL, revealing that most questions require evidence from multiple sessions (up to six) and that evidence statements are positioned diversely within sessions, increasing the challenge to the memory design.
500
14%
200
400
150
300
6%
16%
100
280
27%
50
100
single
single session assistant
Number Sessions
Answer Index)
aingle seasion pr
(b) Distribution the number evidence sessions. Most ques- tions emphasize multi-session reasoning, requiring reading up six sessions to answer.
(c) Distribution the location of the evidence statement within the evidence sessions. Most
crenstedpe-.pdate semperall-reaae
(a) Distribution of question types in LONGMEMEVAL.
evidence statements located at the beginning of the chat.
Figure 9: LONGMEMEVAL challenges chat assistants through its (a) diverse question types, (b) emphases on multi-session reasoning, and (c) diverse evidence locations within sessions.
19

---

Published as a conference paper at ICLR 2025
A.4 EVALUATION METRIC BUILDING
To accurately evaluate the diverse responses of LLMs, use expert-written prompt instruct GPT-4o as the correctness judge. We present the full prompt in Figure 10. To enable the model to handle detailed edge cases as how expert evaluators would do, we design separate prompts for a number of tasks. To ensure the prompt has a high agreement with expert judge, we sample 30 questions per problem type, collect the long-context generation results from GPT-4o and Llama 3.1 8B Instruct, and report the judgment correctness by category.
As shown in Table 6, the prompt-engineered GPT-4o judge achieves reliable performance in evaluating both GPT-4o and Llama-3, 1-..3..1--BBm- Instruct as the generation model. The evaluator's judgment slightly deviates from human experts for the iinll-session-prefereees and abstention problems due to the open-ended nature of the response. Nevertheless, our evaluation prompt still achieves 90% higher accuracy under all settings. We will include this prompt in the benchmark package that we will release to enable consistent comparisons for future work.
temp-reasoning
I will give you question, a correct answer, and a response from model. Please answer yes if the response contains the correct answer. Otherwise, answer no. If the response is equivalent the correct answer or contains all the intermediate steps get the correct answer, you should also answer yes. If the response only contains a subset of the information required by the answer, answer no. In addition, do not penalize off-by-one errors for the number of days. If the question asks for the number of days/weeks/months, etc., and the model makes off- oneeee errors (e.g., predicting 19 days when the answer is 18), the model's response is still correct.
knowledge-update
I will give you question, a correct answer, and a response from a model. Please answer yes if the response contains the correct answer. Otherwise, answer no. If the response contains some previous information along with an updated answer, the response should be considered as correct as long as the updated answer is the required answer.
single-sessoon-preeereee
I will give you question, rubric for desired personalized response, and response from model. Please answer yes if the response satisfies the desired response. Otherwise, answer no. The model does not need to reflect all the points in the rubric. The response correct as long as it recalls and utilizes the user's personal information correctly.
Other question types
I will give you question, a correct answer, and a response from a model. Please answer yes if the response contains the correct answer. Otherwise, answer no. If the response is equivalent to the correct answer or contains all the intermediate steps to get the correct answer, you should also answer yes. the response only contains a subset of the information required by the answer, answer no.
Evaluation instructions for the GPT-4o judge. We provide the question, answer, and the model's ' hypothesis after the instruction and ask GPT-4o to directly generate "yes" or "no".
Figure 10:
20

---

Published as a conference paper at ICLR 2025
Table Meta-evaluation results of prompt-engineered JPT-4ojudge. We observe high evaluation accuracy across all the problem types in LONGMEMEVAL.
Answer Model
Question Type
GPT-4o
-lama-.1--B--inttie
single-session-user
1.00 (30/30)
0.97 (29/30)
single-sesioonasssitaat
1.00 (30/30)
1.00 (30/30)
single-sesion-preeerene
0.90 (27/30)
0.97 (29/30)
multi-session
1.00 (30/30)
1.00 (30/30)
knowledge-update
1.00 (30/30)
1.00 (30/30)
temporal-reasoning
1.00 (30/30)
0.97 (29/30)
abstention
0.97 (29/30)
0.90 (27/30)
Average
0.98
0.97
B A HUMAN STUDY ON COMMERCIAL MEMORY CHATBOTS
We evaluate two commercial memory-augmented chatbots: ChatGPT (OpenAI, 2024) and Coze (Coze, 2024). We randomly selected 97 questions and created short chat history of 3-6 sessions by sampling according to mixture of 50% ShareGPT and 50% simulated sessions. We skip two type questions that the assistants cannot answer: (1) a subset of temporal-reasoning since our manual analysis cannot afford interacting with the (potentially evolving) systems across multiple months, and (2) single-session wwwss-e--ssssssaaaeeeee by the assistant. We also did not evaluate the abstention ability of these two systems because early version of the dataset without any abstention questions was used for this analysis.
since the systems do not remember any information given
Since these systems only support memory features via their web interfaces, human annotators manually interacted with the chat assistants session-by-sessionn ending with a new session where the question was posed. After collecting the model's response, the annotator manually evaluates the answer's correctness. Finally, to start evaluating the next instance from a fresh state, the annotator manually clears the model's memory through the web interface. We distribute the questions across five amnotators. A discussion is performed among the annotators whenever there is a concern with the model's response or with the evidence sessions. All evaluations were conducted in the first two weeks of August 2024.
In Table 7, we present the detailed human evaluation results by problem types. We observe that when the task is memorizing the information from a single session (column IE), both systems can answer considerable number of problems correctly. However, for the other question types where aggregation across multiple sessions is generally required, both systems exhibit significant performance drops. Compared to ChatGPT, we find most of Coze's 's errors are due to failing to record information from some session. On the other hand, ChatGPT generally records the evidence statements immediately after it has been presented the evidence session. However, as the interaction proceeds, ChatGPT often modify this information when it compresses the history, resulting in information loss. This highlights the potential trade-off between reliable personalization and efficiency.
Table 7: Human evaluation results of two systems categorized by evaluated ability types. We use the questions from single- sedle--ession-usee the column. For the temporal reasoning column (TR), we use the questions that do not require reasoning with metadata.
Memory Ability MR KU
System
Model
IE
TR
GPT-4o-mini
1.000
0.647
0.667
0,652
ChatGPT
GPT-4o
0.688
0.441
0.833
0.435
GPT-3.5-turbo
0.625
0.118
0.375
0.043
Coze
GPT-4o
0.813
0.147
0.208
0.391
21

---

Published as a conference paper at ICLR 2025
C UNIFIED MEMORY VIEW
C.I AN ALTERNATIVE MATHEMATICAL FORMULATION
In this section, we provide more nigorous mathematical formulation of the unified memory framework for interested readers. Since we find the corresponding natural language descriptions clear enough for presenting the main arguments while avoiding introducing excessive symbols.
We formulate long-term memory as a massive key-value datastore D = [(k u1), (b2, v2.. where the keys k; can heterogeneous and the values v; might repeat. More concretely, a key could be discrete or continuous. In the discrete case, the key could be a sentence, a paragraph, fact, or an entity, etc. In the continuous case, the key could be e.g., the model's internal representation under some inputs. For the three stages, we formulate them with four functions: I, Q. S, R. The offline index function I(S, D) converts a session S into a series of key-value tuples. Alternatively, the online index function Tonline (S, D) updates D with the key-value tuples extracted from S, potentially removing or editing the existing items in D. The query formulation function Q(g) converts a natural language user query q into representation o' that function S could leverage. The salience scoring function S D) orders D by their relevance to g'. Potential initializations of S include ranking dense text embedding, graph-based ranking, and so on. Finally, the reading function R(M, D*) combines the model M with D', the most relevant part as ranked by the salience scoring function S and generates a response. The methods outlined in the columns of Table could be seen as different ways to instantiate the four functions I, Q S, R.
C.2 EXISTING MEMORY SYSTEMS FROM THE UNIFIED VIEW
In $4, we have presented a unified view of the memory systems with three stages and four control points. In this section, we provide closer look at nine memory-augmented chat systems and view them as specific instances of the unified framework. Specifically, we consider in-context RAG (Shi al., 2024), Memorybant (Zhong al., 2024), LD-Agent (Li al., 2024), CoN (Yu et al., 2023), ChatGPT, Coze, RAPTOR (Sarthi et al., 2024), MemWalker (Chen et al., 2023a), and HippoR AG (Gutiérrez al., 2024). Table provides an overall comparison between the systems, among which we also situate the recommended design identified through the paper's 's empirical study.
Indexing For the value representation, most of the surveyed systems either directly use the sessions themselves or use mixture of them with the extracted facts or summaries. Only three systems (LD-Agent, ChatGPT, and Coze) use only the compressed values to replace the original ones, which may incur information loss. When it comes to the indexing key, we observe three major types of decisions. First, a number of systems (in-context RAG, Memorybank, LD-Agent, and CoN) simply the values as the keys, which wins in terms of simplicity. By comparsion, HippoRAG builds an entity-centric index, and RAPTOR/Memwalke build a hierarchical index using recursive summarization. structure can potentially benefit certain types of queries, they also increase the cost of creating and maintaining the index in online interactions. Specifically, for HippoRAG, RAPTOR, and Memwalker, some level of re-indexing is required when a new session is added to the memory, increasing the computational overhead of these sytems.
is noteworthy that while more complex memory indexing
Retrieval For most of the systems, find that the question is generally used as the query, which is intuitive since the questions are generally short. LD-Agent and HippoRAG extract additional information and leverage them better pinpoint entity-centric knowledge within the index. In terms of the query-key matching process, most systems use a flat retrieval setup, where similarity search is directly conducted between the query and the keys. Hi uses Personalized PageRank (PPR) to leverage its entity graph structure to match passages that feature entities close to the seed entities extracted from the query. Different from these approaches, Mem Walker bridges the retrieval and inference through an interactive reading mechanism where the reader L.MM self-navigates the indexing structure to find the answer. This method brings the additional robustness in case the retriever fails, but also incurs additional latency costs due more LLM inference.
Generation Most of the systems apply a direct reading mechanism where the reader model is provided with the retrieved memory items and directly asked to generate the response. However,
22

---

Published as a conference paper at ICLR 2025
show in the main results, adapting extract-before-rest strategy is important to high performance. On the other hand, the interactive reading mechanism such as that used MemWalker allows the reader to backtrack and search again in case the recalled memory is inadequate.
D MEMORY OPTIMIZATIONS: IMPLEMENTATION DETAILS
In this section, we describe implementations of the memory optimizations in detail.
Value Decomposition We design an LLM-based method for compressing the value, which are either sessions or rounds, into three different formats: summaries, keyphrases, user facts. In Figure 11, we present the corresponding zero-shot and few-shot prompts. Note that few-shot learning is only applied for user fact extraction, and we find it to be unnecessary for the rest two tasks. For in-context examples, we randomly sample ten examples, five from the simulated user sessions and five from ShareGPT. The expected responses are generated by GPT-4o and modified by human annotators. We manually tune the prompts by observing several examples. Llama 3.1 8B Instruct is used for all the extraction experiments. Note that for all the experiments, we only provide the user-side messages for the extraction, skipping the assistant's responses.
Key Expansion To expand the value documents with summaries, keyphrases, or user facts, we follow the same pipeline the previous section first extract these pieces of information from each value. Then, we directly prepend the extracted information to the corresponding key.
Time-Aware Indexing and Query Expansion At the indexing stage, we instruct Ilama 3.1 Instruct to extract the event mentions in the text whose timestamp could be inferred. At the retrieval stage, we instruct an LLM (we explore GPT-4o and Llama 3.1 8B Instruct) to extract the time range for retrieval for the problems that specifically focus range of time. We present the corresponding prompts in Figure 12. Ten human-written in-context examples are additionally provided.
Reading Strategy We present the prompt for reading Figure 13. which is a small variation of Chain-of-Note (CoN). Overall, the prompt shares the same idea with CoN, that the model is asked traverse the documents and extract the evidence before generating the answer to the question. We use greedy search for all the experiments, and set the maximum generation length to 800 tokens.
23

---

Published as a conference paper at ICLR 2025
Summaries
Below is a transcript of a conversation between a human user and an Al assistant. Please summarize the following dialogue concisely possible in short paragraph, extracting the main themes and key information. your summary, focus more on what the user mentioned asked for. Dialogue content: {session or round]
Keyphrases
Below is transcript of conversation between a human user and an Al assistant. Generate a list of keyphrases for the session. Separate each keyphrase with a semicolon. Dialogue {session round]
content:
User Facts
You will be given list of messages from a human user to an Al assistant. Extract all the personal information, life events, experience, and preferences related to the user. Make sure you include all details such as life events, personal experience, preferences, specific numbers, locations, or dates. State each piece of information in a simple sentence. Put these sentences in json list, each element being standalone personal fact about the user. Minimize the coreference across the facts, e.g., replace pronouns with actual entities. there is specific events, personal information, preference mentioned, just generate empty list.
Human user messages: {session}
Personal facts about the user (a list of strings in json format; do not generate anything else);
Figure 11: Zero-shot prompts for extracting information from the value items for indexing. For user fact extraction, we additionally provide ten in-context examples.
Extracting Timestamped Events
Al assistant, well as the time the conversation took place. Extract all events related to the user as long as its date specified could be inferred. If the time some event took place cannot be inferred, do not extract that event. Return the events in a json list where each item contains two fields: "date" and "event". Write date in the form YYYY/M/DDD. If there no specific event, just write an empty list.
You will be given list of messages from human
Query Expansion
You will be given a question from a human user asking about some prvious events, as well as the time the question is asked. Infer a potential time range such that the events happening in this range is likely to help to answer the question (a start date and an end date). Write json dict two fields: "start" and "end". Write date in the form YYYY/MMDDD. the question does not have any temporal referencea, do not attempt to guess time range. Instead, just. say N/A."
Figure 12: Zero-shot prompt for extracting timestamped events from the value items for indexing and the prompt for extracting time range from the user question. For both settings, we additionally provide ten in-context examples.
24

---

Published as a conference paper at ICLR 2025
CoN Prompt
I will give you several history chats between you and user. Please answer the question based on the relevant chat history. Answer the question step by step: first extract all the relevant information, and then reason over the information to get the answer.
History Chats: (istt_history]
Current Date: {question date}
Question: {question} Answer (step by step):
Non-CoN Prompt
I will give you several history chats between you and user. Please answer the question based on the relevant chat history.
History Chats: (chat history)
Current Date: {question_date
Question: {question}
Answer:
Figure 13: The prompt for reading with recalled memory with and without Chain-of-Note.
25

---

Published as a conference paper at ICLR 2025
E EXTENDED ANALYSES
We provide additional analyses on LONGMEMEVAL, including more LLMs, retriever selection, memory optimization, and error analyses of end-to-end retrieval-augmmenee generation.
E. RESULTS ON MORE LLMS
In Table 8, we evaluate five more models on LONGMEMEVAL: the 1B and 3B models of Llama- 3.2 Instruct (Dubey et al., 2024), Qwen-2.5-7B (Team, 2024), Phi-3.5-mini-instrutt (Abdin et al., 2024b), and Misstrl-Nemo--nnstuutt--2-2 the performance is consistent with the results reported in the paper. For instance, long-context performance on LONGMEMEVALs is significantly lower than the oracle retrieval performance. In addition, the fact-based key expansion and CoN consistently improves the performance.
(Mistral AI Team, 2024). Overall, we observe that
Oracle
LONGMEMENALs
LoNGMEMEVALa
Direct
CCN
CuN
K=
V+fact
Mistral-Nema Tnst ruet-2407
0.686
0,286
0.554
0.598
Uwen2.5-7u
0.504
0.144
0.390
0.424
2ad-b.3-mind-nnstruee
0.490
0.352
0.468
Llana Instruet 11ama-3.2-18-Tnstree
0.522
0.636
0.010
0.508
0.466
0.470
0.386
0.398
0.010
0.016
0.312
0.336
0.286
0.312
Table 8: Evaluation results using five additional LLMs of various sizes. "LC" indicates directly reading from the entire history without any memory operations. For the last four columns that have memory operations, we always round as the value granularity.
E.2 ABLATIONS ON RETRIEVER SELECTION
In Table 9, we compare the performance of Stella V5 with two alternative popular retrievers: BM25 (Robertson & Zaragoza, 2009) and Contriever (Izacard et al., 2022). We do not report the results of retrievers that use larger embedding models, as their latency is generally unrealistic for real applications. The retrievers are compared under two settings: (1) vanilla key design where the values themselves are used as the key and (2) the key expansion strategies introduced in the main text. We list the key observations below:
Both dense retrieval embeddings have significantly higher performance than the BM25 sparse retrieval method. The trend is the same across most settings.
The recommended technique of performing key expansion with user fact consistently improves the performance over directly using the value as the key.
Key expansion with summary and keyphrases could improve the performance in some settings. For example, both summary and keyphrase improve the Recall of Contriever when session is used as the value granularity. However, expanding the key with facts still gives the greatest performance gain.
E.3 POST-RETRIEVAL RANK MERGING FOR INDEX EXPANSION
In $5.3, we have introduced and evaluated the key expansion strategy for multi-path retrieval, where important information extracted from the value items and then used augment the key. However, an alternative strategy is to directly create a separate key with the retrieved information and place it in parallel as the original keys. the retrieval stage, the query could used to retrieve from pools defined multiple types of keys and the values from different sequences could be merged subsequently according to their rank. Effectively, this implements the true "multi-path retrieval. We call this strategy "rank merging". In Table 10, we thoroughly evaluate rank merging verse key merging, the strategy we recommended in the main text. We find that rank merging has much lower performance than key merging. One potential reason is that rank merging increases the index size by + 1 times, where m is the number of information pieces extracted from each (key, value) pair. By comparison, key merging highlights the important information while avoiding explording the size of the index.
26

---

Published as a conference paper at ICLR 2025
Table A comparison between three retrievers under different key-value indexing settings.
Value session
Value round
Key Design
Retriever
Metric 5
Metric 10
Metric@5
Metric 10
Recall
NDCG
Recall
NDCG
Recall
NDCG
Recall
NDCG
BM25
0.634
0.516
0.710
0.540
0.472
0.352
0.538
0.372
K=V V
Contriever
0.723
0.634
0.823
0.663
0.589
0.454
0.747
0.495
Stella V5 1.5B
0.720
0.594
0.794
0.615
0.660
0.498
0.784
0.528
BM25
0.626
0.544
0.694
0.565
K = V summary
Contriever
0.732
0.632
0.823
0.657
Stella V5 1.5B
0.689
0.608
0.749
0.624
BM25
0.683
0.560
0.757
0,582
0.554
0,454
0.608
0.473
K = V fact
Contriever
0.762
0.632
0.862
0.658
0.612
0,489
0.756
0.530
Stella V5 1.5B
0.732
0.520
0.862
0,552
0.644
0.498
0.784
0.536
BM25
0.632
0,546
0.711
0.569
0.453
0.391
0.538
0.416
K V keyphrases
Contriever
0.740
0.638
0.849
0.668
0,546
0,450
0.672
0.489
Stella V5 1.5B
0.710
0.587
0.768
0.602
0.478
0.359
0.636
0.410
Table 10: Retrieval and end-to-end QA performance on LONGMEMEVALL with different key designs for indexing. L3.1 = Llama 3.1 Instruct. RM = Rank Merging and KM = Key Merging.
Retrieval
End-to-End QA
Key Design
Metries@5
Metrics
GPT-40
133.1 70B
133.1 8B
Recall NDCG
Recall NDCG
Top-5
Top-10
Top-5
Top-10
Top-5 Top-10
Value Round
K V
0.582
0.481
0.692
0.512
0.615
0.670
0.600
0.624
0.518
0.534
K V + fact (RM)
0.478
0.372
0.568
0.403
0.558
0.596
0.536
0.586
0.448""
00.422
K V keyphrase (RM)
0.386
0.376
0.536
0.329
0.485
0.590
0.460
0.558
0.426
0.466
+ fact (KM)
0.644
0.498
0.784
0.536
0.657
0.720
0.638
0.682
0.566
0.572
K V + keyphrase (KM)
0.478
0.359
0.636
0.410
0.541
0.652
0.538
0.620
0.472
0.524
Value Session
V
0.706
0.617
0.783
0.638
0.670
0.676
0.592
0.570
0.524
0.464
K V + summary (RM)
0.608
0.488
0.684
0.511
0,600
0.620
0,510
0.544
0.468
0.466
V fact (RM)
0.618
0.511
0.754
0.548
0.622
0.688
0.574
0.574
0.468
0.466
V + keyphrase (RM)
0.550
0.435
0.650
0.466
0.560
0.620
0.454
0,506
0.492
0.482
K V + summary (KM)
0.689
0.608
0.749
0.624
0.658
0.666
0.568
0.560
00,188
(1.494
V fact (KM)
0.732
0.620
0.862
0.652
0.714
0.700
0.588
0.584
0.530
0.490
V + keyphrase (KM)
0.710
0.587
0.768
0.602
0.665
0.672
0.590
0.566
0.526
0,508
E.4 STRONG AND WEAK LLMS FOR EXTRACTING TIME RANGES FROM QUERIES
In $5.4, we concluded that a strong model is required to accurately extract the time ranges from the questions, even if in-context examples (with balanced labels) are provided. In this section, further illustrate this finding with examples in Table 11. We find that the Llama 3.1 8B Instruct model can generate a correct temporal range when the time references are clearly defined (example 2). However, in many of the cases where the question does not have any temporal reference (example 1, 4), the model often mistakenly extracts a time range, which erroneously prunes out the search space, leading to low memory recall. By contrast, GPT-4o is able to refuse to generate time range when the question does not have a time reference.
E.5 ERROR ANALYSIS
In this section, we analyze the source of error of various LLMs with our best memory design. Specifically,, we use round as the value granularity, expand the key with extracted user facts, and apply CoN as the reading strategy. Stella v5 1.5B is used as the retriever, and top-10 items are provided to the model with a JSON structured prompt. In Figure 14, we analyze the distribution of the correct and error cases for three different reader LLMs. First, observe that substantial proportion of errors corresponds to correct retrieval yet wrong generation (15%~19% of all instances, and 40%~50% among the emror instances). The proportion is higher when weaker
27

---

Published as a conference paper at ICLR 2025
Table 11: Example of time span extraction outputs from GPT-4o and Llama 3.1 8B Instruct.
Example
Question Date: 2023/05/28
Question: How long had been taking guitar lessons when bought the new guitar amp?
Predicted time range (GPT-4o): date extracted [correct] Predicted time range (Llama 3.1 8B Instruct):
[false positive]
Example 2
Question Date: 2023/04/27
Question: Which airline did fly with the most in March and April? Predicted time range (GPT-4o): 2023/03/01
[correct] Predicted time range (Llama 3.1 8B Instruct): 2023/05501 [correct]
Example
Question Date: 2023/06/28
Question: How many days before the 'Rack Fest' did participate in the 'Turbocharged Tuesdays'
event?
Predicted time range (GPT-4o): No date extracted [correct] Predicted time range (Llama 3.1 8B Instruct): 2023/0661-22223007777 [false positive]
Example 4
Question Date: 2023/03/10
Question: Which seeds were started first, the tomatoes or the marigolds?
Predicted time range (GPT-4o)): No date extracted [correct]
Predicted time range (Llama 3.1 8B Instruct): 2023/03301--02230331 [false positive]
reader LLM is used. This indicates that the reading strategy still has large room of improvement. In addition, we observe that for the reader LLM to generate correct answer, performing correct retrieval is necessary in ~900% of the time. We observe that the rest 10% instances are mostly of the question type knowledge-update and the retriever was only able identify the updated knowledge but failed to retrieved the previous information before update. For these cases, our strict retrieval evaluation criteria will deem that the retrieval has failed. Overall, this result also highlights the high quality of LONGMEMEVAL VAL, as the reader LLM cannot take any shortcut to answer the question correctly without a correct memory recall.
GPT-4o
Liama 3.1 70B Instruct
Liama 3.1 BB Instruct
0.0%
5.6%
4.0%
17.2%
18.8%
52.6%
64.0%
13.2%
62. 6%
14.6%
R, Wrong
Wrong Wrong
Wrong
Figure 14: An analysis of the error distribution of different reader models. We use and to indicate correct Recall 10 and correct answer based on the top- 10 retrieved results.
28


Reviews:
[{"review_id": "wPzynD6sce", "reviewer": "Reviewer_NGvs", "summary": "the paper proposes LONGMEMEVAL, a comprehensive benchmark designed to evaluate five core long-term memory abilities of chat assistants: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. Based on that, the present a unified framework that breaks down the long-term memory design into four design choices across the indexing, retrieval, and reading stages. Built upon key experimental insights, they propose several memory designs including session decomposition for optimizing value granularity, fact-augmented key expansion for enhancing the index structure, and time-aware query expansion for refining the search scope. Generally, i think this is a good paper and demonstrates some insightful observations via comprehensive experiments.", "questions": "1. according to (c) in figure 3, it seems there are no answer located in 7 8 9 10 round.\n2. in figure6, what motivate you to list multi-session subset independently? and why not analyze this in later experiemnts?\n3. why some experiments use both llama 3.1 70b and 8b instruction model, and some only use 8b model?\n\nother typos\n\n1. the answer in (a) of figure 2 is wrong?\n2. best performance of llama 3.1 70b when value = session on table 2 is wrong?\n3. typo in line 468", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. the proposed benchmark is valuable, being diverse, scalable and practical.\n2. the experimental results are solid and comprehensive and the analysis is insightful such as the choices of key, value and query expansion.\n3. the presentation and organization is relatively clear and easy-to-understand.", "weaknesses": "1. lots of human intervention is required to ensure the quality of dataset, such as line 248\n2. most of experiments are conducted using gpt4o and llama3.1 8b instruct models. there is no comprehensive analysis about exsiting models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "chb9FVGHLB", "reviewer": "Reviewer_eitt", "summary": "This paper evaluates long-term memory abilities in dialogue tasks from extraction, multi-session reasoning, temporal ability, knowledge management, and abstention. They found that it's a challenge for existing LLMs to perform sustained interaction with those chat histories. To tackle this issue, this paper presents a unified framework to redefine the process when treating long-term memory. Beyond that, extensive experiments show optimizing in the new proposed framework can improve memory recall and QA ability in their benchmark. The key point of this work is to divide the pipeline workflow into single parts when facing long-term memory dialogue situations. They create larger data to make the long-term memory better represented, then make the model perform better.", "questions": "Please give feedback to Weaknesses -> No.1 and No.2.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. It divides the memory workflow into three phases: indexing, retrieval, and reading. This segmentation not only helps to conceptualize the entire process but also lays the groundwork for systematically improving each stage. Then this paper further enhances this workflow. These modifications represent a thoughtful approach to refining how information is indexed, retrieved, and read from long-term memory. The process is sound and convincing.\n\n2. This paper did a comprehensive evaluation across different existing models and provided detailed insights into how various models perform differently depending on the task—some models excel at reading (i.e., interpreting memory), while others are better at understanding or processing the retrieved information.", "weaknesses": "1. This work was generated by human-AI talk. However they did not mention how they control the hallucination and other issues when talking to AI chatbots (i.e., repeating problems, proportion of effective dialogue turns, etc.)\n\n2. The proposed five aspects to evaluate long-term memory should be discussed more (Which aspect contributes more? Do different aspects have overlaps? What if one aspect performs better but another one fails?).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fXt2YjfXNR", "reviewer": "Reviewer_GckN", "summary": "This paper focuses on the problem of long-term dialogues. To study this problem, this work constructs LongMemEval, a new benchmark for evaluating five long-term memory capabilities of chat assistants, including information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention.  Experimental results on this new benchmark indicate significant challenges in existing LLMs on long-term dialogues. Furthermore, the authors propose a new framework, consisting of several designs, such as session decomposition, fact-augmented key expansion, and time-aware query expansion. Further results show that these designs can improve both memory recall and downstream question answering.", "questions": "1. Why is the proposed framework called a \"unified\" framework? It is very pipeline, not unified. \n2. Since the timestamp is randomly assigned, how to guarantee the evaluation of temporal reasoning?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Long-term dialogue is an important problem in the era of LLMs, which also lacks of comprehensive problem formulations. This work formulates several core memory capabilities in this problem.\n2. Construct a new challenging benchmark for studying this problem.\n3. Propose a unified framework for improving the long-term dialogue capabilities in LLM-based conversational systems.", "weaknesses": "1. The construction of history sessions is not well motivated. The constructed history could be very noisy and the temporal order is just random. Not sure what is the motivation behind this messy historical session construction.\n2. The proposed framework is actually a combination of several existing design. \n3. There are too many terms to remember in this work, which hampers the readability of this paper.\n4. It would be better to include some long-term memory models to make a comparison for better presenting the challenge of the proposed benchmark and the effectiveness of the proposed framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IPhCs00yfx", "reviewer": "Reviewer_LhuP", "summary": "This paper proposes a 500 LongMemEval dataset focusing on evaluating five crucial long-term capabilities of LLMs: (single-session) information extraction, multi-session reasoning, knowledge update, temporal reasoning, and abstention. \nThe authors define each conversation as a key-value pair, where the key is the \"time index\" and the value is the corresponding chat.\nThey propose a three-stage framework for indexing, retrieval, and reading.\nThe framework has four control points: key (in the indexing stage), value (in the indexing stage), query (in the retrieval stage), and reading strategy (in the reading stage).\nThey conduct experiments to test their framework using LongMemEval on GPT-4o and Llama-3 (70B and 8B).", "questions": "Please address the concerns/questions in the above presentation (P1-P3), soundness (S1-S5), contribution (C1-C2), and the following other questions.\n\nOther:\n\n1. The Ethics Statement is missing. \n\n* Since the LongMemEval dataset and the pilot study require human annotators, the ethics statement *should* be included, even though they are not as strongly recommended as the Reproducibility Statement. Moreover, given that this paper studies a three-stage framework for improving a chatbot's long-term memory, could this be exploited by malicious users to produce harmful results? I cannot find it in the appendix, however. \n\n2. Do the authors include the Limitations section? If not, then is this work thoroughly studied? What are the future work? \n\n3. What is the temperature used in this paper? What is the version of GPT-3.5 turbo tested in Figure 4-a?\n\n4. In Figure 4-a, do the author have insights regarding why gpt-4o-mini is better than gpt-4o?\n\n5. Do the authors touch upon the NDCG metric in the discussion section?\n\n6. In Lines 419-420, are the timestamps sorted by ascending or descending order? Do the authors evaluate the performance when the timestamps are not sorted in the preliminary experiments?\n\n7. In Table 2, why K = summary and K = V + summary experiments are not included in this paper? Is it because of the budget-aware manner? Moreover, why is the numeric value (0.590) in QA performance using Llama-3 (70B) in bold? It is smaller than the corresponding value when K = V (0.592).\n\n8. The rounding is confusing.\n\n* In the third contribution (Line 99), the authors report improvement is 7% ~ 11%, but it's 6.7% ~ 11.4% in Lines 480-481. Moreover, the result is for GPT-4o LLM, and this improvement does not hold when Llama-3.1 (8B) is used, as stated in Lines 482 (*We also find that the effectiveness of this method depends on using a strong LLM*). Nevertheless, this strong prerequisite is not mentioned in Line 99, which could be a bit misleading regarding the \"7% ~ 11%\" improvement.\n\n* Can the author provide how to compute the numerical values in Line 467 (*This approach, particularly when using user facts, yielded an average improvement of 4% in retrieval metrics and 5% in final accuracy across all models.*)? Specifically, how to get the result of a 5% improvement in Table 2? Is it $1/3 * [(0.714-0.670)/0.670 + (0.584-0.570)/0.570 + (0.490-0.464)/0.464] = 0.04875$, then 4.9% rounds to 5%? If so, why **only** use Top-5 in GPT-4o and Top-10 in Llama-3.1 (70B and 8B)? Because I can get the 4% result in retrieval metrics if I average all the improvement in \"Recall@5, NDCG@5, Recall@10, and NDCG@10\": $1/4 * [(0.732-0.706)/0.706 + (0.620-0.617)/0.617 + (0.862-0.783)/0.783 + (0.652-0.638)/0.638] = 0.0411$ (4.1%); however, when I try to average all the top-5 and top-10 improvement in these LLMs (6 entries), it does not match 5%.\n\n* Similar to the previous question, can the authors list the steps when computing the average metric in Lines 480-481? How can we get the 11.4% and 6.7% results from Table 3? Is there any additional rounding method involved after the computation?\n\n9. Are Figure 4-b and Figure 7 related in any sense? If so, please explain the following questions.\n* There is a discrepancy between Figure 4-b and Figure 7. Specifically, the oracle of GPT-4o is 0.924 and Llama-3.1 (*70*B, I guess) is 0.848 in Figure 4-b, and they can be found in Figure 7 (JSON + CoN). However, the oracle of Llama-3.1 (8B) is 0.710 in Figure 4-b, which is the natural language with CoN (NL + CoN). Since the JSON input format is not mentioned in Section 3.5, the authors should report when the input format is NL but not JSON. \n* In this vein, if I understand correctly, how can we derive the oracle of those LLMs w/o CoN from Figure 7? These values are not identical (I thought it could be done by looking at NL + Direct Prediction).\n* Is it a coincidence that Llama-3.1 (8B) has no improvement when CoN is used?\n\n10. Other typos and suggestions:\n* To improve the clarity of this paper, including a simple, concrete example in Figure 5 (or mentioned in Sections 4 and 5) would be beneficial. Otherwise, readers must refer to Appendix D (in Figures 11 and 12) to know what keyphrases/user facts/query expansion is exactly.\n* The term \"round\" is not precisely defined in this paper (see Line 377: decomposing sessions into individual rounds). I only find the session is defined in Section 3.1. For instance, can a session be decomposed into 2 (i.e., user and chat assistant) rounds? If more rounds ($>$ 2) are possible, how do you handle this process in this paper?\n* cross-session $\\rightarrow$ multi-session (for consistency). Note that other words need to be revised for consistency, yet it is too cumbersome to list all of them.\n* In Line 65 (verb missing): ... Finally, we the coverage of five core ...\n* In Line 138, the em dash is inconsistent.\n* In Line 161, RAG is not mentioned in retrieval-augmented generation but is later used in Sections 5.2 and 5.3.\n* In Lines 375-376, should use \\citep.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "1. This paper aims to tackle a problem of LLM from the time perspective, which may be a useful method.\n2. Apart from the single-session, the knowledge update and the temporal reasoning are known issues in contemporary LLMs.\n3. They study these phenomena in the long-context, long-term scenario.", "weaknesses": "This paper has several drawbacks regarding presentation, soundness, and contribution. \n\nTL;DR, the major weaknesses are as follows (see **Details** below for further elaboration):\n\n**Weakness 1.** [Presentation and Soundness] **This paper lacks careful proofreading, as it contains an unusually high number of typos and inconsistent writing. The numerical rounding is also confusing.** For example, in the third contribution, the authors report improvement is 7% ~ 11% in Line 99, but it's 6.7% ~ 11.4% in Lines 480-481 (should report the exact value or *at least* add the term \"around\" in Line 99). **Additionally, the flow in each section is not strongly connected, requiring significant revision to reach a publishable standard. Many tables and figures need substantial revisions and adjustments. The authors also need to be crisp in their writing.** For one thing, in the main content, the \"result and discussion\" of their proposed framework only spans two pages (Sections 5.2 to 5.5), including two tables and one figure. Without these, the text is roughly a page only in the main paper, which is insufficient for the ICLR community. \n\n**Weakness 2.** [Presentation and Soundness] The LongMemEval dataset should be tested on other papers' existing baselines (with external systems like RAG). However, no \"baseline section\" is mentioned in this paper, and I cannot find the baseline mentioned in the \"experimental setup\" in Section 5.1 as well as in Tables 2 and 3 and their titles (e.g., \"K = V is the baseline from xxx paper\"). Hence, it is challenging to understand the prior works' baselines based on each individual's assumption. If non-familiar readers try to search for the \"baseline\" keyword, they will find it only appears once in the second contribution (in Line 95). After jumping to Section 5.3, they cannot find the recent papers conducting LLM with RAG baselines within this section either. If no comparable baselines are suitable for this task, see S2 below for suggestions.\n\n**Weakness 3.** [Presentation and Soundness] **The authors claim this paper presents a *unified* framework, as shown in Figure 5, but without a formal mathematical definition, it is difficult to justify whether this is indeed true.** For example, how does the proposed framework encompass the existing RAG frameworks? Moreover, it is also hard for other future researchers to follow/revise/expand their ongoing work, particularly the four control points (CP) in Figure 5. Take the conducted CoN experiment (Section 5.5) as an example: The connection between the case when CoN is applied and Figure 5 is unclear. This figure is not illuminating, and the imprecise definition will let the readers interpret this paper on their own. Note that each separated component can be visualized and formalized in a better way (see Figure 1 in [0] and [1] for visualization; see also Section 3.1 in [1] for formalization). In addition, this paper only displays general terms without an explicit example to showcase their framework in Figure 5, whereas it is common for various RAG papers (see [2] and [3]). \n\n**Weakness 4.** [Contribution] **As for the dataset comparison, LongMemEval needs justification to significantly distinguish it from other related works, such as task-oriented dialogues (as stated in Line 50) and other QA-based datasets (see C2 below).** Specifically, several question types are highly similar to (that is, can be leveraged by) those existing datasets, especially the single-session scenarios. Nonetheless, they are missing in the references. Hence, further analysis is necessary regarding why not to leverage their datasets.\n\n[0] Tree of Thoughts: Deliberate Problem Solving with Large Language Models, Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, Karthik Narasimhan, NIPS 2023\n\n[1] Memory-Based Model Editing at Scale, Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning, Chelsea Finn, ICML 2022\n\n[2] Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection, Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, Hannaneh Hajishirzi, ICLR 2024\n\n[3] REPLUG: Retrieval-Augmented Black-Box Language Models, Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Richard James, Mike Lewis, Luke Zettlemoyer, Wen-tau Yih, NAACL 2024\n\n\n# Details\n\n## Presentation (P)\n\n**P1. The contexts need to be reorganized to provide a clear flow.**\n\n* As shown in Figure 2, the dataset construction pipeline is (a) Question Construction $\\rightarrow$ (b) Evidence Session Construction $\\rightarrow$ (c) History Construction. However, the (a) and (b) steps are merged into a single \"paragraph\" in Line 241 (within Section 3.2: LongMemEval: Benchmark Curation), while the (c) step has its own single \"subsection\" (Section 3.3: History Compilation).\n\n* In the main paper, the results and discussion of their proposed framework only have two pages (Sections 5.2 to 5.5), including the two tables and a figure. More analysis should be moved to the main content. For instance, I find Table 7 in Appendix C informative, so perhaps the authors could consider moving this to the main content.\n\n* There is a discrepancy between Section 4.2 and Figure 5. In Section 4.2, the authors use \"CP 1: Value\" and \"CP 2: Key\" as the paragraph title. However, it is \"CP 1: Key\" and \"CP 2: Value\" in Figure 5. \n\n* What is NL in Figure 7? I think it is natural language (as opposed to JSON format), but \"NL\" is not a common abbreviation and is not mentioned in the main content either. Moreover, this figure reports the result on LongMemEval_s, yet it is not mentioned in Figure 7 and Section 5.5. \n\n**P2. The figures/tables do not follow the order or ICLR format. Many of them require adjustment in format and location in this paper for better reference.**\n\n* Should follow the ICLR format and place the table title on top (see Tables 1, 2, and 3). \n* Tables and Figures are placed on page \"i\" but first mentioned on page \"i+1.\" For example, Table 1 is first mentioned on page 3 (Line 146) but placed on page 2. For another example, Figure 2 is first mentioned on page 5 (Line 242) but placed on page 4.\n* As shown in Figures 2, 3, and 4, it really confuses me to place (a), (b), and (c) at the end. Take Figure 3 as an example: \n```\nLongMemEval challenges chat assistants through its diverse question types (a), emphases on multi-session reasoning (b), \nand diverse evidence locations within sessions (c).\n```\nIt should be:\n```\nLongMemEval challenges chat assistants through its (a) diverse question types, (b) emphases on multi-session reasoning, \nand (c) diverse evidence locations within sessions.\n```\n* The column separator line is inconsistent (Tables 1 and 3). \n* In Table 1, can the authors explain why you do not use \"# data\" but \"# Q\"?\n* In Line 83, the term session has a different meaning compared to Table 1 (session means the total number of LongMemEval dataset in \"with 500 sessions\"). \n* The text in Figure 2 is small. Moreover, the \"Answer: 5\" is wrong in Question Construction (should be 50).\n* In Figure 3-c, the round index 7 ~ 10 is \"invisible,\" compared to the \"barely visible\" number of sessions = 6 in Figure 3-b. If they do have \"height,\" perhaps insert the numeric values above these bars.\n* In Figure 4-b, two \"Llama-3.1 8B Instruct\" are presented in both w/o CoN and w/ CoN. Which one is 70B? \n* Moreover, no explicit evaluation metric (accuracy) is found in Figure 4-b and its caption. While the authors mention \"... compared to answering the questions based on only ...,\" this makes readers need to first observe it is the question answering task, then refer to Section 3.3 to conclude that this table is indeed using accuracy (but not Recall@k and NDCG@k).\n* In Figure 5, presenting \"key 4,\" \"value chunk 2,\" etc. without a simple example is hard to grasp the idea. This should be done by showing different color blocks in \"legend\" (just like Figure 3) and providing a concrete example (possibly re-use the examples in the previous figures). Additionally, Figure 5 does not visualize a \"smooth\" pipeline; there is a break between the\"(1) indexing\" process and the \"(2) retrieval and (3) reading\" process. Specifically, there is no connection between the \"key 4 and key 2\" and the data storage system.\n\n**P3. Too much emphasis on many terms and two (or more) adjectives to describe certain words.** Specifically, the overuse of \"\\textit\" and compound words (excluding common words in NLP like long-term). While I could understand that the authors want to emphasize the *scenario* they aim to test and *why* not conduct more experiments etc., it further adds the difficulty to grasp the main point in this paper. The texts in italics are already overwhelming even without these. Below are several sentences that may distract the readers:\n\n* In Lines 287-288: This design allows us to simulate *realistic* and *extensible* user-AI chat histories with *minimal conflicts*.\n* In Lines 319-320: ... we compare this *online* memory evaluation with *offline reading*, where GPT-4o is prompted to answer with the complete history provided as context. (Note: online is already in italics in Lines 80 and 183)\n* In Line 426: Using LongMemEval_m, we compare different value choices in a *budget-aware* manner.\n* In Line 430: The only exception is with *multi-session reasoning* questions, where fact decomposition consistently improves performance. (Note: multi-session reasoning is already mentioned and abbreviated as MR before)\n* In Lines 476-477: To address this need, we introduce a simple yet effective *time-aware indexing and query expansion* scheme. (Note: \"time-aware indexing and query expansion\" is already in italics in Line 400; moreover, \"time-aware indexing and query expansion\" first appears in Line 98, but only time-aware is in italics)\n* The authors spend 6 lines mentioning that this paper is inspired by the needle-in-the-haystack test (in Lines 292-297); however, it is already stated in the Introduction Section (in Lines 77-78). In my opinion, the authors should remove this paragraph and elaborate more on either (1) the framework formulation or (2) other experiments that the authors found worth noting in the main content (for instance, Table 7).\n* As for the over-use of adjectives/compound words, please refer to Lines 72-79 as an example (others are scattered in the main content, particularly in the Introduction):\n```\nWe introduce LongMemEval, a comprehensive, challenging, and scalable benchmark designed to assess the long-term memory capabilities of chat assistants. LongMemEval consists of 500 human-curated, high-quality questions to test five core memory abilities: (...). Each question requires recalling information hidden within one or more multi-turn task-oriented user-AI dialogues that are LLM-simulated and human-edited. Inspired by the “needle-in-a-haystack” test (Kamradt, 2023), we design an attribute-controlled pipeline to compile a coherent, extensible, and timestamped chat history for each question.\n```\n\n\n## Soundness (S)\n\n**S1. The proposed unified framework is not formalized.** See Weakness 3 above.\n\n**S2. The authors do not compare their proposed framework with other similar baselines such as LLM with RAG or external systems.** While the authors do report 97 data using proprietary LLMs (ChatGPT and Coze) in the pilot study, this subset of data is 5x smaller than the original size of the data (500), not to mention the conversation sessions are extremely short, which is mentioned in Lines 317-318: approximately 10x shorter than LongMemEval_s. What's more, the distribution does not even really match the original LongMemEval_s dataset (see Lines 1030-1034 in Appendix B). Is there at least an implicit baseline comparison between your and other works in this paper in some way (see Weakness 2 above)? Are the works mentioned in Line 393 the baselines?\n\nOn the other hand, *if the nature of this dataset is that no prior baselines are suitable for comparison, then the authors should test their framework using more LLMs to benchmark the LongMemEval dataset for future researchers and demonstrate its effectiveness across various LLMs.* \nIn this scenario, to strengthen the method, it would be necessary to (a) run the experiments multiple times (see [4]) or (b) enlarge the existing LongMemEval dataset. This paper only has 500 data and experiments on two LLMs in its current state: GPT-4o and Llama 3 (70B and 8B). If the budget is an issue, testing other small LLMs (e.g., below 7B) is also a welcome contribution.\n\n**S3. Regarding the soundness in Figure 4-a, the authors only test the closed-source LLMs with a memory system once in a small (97) data.** While the authors already mention (multiple times) that they are interested in testing \"*online* interactions with chatbot\", this topic further narrows down to a very specific setting. Moreover, as these LLM-based memory systems do not have a snapshot (nor do they release a technical report demonstrating the robustness of their memory systems on various reasoning tasks), including this potentially immature result in the main content could be a problem for future reference (despite the evaluation time mentioned in the footnote), because these systems would be constantly improved over time without any notification, increasing the difficulty in terms of reproducibility.\n\n**S4. The dataset contains only 500, which is rather small. Moreover, many questions focus on *single-turn* sessions (31%; see Figure 3-a).** As these settings are constantly tested in previous long-term/short-term datasets (see Contribution C2 below), the authors should create a dataset with more \"multi-session, knowledge update, and temporal reasoning\" for current LLMs.\n\n**S5. In Figure 1, the definition of those question types is unclear.** Specifically, in Figure 1, why not treat the \"knowledge-update\" as a \"multi-session\" example, and vice versa? Moreover, after adding the time and date in each session, they can also be generalized to the \"temporal-reasoning.\" The authors do not define these types of questions clearly and only use \"The other types of questions are multi-session (MR), knowledge-update (KU), and temporal-reasoning (TR)\" in Lines 236-237, and yet the distribution of question types is distinctly shown in the pie chart in Figure 3-a.\n\n## Contribution (C)\n\n**C1. Could the authors explain why it is necessary to differentiate \"human-AI\" and \"human-human\" conversations?** Am I missing something in this paper? If so, could you kindly explain why you are interested in the user-AI setting and further differentiate this in Table 1? For instance, is there a significant difference between them when training an LLM or in some tasks that prefer user-AI datasets over human-human ones? \n\n**C2. Prior works of task-oriented dialogues (TOD) are missing, such as the MultiWoZ dataset [5].** On the other hand, the AirDialogue dataset [6] has a more narrower scope. **As a result, there may be an issue in Lines 48-50**: *Many datasets focus solely on human-human conversations (Xu et al., 2022a; Maharana et al., 2024; Kim et al., 2024), while others omit task-oriented dialogues, which represent a significant portion of chat assistant usage.* **As for the Related Work, there is another issue in Lines 142-144** (*Despite these advancements, existing QA-based benchmarks overlook several memory capabilities critical to long-term user-assistant interactions: synthesizing information across numerous sessions, recalling assistant-side information, ...*) **because the CoQA dataset [7] encompasses these two issues.** The model needs to refer back to the conversation history. Lastly, **the SituatedQA dataset [8] $-$ related to temporal context and retrieval $-$ is missing in the Related Work.**\n\n[4] Self-consistency improves chain of thought reasoning in language models, Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou, ICLR 2023\n\n[5] MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselines, Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, Jindong Chen, NLP4ConvAI 2020\n\n[6] AirDialogue: An Environment for Goal-Oriented Dialogue Research, Wei Wei, Quoc Le, Andrew Dai, Jia Li, EMNLP 2018\n\n[7] CoQA: A Conversational Question Answering Challenge, Siva Reddy, Danqi Chen, Christopher D. Manning, TACL 2019\n\n[8] SituatedQA: Incorporating Extra-Linguistic Contexts into QA, Michael J.Q. Zhang, Eunsol Choi, EMNLP 2021", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)