MemSim: A Bayesian Simulator for Evaluating Memory of LLM-based Personal Assistants
Authors: Zeyu Zhang
Quanyu Dai
Luyu Chen
Zeren Jiang
Rui Li
Jieming Zhu
Xu Chen
Yi Xie
Zhenhua Dong
Ji-Rong Wen
Venue: NeurIPS 2025 poster ()
Keywords: LLM-based Agents
Memory
Personal Assistants
Primary area: evaluation

Abstract: LLM-based agents have been widely applied as personal assistants, capable of memorizing information from user messages and responding to personal queries. However, there still lacks an objective and automatic evaluation on their memory capability, largely due to the challenges in constructing reliable questions and answers (QAs) according to user messages. In this paper, we propose MemSim, a Bayesian simulator designed to automatically construct reliable QAs from generated user messages, simultaneously keeping their diversity and scalability. Specifically, we introduce the Bayesian Relation Network (BRNet) and a causal generation mechanism to mitigate the impact of LLM hallucinations on factual information, facilitating the automatic creation of an evaluation dataset. Based on MemSim, we generate a dataset in the daily-life scenario, named MemDaily, and conduct extensive experiments to assess the effectiveness of our approach. We also provide a benchmark for evaluating different memory mechanisms in LLM-based agents with the MemDaily dataset.

MemSim: A Bayesian Simulator for Evaluating Memory of LLM-based Personal Assistants
Zeyu Zhang Quanyu D Luyu Chen Zeren Jiang', Rui Li Jiiming Zhu?,
Xu - Yi Xie Zhenhua Dong² Ji-Rong Wen
Gaoling School of Artificial Intelligence, Renmin University of China
Huawei Noah's Ark Lab HHuawei Technologies Ltd.
(zeyuzhang, xu. chen) @ruc edu. cn, daiquanyu@huawei. com
Abstract
LLM-based agents have been widely applied as personal assistants, capable of memorizing information from user messages and responding to personal queries. However, there still lacks objective and automatic evaluation on their mem- ory capability, largely due the challenges constructing reliable questions and answers (QAs) according to user messages. In this paper, we propose Mem- Sim, Bayesian simulator designed automatically construct reliable QAs from generated user messages, simultaneously keeping their diversity and scalability. Specifically, we introduce the Bayesian Relation Network (BRNet) and a causal generation mechanism to mitigate the impact of L.LM hallucinations on factual information, facilitating the automatic creation of an evaluation dataset. Based on MemSim, generate dataset in the daily-life scenario, named MemDaily, and conduct extensive experiments to assess the effectiveness of our approach. We also provide benchmark for evaluating different memory mechanisms in LLM-based agents with the MemDaily dataset. To benefit the research community, we have released project at https: /et///gttttute
1 Introduction
In recent years, large language model (LLM) based agents have been extensively deployed across various fields [1 One of their most significant applications is serving as personal assistants 8) where they engage in long-term interactions with users to address wide range of issues [ [0] For LLM-based personal assistants, memory is one of the most significant capability [T]. To perform personal tasks effectively, these agents must be capable of storing factual information from previous messages and recalling relevant details to generate appropriate responses. For example, a user Alice might tell the agent, "I will watch movie at City Cinema this Friday in Hall 3. Row 2, Seat 9. When Friday arrives, she might ask the agent, "Where is my movie seat?" Then, the agent should recall the relevant information (i.c.. the seat number) to generate appropriate response to Alice.
Previous research has proposed methods for constructing the memory L.M --seed agents 12 13 9 14 15). However, there remains a lack of objective and automatic methods to evaluate how well personal assistants can memorize and utilize factual information from previous messages. One conventional solution is collect messages from real-world users, and manually annotate answers to human-designed questions based on these messages. However, it requires substantial human labor
Work under the internship in Huawei Noah's Ark Lab.
Co-first authors.
Corresponding author.
Beijing Key Laboratory of Research on Large Models and Intelligent Governance. Engineering Research Center next-Generation Intelligent Search and Recommendation. MOE.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) Bayesian Relation Network
Endly
Atribote
(b) Causal Generation Mechanism
User Messages
M
Imm Noise Intusion
Choices
Proparty
Answer
Target
Tratestory
Single-hop (Muyy (ond
Comparative (Ai K.PLn) (A4po) Kinka)
Peat-procassing
(A Min) (10)
Hirraishical User Profile
Mutt-hpp
Aggregative
BRNe
mi V.R
Ancestral Sampling
X2 .g.... Plat
[[ K, Powll
(c) MemDaily Dataset
(M. aM..e...t.hh
Trajectury 2,954
(d) MemDaily Benchmark
Memory Mechanism
Apri' fo 2024
April 6th 2024
Apri Bih 2024
Message
Process
Question
LLM Inforance
2
Who leunger
Retrioval Target
Retheved
Alico and Bo0?
Messago
Adaptation Process
Choliees:
26 buthaay is next Monday. Remind me
(ob is 29 years
Alce
David
Reaporas
(ali looka
AnsworiRebteval Targer.
Accuracy
that
vvery young.
Ground Trush
Resporia
Agent Interface
Figure 1: The overview of MemSim framework and MemDaily dataset.
that lacks scalability. Another solution is generate user messages and question-anse answers (QAs) with LLMs. However, the hallucination of LLMs can severely undermine the reliability of generated datasets, particularly complex scenarios [6] Here, we refer to the reliability of dataset as the correctness of its ground truths to factual questions given the corresponding user messages. Our research shows that due to the hallucination of LLMs, the correctness of ground truths generated by vanilla LLMs is less than 90% in most scenarios and can fall below 40% in some complex scenarios (see Section (52).. Moreover, generating diverse user profiles through L.MS is also challenging, as they tend to produce the most plausible profiles that lack diversity.
To address these challenges, we propose MemSim, a Bayesian simulator designed to construct reliable QAs from generated user messages, keeping their diversity and scalability. It can be utilized to evaluate the memory capability of LLM-based personal assistants. Specifically, we introduce the Bayesian Relation Network (BRNet) to generate the simulated users that are represented by their hierarchical profiles. Then, we propose causal generation mechanism to produce various types of user messages and QAs for the comprehensive evaluation on memory mechanisms, Based on MemSim, we create dataset in the daily-life scenario, named MemDaily, and perform extensive experiments to assess the quality of MemDaily. Finally, we construct benchmark to evaluate different memory mechanisms of L.LM-based agents with MemDaily. Our work the first one that evaluates memory of LL---baed personal assistants in an objective and automatic way. Our contributions are summarized as follows:
We analyze the challenges of constructing datasets for objective evaluation on the memory capability of LLM-based personal assistants, focusing on the aspects of reliability, diversity, and scalability.
We propose MemSim, Bayesian simulator designed to generate reliable, diverse and scalable datasets for evaluating the memory of LLM- personal assistants. We design BRNet to generate the simulated users, and propose causal generation mechanism construct messages and QAs.
We create the MemDaily dataset, which can be used to evaluate the memory capability of LLM- based personal assistants. We perform extensive experiments to assess the quality of MemDaily, and provide a benchmark for different memory mechanisms of LLMe-ased agents. To support the research community, we release project available h/ gigghbthhub.... com wwtp:nuusteeiiiee 128 MemSim
The rest of our paper is organized as follows. In Section we review the related works on the evaluation of memory in LLM-based agents and personal assistants. In Section we introduce the details of MemSim, and the generation process of MemDaily. In Section 4 we assess the quality of MemDaily. Section 5 provides benchmark for evaluating different memory mechanisms of LLM-based agents. Finally, y, in Section we discuss the limitations of our work and draw conclusions.
2 Related Works
I.LM-based agents have been extensively utilized across various domains, marking a new era for artificial personal assistants 7 17 18). For LLM-based personal assistants, memory is critical to

---

enable agents to deliver personalized services. This includes storing, managing, and utilizing users' personal and historical data [2 19
For instance, MPC [TO] suggests storing essential factual information in a memory pool with a summarizer for retrieval as needed. MemoryBank [ converts daily events into high-level summaries and organizes them into a hierarchical memory structure for future retrieval. These approaches primarily aim to enhance agents' memory capability,
Previous studies have also attempted to evaluate the memory capability of LLM-based agents. Some studies use subjective methods, employing human evaluators to score the effectiveness of
24 Other studies use objective evaluations by constructing dialogues and question-answer pairs [4 22, 26]. However, there exist limitations in higher cost and lower reliability. Some previous studies construct knowledge-bird basedd question-answerree -nswering datasets assess Retrieval-Augmentt Generation 28). These studies either use knowledge graphs to generate QAs through templates or manually annotate QAs with human input [29 However, most of them focus on common-sense questions rather than personal questions, and they do not include textual user messages and target indexes for retrieval evaluation 32
retrieved memory [TO
35 36.. They are also highly dependent
Our work is the first that evaluate memory of LLM-based personal assistants in an objective and automatic way, which can generate user messages and QAs without human annotators, keeping reliability, diversity and scalability.
on the entities extracted from the given corpus [30 35
3 Methods
3.1
Overview of MemSim
In order to construct reliable QAs from generated user messages, we propose a Bayesian simulator named MemSim shown in Figure 1 First, we develop the Bayesian Relation Network to model the probability distribution of users' relevant entities and attributes, enabling the sampling of diverse hierarchical user profiles. Then, we introduce a causal mechanism to generate user messages and construct reliable QAs We design various types of QAs for comprehensive memory evaluation, incorporating different noises to simulate real-world environments. Based on the constructed QAs and generated user messages, researchers can objectively and automatically evaluate the memory capability of LLM-based personal assistants on factual information from previous messages.
3.2 Bayesian Relation Network
We introduce Bayesian Relation Network (BRNet) to model the probability distribution of users' relevant entities and attributes, where we sample hierarchical profiles to represent simulated users (see Figure [a)). Specifically, we define a two-level structure in BRNet, including the entity level and the attribute level. The entity level represents user-related entities, such as relevant persons, involved events, and the user itself. At the attribute level, each entity comprises several attributes, such as age, gender, and occupation. Here, BRNet actually serves as predefined meta-user. Formally, let A {4i, .... Axr)) ), where N is the number of entities, and N is the number of attributes belonging to the entity AA'. Each attribute A corresponds to a random variable Xj. which can be sampled in a value space. For example, the college's 's (entity A') age (attribute A3) is 28 years old (value 2² ~ X().
A represent different entities, and each entity A'' comprises several attributes
We denote BRNet as a directed graph G = (V, E) at the attribute level, where the vertex set V includes all attributes, i.e., V - [A] .2...., ... A'r The edge set E captures all the direct causal relations among these attributes, defined E = ((A5. XX E XX where X' = UN (Xi x For better demonstration, in this subsection, we simplify the subseripts of the variables in X as 1.2..... EN N. The conditional probability distribution among them can either be explicitly predefined or implicitly represented by LLM's generation with conditional prompts. ensuring that BRNet forms a directed acyclic graph (DAG), which is typical in most scenarios [37]. Additionally, the vertices (i.e., attributes), edges (i.e., causal relations), and conditional probability distributions (i.c., prior knowledge) can be easily scaled to different scenarios.
X²).
X
important to note that we assume the causal structure is loop-free,
So far, we have constructed the BRNet, where the joint probability distribution P(X1, (xx) over all attributes can represent the user distribution in the given scenario. Then, we can sample differ- ent values of attributes on entities from BRNet to represent various user profiles. One straightforward approach is to compute the joint probability distribution and sample from it.
3

---

Assumption 1 (Local Markov Property). BRNer satisfies the local Markov property, which states that
Xx X (e) (par (Xt), VX, X,
where des (Xt) denotes the non-descendant set of X (Xr( denotes the parent set of X., and the notation indicates the variables are conditionally independent.
Because the parents of attribute can be extended to any non-descendant attributes of it by adding new edge if they have a direct causal relation. Given these parent attributes, other non-descendent attributes are conditionally independent of that attribute according Pearl [38)
Theorem (Factorization). The joint probability distribution of BRNet can be expressed as
II X,EX
P(X.)par(Xx)),
where pXr(Xe) denotes the set of parent attributes of X,.
The proof of this theorem is provided in Appendix A.1 However, calculating the joint probability distribution and sampling from it may be impractical our scenarios. First, the joint probability distribution is often nigh-dimensional making its calculation and sampling costly. Second, some conditional probability distributions are difficult represent in explicit forms, particularly when using LLMs for value generation through conditional prompts. To address these issues, we introduce the ancestral sampling process to obtain the values of attributes.
Assumption 2 (Conditional Sampling). In BRNet, attribute can be sampled from the conditional probability distribution given its parent attributes. Specifically, we have
It P (Xil (Xi)) VXX, E X,
where the conditional probability distribution can be expressed in either explicit or implicit forms.
The ancestral sampling algorithm is outlined as follows. First, obtain the topological ordering of BRNet using Kahn's algorithm [39]. Next, we sample all attributes according to this ordering. For top-level attributes without parents, the sampling is performed based their marginal probability distributions. For other variables like X,, we sample their values using the conditional probability distribution ã. ~ P (X, par (Xt)) as specified in Assumption] 2 Finally,
consider each sampling result {म1, T2, ...,..,. T x1) as the attribute-level profiles of a user, which constitute different entities as the entity-level profiles of the user. These two levels represent the user in different grains, which important to generate user messages and QAs subsequently.
Theorem (Ancestral Sampling). For BRNet, the result of ancestral sampling is equivalent to that of sampling from the joint probability distribution. Specificully, we huve
P( 82, ..., ((x1) = P(x1, x2,,..
where I1, T2, F|XI P(X1, X2, ... X1x)) are sampled from the joint probability distribution.
The proof can be found in Appendix A.2 By employing ancestral sampling (38], we eliminate the need compute the joint probability distribution, making the sampling process more efficient and practical. By utilizing BRNet, we introduce prior knowledge of the specific scenario into the graphical structure and sampling process, which can improve the diversity and scalability of user profiles, thereby enhancing the diversity and scalability of whole datasets.
3.3 Causal Generation Mechanism
Based hierarchical user profiles, we propose causal generation mechanism to generate user mes- sages, and construct reliable QAs corresponding to them. Here, causal indicates that the generation of user messages and the construction of QAs are causally dependent on the same informative hints that are also causally derived from hierarchical user profiles. Specifically, we define piece of hint triple (A', A x that provides factual information in structural format. In other words, the hierarchical user profiles provide a structural foundation to get different hints, which then provide a set of relevant information the causation of both user messages and QAs, shown in Figure (Tb).
Construction of Informative Hints. We construct the hints of factual information based on hierar- chical user profiles before creating the user messages and QAs. We select target entity A' at the

---

Table 1: Overview of comprehensive questions and answers.
Types
Descriptions
Examples
Causal Hints
Retrieval Target
Single-hop
Rely on one message to an- swer the question directly.
Q: When is Alice's 's birthday?
(Agg) Ko)=
{mgy)
A: June 1st.
Multi-hop
Require multiple messages
Q: Where is the meeting that I
(At', K6)) (A', (()))))))))))
(mo)) maa)
to answer the question
will attend next week?
jointly.
A: Victoria Conference Center.
Comparative
Compare two entities on a shared attribute with multi-
Q: Who is younger between Alice and Bob?
(Ass K, (6y)) (A(s) K, (,))))
(mo).
ple messages.
A: Bob.
Aggregative Aggregate messages about
Q: How many people are under
{(AGs)) K, P(o)) (maa)d
than entities on common attribute.
35 years old?
A: Three.
Post- processing
Involve extra reasoning steps to answer with multi-
Q: What season was the teacher that I know born in? A: Spring.
(t.KK)) "(a)s ma))
(A', K4),
ple messages.
Table 2: Summary of the MemDaily dataset.
Statistics
Simp.
Cond.
Comp.
Aggr.
Post.
Noisy
Total
Trajectories
500
500
492
462
500
500
2,954
Messages
4215
4195
3144
5536
4438
4475
26,003
Questions
500
500
492
462
500
500
2,954
TPM
15.48
15.49
14.66
14.65
17.07
16.14
15.59
entity-level, and choose (' attributes {Ki, .Y.................... Kf+} } S A' along with their corresponding values ofe} } from the attribute-level profiles. Then, we reformulate them into list of triple hints {ul, .i..... ...
H' (AA', K7, of For some complex types of QAs, we choose more than one target entities, and concatenate their lists of hints. For better demonstration, we re-index the final list of hints as H [(Ag), k)), " where I is the number of hints in the final list.
Construction of User Messages. Based on the j-th hint (Ag); Ko): m(2)) E H, we construct the corresponding user message mij) with LLM, where we have m(g) LLM (Ag) (())) Here, the LLM only serves the purpose of rewriting structural hints, without any reasoning process. For example, if the hint is (my uncle Bob, occupation, driver), the generated user message might be "The occupation of my uncle Bob is driver". We generate user messages for all the hints in H. and we
finally get the list of user messages M [mm))] j-1
Construction of Questions and Answers. In order to evaluate the memory capability of LLMd personal assistants more comprehensively, we propose to construct five representative types of QAs to cover various complexities in woaldworld scenarios, as detailed in Table [ For each question g, we provide three forms of ground truths: (1) the textual answer a that can correctly respond to g, (2) the correct choice a among confusing choices a' (generated by LLM) as a single-choice format, and (3) the correct retrieval target h M that contains the required factual information to the question.
(i.) Single-hop QA. Single-hop QA is the most basic type of QAs, relying on single piece message to directly answer the question. In constructing QA, we randomly select the j-th hint (Ag) KG), "G)) and generate the question q = LLM K()) through LLM rewriting, where the answer is a = -G)- Correspondingly, the retrieval target is h = {m:)]-
(ii.) Multi-hop QA. Multi-hop QA necessitates the use of multiple messages to determine the correct answer, making it more complex than single-hop QA. In constructing Multi-hop QA, we first sample two hints (Ag): KG), U(y)) and (A(s). K (()), v(() from the same bridge entity A (i.e., A'
Ag)
A(a)). We then mask this bridge entity and generate the question q T.I. K (j): P0), (K)) through LLM rewriting, where the answer is U(k)- The target message set is h = {mo) m()) By incorporating additional entities, the questions can be easily extended more hops.
(ii.)) Comparative QA. Comparative QA is an extensive type of multi multi-hop QA, which involves comparing two entities based on shared attribute. We first select two hints (AG)) KGy, f()) and (A(k), K()), P(K)) from different entities with the same meaning attribute K (i.c., A, * Ar and K e
5

---

(G) = K(x)). We then rewrite the question q - (LM((A)) K) by LLM, where the answer a = =f(KK ((A)) is derived from the function f(.). The retrieval target is h = {m maa)
(iv.) Aggregative QA. Aggregative QA is general type of comparative QA, which requires aggregat- ing messages from more than two entities on a shared attribute. For construction, we choose hints ((A()) K, (()))) from different entities with the same meaning attribute K. Then, construct the question q = (LA((()))))
K), where we obtain the answer a f(K, (o (0 The target message set should include all these related references, that is, = {m( )
(v.) Post-pracessing QA. Post-processing QA addresses situations where personal questions require additional reasoning steps for agents to answer, based on the retrieved messages. We first select two hints (A(): K KG); v()) and (A(k): ((z), ti(e)) from the same bridge entity AA.. We then design reasoning factor &o to generate the question = LLM (K)) V(j): K(k), 1²), and derive the answer a -= ((K U(K)) 4)), where v specifies the reasoning process. For example, it could be "the sum of the last five digits of the phone number V(())* Similarly, the retrieval target will be h = {m(z): m(xe): }.
Infusion of Noise in User Messages. We integrate two types of noise in user messages by concatena- tion, in order to simulate real-world circumstances. The first type is entity-side noise, which refers to noisy messages that contain the selected attributes from unselected entities. The second type is attribute-side noise, which involves noisy messages that describe unselected attributes of the selected entities. Both types of noise can impact agents' ability to retrieve messages and generate answers.
Eventually, we formulate the trajectory & (M, q, a, a'; h)) by discarding all hints, where each trajectory serves as a test instance for evaluating the memory capability of LLM-based personal assistants. There are two insights into the causal generation mechanism. First, the factual information of messages and QAs are causally constructed from the shared hints that are sampled from user profiles, where L.MS are only responsible for rewriting based on the given information, rather than imagining or reasoning. This pipeline mitigates the impact of LLM hallucination on the factual information, keeping the reliability of QAs. It can also prevent contradictions among user messages from the same trajectory, because their hints are derived from the same user profile. Second, our method focuses on designing the asymmetric difficulty between constructing QAs (i.e., tattsss-hintttssmsssseee profiles>hing soinnts--mesee messages, question and answer) and solving QAs (i.e., which is critical for the automatic generation of evaluation datasets.
3.4 MemDaily: A Dataset in the Daiey-life Scenario
Based on MemSim, we create a dataset in the daily-life scenario, named MemDaily, which can be used to evaluate the memory capability of LLM-based personal assistants, shown in Figure Tc). Specifically, MemDaily incorporates 11 entities and 73 attributes (see details in Appendix G.I). all of which are representative and closely related users' daily lives. We create sub-datasets of different QA types mentioned previously: (1) Simple (Simp.): single-hop QAs. (2) Conditional (Cond.):
multi-hop QAs with conditions. (3) Comparative (Comp.): comparative QAs. (4) Aggregative (Aggr.): aggregative QAs. (5) Post-processing
(Post.): post-processing QAs. (6) Noisy: multi-hop
QAs with additional irrelevant noisy texts inside questions. The summary of MemDaily is shown in Table 2, where we present the number of trajectories, user messages, questions, and TPM (tokens per message). More details and examples can be found in Appendix G
Experiments of Evaluation
In this section, we evaluate the quality of MemDaily, which can reflect the effectiveness of MemSim. Specifically, the evaluations are conducted in three parts: the user profiles, the user messages, and the constructed QAs. Besides, we also conduct comprehensive case studies in Appendix G
4.1 Evaluation on User Profiles
The generated user profiles are supposed to express both rationality and diversity, which also directly influence the creation of user messages and QAs. Therefore, evaluate these two aspects to reflect their quality. Rationality means that the user profiles should possibly exist in the real world, with no internal contradictions in their descriptions. Diversity indicates that the descriptions among users distinct, covering a wide range of user types.

---

Table 3: Results of the evaluation on user profiles.
Methods
R-Human
R-GPT
SWI-R
SWI-O
SWI-A
IndePL
1.35+0.53
4.32
0.464
0.231
0.347
SeqPL
1.6440.73
4.40
1,471
1.416
1,443
JointPI. 3.02+1.14
4.80
1.425
0.462
0,943
MemSim
4.91+0.30
4.68
3.206
2.895
3.050
Metries. For rationality, recruit six human evaluators score the generated user profiles scale Additionally, we use GPT-4o as a reference for scoring. These two metries are denoted as R-Human and R-GPT. For diversity, calculate the average Shannon-Wienet Index (SWI) [40] on key attributes, using the following formula:
from to 5 based on the guidelines in Appendix H.1
SWI-W
Σ Σ p(T:) Inp(x;), W XvEW
1
where W C X' is the subset of attribute variables. Therefore, we calculate SWI-R, SWI-O, and SWI- A, corresponding to role-relevant attributes, role-irrelevant attributes, and all attributes, respectively.
Baselines. We design several baselines to generate user profiles: (1) JointPL: prompting an LLM to generate attributes jointly. (2) SeqPL: prompting an LLM generate attributes sequentially, conditioned on previous attributes in linear order. (3) IndePL:
prompting an LLM to generate attributes independently. We compare our method with these baselines on generating user profiles.
Results. As shown in Table MemSim outperforms other baselines on R-Human, demonstrating the effectiveness of BRNet an ablation study. However, we also observe inconsistency between R-Human and R-GPT, which may be due to the inaccuracy of the LLM's scoring [1 Furthermore, method achieves the highest diversity compared to the other baselines.
4.2 Evaluation on User Messages
We evaluate the quality of generated user messages multiple aspects, including fluency, rationality, naturalness, informativeness, and diversity. The first four aspects are designed to assess the quality inside a trajectory, while the final one targets the variety across trajectories.
Metries. For the irside-traeectory aspects, human evaluators score user messages on a scale from to based on the guidelines Appendix H.2 denoted as F-Human (fluency), R-Human (rationality). N-Human (naturalness), and I-Human (informativeness)) GPT-4o scores are also available and detailed Appendix [D] To assess the diversity across trajectories, we extract all entities and calculate their average Shannon-Wient Index per 10,000 tokens of user messages, referred to as SWIP
Baselines. We implement several baselines that generate messages under different constraints regarding user profiles and tasks: (1) ZeroCons: no constraints on attributes when prompting LLMs. (2) PartCons: partial attributes of user profiles constrained in prompts for LLMs. (3) SoftCons: full attributes of user profiles are constrained in prompts but they are not forcibly for generation. Our MemSim method imposes the most strict constraints, requiring both the integration of specific attributes into user messages and ensuring that questions are answerable with established ground truths based the shared hints. Generally, higher constraint commonly means sacrifice of fluency and naturalness, because it compulsively imposes certain information to benefit QA constructions.
Results. As shown in Table 4 our method maintains relatively high scores despite the rigorous constraints on constructing reliable QAs. Additionally, MemSim exhibits the highest diversity index, attributed to the BRNet and the causal generation mechanism that produces a wider variety of user messages based on the provided hierarchical user profiles.
4.3 Evaluation on Questions and Answers
The primary challenge for constructing reliable dataset is ensuring the accuracy of ground truths for the constructed questions. To assess the reliability MemDaily, we sample approximately 20% of all the trajectories in MemDaily and employ human evaluators to verify the correctness of their ground truths. Specifically, the evaluators are required to examine three parts of the ground truths: textual

---

Table 4: Results of the evaluation on user messages.
Methods
F-Human
R-Human
N-Human
I-Human
SWIP
ZeroCons
4.9440.24
4.94:0.24
4.85+0.35
2.82+1.15
2.712
PariCons
4.98+0.14
4.94+0.37
4.97=0.18
4.01+1.18
6.047
SoftCons
4.93+0.30
4.80+0.77
4.91+0.42
4.37+0098
5.868
MemSim
4.93+0.30
4.93+0.39
4.90+0.41
3.61+1.19
6.125
answers, single-choice answers, and retrieval targets. and report their accuracy. Due to the page limitation, we put the evaluation details in Appendix ( According to the results, we find MemDaily can ensure the accuracy of the answers provided for constructed questions.
Experiments of Benchmark
In this section, create benchmark based the MemDaily dataset, in order to evaluate the memory capability of LLM-based personal assistants. Our benchmark sets various levels of difficulty by introducing different proportions of question-irrelevant daily-life posts.
5.1 Experimental Settings
Levels of Difficulty. We utilize the MemDaily dataset as the basis of our benchmark. In order to set different levels of difficulty, we collect and randomly incorporate them into user messages by controlling their proportions. Specifically, we denote MemDaily-vaninn
irrelevant posts from social media platforms,
as the vanilla and easiest one without extra additions, and create series of MemDaily-n, where we use 77 to represent the inverse percentage of original user messages. Larger indicates higher level of difficulty in the benchmark. We primarily focus on MemDaily-vanill) and MemDaily- 100 as representatives. We also conduct evaluations on MemDaily-10, MemDaily-50 and MemDaily-2000 putting their experimental results in Appendix E
Baselines. We implement several common memory mechanisms for LLMM-based agents according to previous studies [1I], including (1) Full Memory (FullMem): saves all previous messages and (2) Recent Memory (ReceMem): maintains the most recent k messages and concatenates them into the prompt for LLM inference, also known as short-term memory. (3) Retrieved Memory (RetrMem): stores all previous messages using FAISS [42] and retrieves the top-k relevant messages for inclusion in the prompt for LLM inference, which is commonly used to construct long-term memory. Specifically, we use Llama- 160m [4] to transform message into f68-dimensional embedding and compute relevance scores using cosine similarity [44]. (4) None Memory (NonMem): does not use memory for LLM inference. Additionally, include two special basclines for reference: (5) Noisy Memory (NoisyMem)) receives only untargeted messages. (6) Oracle Memory (OracleMem): receives only targeted messages. Here, the targeted messages indicate the messages in the ground truth retrieval target. For all methods, we use the open-source GLM-4-9B (45] as the foundational model for its excellent ability in long-context scenarios, and experiments more backbones be found in Appendix F
concatenates them into the prompt for L.LM inference.
Metries. We propose to evaluate the memory of LLM-based agents from two perspectives: effective- ness and efficiency. Effectiveness refers to the agent's ability to store and utilize factual information. The metrics for effectiveness include: (1) Accuracy: The correctness of agents' responses, measured by their ability to answer personal questions based the factual information from historical user messages. (2) Recall@5: The percentage of messages in retrieval target successfully retrieved within the t relevant messages. Efficiency mainly assesses the time cost associated with storing and utilizing information from memory. We use two metrics to evaluate efficiency:
(1) Response Time: The time taken for agent to respond after receiving query, covering the retrieval and utilization processes. (2) Adaptation Time: The time required for an agent to store a new message.
5.2 Effectiveness of Memory Mechanisms
Accuracy of factual question-answering, The results of accuracy are presented in Table including the response time for generating answers and adaptation time for storing messages. FullMem and RetrMem demonstrate superior performance compared to other memory mechanisms, achieving high accuracy across both datasets. ReceMem tends to underperform when a large volume of noisy messages is present, target messages may fall outside the memory window. We observe

---

Table 5: Results of accuracy for factual question-answerigg answering.
MemDaily-vanilla
Methods
Simp.
Cond.
Comp.
Aggr.
Post.
Noisy
FullMem
0.976=0.022
0.982=0.017
0.859=0.054
0.320=0.079
0.848=0.045
0.966+0.028
RetrMem
0.898=0.048
0.88240.044
0.771+0.078
0.317+0.061
0.800+0.054
0.786=0.040
ReceMem
0.832=0.080
0.798+0.046
0.631 069
0,257+0.040
0.760=0.051
0.764=0.042
NonMem
0,508=0.032
0.452+0.005
0.157=0.049
0.254+0.055
0.594+0.073
0,380=0.06
NoisyMem
0.512#0.044
0.468+0.055
0.204+0.067
0.239+0.058
0.388=0.045 0.590=0.045
OracleMem
0.966=0.020
0.988+0.013
0.910
0.376+0.057
0. 0.888+0.053
0.984=0.017
MemDaily-100
Methods
Simp.
Cond.
Comp.
Aggr.
Post.
Noisy
FullMem
0.962=0.017
0.938=0.033
0.586+0.00
0.343+0.047
0.804+0.041
0.87220.041
RetrMem
0.892=0.034
0.840=0.036
0.706=0.074
0.320#0.092
0.770=0.055
0.726+0.052
ReceMem
0.500=0.063
0.442=40058
0.104+0.048
0.257:0.054
0.600=0.060
0.386=0.076
NonMem
0.508=0.032
0.454+0.065
0.159#0.052
0.252+0.043
0.594+0.032
0.380=0.057
NoisyMem
0.458=0.071
0.422:0.051
0,261#0.068
0,283+0.041
0.566#0.064
0,348 044
OracleMem
0.966=0.020
0.988+0.016
0.912+0.005
0.372:0.062
0.888+0.038
0.984=0.012
Table 6: Results of recall@5 5 for target message retrieval.
MemDaily-vanilla
Methods
Simp.
Cond.
Comp.
Aggr.
Post.
Noisy
LLM
0,888+0.025
0.851=0.020
0.947=0.018
0.544=0.021
0.80020.028
0,84620.036
Embedding
0.735+0.064
0.717=0.041
0.845#0.022
0.515#0.059
0.693+0.033
0.648+0.018
Recency
0.514#0.052
0.513=0.038
0.698+0.034
0.237=0.026
0.511#0.053
0.50420.047
MemDaily-100
Methods
Simp.
Cond.
Comp.
Aggr.
Post.
Noisy
LLM
0.612+0.021
0,47920.037
0.68340.036
0.290#0.027
0.43920.047
0.430
Embedding
0.698+0.049
0.65320.061
0.778+0.048
0.49020.037
0.56720.042
0.54320.034
Recency
0.002+0.003
0.003=0.004
0.00240.003
0.00010.001
0.00240.003
0.000
that agents excel with simple, conditional, post-processing, and noisy questions but struggle with comparative and aggregative questions. By comparing with OracleMem, we find the primary difficulty possibly lies in retrieving target messages. Even with accurate retrieval, aggregative questions remain challenging, indicating potential bottleneck in textual memory. An interesting phenomenon notice is that NoisyMem shows higher accuracy than NonMem in MemDaiil--vnnill but lower accuracy in MomDaily-100. Similarly, FullMem unexpectedly outperforms OracleMem simple questions in MemDaily. We suspect that LLMs may perform better with memory prompts of medium length, suggesting potential limitation of textual memory mechanisms for LLM-based agents.
Recall of target message retrieval. We implement three retrieval methods to obtain the most relevant messages and compare them with target messages to calculate Recall@ Embedding refers to the retrieval process used in RetrMem. Recency considers the most recent & messages as the result. LLM directly uses the LLM to respond with the top-k relevant messages. The results presented in Table 6 We find that LLM performs best in short-context scenarios, while Embedding achieves higher recall scores in longer contexts. Additionally, notice that separating the retrieval and inference stages may exhibit different performances compared with integrating them.
5.3 Efficiency of Memory Mechanisms
We put the results in Appendix C due to the page limitation. We find that RetrMem consumes the most response time in short-context scenarios, and FullMem also requires more time for inference due to longer memory prompts. However, the response time of FullMem increases significantly faster than that of other methods as the context lengthens. Regarding adaptation time, we observe that RetrMem requires substantially more time because it needs to build indexes in the FAISS system.

---

Conclusions and Limitations
In this paper, we propose MemSim, a Bayesian simulator designed to generate reliable datasets for evaluating the memory capability of LLM-based agents. Utilizing MemSim, we generate MemDaily as dataset in the daily-life scenario, and conduct extensive evaluations assess its quality. We also provide a benchmark analysis. However, as the very initial study, there are several limitations. Firstly, our work focuses on evaluating the memory capability of LLM-based agents on factual information, but does not address higeer-level and abstract information, such as users' hidden preferences. Additionally, our evaluation does not include dialogue forms, which are more complex and challenging ensure reliability. Besides, preventing the misuse of generated data also an important consideration. In future works, we aim to address these issues contribute the social benefit.
different memory mechanisms of LLM-based agents and provide further
Acknowledgment
This work is supported in part by National Natural Science Foundation of China (No. 62422215 and No. 62472427). Huawei Innovation Research Programs, Major Innovation & Planning Inter- disciplinary Platform for the "DoubleFirst Class" Initiative, Renmin University of China, Public Computing Cloud, Renmin University of China, fund for building world-class universities (disci- plines) of Renmin University of China.
References
[1] Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey progress and challenges. arXiv preprint 1tt.::240200688880 2024.
[2] Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang. Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6): 18634, 2024.
[3] Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. arXiv preprint 1xXi::2309008644 2023.
[4] Yingqiang Ge, Yujie Ren, Wenyue Hua, Shuyuan Xu, Juntao Tan, and Yongfeng Zhang. Llm (Ilmao), agents as apps: Envisioning aios, agents and the aios-agent ecosystem. arXiv preprint arXiv:2312.038..1 2023.
[5] Lei Wang, Jingsen Zhang, Xu Chen, Yankai Lin, Ruihua Song, Wayne Xin Zhao, and Ji-Rong Wen. Recagent: novel simulation paradigm for recommender systems. arXiv preprint arXiv:2306.02552. 2023.
[6] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen Ilm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08 155, 2023.
[7] Yuanchun Li, Hao Wen, Weijun Wang. Xiangyu Li, Yizhen Yuan, Guohong Liu, Jiacheng Liu, Wenxing Xu, Xiang Wang, Yi Sun, et al. Personal I1m agents: Insights and survey about the capability, efficiency and security. arXiv preprint arXiv:2401 005459, 2024.
[8] Jiahong Liu, Zexuan Qiu, Zhongyang Li. Quanyu Dai, Jieming Zhu, Minda Hu, Menglin Yang. and Irwin King. A survey of personalized large language models; Progress and future directions. arXiv preprint arXiv:2502.1 1528, 2025.
[9] Junru Lu, Siyu An, Mingbao Lin, Gabriele Pergola, Yulan He, Di Yin, Xing Sun, and Yunsheng Wu. Memochat: Tuning 1lms to use memos for consistent long-range open-domain conversation. arXiv preprint arXiv:2308.0822 2023.
[10] Gibbeum Lee, Volker Hartmann, Jongho Park, Dimitris Papailiopoulos, and Kangwook Lee. Prompted llms as chatbot modules for long open-domain conversation. arXiv preprint Stt.:2305.045333 2023.
10

---

[11] Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai. Jieming Zhu, Zhenhua Dong. and Ji-Rong Wen. A survey on the memory mechanism of large language model based agents. arXiv preprint arXiv:2404. 13501, 2024.
[12] Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. Memorybank: Enhancing large language models with long-term memory. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19724-19731, 2024.
[13] Ali Modarressi, Ayyoob Imani, Mohsen Fayyaz, and Hinrich Schütze. Ret-llm: Towards general read-write memory for large language models, arXiv preprint arXiv:2305. 14322, 2023.
[14] Charles Packer, Vivian Fang, Shishir G Patil, Kevin Lin, Sarah Wooders, and Joseph E Gonzalez. Memgpt: Towards llms as operating systems. arXiv preprint arXi::33102 2023.
[15] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Re- flexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[16] Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng. Haotian Wang. Qian- glong Chen, Weihua Peng. Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint 1rXiv:2311.0521 05532, 2023.
[17] Xiaoyan Zhao, Juntao You, Yang Zhang, Wenjie Wang, Hong Cheng, Fuli Feng, See-Kiong Ng. and Tat-Seng Chua. Nextquill: Causal preference modeling for enhancing Ilm personalization. arXiv preprint arXiv:2506.02366. 2025.
[18] Yilun Qiu, Tianhao Shi, Xiaoyan Zhao, Fengbin Zhu, Yang Zhang, and Fuli Feng. Latent inter-user difference modeling for IIm personalization, arXiv preprint arXiv:2507.208.. 20849, 2025.
[19] Weiran Yao, Shelby Heinecke, Juan Carlos Niebles, Zhiwei Liu, Yihao Feng, Le Xue, Rithesh Murthy, Zeyuan Chen, Jianguo Zhang, Devansh Arpit, al. Retroformer: Retrospective large language agents with policy gradient optimization. arXiv preprint urXiv:2308.01551 2023.
[20] Zeyu Zhang, Yang Zhang, Haoran Tan, Rui Li, and Xu Chen. Explicit vs implicit mem- ory: Exploring multi-hop complex reasoning over personalized information. arXiv preprint arXiv:2508.1 13250, 2025.
[21] Zeyu Zhang, Quanyu Dai, Rui Li, Xiaohe Bo, Xu Chen, and Zhenhua Dong. Learn to memorize: Optimizing llmeb agents with adaptive memory framework. arXiv preprint arXiv:2508.1666 16629, 2025.
[22] Zeyu Zhang, Quanyu Dai, Xu Chen, Rui Li, Zhongyang Li, and Zhenhua Dong. Memengine: A unified and modular library for developing advanced memory of llm-based agents. In Companion Proceedings of the ACM on Web Conference 2025, pages 821-824, 2025.
[23] Rui Li, Quanyu Dai, Zeyu Zhang, Xu Chen, Zhenhua Dong, and Ji- Ji-Rong Wen. Knowtrace: Bootstrapping iterative retrieval- -ugmeeetd generation with structured knowledge tracing. In Proceedings of the 3tt ACM SIGKDD Conference Knowledge Discovery and Data Mining V. pages 1770-1480. 2025.
[24] Lei Liu, Xiaoyan Yang, Yue Shen, Binbin Hu, Zhiqiang Zhang, Jinjie Gu, and Guannan Zhang Think-in-memory: Recalling and post-thinking enable llms with long-term memory. arXiv preprint arXiv:231 108719, 2023.
[25] Chenxu Hu, Jie Fu, Chenzhuang Du, Simian Luo, Junbo Zhao and Hang Zhao. Chatdb: Augmenting llm with databases as their symbolic memory. arXiv preprint ar 0.X3992306.03901, 2023.
[26] Adyasha Maharana, Hong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. Evaluating very long-term conversational memory of Im agents. arXiv preprint 1rXiv:2402.17753, 2024.

---

[27] Yunshi Lan, Gaole He, Jinhao Jiang, Jing Jiang, Wayne Xin Zhao, and Ji- Ji-Rong Wen. A survey on complex knowledge base question answering: Methods, challenges and solutions. arXiv preprint arXiv:2105. 1644, 2021.
[28] Boci Peng, Yun Zhu, Yongchao Liu, Xiaohe Bo, Haizhou Shi, Chuntao Hong, Yan Zhang, and Siliang Tang. Graph retrieval-augmented generation: A survey. arXiv preprint urXiv:2408.0899 2024.
[29] Lingxi Zhang, Jing Zhang, Xirui Ke, Haoyang Li, Xinmei Huang, Zhonghui Shao, Shulin Cao, and Xin Lv. A survey on complex factual question answering. Al Open, 4:1-12, 2023.
[30] Shulin Cao, Jiaxin Shi, Liangming Pan, Lunyiu Nie, Yutong Xiang, Lei Hou, Juanzi Li, Bin He, and Hanwang Zhang. Kqa pro: dataset with explicit compositional programs for complex question answering over knowledge base. arXiv preprint arXiv:2007.0330
2020.
[31] Bowen Jin, Chulin Xie, Jiawei Zhang, Kashob Kumar Roy, Yu Zhang, Suhang Wang, Yu Meng. and Jiawei Han. Graph chain-of- -hain-of-thought: Augmenting large language models by reasoning on graphs. arv preprint arXiv:2404.07 103, 2024.
[32] Wenyu Huang, Guancheng Zhou, Mirella Lapata, Pavlos Vougiouklis, Sebastien Montella, and Jeff Z Pan. Prompting large language models with knowledge graphs for question answering involving long-tail facts. arviv preprint Nttv:2405.06522. 2024.
[33] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: henchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466 2019.
[34] Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Daniel Gui, Ziran Will Jiang, Ziyu Jiang, et al. Crag- conpreeereeenivvee rag benchmark. arXiv preprint arXiv:2406.04744 2024.
[35] Wen-tau Yih, Matthew Richardson, Christopher Meek, Ming- Wei Chang, and Jina Suh. The value of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 201-206, 2016.
[36] Alon Talmor and Jonathan Berant. The web as knowledge-base for answering complex questions. arXiv preprint arXiv: wwXiv:1803.0666 1803006643, 2018.
[37] Christina Heinze-Deml, Marloes H Maathuis, and Nicolai Meinshausen. Causal structure learning. Annual Review of Statistics and Its Application, 5(1):371-391, 2018
[38] Judea Pearl. Probabilistic reasoning in intelligent systems: networks of plausible inference. Elsevier, 2014.
[39] Arthur B Kahn, Topological sorting of large networks. Communications of the ACM, 5(11): 558-562, 1962.
[40] E Kathryn Morris, Tancredi Caruso, François Buscot, Markus Fischer, Christine Hancock, Tanja Maier, Torsten Meiners, Caroline Müller, Elisabeth Obermaier, Daniel Prati, et al. Choosing and using diversity indices: insights for ecological applications from the german biodiversity exploratories. Ecology and evolution, 4(18):5514-352) 2014.
[41] KuanChao Chu, Yi-Pei Chen, and Hideki Nakayama. A better Ilm evaluator for text generation: The impact of prompt output sequencing and optimization. arXiv preprint 0rX1v:2406.09972. 2024.
[42] Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 7(3):535-547, 2019.
[43] Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang. Xiaoxiang Shi, et al. Specinfer: Accelerating generative large language model serving with tree-based speculative inference and verification. arXiv preprint arXiv:2305 2305. 009811, 2023.
12

---

[44] Amit Singhal et al. Modern information retrieval: A brief overview. IEEE Data Eng. Bull., 24 (4):35-43, 2001.
[45] GLM Team, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng. Hanlin Zhao, Hanyu Lai, et al. Chatgim: A family of large language models from glm-130b to glm-4 all tools. arXiv e-prints, pages arXiv-2406, 2024.
13

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction include our main claims, which can reflect the contribution and scope of our paper. Specifically, they are present in the abstract as well as the last but one paragraph introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our work in the last section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model -peccficiiii,nonn. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tes system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess..
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [Yes]
Justification: We provide the full set of assumptions and a complete (and correct) proof in Section and Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We fully disclose all the information needed reproduce our methods in the appendix of our paper. We also provide the source code and datasets (see the link in the introduction section).
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
15

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide the source code and datasets (see the link in the introduction
section).
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc)
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/nips. wwwppppwccggpuiceeeoeoeooooppppe gwwcc//gu/es//eee Soaeeeeeoooeeubiiiicccece ceabbbmssssonni ssinnoliccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We fully disclose all the information needed to reproduce our methods the appendix of our paper. We also provide the source code and datasets (see the link in the introduction section).
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report error bars suitably and correctly defined in Section 4 and Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide sufficient information on the computer resources in Section 5.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification: This research conforms with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss positive societal impacts in Section and Section 6
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification:
We discuss positive societal impacts in Section and Section 6
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Please see dection4ant Section 5
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
18

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Please see Section 4 and Section 5
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: Please see Section Section and the Appendix.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: Please Section Section 5 and the Appendix.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
19

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.MM policy (https: hate:p///eerrippc.cc///ofeeerers for what should or should not be described.
20

---

A Proof in Bayesian Relation Network
A.1 Proof of Theorem
Theorem (Factorization), The joint probability distribution of BRNet can be expressed as
P(X1,X2 X(x1)
II
P(X.lpar(Xe)),
X,EX
where par(X:) denotes the set of parent attributes of X.. Proof. Because BRNet is DAG, we can certainly find a topological ordering
O 01, 02.
Then, we inverse the sequence to get reversed topologically ordering
Ô Ö1 02 ..., 0|z
Then, we utilize the theorem of conditional probability according to the order Õ, and have
1( X2,..,,,1 X1x)) = P(Xo, Xögy.., Xayx)) P(Xág (āags.g XoyN)) )........ P(X
|x
II
P P [ōs+1 0)x1)):
where X [ōl+1 ] means all the variables after ō,+1 in the reversed topologically ordering, and there are no descendant variables inside. According to Assumption [
we have
P(Xa, |ōt+1: oxxl)) P(Xa, pprr (Xx.)).
Finally, rewrite it and obtain
P(X1,X2, X2,,,, = II
P(Xelpar(X)) (X)))
X.Ex
A.2 Proof of Theorem
Theorem 2 (Ancestral Sampling). For BRNet, the result of ancestral sampling is equivalent that of sampling from the joint probability distribution. Specifically, we have
P( 22, x(x1) P(z1, Z2, 2(.
where I1, F2,... F[X| P(X,, X,,..,, X are sampled from the joint probability distribution. Proof. We first calculate the reversed topologically ordering
δ Ōō 02 o
Then, we have
X
02, -
II
P(oa, 1 Jōrt1 =xxI)
X P(ūa, (par((a)))
1=1
where & [00+1 : T means the values of all the variables after ōiti the reversed topologically ordering. According to Assumption 2 we have
P(1), 82, T(pe) II P(wa, (par(xa.)
P(11, 82 ..,, x[x))
21

---

Table Results of the evaluation on questions and answers.
Question Types
Textual Answers
Single-choice Answers
Retrieval Target
Simple
100%
98%
100%
Conditional
100%
100%
100%
Comparative
100%
100%
100%
Aggregative
99%
99%
100%
Post-processing
100%
100%
99%
Noisy
100%
100%
100%
Average
99.8%
99.5%
99.8%
B Evaluation on Questions and Answers
The primary challenge for constructing reliable dataset is ensuring the accuracy of ground truths for the constructed questions. To assess the reliability of MemDaily, we sample approximately 20% of all the trajectories in MemDaily and employ human evaluators to verify the correctness of their ground truths. Specifically, the evaluators are required to examine three parts of the ground truths: textual answers, single-choice answers, and retrieval targets, and report their accuracy. The guidelines of human evaluators are provided in Appendix H.3
Metrics. The accuracy of textual answers assesses whether answer correctly responds to the question based on the user messages within the same trajectory. The accuracy single-choice answers indicates whether the ground truth choice is the sole correct answer for the question, given the user messages, while other choices are incorrect. The accuracy retrieval targets evaluates whether the messages of the retrieval target sufficient and necessary answer the question.
Results. As shown in Table 7 MemDaily significantly ensures the accuracy the answers provided for constructed questions. In the few instances where accuracy is compromised, it is attributed to the rewriting process by LLMs, which occasionally leads to information deviation. The results also demonstrate that MemSim can effectively mitigate the impact of LLM hallucinations on factual information, addressing a critical challenge generating reliable questions and answers for memory evaluation. Another baseline method that directly generates answers through LLMs based on targeted user messages and questions performs much lower reliability. We implement this method and present the results OracleMem in constructed benchmarks in Section 5.2
22

---

C
Benchmark on the Efficiency of Memory Mechanisms
The results of efficiency are presented Table and Table
Table 8: Results of response time for generating answers (seconds per query).
MemDaily-vanilla
Methods
Simp.
Cond.
Comp.
Aggr.
Post.
Noisy
FullMem
0.13940.001
0.141=0.001
0. 0.13000.001
0.154+0.002
0.152+0.002
0.150=0.003
RetrMem
0.290+0.007
0.277=0.000
0.267=0.009
0.236*0.009
0.257=0.004
0.284:0.007
ReceMem
0.12640.001
0.127=0.001
0. 0.500000000
0.125=0.001
0.135+0.001
0.134+0.001
NonMem
0.11840.000
0.11000.000
0.118+0.000
0.118+0.000
0.121#0.001
0.121=0.000
NoisyMem
0.1 0.11840..00
0.118+0.001 1911000.000
0.118=0.000 000
0.12100.001
0.121=0.000
OracleMem
0.12240.001
0.122+0.001
0.122=0.000
0.13110.001
0.19200.02
0.128+0.001
MemDaily-100
Methods
Simp.
Cond.
Comp.
Aggr.
Post.
Noisy
FullMem
1.63240.09
1.68400100 101
1.196=0.077
2.52240. 129
1.78240.
1.79940.
RetrMem
0,20740.00 0200
0.223+0.005
0.228=0.011
0.205+0.008
0.228+0.029
0.28440.023
ReceMem
0.120 000
0.125=0.008
0.121
0.120+0.000
0.12540.001
0.12440.001
NonMem
0.11940.00 001
0.119=0.000
0.119=0.000
0.11940.001
0.12340.000
0.12240.001
NoisyMem
1.57840. 124
1.591+0.18 187
1.15320.073
2.42440. 138
1.71740.095
1.73540. 158
OracleMem
0.12240.001
0.123+0.001
0.123+0.001
0.13240.001
0.130#0.001
0.12940.001
Table 9: Results of adaptation time for storing messages (seconds per message).
MemDaily-vanilla
Methods
Simp.
Cond.
Comp.
Aggr.
Post.
Noisy
RetrMem
0.22240.009
0.182=0.004
0.151+0.009
0. 0.13640.010
0.133+0.004
0.112+0.005
Others
0.000
0.001
0.001
0.001
< 0.001
0.001
MemDaily-100
Methods
Simp.
Cond.
Comp.
Aggr.
Post.
Noisy
RetrMem
0.064+0.008
0.072=0.004
0.066+0.007
0.06440.006
0.056=0.002 <0.001
0.066+0.005
Others <0.001
<0.001
<00001
<0.001
<00001
D Extensive Evaluation on User Messages by GPT-4o
We also let GPT-4o score on user messages as a reference, and the results are shown in Table 10
Table 10: Results of evaluation user messages by GPT-4o.
Methods
F-GPT
R-GPT
N-GPT
I-GPT
ZeroCons
4.04
4.80
4.60
3.04
PartCons
4.28
4.88
4.80
4.28
SoftCons
4.20
5.00
5.00
3.96
MemSim
4.04
4.84
4.68
3.60
23

---

E Extensive Benchmark on More Composite Datasets
E.1 Results on MemDaily-10
The results of accuracy are shown in Table I1 The results of 5 are shown in Table 12 The results of response time are shown in Table 13
The results of adaptation time are shown in Table 14
Table
Results of accuracy on MemDaily-10.
Methods
Simp-
Cond.
Comp.
Aggr.
Post.
Noisy
FullMem
0.962=0.040
0.96620.028
0.665+0.055
0.243+0.072
0.810=0.036
0.922=0.029
RetrMem
0.896=0.033
0.882=0.047
0.759=0.068
0.315#0.045
0.782+0.065
0.764=0.053
ReceMem
0.534=0.047
0.482++0006
0.147=0.049
0.248+0.067
0.604+0.088
0.430=0.044
NonMem
0,510=0.090
0.450+0.078
0.159+0.041
0,254#0.065
0.594#0.032
0.380#0.057
NoisyMem
0.428=0.068
0.40220.055
0.16940.044
0.280
0.58420.090
0,350
OracleMem
0.96620.022
0.988#0.010
0.910=0.031
0.372+0.037
0.888=0.030
0.888=0.030
Table 12: Results of r on MemDaily-10.
Methods
Simp.
Cond.
Comp.
Aggr.
Post.
Noisy
LLM
0.794+0.035
0.872+0.019
0.518+0.027
0.73240.036
0.75620.038
0.84620.036
Embedding
0.704+0.039 0.39
0.833=0.026
0.50640.052
0,643=0,043
0.609#0.027
0,648+0.018
Recency
0.032+0.017
0.011=0.010
0.01340.011
0.030#0.012
0.009#0.007
0.504+0.047
Table 13: Results of response time on MemDaily- 10 (seconds per query).
Methods
Simp.
Cond.
Comp.
Aggr.
Post.
Noisy
FullMem
0.243=0.008
0,243+00000
0.208+0.000 0.33
0.30620.00
0.26310.006
0,262=0.00L
RetrMem
0.21320.002
0.230+0.005
0.24620.000
0.212+0.00 .002
0.240+0.004
0.292=#.016
ReceMem
0.120=0.000
0.121#0.00 0.00
0.120:0.000
0.119:0.002
0.126#0.001
0. 0.124:0.001
NonMem
0.119=00.00
0.119::.001
0.119=0.000
0.117=0.002
0.122+0.000
0.119#0.002
NoisyMem
0,205=0.005
0.207=0.007
0.181 0804
0,25340.010
0.223*0.008
0.222
OracleMem
0.1100000
0.12340.001
0. 0.12240.000
0.131+0.001
0.130+0.001
0. 0.12810.001
Table 14: Results of adaptation time on MemDaily- 10 (seconds per message).
Methods
Simp.
Cond.
Comp.
Aggr.
Post.
Noisy
FullMem
< 0.001
0.001
< 0.001
< 0.001
< 0.001
< 0.001
RetrMem
0.073+0.003
0.079#0.006
0.084+0.006
0.069*0.000
0.073+0.003
0.075=0.006
ReceMem
< 0.001
0.001
< 0.001
< 0.001
< 0.001
< 0.001
NonMem
<< 0.001
0.001
0.001
0.001
0.001
0.001
NoisyMem
0.001
0.001
0.001
< 0.001
< 0.001
0.001
OracleMem
0.001
0.001
0.001
< 0.001
< 0.001
0.001
24

---

E.2 Results of MemDaily-50
The results of accuracy are shown in Table 15 results of response time are shown in Table 17
The results of recall @5 are shown in Table 16 The
The results of adaptation time are shown Table 18
Table 15: Results of accuracy on MemDaily-50.
Methods
Simp.
Cond.
Comp.
Aggr.
Post.
Noisy
FullMem
0.962=0.027
0.948=0.020
0.602+0.065
0.296:0.072
0.80240.046
0.880=0.041
RetrMem
0.886=0.035
0.86440.037
0.724=0.062
0.320:0.071
0.780=0.059
0.748=0.049
ReceMem
0.508=0.042
0.434=+.052
0.108
0.237+0.054
0.588+0.066
0.376
NonMem
0.510=0.001
0.452=00.055
0. 0.15990.039
0.254+0.066
0.594+0.078
0.380=0.055
NoisyMem
0,454=0.040
0.416#0.08$
0.229:0.077
0.272+0.073
0.568=0.078
0,360=0.084
OracleMem
0.966=0.025
0.988+0.010
0.910+0.053
0.376+0.042
0.888=0.032
0.984=0.012
Table 16: Results of recall @5 on MemDaily-50.
Methods
Simp.
Cond.
Comp.
Aggr.
Post.
Noisy
LLM
0.725=00047
0.640=0.053
0.773+0.018
0.373=0.003
0.591#0.033
0.561=0.050
Embedding
0.710=0.041
0.674=0.021
0.790=0.037
0.497=0.039
0.591=0.037
0.564+0.053
Recency
0.01 2.0009
0.005=0.004
0.00620.006
0.001=0.002
0.00320.004
0.001
Table 17: Results of response time on MemDaily-50 (seconds per query).
Methods
Simp.
Cond.
Comp.
Aggr.
Post.
Noisy
FullMem
0.776=0.031
0.783+0.067
0.596+0.022
1.134+0.054
0.841+0.032
0.847=0.062
RetrMem
0.203=0.0033
0.206:0.000
0.215
0.204+0.000
0.229+0.005
0.324=0.020
ReceMem
0. 0.0:0000001
0. 0.121#0.002
0.118+0.000
08100000001
0.123+0.002
0. 0.123=0.00
NonMem
0.118=0.001
0.118:0.002
0.117=0.002
0.118#0.001
0.121#0.001
0.119#0.001
NoisyMem
0.728=0.037
0.73740.04|
0,562+0.027
1.06020.055
0.78740.022
0,79420.058
OracleMem
0.12110.001
0.12240.001
0.121+0.001
0.13110.001
0.12920.001
0.12800.001
Table 18: Results of adaptation time on MemDaily-50 (seconds per message).
Methods
Simp.
Cond.
Comp.
Aggr.
Post.
Noisy
FullMem
0.001
0.001
0.001
< 0.001
0.001
0.001
RetrMem
0.059=0.001
0.057+0.003
0.05740.004
0.060+0.003
0.062+0,003
0,08920.005
ReceMem
< 0.001
0.001
0.001
< 0.001
0.001
0.001
NonMem
< 0.001
0.001
< 0.001
< 0.001
0.001
0.001
NoisyMem
< 0.001
0.001
< 0.001
< 0.001
< 0.001
0.001
OracleMem
0.001
0.001
0.001
< 0.001
0.001
0.001
25

---

E.3 Results of MemDaily-200
The results of accuracy are shown in Table 19
The results of recall @5 are shown in Table 20 The
results of response time are shown in Table 21
The results adaptation time are shown Table 22
Table 19: Results of accuracy on MemDaill-2000
Methods
Simp.
Cond.
Comp.
Aggr.
Post.
Noisy
FullMem
0.93220.040
0.932:2.036
0.56340.061
0.30940.056
0.782+0.045
0.86620.044
RetrMem
0.874=0.052
0.84440.034
0.704=0.061
0.315:0.065
0.766#0.046
0.714=0.052
ReceMem
0.486=0.046
0.420=0.057
0.114+0.036
0.272+0.055
0.570=0.055
0.366#0.06
NonMem
0.470=0.0057
0.454+0.077
0.157=0.045
0.257+0.069
0.592+0.082
0.380=0.044
NoisyMem
0.398=0.052
0.398#0.066
0.282+000058
0,276
0.564+0.037
0,350=0.035
OracleMem
0.990=0.013
0.988+0.013
0.910+0.034
0.374+0.063
0.888=0.056
0.984=0.012
Table 20: Results of recall@5 MemDaail-2200
Methods
Simp.
Cond.
Comp.
Aggr.
Post.
Noisy
LLM
0.457=0.006
0.356=0.051
0.556#0.033
0.1 1.16=00002
0.342+0.004
0.322+0.044
Embedding
0.674=0.052
0.641=0.044
0.753+0.036
0.484=0.050
0.544+0.054
0.508:0.052
Recency
0.001+0.003
0.001=0.002
0.00140.002
0.000=0.001
0.00120.003
0.000*0.000
Table 21: Results of response time on MemDaily-200 (seconds per query).
Methods
Simp.
Cond.
Comp.
Aggr.
Post.
Noisy
FullMem
4.028#0.16 161
3.914#0.221
2.697+0. 100
6.365+0.374
4.252+0.328
4.307=0.288
RetrMem
0.236=0.0223
0.241#0.018
0.238+0.024
0.585+0.230
1.012+0.690
1.252=0:427
ReceMem
0.130=0.002
0. 0.12000.002
0.118=0.001
0.1 191400000
0.124+0.001
0. 0.123=0.001
NonMem
0.139=0.0006
0.11920.001
0.11940.001
0.11720.001
0.121#0.001
0.121#0.001
NoisyMem
3.94720.209
3.83240.22 203
2.63740. 118
6.22110.325
4.158
4.21440.288
OracleMem
0.141=0.003
0.12240.001
0.121+0.001
0.13110.002
0.128=0.002
0.12820.001
Table 22: Results of adaptation time on MemDaily-22 (seconds per message).
Methods
Simp.
Cond.
Comp.
Aggr.
Post.
Noisy
FullMem
<0.001
<0.001
<0.001
<0.001
<0.001
<0.001
RetrMem
0.080=0.011
0.080+0.013
0.08040.010
0.220+0.076
0.26420,089
0.420=0.
ReceMem
<0.001
<0.0011
<0.001
<0.001
<0.001
<0.001
NonMem
<0.001
<0.0011
<0.001
<0.001
<00001
<0.001
NoisyMem
<0.001
<0.001
<0.001
<0.001
<0.001
<0.001
OracleMem
<0.001
<0.001
<0.001
<0.001
<0.001
<0.001
26

---

Supplementary Experiments on More Backbones
To better compare the performance across different models, we conduct supplementary experiments with Qwen2.5-7B on MemDaily-vanilla, The results are shown in Table 23 and Table 24
Table 23: Results of accuracy with Qwen2.5-7B on MemDaily-vanilla
Methods
Simp.
Cond.
Comp.
Aggr.
Post.
Noisy
FullMem
0,980 008
0.974=0.017
0.876=0.029
0.312+0.045
0.853+0.003
0.966=#.006
RetrMem
0.880 005
0.878=0.010
0.709=0.019
0.229+0.027
0.787+0.024
0.799=0.008
ReceMem
0.809+0.023
0.797=0.010
0.512#0.026
0.145#0.012
0.739+0.033
0.743+0.046
NonMem
0.526#0.017
0.424=0.027
0.183=0.005
0.255+0.013
0.62940.033
0.376+0.010
NoisyMem
0.512*0.017
0.424=0.050
0.262=0.017
0.301#0.017
0.637=0.032
0.373=0.020
OracleMem
0.982+0.005
0.976*0.005
0.909=0.023
0.450=0.089
0.880=0.020
0.978+0.008
Table 24: Results of recall with Qwen2.5-7B on MemDaily-vanila..
Methods
Simp.
Cond.
Comp.
Aggr:
Post.
Noisy
FullMem
0.93940.00
0.884+0.007
0.918#0.009
0.601#0.016
0,863+0.003
0.872=0.019
RetrMem
0.72640.013
0.717=0.023
0.84620.008
0.516:0.029
0.691+0.014
0.649=0.0144
ReceMem
0.59640.001
0.52440.025
0.700=0.010
0,223+0.012
0.501#0.023
0.503=0.031
NonMem
0.31920.013
0.391=0.019
0.41420.048
0.18340.008
0.31940.006
0.562+0.017
NoisyMem 0.575=00.055
0.525=0.014
0.725=0.012
0.454
0.487=0.025
0.660=0.0119
OracleMem
0.406:0.021
0.60920.035
0.78220.016
0.56240.025
0.605#0.009
0.585+0.023
Based on the experimental results for Qwen2.5-7B, a clear performance hierarchy emerges across different memory mechanisms, with FullMem and OracleMem consistently achieving the highest accuracy scores, followed by moderate performance from RetrMem and ReceMem, while NonMem and NoisyMem demonstrate substantially degraded performance (0.183-0.637 range). The results reveal task-specific challenges, particularly evident in the Aggregation task where all methods show reduced accuracy, and demonstrate trade-off between accuracy and recall metrics, with some methods like OracleMem showing lower recall despite high accuracy, suggesting potential overfitting or conservative prediction behavior. This is consistent with the conclusions drawn from GLM-4-9B.
27

---

G
Case Studies
In this section, we present several case studies to illustrate the effectiveness of the data generated by MemDaily. First, we will display the hierarchical user profiles generated from BRNet. Next, we will present examples of user messages created by our method. Finally, we will provide examples of questions and answers for each type.
G.1 Case Study on Generated User Profiles
In MemDaily, we incorporate 11 entities that cover types, with 73 attributes of them. The summary of entities and attributes of MemDaily are provided in Table 25
We introduce prior knowledge as several rules according to our scenarios to constrain among attributes. For example, a relative role is highly possible to share the same hometown with the user, because they are likely to come from the same place. All of these constraints expressed in BRNet with causal relations. We generate 50 graphical user profiles and conduct observations, finding that most profiles align well with real- world users without contradictions.
Here a case of user profiles, and we translate them into English for better demonstration:
An example of Generated User Profiles
User Profiles:
(Gender) Male; (Name) Qiang Wang; (Age) 38; (Height) 166cm; (Birthday) December
1st.; (Hometown) Beijing; (Workplace) Shenzhen, Guangdong: (Education) High School; (Occupation) Bank Teller; (Position) Head Teller; (Company) Huayin Financial Service Center; (Hobbies) Model Making; (Personality) Outgoing; (Phone) 13420824898; (Email) wangqiang 220 hhayinfinance.com (ID Number) 6401 68 1 2016598; (Passport Num- ber) NZ0448096; (Bank Card Number) 6222022612177604, (Driver's s License Number) 1401681986120122
College Role 1:
(Gender) Female; (Relationship) Supervisor; (Name) Yalin Zhao; (Age) 44; (Height) 165cm; (Birthday) Febrary 5th.; (Hometown) Chongqing; (Workplace) Shenzhen, Guangdong; (Edu- cation) High School; (Occupation) Bank Teller, (Position) Bank Manager; (Company) Huayin Financial Service Center; (Hobbies) Sports; (Personality) Patient; (Phone) 13651039007; (Email) zhaoyalin0205 sxsfinancccenter.com
College Role 2:
(Gender) Male; (Relationship) Colleague; (Name) Zhihong Sun; (Age) 39; (Height) 164cm; (Birthday) April 24th.; (Hometown) Chengdu, Sichuan; (Workplace) Shenzhen, Guangdong: (Education) High School; (Occupation) Bank Teller; (Position) Senior Teller; (Company) Huayin Financial Service Center; (Hobbies) Attending concerts; (Personality) Enthusiastie; (Phone) 15391 1716112 (Email) sunzhihong0421 @ huayinfinance.com:
Relative Role 1:
(Gender) Male; (Relationship) Cousin: (Name) Wei Zhang; (Age) 36; (Height) 169cm; (Birthday) July 15th.; (Hometown) Beijing; (Workplace) Hangzhou, Zhejiang; (Education) Doctor; (Occupation) Doctor; (Position) Chief Physician; (Company) West Lake Hospital; (Hobbies) Playing Video Games; (Personality) Patient; (Phone) 13225162475; (Email) zhangwei07 15 @ westlakehospiital.cmm:
Relative Role 2:
(Gender) Female: (Relationship) Cousin; (Name) Tingting Li; (Age) 36; (Height) 164cm; (Birthday) June 23rd.; (Hometown) Beijing: (Workplace) Shanghai; (Education) Master; (Occupation) Teacher; (Position) Middle School Language Teacher; (Company) Pudong 1 Middle School; (Hobbies) Yoga; (Personality) Patient; (Phone) 13401551341; (Email) litingting(663ee pdxzyz.com;
Work Event 1:
(Type) Job Fair; (Content) Job Fair for Bank Teller Supervisors in the Shenzhen area, sharing professional experience, recruiting talented individuals, and jointly creating a brilliant future for the banking industry.; (Location) Shenzhen, Guangdong: (Time) At PM the Sunday after next; (Title) Bank Teller Job Fair; (Scale) Around 500 People; (Duration) Eight Weeks;
28

---

An example of Generated User Profile
Work Event 2:
(Type) Academic Exchange Conference; (Content) Discuss the development trends of finan- cial technology, share experiences innovative banking services, and promote communi- cation and cooperation among industry elites.; (Location) Beijing; (Time) Next Saturday at 2 PM; (Title) Financial Technology Elite Forum; (Scale) Around 3000 People; (Duration) Seven days;
Entertainment Event
(Type) Art Exhibition; (Content) Displaying selected model works, exchanging making techniques, experiencing creative handicrafts, and feeling the charm of art.; (Location) Beijing; (Time) At PM on the coming Monday; (Title) Model Art Feast; (Scale) Around (Duration) Seven Days; (Relationship) Live; Entertainment Event 2:
900 People;
(Type) Outdoor Hiking; (Content) Conduct outdoor hiking activities, combined with model making, taking natural scenery along the way, creating outdoor landscape models, and sharing modeling techniques.; (Location) Guangdong. Shenzhen; (Time) The Wednesday evening at seven in two weeks; (Title) Outdoor Hiking Model Creation Journey: (Scale) Around 900 People; (Duration) Seven Days; (Relationship) Eight weeks;
Place:
(Type) Residential Community; (Name) Oasis Home; (Comment) Oasis Home is really a nice place to live, with high green coverage rate and beautiful environment. It's especially great to walk and relax here after work every day. However, the commercial facilities are slightly lacking, and it would be perfect if there were more convenience stores and restaurants.; (Relationship) Use;
Item:
(Type) Sports Shoes; (Name) ASICS Gel-Kar Gel-Kayano 26; (Comment) These ASICS Gel- Gay-no 26 shoes are really great, especially for their stability and support, which is perfect for standing work for long periods. Wearing them, my feet feel much more comfortable. However, it would be perfect if they had better breathability.;
From the case profile in MemDaily, we find that our generated user profiles can greatly align with that in real-world scenarios.
29

---

Table 25: Summary of entities and attributes of MemDaily.
Entity
Attribute
Entity
Attribute
Gender
Name
Name
Age
Age
Height
Height
Birthday
Birthday
Hometown
Hometown
Workplace
Workplace
Education
Relative Roles
Education
Occupation
Occupation
Position
User (self)
Position
Company
Company
Hobbies
Hobbies
Personality
Personality
Phone
Phone
Email
Email
Type
ID Number
Content
Passport Number
Location
Bank Card Number
Work Events
Time
Driver's License Number
Title
Gender
Scale
Relationship
Duration
Name
Type
Age
Content
Height
Location
Birthday
Entertainment Events
Time
Hometown
Title
Workplace
Scale
College Roles
Education
Duration
Occupation
Relationship
Position
Type
Places
Company
Name
Hobbies
Comment
Personality
Relationship
Phone Email
Type
Items
Name
Gender
Comment
Relative Roles
Relationship
Total (7)
Total (73)
30

---

G.2
Case Study on User Messages
Based on the generated user profiles, we further generate user messages without inside contradictory according to Section 3.3 Here is case of message list (translated into English) Table 26
Table 26: A case of user messages.
Index
Message
Time
Place
0
My colleague's email is sunzhi-
April 1, 2024, Monday, 08:07
Guangdong Shenzhen
hung0421 @ @ehuyinfinance.com
My colleague really likes attend con- certs.
April 2024, Tuesday, 07:01
Guangdong Shenzhen
2 My colleague's phone number is 15391721618.
April 2, 2024, Tuesday, 08:23
Guangdong Shenzhen
3
My colleague's birthday is April 21st.
April 2, 2024, Tuesday, 17:02
Guangdong Shenzhen
My colleague's name Zhihong Sun.
April 2024, Wednesday, 07:49
Guangdong Shenzhen
5 Wei Zhang's s email address is zhang-
April 3, 2024, Wednesday, 19:07
Guangdong Shenzhen
weid) (1/771662 westaakhoospiial.com
6
Tingting Li's email address is litingt-
April 2024, Thursday, 07:16
Guangdong Shenzhen
ing0623@ (eo3@@ddxyyzzcctt
Yalin Zhao's email address zhaoy- alin020S wwwinanceeeeteercoo
April 2024, Thursday. 13:38
Guangdong Shenzhen
8
I am going to attend the bank teller job fair.
April 5, 2024, Friday, 16:21
Guangdong Shenzhen
9
The time for the bank teller job fair is o' clock the evening on the Sunday.
April 2024, Saturday, 07:18
Guangdong Shenzhen
10
The location of the bank teller job fair is Guangdong Shenzhen.
April 6, 2024, Saturday, 16:58
Guangdong Shenzhen
11 The main content of the bank teller job fair is the job fair: Shenzhen area bank head teller, sharing professional experi- ence, recruiting talent, creating brilliant bank career together.
April 7, 2024, Sunday, 07:21
Guangdong Shenzhen
12 The time for Financial Technology Elite Forum is two cllck the after- noon next Saturday.
April 7, 2024, Sunday, 21:33
Guangdong Shenzhen
13 The time for the Model Art Banquet is seven o' clock in the evening next Mon- day.
April 8, 2024, Monday, 12:45
Guangdong Shenzhen
14 time Creation Journey seven clolock the evening on the next Wednesday.
Outdoor Hiking Model
April 2024, Tuesday, 07:36
Guangdong Shenzhen
By utilizing our mechanisms, we can ensure that there is no contradiction among user messages. We further demonstrate the list of hints that correspond the above messages in Table
31

---

Table 27: A of the hint list.
Index
Entity
Attribute
Value
0
Colleague Role 2
Email
grnbiihong442l@hhayyiiiinaaam
1
Colleague Role 2
Hobbies
Attend Concerts
2
Colleague Role 2
Phone
15391721618
3
Colleague Role 2
Birthday
April 21st
4
Colleague Role 2
Name
Zhihong Sun
5
Relative Role
Email
6
Relative Role 2
Email
litingting0623 @ @dxxzzd.ddxxmmm
7
Colleague Role
Email
zhaoyalin0205 @ sesffnancecennte.ccm
8
Work Event
Title
Bank Teller Job Fair;
9
Work Event
Time
At PM on the Sunday after next Shenzhen, Guangdong
10
Work Event
Location
11
Work Event
Content
Job Fair for Bank Teller Supervisors in the Shenzhen area, sharing professional experience, recruiting talented individu- als, and jointly creating brilliant future for the banking industry
12
Work Event 2
Time
Next Saturday at 2 PM; (Title) Financial Technology Elite Forum
13
Entertainment Event
Time Time
At PM on the coming Monday
14
Entertainment Event 2
The Wednesday evening at seven in two weeks
32

---

G.3 Case Study on Questions and Answers
In this section, we will show the cases of questions and answers of different types. We leave out the time and place of each message in this section, where they do not influence the QA these cases. We have translated all texts into English for better demonstration.
Simple (Simp.) Simple QAs in single-hop.
Case of Simple Questions and Answers
Messages:
0 My cousin's 's email address is zhangwei07 westlakchospital.com. wtm..
1 My cousin works in Hangzhou, Zhejiang.
2 My cousin is 169 em tall.
3 My cousin is from Beijing.
4 My cousin is 36 years old this year.
5 My sister is of her 36 age as well.
6 My boss 44 years old.
7 My colleague is 39 years old this year.
Question:
How old is my cousin now?
ATewer(Text):
36 years old.
Choices:
A. 35 years old.
B. 37 years old.
C. 34 years old.
D. 36 years old.
Answer(C ChnrerCChocc):: D
:nswer(Retrieval): [4]
Time: April 5, 2024, Friday 07:54
Conditional (Cond.) Conditional QAs in multi-hop.
Case of Conditional Questions and Answers
Messages:
[0) My boss only has high school education.
1 My boss works as bank teller.
2 My boss's contact phone number is 13651039007.
3 My boss 165cm tall.
4 My boss works in Shenzhen, the one in Guangdong.
5 My cousin works in Hangzhou, Zhejiang.
6 My cousin works in Shanghai.
7 My colleague works in Shenzhen, in Guangdong.
Question:
Where does the person with only a high school education work now?
Answer(Text):
Shenzhen, Guangdong.
Choices:
A. Zhuhai, Guangdong.
B. Shenzhen, Guangdong.
C. Shenzhen, Guangzhou.
D. Xiamen, Fujian.
Answer( Choice): B
Retrivaa:: [0,
Time: April 6, 2024, Saturday 07:24
33

---

Comparative (Comp.) Comparative QAs in multi-hop.
Case of Comparative Questions and Answers
Messages:
[0] Yalin Zhao is my boss, who is 44 years old.
1 Wei Zhang my cousin, and he is 36 years old.
2 Tingting Li is my cousin, and she is 36 years old.
3 Zhihong Sun is my colleague, and he is 39 years old.
Question:
Who older, Yalin Zhao or Wei Zhang?
Answer(Text):
Yalin Zhao.
Choices:
A. Yalin Zhao.
B. Wei Zhang.
C. Both are the same age.
D. Neither is correct.
Answer( Answer(Choice): A
Answer(Retrieval): []
Time: April 3, 2024, Wednesday 14:38
Aggregative (Aggr.) Aggregative QAs in multi-hop.
Case of Aggregative Questions and Answers
Messages:
() Wei Zhang is my cousin, and his educational background is a Ph.D.
1 Tingting Li is my cousin, and her educational background is master's degree.
2 Yalin Zhao is my boss, and her educational background is high school.
3 Zhihong Sun is my colleague, and his educational background is high school.
4 Wei Zhang is my cousin, and his hometown is Beijing.
5
Tingting Li is my cousin, and her hometown is Beijing.
6 Yalin Zhao is my boss, and her hometown is Chongqing.
7 Zhihong Sun is my colleague, and his hometown Chengdu, Sichuan.
Question:
How many people have an educational background of high school below?
Answer(Text):
people.
Choices:
A. 3 people.
B. 1 people.
C. 4 people.
D. 2 people.
Answer(Choice): D
Rettiia)) 2,
Time: April 5, 2024, Friday 07:27
34

---

Post-processing (Post.) Multi-hop QAs that requires extra reasoning steps.
Case of Post-processing Questions and Answers
Messages:
[0] My cousin works in Hangzhou, Zhejiang.
1 My cousin likes to play video games.
2 My cousin's birthday is July 15th.
3 My cousin's email address is ahangwidl.ssewssssccppp.....mmm westlakchospial.com.
4 My cousin's phone number is 13225162475.
5 Tingting Li works in Shanghai.
6 Yalin Zhao works in Shenzhen, Guangdong.
7 Zhihong Sun works in Shenzhen, Guangdong.
Question:
Which of the following descriptions matches the work location of the person whose birthday
July 15th?
(ext::
A city with beautiful West Lake scenery and a developed internet industry. Choices:
A. Capital, political and cultural center.
B. International metropolis, economic and financial center
C. A city with beautiful West Lake scenery and developed internet industry.
D. Special economic zone, an important city for technological innovation.
Answer( (CaneerCh C
Retrei:)) [0,
Time: April 6, 2024, Saturday 07:51
Noisy (Nois.) Multi-hop QAs that add extra noise questions.
A Case of Noisy Questions and Answers
Messages:
00) My boss 44 years old this year.
1 My boss is the head of a bank.
2 My boss works Shenzhen, Guangdong.
3 My boss really likes sports.
My boss's phone number is 13651039007.
My cousin really likes to play video games.
6 My cousin likes to practice yoga.
My colleague really likes to attend concerts.
Question:
Oh, the weather has been unpredictable lately, it was hot enough to short sleeves yesterday, but today had to put on a jacket. Speaking of which, my favorite season is autumn, not too cold, not too hot, it's the most comfortable time for walk. By the way, that coffee shop recommended by friend last time seems pretty good, should find some time to try it. What wanted to ask is, what are the hobbies of the person who works in Shenzhen, Guangdong?
Answer(Text):
Sports.
Choices:
A. Traveling.
B. Photography.
C. Sports.
D. Reading.
Answer( CChoice): C
: [2.3]
Time: April 4, 2024, Thursday 18:08
35

---

H
Description of Human Evaluation Guidelines
H.1 Guideline of Evaluation on User Profiles
Guideline: You will see some user profiles in the left column of the questionnaire. Please assess whether these user profiles are reasonable, and the rationality of them ranging from I to 5, Score 1 means the least reasonable, while score 5 means the most reasonable.
Here, reasonableness refers to: (1) Likely to exist in the real world, resembling real user (realistic); (2) No inside conflicts or contradictions (consistent).
Here are some examples of unreasonable cases for reference:
(1) [1 point] The user's age 24, but the related person is his grandson. (Logical error: A 24-yeor-old cannot have grandson.)
(2) [2 points] The user's height is "(1) 175cm (2) 168em (3) 172cm". (Generation error: Multiple
values are given for single attribute that can only have one value, like height.)
(3) [2-4 points] The user's phone number is 01234567891. (Unrealistic: The phone number does not seem real.)
(4) [2-4 points] The user's company name is Shanghai XX Company. (Unrealistic: The company name XX does not seem real.)
Tips: If there are no obvious unreasonable aspects, score of can be given; if there are serious errors, a score of 1-2 can be given; for other unrealistic elements, points can be deducted accordingly.
H22 Guideline of Evaluation on User Messages
Guideline: You will see some messages in the left column of the questionnaire. These messages what the user said to the personal assistant while using it, i.e., the recorded user messages. Please assess the fluency, rationality, naturalness, and informativeness of these user messages, and score them ranging from to 5.
[Fluency] The fluency of user messages refers to the correctness of the message text in terms of words, sentences, and grammar; whether the message text is coherent and conforms to language and expression habits, allowing for colloquial expressions. Score 1 means the least fluent, while score 5 means the most fluent.
Here are some examples that lack of fluency for reference: (1) [1-2 point] Today day day day upwards to juggle night, I ate meat pork and or but rice. (Hardly understand due to lack of fluency.)
(2) [2-3 point] This night, pork and rice, delicious. (Requires effort guess due to lack of fluency, but can realize what it means.)
Tips: No obvious issues in fluency can be given score of 5; serious errors can receive score of 2; other elements affecting fluency can lead to a deduction of points as appropriate.
[Rationality] The rationality of the user message refers to: (I) it can be existed in the real world (2) without inside conflict and contradiction. Score means the least rational, while score means the most rational.
Here are some examples that lack of fluency for reference: (1) [1 point] I am 24 years old, and my grandson is 2 years old. (It is impossible for 24- to have a grandson.)
(2) [2-3 point] Today is Monday, tomorrow is Wednesday. (Tomorrow cannot be Wednesday as the day after Monday is Tuesday.)
Tips: If there are no obvious unreasonable points, a score of 5 can be given; for serious errors, score of 1-2 can be given; for other unreasonable elements, corresponding points can be deducted discretion.
[Naturalnes]] The naturalness of a user message refers to whether the message closely resembles real user message. Score means the least natural, while score 5 means the most natural.
[nformativeness]] The informativeness of user messages refers to whether these messages can provide rich and valuable information points. Information points are those points that can be queried about. Score means the least informative, while score 5 means the most informative.
36

---

The following are some examples:
(1) [Low Informativeness] How is the weather today?
(2) [Medium Informativeness] How is the weather today? I plan to to the park this afternoon.
(3) [High Informativeness] Today's 's weather overcast turning to cloudy, won't t rain, plan to go to the park this afternoon.
Highlight: You should have general sense of the informativeness in the user's message during the pre-evaluation phase.
Additional Requirement: You should indicate the reason at the above critical points for deduction. If no major points for deduction exist, then there is no need to fill this requirement.
H.3 Guideline of Evaluation on Questions and Answers
Guideline: In the left column of the questionnaire, you will see (1) list of user messages (2) question (3) the textual answer (4) the multiple choices with the correct answer (5) the index list of retrieval targets. You should check the three aspects of the QAs, including the accuracy of textual answers, the accuracy of multiple-choice answers, and the accuracy of retrieval targets.
[Accuracy of Textual Answers] You need to check whether the textual answer is correct relative the question based the user's message list. it correct, please select the button [Correct], otherwise, please select the button [Incorrect].
[Accuracy of Retrieval Targets] Please judge the correctness of the retrieval targets in the Q&A. Retrieval targets refer to which messages (given in index form) from the user's message list are needed to obtain the textual answer to the question. Determine whether the retrieval targets correct. If it is uniquely correct, please select the button [Correct], otherwise, please select the button [Incorrect].
Additional Requirement: You should indicate the reason for choosing [Incorrect]. If all of the above are correct, then there is no need to fill in this requirement.
37


Reviews:
[{"review_id": "UFmDrY5pwL", "reviewer": "Reviewer_CQvC", "summary": "This paper introduces MemSim, a Bayesian simulator that automatically generates diverse and scalable question-answer pairs. These pairs are designed for evaluating the memory of LLM-based personal assistants. The authors highlight the lack of objective and automatic evaluation methods for this task.\n\nMemSim uses a Bayesian Relation Network (BRNet) and a causal generation mechanism. This helps reduce hallucinations from LLMs when creating factual information. Based on MemSim, a new dataset called MemDaily is created, simulating daily-life scenarios. The paper provides a benchmark using MemDaily to evaluate different memory mechanisms in LLM-based agents. The authors state their work is the first to offer an objective and automatic way to evaluate LLM memory.", "questions": "1.  How do variations in the prompts used for data generation (e.g., user messages, facts, questions) impact the quality and characteristics of the generated dataset? Furthermore, what is the impact of using different base LLMs?\n2.  Could the proposed MemSim methodology be adapted to generate memory-specific evaluation data based on existing datasets, rather than purely synthetic ones?\n3.  In the Bayesian Relation Network (BRNet), is the directed graph structure manually designed (e.g., based on pre-defined rules) or is it automatically learned/generated? If the attribute set is very large and causal relationships are complex, would the graph construction process become challenging?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "### Strengths\n\n1. Addresses a Key Gap: This paper fills a critical void by introducing an objective and automatic evaluation method for LLM-based personal assistant memory.\n2. Novel, Scalable Data Generation: MemSim, with its BRNet and causal mechanism, offers an innovative approach to automatically and scalably generate reliable evaluation data, mitigating LLM hallucinations.\n3. Valuable Dataset & Open-Source: The new MemDaily dataset is a significant contribution. Its open-source release promotes reproducibility and future research.\n\n### Weaknesses\n\n1. Limited Cross-Dataset Comparison: The paper lacks explicit comparisons with existing public datasets, making it difficult to fully contextualize performance. While the authors note the lack of previous datasets for this specific task, comparison with less related existing datasets would also be valuable.\n2. Insufficient Detail on Data Generation Prompts: The paper does not provide the specific prompts used for data generation, nor does it analyze the sensitivity of data reliability to different LLMs or prompting strategies.", "was_revised": true, "final_justification": "The rebuttal addressed most of my earlier concerns. My main reservation is about long-term impact: with rapidly advancing LLMs and abundant real-world data in industry, the necessity of such synthetic datasets may be limited. I therefore keep my score of 4.", "rebuttal": "Dear reviewer CQvC,\n\nThanks so much for your precious time in reading and reviewing our paper, and we are encouraged by your positive feedback. In the following, we try to alleviate your concerns one by one:\n\n\n\n**For Weakness 1: \"Limited Cross-Dataset Compariso... related existing datasets would also be valuable.\"**\n\n**Response:** \n\nThanks for your valuable question. As for the unique advantages compared with existing datasets, there are two critical advantages in our work:\n\n**(1) Automatic Data Generation without Human Annotation (Compared with Other Long-term QA Datasets):**\n\nPrevious approaches usually adopt to the pipeline like \"message --> question --> answer\". They generate or collect some user messages, and then let an LLM generate questions based on these messages. Finally, they make the LLM generate correct answers based on the user messages and questions. Although this method is simple, the accuracy of the answers depends on the performance of the LLM, which **makes the difficulty of constructing and solving the Q&A the same**. Therefore, these approaches require further human annotation to check whether the answer is correct, such as PerLTQA, LOCOMO, and LeMon.\n\nIn contrast, our proposed approach takes the pipeline like \"prior knowledge --> question & answer --> message\". We generate questions and answers based on constructed prior information (such as user attributes). Then, we create user messages by injecting answers with other information. This construction method makes it easier to construct Q&A than to solve them. By this means, we can ensure the correct answer is contained and well-located in the user messages.\n\nThe feature of \"automatic\" makes the evaluation extendable to other specific scenarios without expensive human annotators.\n\n**(2) User Messages as Information Foundations for** **QAs** **(Compared with Other KBQA Datasets)**\n\nIn Conventional KBQAs evaluations, a knowledge graph is typically provided as retrieval support [2], LLMs can also be evaluated on general knowledge using common-sense questions, such as HotpotQA. However, for LLM-based personal assistants, users do not provide a knowledge graph to the personal assistant. Instead, these scenarios need to convey factual information in the form of user messages. This makes it challenging to directly evaluate LLM-based agents using existing KBQA data, as it requires reliably injecting structured information into user messages. That is also the problem that our causal generation mechanism aims to address.\n\n**(3) Evaluation on memorizing certain critical information (Compared with Other Memory-based Conversation Tasks)**\n\nSome previous works like [1] focus on utilizing a long/short memory to improve long-term conversation tasks. These works can reflect \"the effectiveness of memory mechanisms for long-term conversation tasks\" by improving their performances on these tasks, but not take a common and direct evaluation on \"**how memory mechanisms can memorize certain critical information**\", which is the key point in our work. The task improvement by memory usage is not identical to the performance that the memory can exactly memorize critical information.\n\n\n\n**For Weakness 2: \"Limited Cross-Dataset Compariso... related existing datasets would also be valuable.\"**\n\n**Response:** \n\nThank you for your feedback. Due to our data construction mechanism's complexity and dependency on specific question types, we've included prompts in our code repository (anonymous link in abstract's last line).\n\nFor MemDaily, we used rule-based mechanisms to constrain user profile generation, creating realistic BRNets reflecting real-world situations. Rather than relying solely on fixed templates, we integrate dynamic rule constraints (e.g., fathers are typically ~24 years older than children, using normal distribution). We constructed five question types for comprehensive memory evaluation: Single-hop, Multi-hop, Comparative, Aggregative, and Postprocessing. Each employs different sampling mechanisms and prompts during Causal Generation, adding prompt construction complexity.\n\nRegarding prompting strategy impacts on data generation, we conducted comparative analysis in Sections 4.1-4.2 (Tables 3-4). We tested various strategies including SeqPL (sequential entity attribute generation, similar to MDP methods) and JointPL (simultaneous generation of all attributes), applying different constraint levels to compare generation accuracy and sensitivity. Results show MemSim's prompting strategy achieves best performance with reliable data quality.\n\nFor LLM impact on data generation, we tested Llama 3 and GPT-4. Llama 3 showed poor stability and failed to produce reliable data in our scenarios. While GPT-4 demonstrated excellent performance and stable, reliable data generation, its closed-source nature makes it expensive and time-consuming, leading us not to adopt it. We selected GLM-4 as our final backbone due to superior data reliability. Future work will investigate data reliability across various LLMs and scenarios.\n\n\n\n**For Question 1: \"How do variations in the prompts used for data generation (e.g., user messages, facts, questions) impact the quality and characteristics of the generated dataset? Furthermore, what is the impact of using different base LLMs?\"**\n\n**Response:** \n\nThank you for this important question. We conducted comprehensive comparative analysis of different prompting strategies' impact on data generation in Sections 4.1 and 4.2.\n\nFor user profile generation, we compared four strategies: (1) JointPL: generating attributes jointly; (2) SeqPL: generating attributes sequentially based on previous ones; (3) IndePL: generating attributes independently; and (4) MemSim, our BRNet-based approach. Table 3 shows MemSim outperforms other baselines on R-Human, demonstrating BRNet's effectiveness in generating high-quality user profiles.\n\nFor user message generation, we evaluated four constraint strategies: (1) ZeroCons: no attribute constraints; (2) PartCons: partial attribute constraints; (3) SoftCons: full attribute constraints without enforcement; and (4) MemSim, our Causal Generation Mechanism. Figure 3 shows our method maintains high scores despite rigorous constraints on constructing reliable question-answer pairs. MemSim exhibits the highest diversity index due to BRNet and the causal generation mechanism producing varied user messages from hierarchical user profiles.\n\nRegarding different base LLMs, we tested Llama 3 and GPT-4. Llama 3 showed poor stability and failed to produce reliable quality data. GPT-4 performed excellently and generated stable, reliable data, but being closed-source, proved expensive and time-consuming. We selected GLM-4 as our backbone for superior data reliability and cost-effectiveness. Future research will investigate data reliability across various LLMs and scenarios.\n\n\n\n**For Question 2: \"Could the proposed MemSim methodology be adapted to generate memory-specific evaluation data based on existing datasets, rather than purely synthetic ones?\"**\n\n**Response:** \n\nThank you for this insightful question. Yes, the proposed MemSim methodology can indeed be adapted to generate memory-specific evaluation data based on existing datasets, rather than relying solely on synthetic generation.\n\nYou can leverage existing datasets by extracting entities and relationships using LLMs or other methods, then construct a BRNet following the approach described in Section 3.2 of our paper. Once the BRNet is established, you can perform sampling and generation on this network to create corresponding user messages, questions, and answers that are grounded in the original dataset.\n\nIn fact, we have successfully applied MemSim to real-world business scenarios and achieved promising results, which demonstrates the practical adaptability and effectiveness of our methodology beyond purely synthetic data generation.\n\n\n\n**For Question 3: \"In the Bayesian Relation Network (BRNet) ... automatically learned/generated ... challenging?\"**\n\n**Response:** \n\nThank you for this excellent question. BRNet construction supports both rule-based and automatic approaches, each suited for different scenarios. Rule-based construction works well when relationships are clear and simple with well-defined rules. It produces reliable, low-noise data since causal relationships are explicitly defined using domain knowledge. However, it becomes challenging and unscalable when attribute sets are large and causal relationships grow complex. For complex scenarios, we use LLM-based automatic extraction to identify entities and relationships for autonomous BRNet construction. This approach handles large-scale scenarios and cases where explicit rules cannot be easily designed, capturing complex relational patterns that are difficult to extract manually. The automatic approach may introduce noise during LLM extraction, requiring human verification and correction to ensure accuracy. This creates a trade-off between scalability and precision when selecting the construction method. When attribute sets are very large with complex causal relationships, automatic construction becomes the appropriate choice over rule-based methods.\n\n\n\n**For Paper Formatting Concerns: \"The text font in Figure 1 is too small.\"**\n\n**Response:** \n\nThank you for your valuable feedback. We will increase the font size in Figure 1 in the revised version of the paper.\n\n\n\n**We sincerely thank you for your time to review our paper, and we also thanks for your insightful comments, which, we believe, are very important to improve our paper. We hope our responses can address your concerns. If you have further questions, we are very happy to discuss them.**\n\n\n\nReferences:\n\n[1] Li, Hao, et al. \"Hello Again! LLM-powered Personalized Agent for Long-term Dialogue.\" arXiv preprint arXiv:2406.05925 (2024).\n\n[2] Lan, Yunshi, et al. \"Complex knowledge base question answering: A survey.\" IEEE TKDE 35.11 (2022): 11196-11215."}, {"review_id": "sZlYInXIVT", "reviewer": "Reviewer_GeDE", "summary": "MemSim is a Bayesian simulator that automatically generate reliable, diverse, and scalable evaluation datasets MemDaily.  MemDaily assesses the memory capabilities of LLM-based personal assistants. MemSim uses BRNet, which models hierarchical user profiles by sampling entities and their attributes in a causally structured, loop-free graph. A causal generation mechanism uses these profiles to produce user messages and construct QA pairs to reduce hallucination. MemDaily comprises six QA types, reflecting real-world query complexity. Extensive human and automated evaluations demonstrate that MemDaily maintains high accuracy, fluency, and diversity. The authors benchmark various memory mechanisms on this dataset, including full memory, recent memory, and retrieved memory, measuring both effectiveness and efficiency.", "questions": "see Weaknesses", "limitations": "NA", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1.\tMemSim’s pipeline is automatic, which saves expensive human annotation. It generates QAs from simulated user profiles, which enables large-scale and reproducible memory evaluation.\n2.\tThe proposed method derives user messages and QAs from the same hints sampled from BRNet. This method effectively reduces hallucination as shown in Table 5.\n3.\tThe authors propose a comprehensive QA taxonomy, including the 6 QA types (simple, conditional, comparative, aggregative, post-processing, and noisy). This taxonomy covers a wide range of reasoning scenarios, which provides a better and more detailed evaluation guideline for memory mechanisms.\n4.\tMemDaily dataset has a high diversity in the simulated profiles and messages, as proven by its Shannon–Wiener score. It shows that BRNet’s hierarchical sampling and the causal mechanism are useful in generating varied user profiles.\n5.\tThis papers benchmarks the memory mechanisms in an end-to-end manner, including evaluateing multiple memory strategies on both vanilla and noisy variants of the dataset. The authors conduct extensive experiments and provide clear metrics for effectiveness and efficiency.\n\n\nWeaknesses:\n1.\tThe experiments uses MemDaily to evaluate only GLM-4-9B. It would be better to evaluate models from different families or models of different sizes. A comparison of the results from different models can better showcase the benchmark’s ability to evaluate models.\n2.\tAlthough you mention in check point 14 (Line824 to 838) that you have included full text instructions given to participants, I could not  find any from the paper, including appendix. This brings that problem that your human evaluation to user profile (line236 - 237) and user messages (Line 254 to 255) become very unclear. Human evaluators give scores scale from 1 – 5, but what is a score 1 profile like? What is a score 5 profile like? What aspects should the evaluator look into a profile? Would evaluator A’s score 3 be the same as evaluator B’s score 3? Without the guideline, your human evaluations are very weak.\n3.\tSome generated questions are still very artificial, especially the multi-hop questions and post-processing questions. For instance, in appendix F3 (Line 534 - 535), “where does the person with only a high school education work now?” or (line 540-541),” Which of the following descriptions matches the work location of the person whose birthday is July 15th?” I hardly think a question like this will appear from any real user. There is still a gap in the distribution of actual encountered questions and the generated ones.\n4.\tYour method of controlling dataset quality is through adding more noise from social media. However, since your useful information are model generated, they naturally have a different distribution from the real message from social media. And hence the negative samples inserted are not high-quality negatives. Hence the difference of performance between different difficulties are not very obvious (table 11 – table 22)\n5.\tThere are some further areas of improvement. \nLike a) including a memory editing mechanism in the benchmark through dynamic dialogue\nAnd b) your college and relative entities are a little toyed. (line 518-522, table 23). In real life a person’s relationship is far beyond colleagues and relatives. It would be better to set up the colleges/relatives also as individual entities, and each entity can be a (self), the user entity can be connected to many other entities with more complex relationships.", "was_revised": true, "final_justification": "My preference is positive and main concern is about the evaluation, such as the generated questions. During rebuttal, the authors have resovled most of my concerns. So, I keep the positive view.", "rebuttal": "Dear reviewer GeDE,\n\nThanks so much for your precious time in reading and reviewing our paper, and we are encouraged by your positive feedback. In the following, we try to alleviate your concerns one by one:\n\n\n\n**For Weakness 1: \"The experiments uses MemDaily to evaluate only GLM-4-9B... ability to evaluate models.\"**\n\n**Response:** \n\nThank you for your suggestion. To better compare the performance across different models, we have additionally included evaluation results for Qwen2.5. The experimental results are as follows.\n\nQwen2.5-7B (Accuracy):\n\n| Methods   | Simp.       | Cond.       | Comp.       | Aggr.       | Post.       | Noisy       |\n| --------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |\n| FullMem   | 0.980±0.008 | 0.974±0.017 | 0.876±0.029 | 0.312±0.045 | 0.853±0.003 | 0.966±0.006 |\n| RetrMem   | 0.880±0.005 | 0.878±0.010 | 0.709±0.019 | 0.229±0.027 | 0.787±0.024 | 0.799±0.008 |\n| ReceMem   | 0.809±0.023 | 0.797±0.010 | 0.512±0.026 | 0.145±0.012 | 0.739±0.033 | 0.743±0.046 |\n| NonMem    | 0.526±0.017 | 0.424±0.027 | 0.183±0.005 | 0.255±0.013 | 0.629±0.033 | 0.376±0.010 |\n| NoisyMem  | 0.512±0.017 | 0.424±0.050 | 0.262±0.017 | 0.301±0.017 | 0.637±0.032 | 0.373±0.020 |\n| OracleMem | 0.982±0.005 | 0.976±0.005 | 0.909±0.023 | 0.450±0.089 | 0.880±0.020 | 0.978±0.008 |\n\nQwen2.5-7B (Recall):\n\n| Methods   | Simp.       | Cond.       | Comp.       | Aggr.       | Post.       | Noisy       |\n| --------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |\n| FullMem   | 0.939±0.006 | 0.884±0.007 | 0.918±0.009 | 0.601±0.016 | 0.863±0.003 | 0.872±0.019 |\n| RetrMem   | 0.726±0.013 | 0.717±0.023 | 0.846±0.008 | 0.516±0.029 | 0.691±0.014 | 0.649±0.014 |\n| ReceMem   | 0.596±0.001 | 0.524±0.025 | 0.700±0.010 | 0.223±0.012 | 0.501±0.023 | 0.503±0.031 |\n| NonMem    | 0.319±0.013 | 0.391±0.019 | 0.414±0.048 | 0.183±0.008 | 0.319±0.006 | 0.562±0.017 |\n| NoisyMem  | 0.575±0.025 | 0.525±0.014 | 0.725±0.012 | 0.454±0.007 | 0.487±0.025 | 0.660±0.019 |\n| OracleMem | 0.406±0.021 | 0.609±0.035 | 0.782±0.016 | 0.562±0.025 | 0.605±0.009 | 0.585±0.023 |\n\nBased on the experimental results for Qwen2.5-7B, a clear performance hierarchy emerges across different memory mechanisms, with FullMem and OracleMem consistently achieving the highest accuracy scores, followed by moderate performance from RetrMem and ReceMem, while NonMem and NoisyMem demonstrate substantially degraded performance (0.183-0.637 range). The results reveal task-specific challenges, particularly evident in the Aggregation task where all methods show reduced accuracy, and demonstrate a trade-off between accuracy and recall metrics, with some methods like OracleMem showing lower recall despite high accuracy, suggesting potential overfitting or conservative prediction behavior. This is broadly consistent with the conclusions drawn from GLM-4-9B.\n\n\n**For Weakness 2: \"Although you mention in check point 14 (Line824 to 838) that you have included full text instruction ... are very weak.\"**\n\n**Response:** \n\nThank you very much for your valuable feedback. You are absolutely right, and we apologize for this omission. Below is the description of our human evaluation guidelines, which clearly defines what each score represents and ensures consistent scoring standards across different evaluators. We will include this description in the revised version of the paper.\n\n**Guideline of Evaluation on User Profiles.**\n\n*Guideline: You will see some user profiles in the left column of the questionnaire. Please assess whether these user profiles are reasonable, and rate the rationality of them ranging from 1 to 5. Score 1 means the least reasonable, while score 5 means the most reasonable. Here, reasonableness refers to: (1) Likely to exist in the real world, resembling a real user (realistic); (2) No inside conflicts or contradictions (consistent).*\n\n*Here are some examples of unreasonable cases for reference:*\n\n*(1) [1 point] The user's age is 24, but the related person is his grandson. (Logical error: A 24-year-old cannot have a grandson.)*\n\n*(2) [2 points] The user's height is \"(1) 175cm (2) 168cm (3) 172cm\". (Generation error: Multiple values are given for a single attribute that can only have one value, like height.)*\n\n*(3) [24 points] The user's phone number is 01234567891. (Unrealistic: The phone number does not seem real.)*\n\n......\n\n*Tips: If there are no obvious unreasonable aspects, a score of 5 can be given; if there are serious errors, a score of 12 can be given; for other unrealistic elements, points can be deducted accordingly.*\n\n**Guideline of Evaluation on User Messages.**\n\n*Guideline: You will see some messages in the left column of the questionnaire. These messages are what the user said to the personal assistant while using it, i.e., the recorded user messages. Please assess the fluency, rationality, naturalness, and informativeness of these user messages, and score them ranging from 1 to 5.*\n\n*[Fluency] The fluency of user messages refers to the correctness of the message text in terms of words, sentences, and grammar*\n\n...\n\n*Additional Requirement: You should indicate the reason at the above critical points for deduction. If no major points for deduction exist, then there is no need to fill in this requirement.*\n\n**For Weakness 3: \"Some generated questions are still very artificial... gap in the distribution of actual encountered questions and the generated ones\"**\n\n**Response:** \n\nThank you very much for your valuable feedback. We acknowledge the gap between generated data and real-world scenarios, which stems from two main factors. First, LLM-synthesized data sources may create distributional bias compared to real-world scenarios. A solution is generating data based on real datasets by extracting entities and relationships from actual datasets, constructing a BRNet using our Section 3.2 methodology, then performing sampling and generation to create more reality-grounded user messages, questions, and answers. Second, rule-based generation processes may also cause distributional bias. We could automatically extract relationships and entities from source data using LLMs and construct corresponding Causal Generation Mechanisms for subsequent generation. Rule-based construction suits scenarios with clear rules and simple relationships, producing lower-noise data. LLM-based automatic extraction works better for large-scale scenarios or cases where explicit rules cannot be easily designed, though it may introduce noise during extraction, requiring human verification to correct the resulting BRNet. We will further investigate methods for generating data based on real-world scenarios to better bridge the gap between synthetic and authentic question distributions.\n\n**For Weakness 4: \"Your method of controlling dataset quality is through adding more noise ... difficulties are not very obvious (table 11 – table 22) .\"**\n\n**Response:** \n\nThank you for your valuable comment. You are absolutely right that the noise we inject does not have exactly the same distribution as the original user messages, but we believe it has reasonable justification for the following reasons:\n\n1. **Similar semantic distribution.** In our Daily Life scenario, user messages primarily focus on human daily activities, while the noise we introduce comes from news trending topics, which also belong to human daily life. The content expressed by both sources is generally similar in nature.\n2. **Simulation of multi-source data in real scenarios.** In real-world scenarios, an agent's memory typically includes not only user-provided content but also additional information sources, such as important information from other applications. These information sources may not have exactly the same distribution as user-provided content, but they remain important. Furthermore, long-term information storage and retrieval is a key focus of current memory mechanisms and should be our primary emphasis.\n3. **Scalability and efficiency considerations.** Constructing noise with perfectly consistent distribution would require generation based on the original profiles, carefully avoiding potential conflicts between user messages. This is difficult to achieve at scale and lacks reliability. Using out-of-distribution information avoids this issue and is more efficient.\n4. **Consistency with existing work.** Most existing work adopts similar approaches for noise injection. For example, [1] also uses long documents to construct needle-in-a-haystack tasks.\n\nWe acknowledge that this approach has limitations, but we believe it provides a reasonable balance between realism, scalability, and evaluation effectiveness for assessing long-term memory capabilities.\n\n**For Weakness 5: \"There are some further areas of improvement ... with more complex relationships.\"**\n\n**Response:** \n\nThank you for your valuable suggestions. Moving forward, we plan to incorporate a counterfactual generation mechanism into the memory system, which will generate user messages with different answers to the same question, thereby implementing a memory editing mechanism. Additionally, we plan to expand our dataset based on real-world data. We are considering leveraging actual recommendation datasets, automatically extracting relationships and entities through LLMs, and constructing corresponding Causal Generation Mechanisms to produce richer and more comprehensive user messages, questions, and answers.\n\n**We sincerely thank you for your time to review our paper, and we also thanks for your insightful comments, which, we believe, are very important to improve our paper. We hope our responses can address your concerns. If you have further questions, we are very happy to discuss them.**\n\nReferences:\n\n[1] Wu, Di, et al. \"Longmemeval: Benchmarking chat assistants on long-term interactive memory.\" *arXiv:2410.10813* (2024)."}, {"review_id": "xVtRLLPL4e", "reviewer": "Reviewer_rBqV", "summary": "This paper introduces MemSim, a Bayesian simulator designed to automatically generate reliable datasets for evaluating the memory of LLM-based personal assistants, thereby addressing the high cost of human annotation and the hallucination issues in existing automatic methods. MemSim leverages a Bayesian Relation Network to model user profiles with causal dependencies and a novel generation mechanism to create user messages and question-answer pairs from shared, structured \"hints.\" Using this simulator, the authors created the MemDaily dataset to test various memory mechanisms, revealing that current LLM assistants struggle significantly with complex comparative and aggregative questions, especially when faced with noisy or irrelevant information.", "questions": "1: Your current evaluation treats memory as static fact retrieval, but real personal assistants need to handle evolving user information, conflicting updates, and temporal dependencies. How would you extend MemDaily to evaluate these dynamic aspects of memory management?\n\n2: You claim the approach is scalable, but BRNet requires manual specification of entities, attributes, and causal relationships for each domain. Can you provide concrete evidence of how easily this transfers to new domains (e.g., professional contexts, healthcare scenarios)? What would be required to adapt MemSim to evaluate memory in coding assistants or research agents?\n\n3: While you position this as an evaluation methodology paper, the core technical contributions appear to be primarily about reliable dataset generation with BRnet sampling and causal strurcual prompting method rather than novel evaluation metrics or frameworks. What makes this fundamentally different from a benchmark paper that happens to introduce a novel dataset generation method?", "limitations": "yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1: The paper addresses a genuine and important problem in LLM agent evaluation. The technical approach is well-motivated and the experimental evaluation is comprehensive, including both dataset quality assessment and memory mechanism benchmarking.\n\n2: The combination of Bayesian networks for user profile generation and causal generation mechanisms is novel for this application domain.\n\nWeakness:\n1: While the authors claim scalability, the approach requires manual design of entity-attribute relationships and causal structures for each domain, which may limit generalizability.\n\n2: Disconnect from Recent Advances: The paper doesn't connect with recent developments in agentic systems like tool use, reflection mechanisms, or planning capabilities that are increasingly important for personal assistants with memories. (e.g. only tests retrieval and basic reasoning, missing evaluation of how memory integrates with other agent capabilities like tool calling or multi-step planning). Also as for memory systems, modern memory systems often incorporate semantic similarity, knowledge graphs, or learned embeddings. The lack of such comparison limits the scope of this paper and I am skeptical about its value to the community given the rapid advancement in modern AI Agents.", "was_revised": true, "final_justification": "reviewers has addressed most of my concern; didn't give a higher score since I think i was looking for more connections with other agentic components", "rebuttal": "Dear reviewer rBqV,\n\nThanks so much for your precious time in reading and reviewing our paper, and we are encouraged by your positive feedback. In the following, we try to alleviate your concerns one by one:\n\n\n\n**For Weakness 1: \"While the authors claim scalability, the approach requires manual design of entity-attribute relationships and causal structures for each domain, which may limit generalizability.\"**\n\n**Response:** \n\nThank you for your valuable question. While BRNet does require entity-attribute relationships and causal structures, this process can be automated by leveraging LLMs to extract the corresponding entities, attributes, and features, thereby enabling automatic construction of the required datasets. For example, with novel data, we can extract character roles, attributes, and relationships to construct the BRNet, derive questions and answers based on these relationship chains, and generate statements as memory observations for the agent.\n\nIn our paper, we focus on designing the generative paradigm of BRNet. The construction of relationships required by BRNet from raw information can be achieved through LLM-based extraction or direct rule-based summarization. The rule-based construction method is suitable for scenarios with clear rules and simple relationships, producing data with lower noise that is more robust. The LLM-based automatic extraction of relationships and entities is applicable to large-scale scenarios or cases where explicit rules cannot be designed, though it may introduce noise during the LLM extraction process, requiring additional human correction of the resulting BRNet.\n\nWe believe this flexibility in construction approaches actually enhances the scalability of our method, as it can adapt to different domain requirements and data availability constraints.\n\n\n\n**For Weakness 2: \"Disconnect from Recent Advances ... rapid advancement in modern AI Agents.\"**\n\n**Response:** \n\nThank you for your insightful question. We strongly agree that agents require not only memory but also additional mechanisms such as tool use, reflection mechanisms, and planning capabilities to assist in memory implementation. However, the evaluation dataset construction method proposed in this paper focuses more on a model-agnostic setting, meaning that any memory system can be evaluated on the dataset constructed by our method. The research on how other mechanisms affect memory that you mentioned can also be implemented on our dataset by simply fixing the memory method and varying different additional capabilities to compare their performance. For example, to study the impact of reasoning capabilities on memory, one could fix the memory method as RAG-based long-term memory and then compare reasoning structures such as CoT, ToT, and GoT. We would be very interested to see future work explore the influence of other agent capabilities on memory.\n\nFurthermore, regarding the memory technologies you mentioned, such as semantic similarity, knowledge graphs, and learned embeddings, these are all components within memory systems that still require the memory framework of storage, retrieval, and utilization to function [1, 2]. Our model-agnostic setting can precisely accommodate these memory components you mentioned by fixing other parts and then modifying the component under study to compare performance. In the future, we will also optimize and improve the evaluation of the integration between other agent capabilities and memory.\n\nWe believe our framework provides a solid foundation for systematic evaluation that can evolve with the rapid advances in agentic systems, and we welcome the community to build upon this work to explore these important research directions.\n\n\n\n**For Question 1: \"Your current evaluation treats memory as static fact retrieval ... dynamic aspects of memory management?\"**\n\n**Response:** \n\nThank you very much for your insightful question. In fact, our proposed evaluation method is model-agnostic for memory systems and provides a comprehensive evaluation of memory systems as a whole. The corresponding methods need to complete the processes of storage, retrieval, and utilization, rather than simple static retrieval. Specifically, in the anonymous code repository referenced at the end of the abstract, we have constructed a complete temporal data flow evaluation where user messages are given sequentially with timestamps, as shown in `benchmark/Timeflow.py`. Therefore, our evaluation has already implemented dynamic extension with temporal information.\n\nRegarding evolving user information and conflicting updates, these can be addressed through the question generation process of the Causal Generation Mechanism by inserting different timestamps for conflicting information, which can seamlessly integrate with `Timeflow.py`. This design allows our framework to naturally handle the dynamic aspects of memory management that you mentioned, including temporal dependencies and information conflicts that occur over time.\n\nWe believe this temporal evaluation capability demonstrates that our approach goes beyond static fact retrieval and can effectively assess how memory systems handle the complexities of real-world personal assistant scenarios.\n\n\n\n**For Question 2: \"You claim the approach is scalable, but BRNet requires manual specification ... or research agents?\"**\n\n**Response:** \n\nThank you for your question. For BRNet, construction can be achieved through both rule-based approaches and LLM-based extraction of entities and relationships for autonomous construction. The rule-based construction method is suitable for scenarios with clear rules and simple relationships, producing data with lower noise that is more robust. The LLM-based automatic extraction of relationships and entities is applicable to large-scale scenarios or cases where explicit rules cannot be designed, though it may introduce noise during the LLM extraction process, requiring additional human correction of the resulting BRNet. As you mentioned, if the attribute set is very large and causal relationships are complex, rule-based approaches would be very challenging, so adopting automatic entity and relationship construction is the appropriate choice.\n\nFor example, in financial and healthcare domains, we can use LLMs to extract key entities, attributes, and corresponding relationships based on large-scale domain expert knowledge, and integrate specific application scenarios (such as Alice's situation) to construct BRNet. Through sampling, we can obtain corresponding questions and answers, generate corresponding statements (user messages), and produce the final evaluation dataset.\n\nFor coding assistants, we can consider code files as the raw dataset, extract different classes as entities, extract different methods as attributes, and derive relationships between various classes and attributes to construct the corresponding memory evaluation dataset. For research agents, we can use knowledge from a specific domain as raw data, extract nouns as entities, their features as attributes, and extract relationships between them, applying MemSim to obtain the final evaluation dataset.\n\nThis flexibility in construction approaches demonstrates the scalability of our method across diverse domains, from personal scenarios to professional contexts, by adapting the entity-attribute-relationship extraction process to domain-specific characteristics while maintaining the same underlying evaluation framework.\n\n\n\n**For Question 3: \"While the authors claim scalability, the approach requires manual design of entity-attribute relationships and causal structures for each domain, which may limit generalizability.\"**\n\n**Response:** \n\nThank you for your question. I agree with your observation that our core contribution is proposing a data generation method based on BRNet and the Causal Generation Mechanism. However, we focus on designing memory data generation methods from an abstract methodological perspective, rather than concentrating on a specific dataset (such as the Daily Life scenario) and its evaluation, although it can be conveniently transferred and applied to generate new datasets. As we mentioned above, our method can also be applied to other domains, such as finance and healthcare. Additionally, we analyze and explore the current performance of explicit memory systems, discussing the advantages and disadvantages of different models. Therefore, we believe this paper differs from a typical benchmark paper.\n\nOur work provides a systematic framework for generating evaluation data for memory systems across domains, rather than simply presenting a fixed benchmark. The methodology we propose enables researchers to create domain-specific memory evaluation datasets using the same principled approach, which we believe constitutes a valuable methodological contribution to the field of memory evaluation for AI agents.\n\n\n\n**We sincerely thank you for your time to review our paper, and we also thanks for your insightful comments, which, we believe, are very important to improve our paper. We hope our responses can address your concerns. If you have further questions, we are very happy to discuss them.**\n\n\n\n**References:**\n\n[1] Zhang, Zeyu, et al. \"A survey on the memory mechanism of large language model based agents.\" *ACM Transactions on Information Systems* (2024).\n\n[2] Zhang, Zeyu, et al. \"MemEngine: A Unified and Modular Library for Developing Advanced Memory of LLM-based Agents.\" *Companion Proceedings of the ACM on Web Conference 2025*. 2025."}, {"review_id": "EmldcOcaw2", "reviewer": "Reviewer_k9DJ", "summary": "This paper presents MemSim, an automated framework for evaluating the memory capabilities of large language model (LLM)-based personal assistants.  By constructing a Bayesian Relation Network (BRNet), the authors generate logically coherent and diverse user profiles.  These profiles serve as the foundation for a causal generation mechanism that simultaneously produces user messages and corresponding question-answer pairs from shared structured hints, mitigating hallucinations common in LLM-generated content.  Based on this approach, the authors build MemDaily, a comprehensive evaluation dataset that spans various memory task types, and introduce a benchmarking framework to evaluate the memory capability of LLM-based personal assistants.", "questions": "1.It would be helpful if the authors could elaborate on the motivation for choosing BRNet over other data modeling methods (e.g., knowledge graph-based sampling). The paper would be strengthened by either (1) providing theoretical or empirical evidence that BRNet yields higher diversity or consistency in user profiles, or (2) including a comparative experiment demonstrating that datasets generated with BRNet outperform those generated by alternative methods in terms of QA reliability and coverage.\n\n2.Although the authors designed multiple internal baselines to compare dataset attributes, all of these baselines rely solely on their self-constructed simulation environment.  The lack of horizontal comparison with existing benchmarks—such as the EpisodicQA-generated datasets [1]—limits the persuasiveness of the proposed method.  [1] Huet A, Houidi Z B, Rossi D. Episodic Memories Generation and Evaluation Benchmark for Large Language Models. arXiv preprint arXiv:2501.13121, 2025.\n\n3.It may be beneficial to include datasets that are more reflective of real-world scenarios—such as multi-turn dialogue settings (e.g., a user says “I’m going to see the dentist” today, and three days later mentions “The doctor reminded me to check for cavities”)—to better evaluate whether the method can maintain and retrieve information consistently in context.\n\n4.The current benchmark evaluation (Section 5.2) is conducted solely using the GLM-4-9B model, which may limit the generalizability of the findings. It is recommended to include evaluations using multiple LLMs (e.g., LLaMA) to provide a more comprehensive assessment of the dataset’s robustness and applicability.", "limitations": "The evaluation protocol relies heavily on subjective human ratings and lacks a standardized, reproducible scoring framework, which may undermine the consistency and generalizability of the results.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1.The introduction of the Bayesian Relation Network (BRNet) to simulate user profiles and model causal dependencies among attributes through a graph structure enhances the diversity and consistency of user information.\n\n2.The MemDaily dataset constructed based on MemSim covers six common types of memory tasks encountered in real-world scenarios, providing a unified benchmark for subsequent related research. \n\nWeaknesses:\n\n1.While the paper presents a systematic approach to constructing a factual QA dataset, the evaluation scope is limited to explicit and structured factual memory.  It overlooks higher-level memory types, such as dynamically evolving memory representations from long-term interactions, as well as implicit forms of memory like user preferences, intentions, and emotional tendencies.\n\n2.The authors attempt to mitigate hallucination issues in LLMs through a causality-based generation mechanism guided by structured hints.   However, the generation of user messages and questions still heavily relies on the language modeling capabilities of LLMs, and no real user dialogues or log data are used for validation.   As a result, the evaluation may not accurately reflect the memory capabilities of LLMs in open-domain settings.\n\n3.The paper relies heavily on human subjective ratings, which are treated as the primary evaluation metric.   This approach is somewhat unconvincing due to its strong subjectivity and limited reproducibility.   A more robust evaluation strategy would prioritize structural consistency or other quantitative metrics, using human judgment as a supplementary measure.   Besides, the paper should clearly specify the human evaluation process, including the criteria, procedures, and consistency scores", "was_revised": true, "final_justification": "I have read the authors' response. Thanks for the rebuttal. The response addressed some of my concerns. I will raise my rating accordingly. Considering the overall quality (novelty, experiments, method, idea), I revised my rating score as 4.", "rebuttal": "Dear Reviewer k9DJ,\n\nThanks so much for your precious time in reading and reviewing our paper. However, I believe there may be some misunderstandings about our paper, and I hope the following rebuttal can make a clarification and change your perspective on our work. In the following, we try to alleviate your concerns one by one.\n\n\n**For Weakness 1: \"While the paper presents a systematic approach ... tendencies.\"**\n\n**Response:** \n\nThank you very much for your valuable comments. In this paper, we indeed primarily focus on users' factual information rather than implicit preferences. However, our proposed MemSim framework can be readily extended to accommodate higher-level memory types. For instance, in user preference scenarios, we can similarly construct graph structures based on BRNet to represent relationships between user preferences, and leverage Causal Generation Mechanisms to generate question-answer pairs based on users' implicit preferences. Meanwhile, we can utilize LLMs to rewrite implicit preferences into concrete factual information, thereby making high-level preferences explicit and enabling the evaluation of higher-level memory.\n\nRegarding evolving memory representations from long-term interactions, we have implemented a model-agnostic dynamic comprehensive evaluation of memory, where the corresponding methods need to complete the processes of storage, retrieval, and utilization. Specifically, in the anonymous code repository referenced at the end of the abstract, we have constructed a complete temporal data flow evaluation where user messages are provided sequentially with timestamps, as demonstrated in `benchmark/Timeflow.py`. In future work, we will further refine the generation pipeline for higher-level memory types.\n\n\n**For Weakness 2: \"The authors attempt to mitigate hallucination ... open-domain settings.\"**\n\n**Response:** \n\nThank you for your valuable comments. We agree with your point that hallucinations arising from LLM generation capabilities cannot be completely avoided. However, our proposed method can mitigate the degree of hallucination during the data generation process, as difficult problems are more prone to hallucinations, and our approach can decompose the challenging generation process into simpler multi-step procedures through causal mechanisms.\n\nFurthermore, we conducted evaluation experiments on the generated data in Section 4. Through these experiments, we found that the reliability of our method shows significant improvement, and the generated question-answer pairs also demonstrate high accuracy. Additionally, other existing papers [2] also rely on LLM-based generation and cannot completely eliminate LLM hallucinations, but can only mitigate them as much as possible.\n\n\n**For Weakness 3: \"The paper relies heavily on human subjective ratings ... consistency scores.\"**\n\n**Response:** \n\nThank you for your comments. Regarding dataset generation evaluation, many works use human verification methods to assess data quality, such as [3]. We use both human annotators and LLM-based evaluations for objective scoring.\n\nHere are the detailed human evaluation steps. We recruited six well-educated human experts to score multiple aspects of our Memdaily dataset. We designed a standard evaluation pipeline with clear instructions, fair scoring, and reasonable comparisons. Our evaluations focus on user profiles, user messages, and QAs (mentioned in Section 4).\n\nOur pipeline includes five steps: (1) Human evaluator recruitment (2) Guideline and questionnaire design (3) Web page construction and deployment (4) Pre-evaluation (5) Formal evaluation.\n\nWe recruited six human experts as evaluators, all holding at least bachelor's degrees to ensure correct understanding of evaluation questions and reasonable feedback. We designed detailed guidelines instructing evaluators on the evaluation process. The guideline includes three parts corresponding to the three evaluation aspects in Section 4.\n\nDue to response length limitations, we will provide a description of the human evaluation. The guideline clearly defines score meanings and ensures consistent scoring standards across evaluators. For consistency scores, see means and standard deviations in Tables 3 and 4.\n\n\n**For Questions 1 : \"It would be helpful if the authors could elaborate on the motivation ... coverage.\"**\n\n**Response:** \n\nThank you for your question. Regarding knowledge graph-based sampling, in our paper, we just need conditional probability distributions among variables, and utilize ancestral sampling to obtain different user profiles, instead of calculating the complex and high-dimensional joint probability distribution. These conditional probability distributions should be introduced into BRNet as prior knowledge, before sampling user profiles. For BRNet, we also provide comparisons of the user profiles it generates in Table 3. As shown in Table 3, MemSim outperforms other baselines on Rationality, demonstrating the effectiveness of BRNet as an ablation study.\n\nRegarding QA reliability and coverage, we have included the evaluation results in Appendix B of our paper. We sample approximately 20% of all the trajectories in MemDaily and employ human evaluators to verify the correctness of their ground truths. Specifically, the evaluators are required to examine the ground truths and report their accuracy in Table 7. We find that MemDaily significantly ensures the accuracy of the answers provided for constructed questions.\n\n**For Questions 2: \"Although the authors designed multiple ... EpisodicQA-generated datasets ...\"**\n\n**Response:** \n\nThank you for your comments. We will add comparative content with the following two papers in the revised version. MemSim proposes a complete Bayesian simulator specifically designed to automatically construct reliable question-answer pairs for evaluating LLM memory capabilities, while ensuring diversity and scalability. In contrast, the Episodic Memories Generation research is still in the conceptual exploration stage, primarily focusing on how to integrate episodic memory capabilities into LLMs, but lacks a mature automated evaluation system. MemSim provides an operational technical solution rather than just a theoretical framework by introducing Bayesian Relation Networks (BRNet) and causal generation mechanisms.\n\n\n**For Questions 3: \"It may be beneficial to include datasets that are more reflective of real-world scenarios ... consistently in context.\"**\n\n**Response:** \n\nThank you for your suggestion. In fact, our method can also be applied to extract information from real-world data. You can extract entities and relationships from existing datasets using methods such as LLMs, construct BRNet according to the method described in Section 3.2 of our paper, and then perform sampling and generation on the BRNet to construct corresponding user messages, questions, and answers. We have also applied MemSim to real business scenarios and achieved promising results. In future work, we plan to supplement relevant datasets that are abstracted and extracted from real-world data.\n\n**For Questions 4: \"The current benchmark ... solely using the GLM-4-9B model ... applicability.\"**\n\n**Response:** \n\nThank you for your suggestion. To better compare the performance across different models, we have additionally included evaluation results for Qwen2.5. The experimental results are as follows.\n\nQwen2.5-7B (Accuracy):\n\n| Methods   | Simp.       | Cond.       | Comp.       | Aggr.       | Post.       | Noisy       |\n| --------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |\n| FullMem   | 0.980±0.008 | 0.974±0.017 | 0.876±0.029 | 0.312±0.045 | 0.853±0.003 | 0.966±0.006 |\n| RetrMem   | 0.880±0.005 | 0.878±0.010 | 0.709±0.019 | 0.229±0.027 | 0.787±0.024 | 0.799±0.008 |\n| ReceMem   | 0.809±0.023 | 0.797±0.010 | 0.512±0.026 | 0.145±0.012 | 0.739±0.033 | 0.743±0.046 |\n| NonMem    | 0.526±0.017 | 0.424±0.027 | 0.183±0.005 | 0.255±0.013 | 0.629±0.033 | 0.376±0.010 |\n| NoisyMem  | 0.512±0.017 | 0.424±0.050 | 0.262±0.017 | 0.301±0.017 | 0.637±0.032 | 0.373±0.020 |\n| OracleMem | 0.982±0.005 | 0.976±0.005 | 0.909±0.023 | 0.450±0.089 | 0.880±0.020 | 0.978±0.008 |\n\nQwen2.5-7B (Recall):\n\n| Methods   | Simp.       | Cond.       | Comp.       | Aggr.       | Post.       | Noisy       |\n| --------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |\n| FullMem   | 0.939±0.006 | 0.884±0.007 | 0.918±0.009 | 0.601±0.016 | 0.863±0.003 | 0.872±0.019 |\n| RetrMem   | 0.726±0.013 | 0.717±0.023 | 0.846±0.008 | 0.516±0.029 | 0.691±0.014 | 0.649±0.014 |\n| ReceMem   | 0.596±0.001 | 0.524±0.025 | 0.700±0.010 | 0.223±0.012 | 0.501±0.023 | 0.503±0.031 |\n| NonMem    | 0.319±0.013 | 0.391±0.019 | 0.414±0.048 | 0.183±0.008 | 0.319±0.006 | 0.562±0.017 |\n| NoisyMem  | 0.575±0.025 | 0.525±0.014 | 0.725±0.012 | 0.454±0.007 | 0.487±0.025 | 0.660±0.019 |\n| OracleMem | 0.406±0.021 | 0.609±0.035 | 0.782±0.016 | 0.562±0.025 | 0.605±0.009 | 0.585±0.023 |\n\nBased on the experimental results for Qwen2.5-7B, a clear performance hierarchy emerges across different memory mechanisms, with FullMem and OracleMem consistently achieving the highest accuracy scores. This is broadly consistent with the conclusions drawn from GLM-4-9B.\n\n**We sincerely thank you for your time to review our paper and comments on it. I hope the rebuttal can make a clarification of the misunderstandings and change your perspective on our work. If you have further questions, we are very happy to discuss them.**\n\n**References:**\n\n[1] Huang, Lei, et al. \"A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.\" *ACM TOIS* 43.2 (2025): 1-55.\n\n[2] Wu, Di, et al. \"Longmemeval: Benchmarking chat assistants on long-term interactive memory.\" *arXiv:2410.10813* (2024).\n\n[3] Maharana, Adyasha, et al. \"Evaluating very long-term conversational memory of llm agents.\" *arXiv:2402.17753* (2024)."}]

Author rebuttal:

Meta-review: The paper proposes MemSim, a Bayesian simulator designed to automatically construct reliable QAs from generated user messages, with the introduction of the Bayesian Relation Network (BRNet) and a causal generation mechanism to mitigate the impact of LLM hallucinations on factual information, facilitating the automatic creation of an evaluation dataset. Therefore, the paper builds MemDaily and conducts comprehensive evaluation on this dataset to evalaute the various memory mechanisms of LLMs, including full memory, recent memory, and retrieved memory, measuring both effectiveness and efficiency.

Strengths:

1. The pipeline is automatic and easily be extended to other domains or datasets.
2. The proposed method derives user messages and QAs from the same hints sampled from BRNet. This method effectively reduces hallucination (` CQvC`, `GeDE`).
3. The authors propose a comprehensive QA taxonomy, including the 6 QA types (simple, conditional, comparative, aggregative, post-processing, and noisy). This taxonomy covers a wide range of reasoning scenarios, which provides a better and more detailed evaluation guideline for memory mechanisms (`k9DJ`, `GeDE`).

Weakness:

1. Some details are not clear, including data generation prompts (`CQvC`, `GeDE`), human evaluation (`k9DJ`, `GeDE`). 
2. The synthesized data can not well reflect the real-data distritbuion (`GeDE`), some generated questions are still very artificial.
3. Lack of analysis on the impact of model size and different model family as the paper mainly conduct experiments on GLM-4-9B (` CQvC`, `GeDE`).
4. The paper does not consider complex memory applications, i.e., memory updating or more agentic components such as reflection, planning and tool use (`GeDE`, `rBqV`).

Generally, the author provide detailed response during rebuttal and all reviewers acknowledged that most of concerns are addressed.
Decision: Accept (poster)