MemSim: A Bayesian Simulator for Evaluating Memory of LLM-based Personal Assistants

## Abstract

Abstract LLM-based agents have been widely applied as personal assistants, capable of memorizing information from user messages and responding to personal queries.
However, there still lacks an objective and automatic evaluation on their memory capability, largely due to the challenges in constructing reliable questions and answers (QAs) according to user messages.
In this paper, we propose MemSim, a Bayesian simulator designed to automatically construct reliable QAs from generated user messages, simultaneously keeping their diversity and scalability.
Specifically, we introduce the Bayesian Relation Network (BRNet) and a causal generation mechanism to mitigate the impact of LLM hallucinations on factual information, facilitating the automatic creation of an evaluation dataset.
Based on MemSim, we generate a dataset in the daily-life scenario, named MemDaily, and conduct extensive experiments to assess the effectiveness of our approach.
We also provide a benchmark for evaluating different memory mechanisms in LLM-based agents with the MemDaily dataset.
To benefit the research community, we have released our project at https://github.com/nuster1128/MemSim .

## 1 Introduction

In recent years, large language model (LLM) based agents have been extensively deployed across various fields [, , , , , ]. One of their most significant applications is serving as personal assistants [], where they engage in long-term interactions with users to address a wide range of issues [, ].
For LLM-based personal assistants, memory is one of the most significant capability [].
To perform personal tasks effectively, these agents must be capable of storing factual information from previous messages and recalling relevant details to generate appropriate responses.
For example, a user Alice might tell the agent, "I will watch a movie at City Cinema this Friday in Hall 3, Row 2, Seat 9." When Friday arrives, she might ask the agent, "Where is my movie seat?" Then, the agent should recall the relevant information (i.e., the seat number) to generate an appropriate response to Alice.

Previous research has proposed methods for constructing the memory of LLM-based agents [, , , , ]. However, there remains a lack of objective and automatic methods to evaluate how well personal assistants can memorize and utilize factual information from previous messages, which is crucial for developing memory mechanisms.
One conventional solution is to collect messages from real-world users, and manually annotate answers to human-designed questions based on these messages. However, it requires substantial human labor that lacks scalability.
Another solution is to generate user messages and question-answers (QAs) with LLMs.
However, the hallucination of LLMs can severely undermine the reliability of generated datasets, particularly in complex scenarios [].
Here, we refer to the reliability of a dataset as the correctness of its ground truths to factual questions given the corresponding user messages.
Our research shows that due to the hallucination of LLMs, the correctness of ground truths generated by vanilla LLMs is less than 90% in most scenarios and can fall below 40% in some complex scenarios (see Section 5.2). For instance, when posing aggregative questions like "How many people are under the age of 35?," they often provide incorrect answers due to hallucinations. Moreover, generating diverse user profiles through LLMs is also challenging, as they tend to produce the most plausible profiles that lack diversity.

To address these challenges, we propose MemSim, a Bayesian simulator designed to construct reliable QAs from generated user messages, simultaneously keeping their diversity and scalability, which can be utilized to evaluate the memory capability of LLM-based personal assistants.
Specifically, we introduce the Bayesian Relation Network (BRNet) to generate the simulated users that are represented by their hierarchical profiles. Then, we propose a causal generation mechanism to produce various types of user messages and QAs for the comprehensive evaluation on memory mechanisms.
By using BRNet, we improve the diversity and scalability of generated datasets, and our framework can effectively mitigate the impact of LLM hallucinations on factual information, which makes the constructed QAs more reliable.
Based on MemSim, we create a dataset in the daily-life scenario, named MemDaily, and perform extensive experiments in multiple aspects to assess the quality of MemDaily.
Finally, we construct a benchmark to evaluate different memory mechanisms of LLM-based agents with MemDaily. Our work is the first one that evaluates memory of LLM-based personal assistants in an objective and automatic way.
Our contributions are summarized as follows:

$\bullet$ We analyze the challenges of constructing datasets for objective evaluation on the memory capability of LLM-based personal assistants, focusing on the aspects of reliability, diversity, and scalability.

$\bullet$ We propose MemSim, a Bayesian simulator designed to generate reliable, diverse and scalable datasets for evaluating the memory of LLM-based personal assistants.
We design BRNet to generate the simulated users, and propose a causal generation mechanism to construct user messages and QAs.

$\bullet$ We create a dataset in the daily-life scenario based on our framework, named MemDaily, which can be used to evaluate the memory capability of LLM-based personal assistants. We perform extensive experiments to assess the quality of MemDaily in multiple aspects, and provide a benchmark for different memory mechanisms of LLM-based agents. To support the research community, we have made our project available at [https://github.com/nuster1128/MemSim](https://github.com/nuster1128/MemSim).

The rest of our paper is organized as follows.
In Section 2, we review the related works on the evaluation of memory in LLM-based agents and personal assistants.
In Section 3, we introduce the details of MemSim, and the generation process of MemDaily.
In Section 4, we assess the quality of MemDaily.
Section 5 provides a benchmark for evaluating different memory mechanisms of LLM-based agents. Finally, in Section 6, we discuss the limitations of our work and draw conclusions.

## 2 Related Works

LLM-based agents have been extensively utilized across various domains, marking a new era for artificial personal assistants []. For LLM-based personal assistants, memory is a critical component that enables agents to deliver personalized services.
This includes storing, managing, and utilizing users’ personal and historical data [, , , ].
For instance, MPC [] suggests storing essential factual information in a memory pool with a summarizer for retrieval as needed.
MemoryBank [] converts daily events into high-level summaries and organizes them into a hierarchical memory structure for future retrieval. These approaches primarily aim to enhance agents’ memory capability.

Previous studies have also attempted to evaluate the memory capability of LLM-based agents, but there still exist limitations. Some studies use subjective methods, employing human evaluators to score the effectiveness of retrieved memory [, , ]. However, this approach can be costly due to the need for evaluators and may introduce biases from varying annotators. Other studies use objective evaluations by constructing dialogues and question-answer pairs [, , ], but these methods still require human involvement for creating or editing the QAs. Therefore, how to construct reliable QAs according to user messages automatically is significant for the objective evaluation.

Some previous studies construct knowledge-based question-answering (KBQA) datasets to assess Retrieval-Augmented Generation (RAG) [, ], which is relative to the data generation for memory evaluation. These studies typically either use knowledge graphs to generate QAs through templates or manually annotate QAs with human input [, , , , , ]. However, most of these efforts focus on common-sense questions rather than personal questions whose answers are only determined by the user messages in the same trajectory. They do not include textual user messages and target indexes for retrieval evaluation [, , , , ]. Additionally, they are highly dependent on the entities extracted from the given corpus, which limits their scalability [, ].
Our work is the first one that evaluate memory of LLM-based personal assistants in an objective and automatic way, which can generate user messages and QAs without human annotators, keeping reliability, diversity and scalability.

## 3 Methods

Our final goal is to evaluate memory mechanisms of LLM-based personal assistants in an objective and automatic way. The whole pipeline is demonstrated in Figure 1. First of all, we propose MemSim that can simulate users and generate evaluation datasets, mainly including the Bayesian Relation Network and a causal generation mechanism. Then, we employ MemSim to create a dataset in the daily-life scenario, named MemDaily. Finally, we construct a benchmark that evaluates different memory mechanisms of LLM-based agents based on MemDaily. In this section, we will deliver the details of MemSim and MemDaily, while the evaluation benchmark will be presented in Section 5.

### 3.1 Overview of MemSim

In order to construct reliable QAs from generated user messages, we propose a Bayesian simulator named MemSim, which includes two primary components.
First, we develop the Bayesian Relation Network to model the probability distribution of users’ relevant entities and attributes, enabling the sampling of diverse hierarchical user profiles.
Then, we introduce a causal mechanism to generate user messages and construct reliable QAs based on these sampled profiles.
We design various types of QAs for comprehensive memory evaluation, including single-hop, multi-hop, comparative, aggregative, and post-processing QAs, incorporating different noises to simulate real-world environments.
Based on the constructed QAs and generated user messages, researchers can objectively and automatically evaluate the memory capability of LLM-based personal assistants on factual information from previous messages, which can be helpful in developing advanced memory mechanisms.

Figure: Figure 1: Overview of MemSim and MemDaily.
Refer to caption: x1.png

### 3.2 Bayesian Relation Network

We introduce Bayesian Relation Network (BRNet) to model the probability distribution of users’ relevant entities and attributes, where we sample hierarchical profiles to represent simulated users (see Figure 1(a)).
Specifically, we define a two-level structure in BRNet, including the entity level and the attribute level. The entity level represents user-related entities, such as relevant persons, involved events, and the user itself. At the attribute level, each entity comprises several attributes, such as age, gender, and occupation.
Here, BRNet actually serves as a predefined meta-user.
Formally, let $\mathcal{A}^{1},\ldots,\mathcal{A}^{N}$ represent different entities, and each entity $\mathcal{A}^{i}$ comprises several attributes $\{A_{1}^{i},A_{2}^{i},\ldots,A_{N^{i}}^{i}\}$, where $N$ is the number of entities, and $N^{i}$ is the number of attributes belonging to the entity $\mathcal{A}^{i}$.
Each attribute $A^{i}_{j}$ corresponds to a random variable $X^{i}_{j}$, which can be sampled in a value space. For example, the college’s (entity $\mathcal{A}^{i}$) age (attribute $A^{i}_{j}$) is 28 years old (value $x^{i}_{j}\sim X^{i}_{j}$).

We denote BRNet as a directed graph $G=\langle V,E\rangle$ at the attribute level, where the vertex set $V$ includes all attributes, i.e., $V=\bigcup_{i=1}^{N}\{A_{1}^{i},A_{2}^{i},\ldots,A_{N^{i}}^{i}\}$.
The edge set $E$ captures all the direct causal relations among these attributes, defined as $E=\{\langle A^{i}_{j},A^{k}_{l}\rangle\mid\forall X^{i}_{j},X^{k}_{l}\in
\mathcal{X},X^{i}_{j}\rightarrow X^{k}_{l}\}$, where $\mathcal{X}=\bigcup_{i=1}^{N}\{X_{1}^{i},X_{2}^{i},\ldots,X_{N^{i}}^{i}\}$.
For better demonstration, in this subsection, we simplify the subscripts of the variables in $\mathcal{X}$ as $1,2,...,\sum_{i=1}^{N}N_{i}$.
The conditional probability distribution among them can either be explicitly predefined or implicitly represented by LLM’s generation with conditional prompts.
It is important to note that we assume the causal structure is loop-free, ensuring that BRNet forms a directed acyclic graph (DAG), which is typical in most scenarios [].
Additionally, the vertices (i.e., attributes), edges (i.e., causal relations), and conditional probability distributions (i.e., prior knowledge) can be easily scaled to accommodate different scenarios.

So far, we have constructed the BRNet, where
the joint probability distribution $P(X_{1},X_{2},\ldots,X_{|\mathcal{X}|})$ over all attributes can represent the user distribution in the given scenario. Then, we can sample different values of attributes on entities from BRNet to represent various user profiles.
One straightforward approach is to compute the joint probability distribution and sample from it.

###### Assumption 1 (Local Markov Property) .

BRNet satisfies the local Markov property, which states that

$$ $X_{t}\perp\!\!\!\perp X_{\overline{des}(X_{t})}|par(X_{t}),\forall X_{t}\in \mathcal{X},$ $$

where $\overline{des}(X_{t})$ denotes the non-descendant set of $X_{t}$, $par(X_{t})$ denotes the parent set of $X_{t}$, and the notation $\cdot\perp\!\!\!\perp\cdot|\cdot$ indicates the variables are conditionally independent.

Because the parents of an attribute can be extended to any non-descendant attributes of it by adding a new edge if they have a direct causal relation. Therefore, given these parent attributes, other non-descendent attributes are conditionally independent of that attribute.

###### Theorem 1 (Factorization) .

The joint probability distribution of BRNet can be expressed as

$$ $P(X_{1},X_{2},...,X_{|\mathcal{X}|})=\prod_{X_{t}\in\mathcal{X}}P(X_{t}|par(X_ {t})),$ $$

where $par(X_{t})$ denotes the set of parent attributes of $X_{t}$.

The proof of Theorem 1 is provided in Appendix A.1.
However, calculating the joint probability distribution and sampling from it may be impractical in our scenarios.
First, the joint probability distribution is often high-dimensional, making its calculation and sampling costly.
Second, some conditional probability distributions are difficult to represent in explicit forms, particularly when using LLMs for value generation through conditional prompts.
To address these issues, we introduce the ancestral sampling process to obtain the values of attributes.

###### Assumption 2 (Conditional Sampling) .

In BRNet, an attribute can be sampled from the conditional probability distribution given its parent attributes. Specifically, we have

$$ $\tilde{x}_{t}\sim P\left(X_{t}|par\left(X_{t}\right)\right),\forall X_{t}\in \mathcal{X},$ $$

where the conditional probability distribution can be expressed in either explicit or implicit forms.

The ancestral sampling algorithm is outlined as follows.
First, we obtain the topological ordering of BRNet using Kahn’s algorithm []. Next, we sample all attributes according to this ordering.
For top-level attributes without parents, the sampling is performed based on their marginal probability distributions.
For other variables like $X_{t}$, we sample their values using the conditional probability distribution $\tilde{x}_{t}\sim P\left(X_{t}|par\left(X_{t}\right)\right)$ as specified in Assumption 2.
Finally, we consider each sampling result $\{\tilde{x}_{1},\tilde{x}_{2},\ldots,\tilde{x}_{|\mathcal{X}|}\}$ as the attribute-level profiles of a user, which constitute different entities as the entity-level profiles of the user. These two levels represent the user in different grains, which are important to generate user messages and QAs subsequently.

###### Theorem 2 (Ancestral Sampling) .

For BRNet, the result of ancestral sampling is equivalent to that of sampling from the joint probability distribution. Specifically, we have

$$ $P(\tilde{x}_{1},\tilde{x}_{2},...,\tilde{x}_{|\mathcal{X}|})=P(x_{1},x_{2},... ,x_{|\mathcal{X}|}),$ $$

where $x_{1},x_{2},...,x_{|\mathcal{X}|}\sim P(X_{1},X_{2},...,X_{|\mathcal{X}|})$ are sampled from the joint probability distribution.

The proof can be found in Appendix A.2.
By employing ancestral sampling, we eliminate the need to compute the joint probability distribution, making the sampling process more efficient and practical.
By utilizing BRNet, we introduce prior knowledge of the specific scenario into the graphical structure and sampling process, which can improve the diversity and scalability of user profiles, thereby enhancing the diversity and scalability of whole datasets.

###### Assumption 1 (Local Markov Property) .

###### Theorem 1 (Factorization) .

###### Assumption 2 (Conditional Sampling) .

###### Theorem 2 (Ancestral Sampling) .

### 3.3 Causal Generation Mechanism

Based on hierarchical user profiles, we propose a causal generation mechanism to generate user messages, and construct reliable QAs corresponding to them. Here, causal indicates that the generation of user messages and the construction of QAs are causally dependent on the same informative hints that are also causally derived from hierarchical user profiles. Specifically, we define a piece of hint as a triple $(\mathcal{A}^{i},A^{i}_{j},x^{i}_{j})$ that provides factual information in a structural format.
In other words, the hierarchical user profiles provide a structural foundation to get different hints, which then provide a set of relevant information as the causation of both user messages and QAs, shown in Figure 1(b).

Construction of Informative Hints.
We construct the hints of factual information based on hierarchical user profiles before creating the user messages and QAs.
We select a target entity $\mathcal{A}^{t}$ at the entity-level, and choose $l^{t}$ attributes $\{K_{1}^{t},K_{2}^{t},\ldots,K_{l^{t}}^{t}\}\subseteq\mathcal{A}^{t}$ along with their corresponding values $\{v_{1}^{t},v_{2}^{t},\ldots,v_{l^{t}}^{t}\}$ from the attribute-level profiles. Then, we reformulate them into a list of triple hints $H^{t}=\left[(\mathcal{A}^{t},K_{i}^{t},v_{i}^{t})\right]_{i=1}^{l^{t}}$. For some complex types of QAs, we choose more than one target entities, and concatenate their lists of hints. For better demonstration, we re-index the final list of hints as $H=\left[(\mathcal{A}_{(j)},K_{(j)},v_{(j)})\right]_{j=1}^{l}$, where $l$ is the number of hints in the final list.

Construction of User Messages.
Based on the $j$-th hint $(\mathcal{A}_{(j)},K_{(j)},v_{(j)})\in H$, we construct the corresponding user message $m_{(j)}$ with LLM, where we have $m_{(j)}=LLM(\mathcal{A}_{(j)},K_{(j)},v_{(j)})$. Here, the LLM only serves the purpose of rewriting structural hints, without any reasoning process.
For example, if the hint is $(\textit{my uncle Bob},\textit{occupation},\textit{driver})$, the generated user message might be "The occupation of my uncle Bob is a driver". We generate user messages for all the hints in $H$, and we finally get the list of user messages $M=\left[m_{(j)}\right]_{j=1}^{l}$.

Construction of Questions and Answers.
In order to evaluate the memory capability of LLM-based personal assistants more comprehensively, we propose to construct five representative types of QAs to cover various complexities in real-world scenarios, as detailed in Table 1. For each question $q$, we provide three forms of ground truths: (1) the textual answer $a$ that can correctly respond to $q$, (2) the correct choice $a$ among confusing choices $a^{\prime}$ (generated by LLM) as a single-choice format, and (3) the correct retrieval target $h\subseteq M$ that contains the required factual information to the question.

**Table 1: Overview of comprehensive questions and answers.**
| Types | Descriptions | Examples | Causal Hints | Retrieval Target \bigstrut |
| --- | --- | --- | --- | --- |
| Single-hop | Rely on one message to answer the question directly. | Q: When is Alice’s birthday ?<br>A: June 1st. | $(\mathcal{A}_{(j)},K_{(j)},v_{(j)})$ | $\{m_{(j)}\}$ \bigstrut |
| Multi-hop | Require multiple messages to answer the question jointly. | Q: Where is the meeting that I will attend next week?<br>A: Victoria Conference Center. | $(\mathcal{A}^{t},K_{(j)},x_{(j)})$, $(\mathcal{A}^{t},K_{(k)},x_{(k)})$ | $\{m_{(j)},m_{(k)}\}$ \bigstrut |
| Comparative | Compare two entities on a shared attribute with multiple messages. | Q: Who is younger between Alice and Bob?<br>A: Bob. | $(\mathcal{A}_{(j)},K,v_{(j)})$, $(\mathcal{A}_{(k)},K,v_{(k)})$ | $\{m_{(j)},m_{(k)}\}$ \bigstrut |
| Aggregative | Aggregate messages about more than two entities on a common attribute. | Q: How many people are under 35 years old?<br>A: Three. | $\{(\mathcal{A}_{(j_{k})},K,v_{(j_{k})})\}_{k=1}^{d}$ | $\{m_{(j_{k})}\}_{k=1}^{d}$ \bigstrut |
| Post-processing | Involve extra reasoning steps to answer with multiple messages. | Q: What season was the teacher that I know born in?<br>A: Spring. | $(\mathcal{A}^{t},K_{(j)},v_{(j)})$, $(\mathcal{A}^{t},K_{(k)},v_{(k)})$ | $\{m_{(j)},m_{(k)}\}$ \bigstrut |

(i.) Single-hop QA.
Single-hop QA is the most basic type of QAs, relying on a single piece message to directly answer the question.
In constructing QA, we randomly select the $j$-th hint $(\mathcal{A}_{(j)},K_{(j)},v_{(j)})$ and generate the question $q=LLM(\mathcal{A}_{(j)},K_{(j)})$ through LLM rewriting, where the answer is $a=v_{(j)}$. Correspondingly, the retrieval target is $h=\{m_{(j)}\}$.

(ii.) Multi-hop QA.
Multi-hop QA necessitates the use of multiple messages to determine the correct answer, making it more complex than single-hop QA. In constructing Multi-hop QA, we first sample two hints $(\mathcal{A}_{(j)},K_{(j)},v_{(j)})$ and $(\mathcal{A}_{(k)},K_{(k)},v_{(k)})$ from the same bridge entity $\mathcal{A}^{t}$ (i.e., $\mathcal{A}^{t}=\mathcal{A}_{(j)}=\mathcal{A}_{(k)}$). We then mask this bridge entity and generate the question $q=LLM(K_{(j)},v_{(j)},K_{(k)})$ through LLM rewriting, where the answer is $a=v_{(k)}$. The target message set is $h=\{m_{(j)},m_{(k)}\}$. By incorporating additional entities, the questions can be easily extended to more hops.

(iii.) Comparative QA.
Comparative QA is an extensive type of multi-hop QA, which involves comparing two entities based on a shared attribute. We first select two hints $(\mathcal{A}_{(j)},K_{(j)},v_{(j)})$ and $(\mathcal{A}_{(k)},K_{(k)},v_{(k)})$ from different entities with the same meaning attribute $K$ (i.e., $\mathcal{A}_{j}\neq\mathcal{A}_{k}$ and $K\cong K_{(j)}\cong K_{(k)}$). We then rewrite the question $q=LLM(\mathcal{A}_{(j)},\mathcal{A}_{(k)},K)$ by LLM, where the answer $a=f(K,v_{(j)},v_{(k)})$ is derived from the function $f(\cdot)$. The retrieval target is $h=\{m_{(j)},m_{(k)}\}$.

(iv.) Aggregative QA.
Aggregative QA is a general type of comparative QA, which requires aggregating messages from more than two entities on a shared attribute. For construction, we choose $d$ hints $\{(\mathcal{A}_{(j_{k})},K,v_{(j_{k})})\}_{k=1}^{d}$ from different entities with the same meaning attribute $K$. Then, we construct the question $q=LLM(\{\mathcal{A}_{(j_{k})}\}_{j=1}^{d},K)$, where we obtain the answer $a=f(K,\{v_{(j_{k})}\}_{k=1}^{d})$. The target message set should include all these related references, that is, $h=\{m_{(j_{k})}\}_{k=1}^{d}$.

(v.) Post-processing QA.
Post-processing QA addresses situations where personal questions require additional reasoning steps for agents to answer, based on the retrieved messages.
We first select two hints $(\mathcal{A}_{(j)},K_{(j)},v_{(j)})$ and $(\mathcal{A}_{(k)},K_{(k)},v_{(k)})$ from the same bridge entity $\mathcal{A}^{t}$. We then design a reasoning factor $\psi$ to generate the question $q=LLM(K_{(j)},v_{(j)},K_{(k)},\psi)$, and derive the answer $a=f(K_{(k)},v_{(k)},\psi)$, where $\psi$ specifies the reasoning process. For example, it could be "the sum of the last five digits of the phone number $v_{(k)}$". Similarly, the retrieval target will be $h=\{m_{(j)},m_{(k)}\}$.

Infusion of Noise in User Messages.
We integrate two types of noise in user messages by concatenation, in order to simulate real-world circumstances.
The first type is entity-side noise, which refers to noisy messages that contain the selected attributes from unselected entities.
The second type is attribute-side noise, which involves noisy messages that describe unselected attributes of the selected entities.
Both types of noise can impact agents’ ability to retrieve messages and generate answers.

Eventually, we formulate the trajectory $\xi=(M,q,a,a^{\prime},h)$ by discarding all hints, where each trajectory serves as a test instance for evaluating the memory capability of LLM-based personal assistants. There are two insights into the causal generation mechanism. First, the factual information of messages and QAs are causally constructed from the shared hints that are sampled from user profiles, where LLMs are only responsible for rewriting based on the given information, rather than imagining or reasoning. This pipeline mitigates the impact of LLM hallucination on the factual information, keeping the reliability of QAs. It can also prevent contradictions among user messages from the same trajectory, because their hints are derived from the same user profile. Second, our method focuses on designing the asymmetric difficulty between constructing QAs (i.e., profiles$\rightarrow$hints$\rightarrow$messages, question and answer) and solving QAs (i.e., messages$|$question$\rightarrow$answer), which is critical for the automatic generation of evaluation datasets.

### 3.4 MemDaily: A Dataset in the Daily-life Scenario

Based on MemSim, we create a dataset in the daily-life scenario, named MemDaily, which can be used to evaluate the memory capability of LLM-based personal assistants, shown in Figure 1(c). Specifically, MemDaily incorporates 11 entities and 73 attributes (see details in Appendix D.1), all of which are representative and closely related to users’ daily lives.
We create 6 sub-datasets of different QA types mentioned previously: (1) Simple (Simp.): single-hop QAs. (2) Conditional (Cond.): multi-hop QAs with conditions. (3) Comparative (Comp.): comparative QAs. (4) Aggregative (Aggr.): aggregative QAs. (5) Post-processing (Post.): post-processing QAs. (6) Noisy: multi-hop QAs with additional irrelevant noisy texts inside questions.
The summary of MemDaily is shown in Table 2, where we present the number of trajectories, user messages, questions, and TPM (tokens per message). More details and examples can be found in Appendix D.

**Table 2: Summary of the MemDaily dataset.**
| Statistic | Simp. | Cond. | Comp. | Aggr. | Post. | Noisy | Total \bigstrut |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Trajectories | 500 | 500 | 492 | 462 | 500 | 500 | 2,954 \bigstrut[t] |
| Messages | 4215 | 4195 | 3144 | 5536 | 4438 | 4475 | 26,003 |
| Questions | 500 | 500 | 492 | 462 | 500 | 500 | 2,954 |
| TPM | 15.48 | 15.49 | 14.66 | 14.65 | 17.07 | 16.14 | 15.59 \bigstrut[b] |

## 4 Evaluations

In this section, we evaluate the quality of MemDaily, which can reflect the effectiveness of MemSim. Specifically, the evaluations are conducted in three parts: the user profiles, the user messages, and the constructed QAs. Besides, we also conduct comprehensive case studies in Appendix D.

### 4.1 Evaluation on User Profiles

The generated user profiles are supposed to express both rationality and diversity, which also directly influence the creation of user messages and QAs. Therefore, we evaluate these two aspects to reflect their quality.
Rationality means that the user profiles should possibly exist in the real world, with no internal contradictions in their descriptions.
Diversity indicates that the descriptions among users are distinct, covering a wide range of user types.

Metrics.
For rationality, we recruit six human evaluators to score the generated user profiles on a scale from 1 to 5.
Additionally, we use GPT-4o (^1^11[https://openai.com/index/hello-gpt-4o/](https://openai.com/index/hello-gpt-4o/)) as a reference for scoring. These two metrics are denoted as R-Human and R-GPT.
For diversity, we calculate the average Shannon-Wiener Index (SWI) [] on key attributes, using the following formula:

$$ $\displaystyle\text{SWI-$\mathcal{W}$}=-\frac{1}{|\mathcal{W}|}\sum_{X_{k}\in \mathcal{W}}\sum_{x_{i}\in X_{k}}p(x_{i})\ln p(x_{i}),$ $$

where $\mathcal{W}\subseteq\mathcal{X}$ is the subset of attribute variables. Therefore, we calculate SWI-R, SWI-O, and SWI-A, corresponding to role-relevant attributes, role-irrelevant attributes, and all attributes, respectively.

Baselines. We design several baselines to generate user profiles: (1) JointPL: prompting an LLM to generate attributes jointly. (2) SeqPL: prompting an LLM to generate attributes sequentially, conditioned on previous attributes in linear order. (3) IndePL: prompting an LLM to generate attributes independently. We compare our method with these baselines on generating user profiles.

Results.
As shown in Table 3, MemSim outperforms other baselines on R-Human, demonstrating the effectiveness of BRNet as an ablation study. However, we also observe an inconsistency between R-Human and R-GPT, which may be due to the inaccuracy of the LLM’s scoring []. Furthermore, our method achieves the highest diversity compared to the other baselines.

**Table 3: Results of the evaluation on user profiles.**
| Methods | R-Human | R-GPT | SWI-R | SWI-O | SWI-A \bigstrut |
| --- | --- | --- | --- | --- | --- |
| IndePL | 1.35±0.53 | 4.32 | 0.464 | 0.231 | 0.347 \bigstrut[t] |
| SeqPL | 1.64±0.73 | 4.40 | 1.471 | 1.416 | 1.443 |
| JointPL | 3.02±1.14 | 4.80 | 1.425 | 0.462 | 0.943 |
| MemSim | 4.91±0.30 | 4.68 | 3.206 | 2.895 | 3.050 \bigstrut[b] |

### 4.2 Evaluation on User Messages

We evaluate the quality of generated user messages in multiple aspects, including fluency, rationality, naturalness, informativeness, and diversity. The first four aspects are designed to assess the quality inside a trajectory, while the final one targets the variety across trajectories.

Metrics. For the inside-trajectory aspects, human evaluators score user messages on a scale from 1 to 5, denoted as F-Human (fluency), R-Human (rationality), N-Human (naturalness), and I-Human (informativeness). GPT-4o scores are also available and detailed in Appendix B. To assess the diversity across trajectories, we extract all entities and calculate their average Shannon-Wiener Index per 10,000 tokens of user messages, referred to as SWIP.

Baselines.
We implement several baselines that generate messages under different constraints regarding user profiles and tasks: (1) ZeroCons: no constraints on attributes when prompting LLMs. (2) PartCons: partial attributes of user profiles are constrained in prompts for LLMs. (3) SoftCons: full attributes of user profiles are constrained in prompts but they are not forcibly for generation. Our MemSim method imposes the most strict constraints, requiring both the integration of specific attributes into user messages and ensuring that questions are answerable with established ground truths based on the shared hints. Generally, higher constraint commonly means sacrifice of fluency and naturalness, because it compulsively imposes certain information to benefit QA constructions.

Results.
As shown in Table 4, our method maintains relatively high scores despite the rigorous constraints on constructing reliable QAs.
Additionally, MemSim exhibits the highest diversity index, attributed to the BRNet and the causal generation mechanism that produces a wider variety of user messages based on the provided hierarchical user profiles.

**Table 4: Results of the evaluation on user messages.**
| Methods | F-Human | R-Human | N-Human | I-Human | SWIP \bigstrut |
| --- | --- | --- | --- | --- | --- |
| ZeroCons | 4.94±0.24 | 4.94±0.24 | 4.85±0.35 | 2.82±1.15 | 2.712 \bigstrut[t] |
| PartCons | 4.98±0.14 | 4.94±0.37 | 4.97±0.18 | 4.01±1.18 | 6.047 |
| SoftCons | 4.93±0.30 | 4.80±0.77 | 4.91±0.42 | 4.37±0.98 | 5.868 |
| MemSim | 4.93±0.30 | 4.93±0.39 | 4.90±0.41 | 3.61±1.19 | 6.125 \bigstrut[b] |

### 4.3 Evaluation on Questions and Answers

The primary challenge for constructing a reliable dataset is ensuring the accuracy of ground truths for the constructed questions. To assess the reliability of MemDaily, we sample approximately 20% of all the trajectories in MemDaily and employ human evaluators to verify the correctness of their ground truths.
Specifically, the evaluators are required to examine three parts of the ground truths: textual answers, single-choice answers, and retrieval targets, and report their accuracy.

Metrics.
The accuracy of textual answers assesses whether an answer correctly responds to the question based on the user messages within the same trajectory.
The accuracy of single-choice answers indicates whether the ground truth choice is the sole correct answer for the question, given the user messages, while other choices are incorrect.
The accuracy of retrieval targets evaluates whether
the messages of the retrieval target are sufficient and necessary to answer the question.

Results.
As shown in Table 5, MemDaily significantly ensures the accuracy of the answers provided for constructed questions. In the few instances where accuracy is compromised, it is attributed to the rewriting process by LLMs, which occasionally leads to information deviation.
The results also demonstrate that MemSim can effectively mitigate the impact of LLM hallucinations on factual information, addressing a critical challenge in generating reliable questions and answers for memory evaluation.
Another baseline method that directly generates answers through LLMs based on targeted user messages and questions performs much lower reliability. We implement this method and present the results as OracleMem in our constructed benchmarks in Section 5.2.

**Table 5: Results of the evaluation on questions and answers.**
| Question Types | Textual Answers | Single-choice Answers | Retrieval Target \bigstrut |
| --- | --- | --- | --- |
| Simple | 100% | 98% | 100% \bigstrut[t] |
| Conditional | 100% | 100% | 100% |
| Comparative | 100% | 100% | 100% |
| Aggregative | 99% | 99% | 100% |
| Post-processing | 100% | 100% | 99% |
| Noisy | 100% | 100% | 100% \bigstrut[b] |
| Average | 99.8% | 99.5% | 99.8% \bigstrut |

## 5 Benchmark

In this section, we create a benchmark based on the MemDaily dataset, in order to evaluate the memory capability of LLM-based personal assistants. Our benchmark sets various levels of difficulty by introducing different proportions of question-irrelevant daily-life posts.

### 5.1 Experimental Settings

Levels of Difficulty.
We utilize the MemDaily dataset as the basis of our benchmark. In order to set different levels of difficulty, we collect question-irrelevant posts from social media platforms, and randomly incorporate them into user messages by controlling their proportions.
Specifically, we denote MemDaily-vanilla as the vanilla and easiest one without extra additions, and create a series of MemDaily-$\eta$, where we use $\eta$ to represent the inverse percentage of original user messages.
Larger $\eta$ indicates a higher level of difficulty in the benchmark.
We primarily focus on MemDaily-vanilla and MemDaily-100 as representatives. We also conduct evaluations on MemDaily-10, MemDaily-50, and MemDaily-200, putting their experimental results in Appendix C.

Baselines.
We implement several common memory mechanisms for LLM-based agents according to previous studies [], including
(1) Full Memory (FullMem): saves all previous messages as a list and concatenates them into the prompt for LLM inference.
(2) Recent Memory (ReceMem): maintains the most recent $k$ messages and concatenates them into the prompt for LLM inference, also referred to as short-term memory.
(3) Retrieved Memory (RetrMem): stores all previous messages using FAISS [] and retrieves the top-$k$ relevant messages for inclusion in the prompt for LLM inference, which is commonly used to construct long-term memory.
Specifically, we use Llama-160m [] to transform a message into a 768-dimensional embedding and compute relevance scores using cosine similarity [].
(4) None Memory (NonMem): does not use memory for LLM inference.
Additionally, we include two special baselines for reference:
(5) Noisy Memory (NoisyMem): receives only untargeted messages.
(6) Oracle Memory (OracleMem): receives only targeted messages. Here, the targeted messages indicate the messages in the ground truth retrieval target.
For all methods, we use the open-source GLM-4-9B [] as the foundational model for inference, as a result of its excellent ability in long-context scenarios.

Metrics.
We propose to evaluate the memory of LLM-based agents from two perspectives: effectiveness and efficiency.
Effectiveness refers to the agent’s ability to store and utilize factual information.
The metrics for effectiveness include: (1) Accuracy: The correctness of agents’ responses, measured by their ability to answer personal questions based on the factual information from historical user messages.
(2) Recall@5: The percentage of messages in retrieval target successfully retrieved within the top-5 relevant messages.
Efficiency mainly assesses the time cost associated with storing and utilizing information from memory.
We use two metrics to evaluate efficiency: (1) Response Time: The time taken for an agent to respond after receiving a query, covering the retrieval and utilization processes. (2) Adaptation Time: The time required for an agent to store a new message.

### 5.2 Effectiveness of Memory Mechanisms

**Table 6: Results of accuracy for factual question-answering.**
| MemDaily-vanilla    \bigstrut |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Methods | Simp. | Cond. | Comp. | Aggr. | Post. | Noisy \bigstrut |
| FullMem | 0.976±0.022 | 0.982±0.017 | 0.859±0.054 | 0.320±0.079 | 0.848±0.045 | 0.966±0.028 \bigstrut[t] |
| RetrMem | 0.898±0.048 | 0.882±0.040 | 0.771±0.078 | 0.317±0.061 | 0.800±0.054 | 0.786±0.040 |
| ReceMem | 0.832±0.080 | 0.798±0.046 | 0.631±0.069 | 0.257±0.040 | 0.760±0.051 | 0.764±0.042 |
| NonMem | 0.508±0.032 | 0.452±0.059 | 0.157±0.049 | 0.254±0.055 | 0.594±0.073 | 0.380±0.060 \bigstrut[b] |
| NoisyMem | 0.512±0.044 | 0.468±0.054 | 0.204±0.067 | 0.239±0.058 | 0.590±0.045 | 0.388±0.048 \bigstrut[t] |
| OracleMem | 0.966±0.020 | 0.988±0.013 | 0.910±0.032 | 0.376±0.057 | 0.888±0.053 | 0.984±0.017 \bigstrut[b] |
| MemDaily-100    \bigstrut |  |  |  |  |  |  |
| Methods | Simp. | Cond. | Comp. | Aggr. | Post. | Noisy \bigstrut |
| FullMem | 0.962±0.017 | 0.938±0.033 | 0.586±0.076 | 0.343±0.047 | 0.804±0.041 | 0.872±0.041 \bigstrut[t] |
| RetrMem | 0.892±0.034 | 0.840±0.036 | 0.706±0.074 | 0.320±0.092 | 0.770±0.055 | 0.726±0.052 |
| ReceMem | 0.500±0.063 | 0.442±0.058 | 0.104±0.048 | 0.257±0.054 | 0.600±0.060 | 0.386±0.076 |
| NonMem | 0.508±0.032 | 0.454±0.065 | 0.159±0.052 | 0.252±0.043 | 0.594±0.032 | 0.380±0.057 \bigstrut[b] |
| NoisyMem | 0.458±0.071 | 0.422±0.051 | 0.261±0.068 | 0.283±0.041 | 0.566±0.064 | 0.348±0.044 \bigstrut[t] |
| OracleMem | 0.966±0.020 | 0.988±0.016 | 0.912±0.045 | 0.372±0.062 | 0.888±0.038 | 0.984±0.012 \bigstrut[b] |

Accuracy of factual question-answering.
The results of accuracy are presented in Table 6. FullMem and RetrMem demonstrate superior performance compared to other memory mechanisms, achieving high accuracy across both datasets. ReceMem tends to underperform when a large volume of noisy messages is present, as target messages may fall outside the memory window.
We observe that agents excel with simple, conditional, post-processing, and noisy questions but struggle with comparative and aggregative questions.
By comparing with OracleMem, we find the primary difficulty possibly lies in retrieving target messages. Even with accurate retrieval, aggregative questions remain challenging, indicating a potential bottleneck in textual memory. An interesting phenomenon we notice is that NoisyMem shows higher accuracy than NonMem in MemDaily-vanilla but lower accuracy in MemDaily-100.
Similarly, FullMem unexpectedly outperforms OracleMem on simple questions in MemDaily. We suspect that LLMs may perform better with memory prompts of medium length, suggesting a potential limitation of textual memory mechanisms for LLM-based agents.

**Table 7: Results of recall@5 for target message retrieval.**
| MemDaily-vanilla    \bigstrut |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Methods | Simp. | Cond. | Comp. | Aggr. | Post. | Noisy \bigstrut |
| LLM | 0.888±0.025 | 0.851±0.020 | 0.947±0.018 | 0.544±0.021 | 0.800±0.028 | 0.846±0.036 \bigstrut[t] |
| Embedding | 0.735±0.064 | 0.717±0.041 | 0.845±0.022 | 0.515±0.059 | 0.693±0.033 | 0.648±0.018 |
| Recency | 0.514±0.052 | 0.513±0.038 | 0.698±0.034 | 0.237±0.026 | 0.511±0.053 | 0.504±0.047 \bigstrut[b] |
| MemDaily-100    \bigstrut |  |  |  |  |  |  |
| Methods | Simp. | Cond. | Comp. | Aggr. | Post. | Noisy \bigstrut |
| LLM | 0.612±0.021 | 0.479±0.037 | 0.683±0.036 | 0.290±0.027 | 0.439±0.047 | 0.430±0.059 \bigstrut[t] |
| Embedding | 0.698±0.049 | 0.653±0.061 | 0.778±0.048 | 0.490±0.037 | 0.567±0.042 | 0.543±0.034 |
| Recency | 0.002±0.003 | 0.003±0.004 | 0.002±0.003 | 0.000±0.001 | 0.002±0.003 | <0.001 \bigstrut[b] |

Recall of target message retrieval.
We implement three retrieval methods to obtain the most relevant messages and compare them with target messages to calculate Recall@5. Embedding refers to the retrieval process used in RetrMem. Recency considers the most recent $k$ messages as the result. LLM directly uses the LLM to respond with the top-$k$ relevant messages. The results are presented in Table 7. We find that LLM performs best in short-context scenarios, while Embedding achieves higher recall scores in longer contexts. Additionally, we notice that separating the retrieval and inference stages may exhibit different performances compared with integrating them.

### 5.3 Efficiency of Memory Mechanisms

The results of efficiency are presented in Table 8 and Table 9. We find that RetrMem consumes the most response time in short-context scenarios, and FullMem also requires more time for inference due to longer memory prompts. However, the response time of FullMem increases significantly faster than that of other methods as the context lengthens. Regarding adaptation time, we observe that RetrMem requires substantially more time because it needs to build indexes in the FAISS system.

**Table 8: Results of response time for generating answers (seconds per query).**
| MemDaily-vanilla    \bigstrut |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Methods | Simp. | Cond. | Comp. | Aggr. | Post. | Noisy \bigstrut |
| FullMem | 0.139±0.001 | 0.141±0.001 | 0.132±0.001 | 0.154±0.002 | 0.152±0.002 | 0.150±0.003 \bigstrut[t] |
| RetrMem | 0.290±0.007 | 0.277±0.007 | 0.267±0.009 | 0.236±0.009 | 0.257±0.004 | 0.284±0.007 |
| ReceMem | 0.126±0.001 | 0.127±0.001 | 0.125±0.000 | 0.125±0.001 | 0.135±0.001 | 0.134±0.001 |
| NonMem | 0.118±0.000 | 0.119±0.000 | 0.118±0.000 | 0.118±0.000 | 0.121±0.001 | 0.121±0.000 \bigstrut[b] |
| NoisyMem | 0.118±0.000 | 0.119±0.000 | 0.118±0.001 | 0.118±0.000 | 0.121±0.001 | 0.121±0.000 \bigstrut[t] |
| OracleMem | 0.122±0.001 | 0.122±0.001 | 0.122±0.000 | 0.131±0.001 | 0.129±0.002 | 0.128±0.001 \bigstrut[b] |
| MemDaily-100    \bigstrut |  |  |  |  |  |  |
| Methods | Simp. | Cond. | Comp. | Aggr. | Post. | Noisy \bigstrut |
| FullMem | 1.632±0.097 | 1.648±0.101 | 1.196±0.077 | 2.522±0.129 | 1.782±0.136 | 1.799±0.102 \bigstrut[t] |
| RetrMem | 0.207±0.020 | 0.223±0.005 | 0.228±0.011 | 0.205±0.008 | 0.228±0.029 | 0.284±0.022 |
| ReceMem | 0.120±0.000 | 0.125±0.008 | 0.121±0.001 | 0.120±0.000 | 0.125±0.001 | 0.124±0.001 |
| NonMem | 0.119±0.001 | 0.119±0.000 | 0.119±0.000 | 0.119±0.001 | 0.123±0.000 | 0.122±0.001 \bigstrut[b] |
| NoisyMem | 1.578±0.124 | 1.591±0.187 | 1.153±0.073 | 2.424±0.138 | 1.717±0.095 | 1.735±0.158 \bigstrut[t] |
| OracleMem | 0.122±0.001 | 0.123±0.001 | 0.123±0.001 | 0.132±0.001 | 0.130±0.001 | 0.129±0.001 \bigstrut[b] |

**Table 9: Results of adaptation time for storing messages (seconds per message).**
| MemDaily-vanilla    \bigstrut |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Methods | Simp. | Cond. | Comp. | Aggr. | Post. | Noisy \bigstrut |
| RetrMem | 0.222±0.009 | 0.182±0.004 | 0.151±0.009 | 0.136±0.010 | 0.133±0.004 | 0.112±0.005 \bigstrut[t] |
| Others | < 0.001 | < 0.001 | < 0.001 | < 0.001 | < 0.001 | < 0.001 \bigstrut[b] |
| MemDaily-100    \bigstrut |  |  |  |  |  |  |
| Methods | Simp. | Cond. | Comp. | Aggr. | Post. | Noisy \bigstrut |
| RetrMem | 0.064±0.008 | 0.072±0.004 | 0.066±0.007 | 0.064±0.006 | 0.056±0.002 | 0.066±0.005 \bigstrut[t] |
| Others | <0.001 | <0.001 | <0.001 | <0.001 | <0.001 | <0.001 \bigstrut[b] |

## 6 Limitations and Conclusions

In this paper, we propose MemSim, a Bayesian simulator designed to generate reliable datasets for evaluating the memory capability of LLM-based agents.
MemSim comprises two primary components: The bayesian Relation Network and the causal generation mechanism.
Utilizing MemSim, we generate MemDaily as a dataset in the daily-life scenario, and conduct extensive evaluations to assess its quality to reflect the effectiveness of MemSim.
Additionally, we provide a benchmark on different memory mechanisms of LLM-based agents and provide further analysis.
However, as the very initial study, there are several limitations. Firstly, our work focuses on evaluating the memory capability of LLM-based agents on factual information, but does not address higher-level and abstract information, such as users’ hidden hobbies and preferences. Additionally, our evaluation does not include dialogue forms, which are more complex and challenging to ensure reliability. In future works, we aim to address these two issues within the benchmark.

## Appendix A Proof in Bayesian Relation Network

### A.1 Proof of Theorem 1

###### Theorem 1 (Factorization) .

The joint probability distribution of BRNet can be expressed as

$$ $P(X_{1},X_{2},...,X_{|\mathcal{X}|})=\prod_{X_{t}\in\mathcal{X}}P(X_{t}|par(X_ {t})),$ $$

where $par(X_{t})$ denotes the set of parent attributes of $X_{t}$.

###### Proof .

Because BRNet is DAG, we can certainly find a topological ordering

$$ $O=\left[o_{1},o_{2},...,o_{|\mathcal{X}|}\right].$ $$

Then, we inverse the sequence to get a reversed topologically ordering

$$ $\tilde{O}=\left[\tilde{o}_{1},\tilde{o}_{2},...,\tilde{o}_{|\mathcal{X}|} \right].$ $$

Then, we utilize the theorem of conditional probability according to the order $\tilde{O}$, and we have

$$ $\displaystyle P(X_{1},X_{2},...,X_{|\mathcal{X}|})$ $\displaystyle=P(X_{\tilde{o}_{1}}|X_{\tilde{o}_{2}},...,X_{\tilde{o}_{| \mathcal{X}|}})\cdot P(X_{\tilde{o}_{2}}|X_{\tilde{o}_{3}},...,X_{\tilde{o}_{| \mathcal{X}|}})\ldots P(X_{\tilde{o}_{|\mathcal{X}|}}).$ $\displaystyle=\prod_{i=1}^{|\mathcal{X}|}P(X_{\tilde{o}_{i}}|\mathbf{X}\left[ \tilde{o}_{i+1}:\tilde{o}_{|\mathcal{X}|}\right]),$ $$

where $\mathbf{X}\left[\tilde{o}_{i+1}:\tilde{o}_{|\mathcal{X}|}\right]$ means all the variables after $\tilde{o}_{i+1}$ in the reversed topologically ordering, and there are no descendant variables inside.
According to Assumption 1, we have

$$ $P(X_{\tilde{o}_{i}}|\mathbf{X}\left[\tilde{o}_{i+1}:\tilde{o}_{|\mathcal{X}|} \right])=P(X_{\tilde{o}_{i}}|par(X_{\tilde{o}_{i}})).$ $$

Finally, we rewrite it and obtain

$$ $P(X_{1},X_{2},...,X_{|\mathcal{X}|})=\prod_{X_{t}\in\mathcal{X}}P(X_{t}|par(X_ {t})).$ $$

###### Theorem 1 (Factorization) .

###### Proof .

### A.2 Proof of Theorem 2

###### Theorem 2 (Ancestral Sampling) .

For BRNet, the result of ancestral sampling is equivalent to that of sampling from the joint probability distribution. Specifically, we have

$$ $P(\tilde{x}_{1},\tilde{x}_{2},...,\tilde{x}_{|\mathcal{X}|})=P(x_{1},x_{2},... ,x_{|\mathcal{X}|}),$ $$

where $x_{1},x_{2},...,x_{|\mathcal{X}|}\sim P(X_{1},X_{2},...,X_{|\mathcal{X}|})$ are sampled from the joint probability distribution.

###### Proof .

We first calculate the reversed topologically ordering

$$ $\tilde{O}=\left[\tilde{o}_{1},\tilde{o}_{2},...,\tilde{o}_{|\mathcal{X}|} \right].$ $$

Then, we have

$$ $\displaystyle P(\tilde{x}_{1},\tilde{x}_{2},...,\tilde{x}_{|\mathcal{X}|})$ $\displaystyle=\prod_{i=1}^{|\mathcal{X}|}P(\tilde{x}_{\tilde{o}_{i}}|\tilde{ \mathbf{x}}\left[\tilde{o}_{i+1}:\tilde{o}_{|\mathcal{X}|}\right])$ $\displaystyle=\prod_{i=1}^{|\mathcal{X}|}P(\tilde{x}_{\tilde{o}_{i}}|par( \tilde{x}_{\tilde{o}_{i}})).$ $$

where $\tilde{\mathbf{x}}\left[\tilde{o}_{i+1}:\tilde{o}_{|\mathcal{X}|}\right]$ means the values of all the variables after $\tilde{o}_{i+1}$ in the reversed topologically ordering. According to Assumption 2, we have

$$ $\displaystyle P(\tilde{x}_{1},\tilde{x}_{2},...,\tilde{x}_{|\mathcal{X}|})$ $\displaystyle=\prod_{i=1}^{|\mathcal{X}|}P(x_{\tilde{o}_{i}}|par(x_{\tilde{o}_ {i}}))$ $\displaystyle=P(x_{1},x_{2},...,x_{|\mathcal{X}|}).$ $$

###### Theorem 2 (Ancestral Sampling) .

###### Proof .

## Appendix B Extensive Evaluation on User Messages by GPT-4o

We also let GPT-4o score on user messages as a reference, and the results are shown in Table 10.

**Table 10: Results of evaluation on user messages by GPT-4o.**
| Methods | F-GPT | R-GPT | N-GPT | I-GPT \bigstrut |
| --- | --- | --- | --- | --- |
| ZeroCons | 4.04 | 4.80 | 4.60 | 3.04 \bigstrut[t] |
| PartCons | 4.28 | 4.88 | 4.80 | 4.28 |
| SoftCons | 4.20 | 5.00 | 5.00 | 3.96 |
| MemSim | 4.04 | 4.84 | 4.68 | 3.60 \bigstrut[b] |

## Appendix C Extensive Benchmark on More Composite Datasets

### C.1 Results on MemDaily-10

The results of accuracy are shown in Table 11. The results of recall@5 are shown in Table 12. The results of response time are shown in Table 13. The results of adaptation time are shown in Table 14.

**Table 11: Results of accuracy on MemDaily-10.**
| Methods | Simp. | Cond. | Comp. | Aggr. | Post. | Noisy \bigstrut |
| --- | --- | --- | --- | --- | --- | --- |
| FullMem | 0.962±0.040 | 0.966±0.028 | 0.665±0.058 | 0.243±0.072 | 0.810±0.036 | 0.922±0.029 \bigstrut[t] |
| RetrMem | 0.896±0.033 | 0.882±0.047 | 0.759±0.068 | 0.315±0.045 | 0.782±0.065 | 0.764±0.053 |
| ReceMem | 0.534±0.047 | 0.482±0.064 | 0.147±0.049 | 0.248±0.067 | 0.604±0.088 | 0.430±0.048 |
| NonMem | 0.510±0.090 | 0.450±0.078 | 0.159±0.041 | 0.254±0.065 | 0.594±0.032 | 0.380±0.057 \bigstrut[b] |
| NoisyMem | 0.428±0.068 | 0.402±0.059 | 0.169±0.046 | 0.280±0.046 | 0.584±0.090 | 0.350±0.077 \bigstrut[t] |
| OracleMem | 0.966±0.022 | 0.988±0.010 | 0.910±0.031 | 0.372±0.037 | 0.888±0.030 | 0.888±0.030 \bigstrut[b] |

**Table 12: Results of recall@5 on MemDaily-10.**
| Methods | Simp. | Cond. | Comp. | Aggr. | Post. | Noisy \bigstrut |
| --- | --- | --- | --- | --- | --- | --- |
| LLM | 0.794±0.035 | 0.872±0.019 | 0.518±0.027 | 0.732±0.036 | 0.756±0.038 | 0.846±0.036 \bigstrut[t] |
| Embedding | 0.704±0.039 | 0.833±0.026 | 0.506±0.052 | 0.643±0.043 | 0.609±0.027 | 0.648±0.018 |
| Recency | 0.032±0.017 | 0.011±0.010 | 0.013±0.011 | 0.030±0.012 | 0.009±0.007 | 0.504±0.047 \bigstrut[b] |

**Table 13: Results of response time on MemDaily-10 (seconds per query).**
| Methods | Simp. | Cond. | Comp. | Aggr. | Post. | Noisy \bigstrut |
| --- | --- | --- | --- | --- | --- | --- |
| FullMem | 0.243±0.008 | 0.243±0.008 | 0.208±0.003 | 0.306±0.008 | 0.263±0.006 | 0.262±0.010 \bigstrut[t] |
| RetrMem | 0.213±0.002 | 0.230±0.005 | 0.246±0.008 | 0.212±0.002 | 0.240±0.004 | 0.292±0.014 |
| ReceMem | 0.120±0.000 | 0.121±0.000 | 0.120±0.000 | 0.119±0.002 | 0.126±0.001 | 0.124±0.001 |
| NonMem | 0.119±0.000 | 0.119±0.001 | 0.119±0.000 | 0.117±0.002 | 0.122±0.000 | 0.119±0.002 \bigstrut[b] |
| NoisyMem | 0.205±0.005 | 0.207±0.007 | 0.181±0.004 | 0.253±0.010 | 0.223±0.005 | 0.222±0.006 \bigstrut[t] |
| OracleMem | 0.121±0.001 | 0.123±0.001 | 0.122±0.000 | 0.131±0.001 | 0.130±0.001 | 0.128±0.001 \bigstrut[b] |

**Table 14: Results of adaptation time on MemDaily-10 (seconds per message).**
| Methods | Simp. | Cond. | Comp. | Aggr. | Post. | Noisy \bigstrut |
| --- | --- | --- | --- | --- | --- | --- |
| FullMem | < 0.001 | < 0.001 | < 0.001 | < 0.001 | < 0.001 | < 0.001 \bigstrut[t] |
| RetrMem | 0.073±0.003 | 0.079±0.006 | 0.084±0.006 | 0.069±0.003 | 0.073±0.003 | 0.075±0.006 |
| ReceMem | < 0.001 | < 0.001 | < 0.001 | < 0.001 | < 0.001 | < 0.001 |
| NonMem | < 0.001 | < 0.001 | < 0.001 | < 0.001 | < 0.001 | < 0.001 \bigstrut[b] |
| NoisyMem | < 0.001 | < 0.001 | < 0.001 | < 0.001 | < 0.001 | < 0.001 \bigstrut[t] |
| OracleMem | < 0.001 | < 0.001 | < 0.001 | < 0.001 | < 0.001 | < 0.001 \bigstrut[b] |

### C.2 Results of MemDaily-50

The results of accuracy are shown in Table 15. The results of recall@5 are shown in Table 16. The results of response time are shown in Table 17. The results of adaptation time are shown in Table 18.

**Table 15: Results of accuracy on MemDaily-50.**
| Methods | Simp. | Cond. | Comp. | Aggr. | Post. | Noisy \bigstrut |
| --- | --- | --- | --- | --- | --- | --- |
| FullMem | 0.962±0.027 | 0.948±0.020 | 0.602±0.065 | 0.296±0.072 | 0.802±0.046 | 0.880±0.041 \bigstrut[t] |
| RetrMem | 0.886±0.035 | 0.864±0.037 | 0.724±0.062 | 0.320±0.071 | 0.780±0.059 | 0.748±0.049 |
| ReceMem | 0.508±0.042 | 0.434±0.052 | 0.108±0.044 | 0.237±0.054 | 0.588±0.066 | 0.376±0.099 |
| NonMem | 0.510±0.061 | 0.452±0.055 | 0.159±0.039 | 0.254±0.066 | 0.594±0.078 | 0.380±0.055 \bigstrut[b] |
| NoisyMem | 0.454±0.040 | 0.416±0.083 | 0.229±0.071 | 0.272±0.073 | 0.568±0.078 | 0.360±0.084 \bigstrut[t] |
| OracleMem | 0.966±0.025 | 0.988±0.010 | 0.910±0.053 | 0.376±0.042 | 0.888±0.032 | 0.984±0.012 \bigstrut[b] |

**Table 16: Results of recall@5 on MemDaily-50.**
| Methods | Simp. | Cond. | Comp. | Aggr. | Post. | Noisy \bigstrut |
| --- | --- | --- | --- | --- | --- | --- |
| LLM | 0.725±0.047 | 0.640±0.053 | 0.773±0.018 | 0.373±0.031 | 0.591±0.039 | 0.561±0.050 \bigstrut[t] |
| Embedding | 0.710±0.041 | 0.674±0.021 | 0.790±0.037 | 0.497±0.039 | 0.591±0.037 | 0.564±0.053 |
| Recency | 0.011±0.009 | 0.005±0.004 | 0.006±0.006 | 0.001±0.002 | 0.003±0.004 | 0.001±0.003 \bigstrut[b] |

**Table 17: Results of response time on MemDaily-50 (seconds per query).**
| Methods | Simp. | Cond. | Comp. | Aggr. | Post. | Noisy \bigstrut |
| --- | --- | --- | --- | --- | --- | --- |
| FullMem | 0.776±0.031 | 0.783±0.067 | 0.596±0.021 | 1.134±0.054 | 0.841±0.032 | 0.847±0.062 \bigstrut[t] |
| RetrMem | 0.203±0.003 | 0.206±0.004 | 0.215±0.004 | 0.204±0.003 | 0.229±0.005 | 0.324±0.020 |
| ReceMem | 0.120±0.001 | 0.121±0.002 | 0.118±0.000 | 0.118±0.001 | 0.123±0.002 | 0.123±0.001 |
| NonMem | 0.118±0.001 | 0.118±0.002 | 0.117±0.002 | 0.118±0.001 | 0.121±0.001 | 0.119±0.001 \bigstrut[b] |
| NoisyMem | 0.728±0.037 | 0.737±0.041 | 0.562±0.027 | 1.060±0.055 | 0.787±0.028 | 0.794±0.058 \bigstrut[t] |
| OracleMem | 0.121±0.001 | 0.122±0.001 | 0.121±0.001 | 0.131±0.001 | 0.129±0.001 | 0.128±0.001 \bigstrut[b] |

**Table 18: Results of adaptation time on MemDaily-50 (seconds per message).**
| Methods | Simp. | Cond. | Comp. | Aggr. | Post. | Noisy \bigstrut |
| --- | --- | --- | --- | --- | --- | --- |
| FullMem | < 0.001 | < 0.001 | < 0.001 | < 0.001 | < 0.001 | < 0.001 \bigstrut[t] |
| RetrMem | 0.059±0.001 | 0.057±0.003 | 0.057±0.004 | 0.060±0.003 | 0.062±0.003 | 0.089±0.005 |
| ReceMem | < 0.001 | < 0.001 | < 0.001 | < 0.001 | < 0.001 | < 0.001 |
| NonMem | < 0.001 | < 0.001 | < 0.001 | < 0.001 | < 0.001 | < 0.001 \bigstrut[b] |
| NoisyMem | < 0.001 | < 0.001 | < 0.001 | < 0.001 | < 0.001 | < 0.001 \bigstrut[t] |
| OracleMem | < 0.001 | < 0.001 | < 0.001 | < 0.001 | < 0.001 | < 0.001 \bigstrut[b] |

### C.3 Results of MemDaily-200

The results of accuracy are shown in Table 19. The results of recall@5 are shown in Table 20. The results of response time are shown in Table 21. The results of adaptation time are shown in Table 22.

**Table 19: Results of accuracy on MemDaily-200.**
| Methods | Simp. | Cond. | Comp. | Aggr. | Post. | Noisy \bigstrut |
| --- | --- | --- | --- | --- | --- | --- |
| FullMem | 0.932±0.040 | 0.932±0.036 | 0.563±0.061 | 0.309±0.056 | 0.782±0.045 | 0.866±0.044 \bigstrut[t] |
| RetrMem | 0.874±0.052 | 0.844±0.034 | 0.704±0.061 | 0.315±0.065 | 0.766±0.046 | 0.714±0.052 |
| ReceMem | 0.486±0.046 | 0.420±0.057 | 0.114±0.036 | 0.272±0.054 | 0.570±0.055 | 0.366±0.051 |
| NonMem | 0.470±0.057 | 0.454±0.077 | 0.157±0.045 | 0.257±0.069 | 0.592±0.082 | 0.380±0.048 \bigstrut[b] |
| NoisyMem | 0.398±0.052 | 0.398±0.068 | 0.282±0.058 | 0.276±0.068 | 0.564±0.037 | 0.350±0.035 \bigstrut[t] |
| OracleMem | 0.990±0.013 | 0.988±0.013 | 0.910±0.034 | 0.374±0.063 | 0.888±0.056 | 0.984±0.012 \bigstrut[b] |

**Table 20: Results of recall@5 on MemDaily-200.**
| Methods | Simp. | Cond. | Comp. | Aggr. | Post. | Noisy \bigstrut |
| --- | --- | --- | --- | --- | --- | --- |
| LLM | 0.457±0.066 | 0.356±0.051 | 0.556±0.035 | 0.176±0.022 | 0.342±0.048 | 0.322±0.043 \bigstrut[t] |
| Embedding | 0.674±0.052 | 0.641±0.044 | 0.753±0.036 | 0.484±0.050 | 0.544±0.054 | 0.508±0.052 |
| Recency | 0.001±0.003 | 0.001±0.002 | 0.001±0.002 | 0.000±0.001 | 0.001±0.003 | 0.000±0.000 \bigstrut[b] |

**Table 21: Results of response time on MemDaily-200 (seconds per query).**
| Methods | Simp. | Cond. | Comp. | Aggr. | Post. | Noisy \bigstrut |
| --- | --- | --- | --- | --- | --- | --- |
| FullMem | 4.028±0.161 | 3.914±0.213 | 2.697±0.100 | 6.365±0.374 | 4.252±0.328 | 4.307±0.283 \bigstrut[t] |
| RetrMem | 0.236±0.023 | 0.241±0.018 | 0.238±0.024 | 0.585±0.230 | 1.012±0.690 | 1.252±0.427 |
| ReceMem | 0.130±0.002 | 0.120±0.002 | 0.118±0.001 | 0.119±0.001 | 0.124±0.001 | 0.123±0.001 |
| NonMem | 0.139±0.006 | 0.119±0.001 | 0.119±0.001 | 0.117±0.001 | 0.121±0.001 | 0.121±0.001 \bigstrut[b] |
| NoisyMem | 3.947±0.209 | 3.832±0.203 | 2.637±0.118 | 6.221±0.325 | 4.158±0.226 | 4.214±0.288 \bigstrut[t] |
| OracleMem | 0.141±0.003 | 0.122±0.001 | 0.121±0.001 | 0.131±0.002 | 0.128±0.002 | 0.128±0.001 \bigstrut[b] |

**Table 22: Results of adaptation time on MemDaily-200 (seconds per message).**
| Methods | Simp. | Cond. | Comp. | Aggr. | Post. | Noisy \bigstrut |
| --- | --- | --- | --- | --- | --- | --- |
| FullMem | <0.001 | <0.001 | <0.001 | <0.001 | <0.001 | <0.001 \bigstrut[t] |
| RetrMem | 0.080±0.011 | 0.080±0.013 | 0.080±0.010 | 0.220±0.076 | 0.264±0.089 | 0.420±0.120 |
| ReceMem | <0.001 | <0.001 | <0.001 | <0.001 | <0.001 | <0.001 |
| NonMem | <0.001 | <0.001 | <0.001 | <0.001 | <0.001 | <0.001 \bigstrut[b] |
| NoisyMem | <0.001 | <0.001 | <0.001 | <0.001 | <0.001 | <0.001 \bigstrut[t] |
| OracleMem | <0.001 | <0.001 | <0.001 | <0.001 | <0.001 | <0.001 \bigstrut[b] |

## Appendix D Case Studies

In this section, we present several case studies to illustrate the effectiveness of the data generated by MemDaily. First, we will display the hierarchical user profiles generated from BRNet. Next, we will present examples of user messages created by our method. Finally, we will provide examples of questions and answers for each type.

### D.1 Case Study on Generated User Profiles

In MemDaily, we incorporate 11 entities that cover 7 types, with 73 attributes of them. The summary of entities and attributes of MemDaily are provided in Table 23.

We introduce prior knowledge as several rules according to our scenarios to constrain among attributes. For example, a relative role is highly possible to share the same hometown with the user, because they are likely to come from the same place. All of these constraints are expressed in BRNet with causal relations. We generate 50 graphical user profiles and conduct observations, finding that most profiles align well with real-world users without contradictions.

Here is a case of user profiles, and we translate them into English for better demonstration:

From the case profile in MemDaily, we find that our generated user profiles can greatly align with that in real-world scenarios.

**Table 23: Summary of entities and attributes of MemDaily.**
| Entity | Attribute | Entity | Attribute \bigstrut |
| --- | --- | --- | --- |
| User (self) | Gender | Relative Roles | Name \bigstrut[t] |
|  | Name |  | Age |
|  | Age |  | Height |
|  | Height |  | Birthday |
|  | Birthday |  | Hometown |
|  | Hometown |  | Workplace |
|  | Workplace |  | Education |
|  | Education |  | Occupation |
|  | Occupation |  | Position |
|  | Position |  | Company |
|  | Company |  | Hobbies |
|  | Hobbies |  | Personality |
|  | Personality |  | Phone |
|  | Phone |  | Email \bigstrut[b] |
|  | Email | Work Events | Type \bigstrut[t] |
|  | ID Number |  | Content |
|  | Passport Number |  | Location |
|  | Bank Card Number |  | Time |
|  | Driver’s License Number |  | Title \bigstrut[b] |
| College Roles | Gender |  | Scale \bigstrut[t] |
|  | Relationship |  | Duration \bigstrut[b] |
|  | Name | Entertainment Events | Type \bigstrut[t] |
|  | Age |  | Content |
|  | Height |  | Location |
|  | Birthday |  | Time |
|  | Hometown |  | Title |
|  | Workplace |  | Scale |
|  | Education |  | Duration \bigstrut[b] |
|  | Occupation | Places | Relationship \bigstrut[t] |
|  | Position |  | Type |
|  | Company |  | Name |
|  | Hobbies |  | Comment \bigstrut[b] |
|  | Personality | Items | Relationship \bigstrut[t] |
|  | Phone |  | Type |
|  | Email |  | Name \bigstrut[b] |
| Relative Roles | Gender |  | Comment \bigstrut |
|  | Relationship | Total (7) | Total (73) \bigstrut |

### D.2 Case Study on User Messages

Based on the generated user profiles, we further generate user messages without inside contradictory according to Section 3.3. Here is a case of message list (translated into English) in Table 24.

**Table 24: A case of user messages.**
| Index | Message | Time | Place |
| --- | --- | --- | --- |
| 0 | My colleague’s email is sunzhihong0421@huayinfinance.com. | April 1, 2024, Monday, 08:07 | Guangdong Shenzhen \bigstrut[t] |
| 1 | My colleague really likes to attend concerts. | April 2, 2024, Tuesday, 07:01 | Guangdong Shenzhen |
| 2 | My colleague’s phone number is 15391721618. | April 2, 2024, Tuesday, 08:23 | Guangdong Shenzhen |
| 3 | My colleague’s birthday is on April 21st. | April 2, 2024, Tuesday, 17:02 | Guangdong Shenzhen |
| 4 | My colleague’s name is Zhihong Sun. | April 3, 2024, Wednesday, 07:49 | Guangdong Shenzhen |
| 5 | Wei Zhang’s email address is zhangwei0715@westlakehospital.com. | April 3, 2024, Wednesday, 19:07 | Guangdong Shenzhen |
| 6 | Tingting Li’s email address is litingting0623@pdxzyz.com. | April 4, 2024, Thursday, 07:16 | Guangdong Shenzhen |
| 7 | Yalin Zhao’s email address is zhaoyalin0205@szfinancecenter.com. | April 4, 2024, Thursday, 13:38 | Guangdong Shenzhen |
| 8 | I am going to attend the bank teller job fair. | April 5, 2024, Friday, 16:21 | Guangdong Shenzhen |
| 9 | The time for the bank teller job fair is at seven o’clock in the evening on the next Sunday. | April 6, 2024, Saturday, 07:18 | Guangdong Shenzhen |
| 10 | The location of the bank teller job fair is in Guangdong Shenzhen. | April 6, 2024, Saturday, 16:58 | Guangdong Shenzhen |
| 11 | The main content of the bank teller job fair is the job fair: Shenzhen area bank head teller, sharing professional experience, recruiting talent, creating a brilliant bank career together. | April 7, 2024, Sunday, 07:21 | Guangdong Shenzhen |
| 12 | The time for the Financial Technology Elite Forum is at two o’clock in the afternoon next Saturday. | April 7, 2024, Sunday, 21:33 | Guangdong Shenzhen |
| 13 | The time for the Model Art Banquet is at seven o’clock in the evening next Monday. | April 8, 2024, Monday, 12:45 | Guangdong Shenzhen |
| 14 | The time for the Outdoor Hiking Model Creation Journey is at seven o’clock in the evening on the next Wednesday. | April 9, 2024, Tuesday, 07:36 | Guangdong Shenzhen \bigstrut[b] |

By utilizing our mechanisms, we can ensure that there is no contradiction among user messages.
We further demonstrate the list of hints that correspond to the above messages in Table 25.

**Table 25: A case of the hint list.**
| Index | Entity | Attribute | Value \bigstrut |
| --- | --- | --- | --- |
| 0 | Colleague Role 2 | Email | sunzhihong0421@huayinfinance.com \bigstrut[t] |
| 1 | Colleague Role 2 | Hobbies | Attend Concerts |
| 2 | Colleague Role 2 | Phone | 15391721618 |
| 3 | Colleague Role 2 | Birthday | April 21st |
| 4 | Colleague Role 2 | Name | Zhihong Sun |
| 5 | Relative Role 1 | Email | zhangwei0715@westlakehospital.com |
| 6 | Relative Role 2 | Email | litingting0623@pdxzyz.com |
| 7 | Colleague Role 1 | Email | zhaoyalin0205@szfinancecenter.com |
| 8 | Work Event 1 | Title | Bank Teller Job Fair; |
| 9 | Work Event 1 | Time | At 7 PM on the Sunday after next |
| 10 | Work Event 1 | Location | Shenzhen, Guangdong |
| 11 | Work Event 1 | Content | Job Fair for Bank Teller Supervisors in the Shenzhen area, sharing professional experience, recruiting talented individuals, and jointly creating a brilliant future for the banking industry |
| 12 | Work Event 2 | Time | Next Saturday at 2 PM; (Title) Financial Technology Elite Forum |
| 13 | Entertainment Event 1 | Time | At 7 PM on the coming Monday |
| 14 | Entertainment Event 2 | Time | The Wednesday evening at seven in two weeks \bigstrut[b] |

### D.3 Case Study on Questions and Answers

In this section, we will show the cases of questions and answers of different types. We leave out the time and place of each message in this section, where they do not influence the QA in these cases.
We have translated all texts into English for better demonstration.

Simple (Simp.) Simple QAs in single-hop.

Conditional (Cond.) Conditional QAs in multi-hop.

Comparative (Comp.) Comparative QAs in multi-hop.

Aggregative (Aggr.) Aggregative QAs in multi-hop.

Post-processing (Post.) Multi-hop QAs that requires extra reasoning steps.

Noisy (Nois.) Multi-hop QAs that add extra noise in questions.

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 3
presentation: 2
questions: - It is kind of confusing that the author mentions Baysian Relation Network. It seems to be a Probabilistic Graphical Model. Why the proof in Sec3.2 can give the conclusion:"we introduce prior knowledge of the specific scenario into the graphical structure and sampling process, which can improve the diversity and scalability of user profiles,"
- Where is the BRNet coming from? How is user profiles generated?
soundness: 2
strengths: - The paper adopts BRNet to sample user profile graphs and build question-answer pairs according to these graphs based on rules. It reduces the hallucinations compared to methods that construct samples directly.
-  Human evaluation seems to validate the effectiveness of rationality and diversity for generated results from the proposed MemSim.
summary: - The paper proposed MemSim, a simulator for automatically constructing diverse and rational QA pairs based on the Bayesian Relation Network. 
- The paper also constructs a MemDaily Dataset using MemSim to evaluate the memory mechanisms of LLMs.
weaknesses: - The human evaluation lacks a detailed description. How can an annotator give a diversity score to a sample without comparing all the samples in the produced dataset?
- The tables are hard to understand, no detailed description about the metric abbreviation. For example in Table 6,7.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1) See above
2) any prompts or implemention details for baselines and experiments?
3) notions are not clear.
soundness: 2
strengths: 1. The diversity, scalability and reliability of generated datasets can be improved since it mainly based on LLMs to automatically construct the data, and the user descriptions (a.k.a, messages) related to user profile is highly controllable.
2. The dataset considers several practical usage cases, including single-hop QA, multi-hop QA, comparative QA, aggregative QA and  post-processing QA.
summary: The paper proposes a data genertor based on Bayesian Relation Network and a causal generation mechanism, aiming to using LLMs to simulate users (i.e., generate user profiles / attrubutes) and generate evaluation datasets (i.e., generate lots of user descriptions based on previous sampled profile). Furthermore, the paper evaluate the quality of collected dataset -- MemDaily and provide performance analysis using GLM4-9B.
weaknesses: 1. lots of statements are not convincing. There are several examples: 1) line 64, the author claims the work is first work to evaluate memory of LLM-based personal assistants in an objective and automatic way. there are many studies evaluation memory usages in objective and automatic ways, such as [1]; 2) line 99, the authors keep emphasising the importance of "objective" evaluation, since they think the human annotator introduce bias. However, my personal concern is LLM also has bias, just like human. In this way, I would not say the objective evaluation is guaranteed.

2. baselines are too weak and experimental results are not convincing. The baselines used in both section 4.1 and 4.2 are too weak, and there are no implemention details. For examples, it is not hard to deign multi-stage but easier prompting strategy to generate user profiles instead of designing such complext sampling mechanisms. Even in this way, the performance gap in table 4 is not significant. The simple baseline leads to better performanc in most metrics. No detailed analysis is provided.

3. the value of dataset is not significant, and the unique advantages compared with existing datasets are not clear. there are several observations: 1) according to table 2, the TPM is around 15 and total messages is around 4000, then for each test instance, if we consider all user message as retrieval pool (note this is also not indicated in the paper), the max token number should be 15*4000 = 60000 token, while most of existing long-term memory benchmark consider much longer context, not even to mention if the pool becomes much smaller if the retrieval pool does not use all messages (so here many detailed statistics are missing). 2) user messages are generated using prompting given detailed user atttributes. Despite it can reduce hallucinations of LLMs, it also poses many constraints of LLMs and make the expression is not natural with real-world interections, such as the user may not explicitly talk about these attributes; 3) according to table 6, the simpliest baseline (RetrMem or FullMem) can achieve almost 80% accuracy, and FullMem can achieve 95%, which further support my claim that the dataset is relatively easy and not to mention the author does not use the existing SOTA model such as gpt4o or others.

[1] Hello Again! LLM-powered Personalized Agent for Long-term Dialogue

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. In section 3.2, how is the joint probability distribution of $X$ and the conditional probability distribution of $x$  determined? Do they need to be manually defined?
2. In MemDaily, there are only 11 entities and 73 attributes. Are these hand-crafted? If it is hand-crafted, how does the dataset scale up?
soundness: 3
strengths: 1. This paper is logically clear. It illustrates the BRNet with theoretical proof and uses multiple symbols to accurately describe causal generation mechanism. 
2. Using BRNet, this method eliminates the hallucination problem caused by LLM generation. Additionally, the causal generation mechanism guarantees the diversity of the datasets. 
3. Using this dataset, this paper evaluates different memory mechanisms of LLM agents and analyzes the results, which are insightful for future agent design.
summary: This paper proposes a method for automatically constructing reliable, diverse, and scalable QAs, MemSim. Based on a Bayesian simulator, MemSim first builds a Bayesian relation network and then designs a causal generation mechanism to produce various types of QAs, including single-hop, multi-hop, comparative, aggregative, and post-processing. Finally, this paper uses MemSim to generate a dataset named MemDaily to evaluate the memory mechanisms in LLM agents.
weaknesses: 1. There is a lack of comparison between MemSim and existing methods for QA generation. For example, generate personal questions through LLMs or build a personal KB and let the LLM generate messages based on the entities and relations. And then, evaluate the datasets using metrics in section 4.
2. The difference between BRNet and a KB is not clear. In section 2, this paper claims that KBQA mainly focuses on common-sense questions. However, building a personal or anonymous KB and allowing LLMs to generate datasets based on triples in the KB is also feasible for personal questions.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. In section 4.1 -- evaluation of user profiles, it will be good to mention the number of the total generated user profiles. In addition, Is each user profile evaluated by a single evaluator or multiple evaluators? 
2. When constructing the MemDaily dataset, it is not clear to me how the trajectory is constructed. 
3. It takes a while to understand the sentence in line 430 -- "Another baseline method that directly ... performs much lower reliability. We implement this method ... as OracleMem ...". The results of OracleMem are compared with the results in Table 5, rather than the other methods in Table 6. It will be better to make this clear. 
4. When evaluating the MemDaily dataset in section 4.3, how is the retrieval target obtained? According to section 4.3, the retrieval target seems to be part of the groundtruth when constructing the dataset. 
5. Related to question 4, in Table 6, are there any insights on why the performance of the OracleMem is much worse in some types of QAs? The OracleMem uses the targeted user message, which is not available when testing the memory machanism of other LLM-based agents. Therefore, shall we say that the results of the OracleMem are the upper bound of the model performance in this dataset? For the Aggregative type of question, the accuracy is 0.376. Is there other way that can further improve this performance? 
6. In line 485, "LLM directly uses the LLM to ... ", are the candidate messages and the question provided to LLM and let LLM decide the top-k relevant message?
soundness: 4
strengths: * This paper is well-written. The challenges of constructing dataset with considering the reliability, diversity and scalability set up a good motivation for the paper. 
* Both theoretical and experimental proofs are included to validate the effectiveness of the proposed method. 
* When generating the QAs, different types of QAs are considered. This ensures the diversity and coverage of the QAs. 
* The experiments are comprehensive: 1) Variations of the MemDaily dataset are also considered. 2) Different memory mechanisms are evaluated on the proposed dataset.
summary: This paper presents a novel method -- MemSim, to construct reliable QA datasets, to evaluate the memory capability of LLM-based personal assistants. A Bayesian Relation Network and a Causal Generation Mechanism are introduced to ensure the diversity, reliability, and scalability of the generated datasets. Based on MemSim, a dataset named MemDaily is constructed. Extensive experiments are conducted to assess the quality of the dataset, as well as evaluate the different memory mechanisms of LLM-based agents.
weaknesses: In the proposed method, the user messages are factual statements. And the constructed question-answers mainly focus on the entities/attributes. Each message in the trajectory seems to be independent. And there is no coreference between messages, no ambiguity of the user message. These greatly simplify the problem of evaluating personal assistants in a real-world scenario.

Meta-review: The authors present a framework to generate new questions from user messages to evaluate model memory capability.
The work seems technically sound, with questions about data generation and baselines addressed during rebuttal. The overall novelty and impact of the work however was not very obvious -- it seems like a fairly complex / bespoke approach that wasn't very well explained or fully justified in its complexity and broad applicability.

Decision: Reject