MemSim: A Bayesian Simulator for Evaluating Memory of LLM-based Personal Assistants

Authors: Zeyu Zhang, Quanyu Dai, Luyu Chen, Zeren Jiang, Rui Li, Jieming Zhu, Xu Chen, Yi Xie, Zhenhua Dong, Ji-Rong Wen
Primary area: datasets and benchmarks
Keywords: LLM-based agent, memory, evaluation, personal assistant

Abstract:
LLM-based agents have been widely applied as personal assistants, capable of memorizing information from user messages and responding to personal queries. However, there still lacks an objective and automatic evaluation on their memory capability, largely due to the challenges in constructing reliable questions and answers (QAs) according to user messages. In this paper, we propose MemSim, a Bayesian simulator designed to automatically construct reliable QAs from generated user messages, simultaneously keeping their diversity and scalability. Specifically, we introduce the Bayesian Relation Network (BRNet) and a causal generation mechanism to mitigate the impact of LLM hallucinations on factual information, facilitating the automatic creation of an evaluation dataset. Based on MemSim, we generate a dataset in the daily-life scenario, named MemDaily, and conduct extensive experiments to assess the effectiveness of our approach. We also provide a benchmark for evaluating different memory mechanisms in LLM-based agents with the MemDaily dataset.

Decision: reject
Reviewer scores: 5, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1730518078877

thread_type:
official_review

<thread_object>
creator: reviewer_cyvc

<content>
summary:
This paper presents a novel method -- MemSim, to construct reliable QA datasets, to evaluate the memory capability of LLM-based personal assistants. A Bayesian Relation Network and a Causal Generation Mechanism are introduced to ensure the diversity, reliability, and scalability of the generated datasets. Based on MemSim, a dataset named MemDaily is constructed. Extensive experiments are conducted to assess the quality of the dataset, as well as evaluate the different memory mechanisms of LLM-based agents.

strengths:
* This paper is well-written. The challenges of constructing dataset with considering the reliability, diversity and scalability set up a good motivation for the paper. 
* Both theoretical and experimental proofs are included to validate the effectiveness of the proposed method. 
* When generating the QAs, different types of QAs are considered. This ensures the diversity and coverage of the QAs. 
* The experiments are comprehensive: 1) Variations of the MemDaily dataset are also considered. 2) Different memory mechanisms are evaluated on the proposed dataset.

weaknesses:
In the proposed method, the user messages are factual statements. And the constructed question-answers mainly focus on the entities/attributes. Each message in the trajectory seems to be independent. And there is no coreference between messages, no ambiguity of the user message. These greatly simplify the problem of evaluating personal assistants in a real-world scenario.

questions:
1. In section 4.1 -- evaluation of user profiles, it will be good to mention the number of the total generated user profiles. In addition, Is each user profile evaluated by a single evaluator or multiple evaluators? 
2. When constructing the MemDaily dataset, it is not clear to me how the trajectory is constructed. 
3. It takes a while to understand the sentence in line 430 -- "Another baseline method that directly ... performs much lower reliability. We implement this method ... as OracleMem ...". The results of OracleMem are compared with the results in Table 5, rather than the other methods in Table 6. It will be better to make this clear. 
4. When evaluating the MemDaily dataset in section 4.3, how is the retrieval target obtained? According to section 4.3, the retrieval target seems to be part of the groundtruth when constructing the dataset. 
5. Related to question 4, in Table 6, are there any insights on why the performance of the OracleMem is much worse in some types of QAs? The OracleMem uses the targeted user message, which is not available when testing the memory machanism of other LLM-based agents. Therefore, shall we say that the results of the OracleMem are the upper bound of the model performance in this dataset? For the Aggregative type of question, the accuracy is 0.376. Is there other way that can further improve this performance? 
6. In line 485, "LLM directly uses the LLM to ... ", are the candidate messages and the question provided to LLM and let LLM decide the top-k relevant message?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730645897576

thread_type:
official_review

<thread_object>
creator: reviewer_o28w

<content>
summary:
This paper proposes a method for automatically constructing reliable, diverse, and scalable QAs, MemSim. Based on a Bayesian simulator, MemSim first builds a Bayesian relation network and then designs a causal generation mechanism to produce various types of QAs, including single-hop, multi-hop, comparative, aggregative, and post-processing. Finally, this paper uses MemSim to generate a dataset named MemDaily to evaluate the memory mechanisms in LLM agents.

strengths:
1. This paper is logically clear. It illustrates the BRNet with theoretical proof and uses multiple symbols to accurately describe causal generation mechanism. 
2. Using BRNet, this method eliminates the hallucination problem caused by LLM generation. Additionally, the causal generation mechanism guarantees the diversity of the datasets. 
3. Using this dataset, this paper evaluates different memory mechanisms of LLM agents and analyzes the results, which are insightful for future agent design.

weaknesses:
1. There is a lack of comparison between MemSim and existing methods for QA generation. For example, generate personal questions through LLMs or build a personal KB and let the LLM generate messages based on the entities and relations. And then, evaluate the datasets using metrics in section 4.
2. The difference between BRNet and a KB is not clear. In section 2, this paper claims that KBQA mainly focuses on common-sense questions. However, building a personal or anonymous KB and allowing LLMs to generate datasets based on triples in the KB is also feasible for personal questions.

questions:
1. In section 3.2, how is the joint probability distribution of $X$ and the conditional probability distribution of $x$  determined? Do they need to be manually defined?
2. In MemDaily, there are only 11 entities and 73 attributes. Are these hand-crafted? If it is hand-crafted, how does the dataset scale up?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730648620997

thread_type:
official_review

<thread_object>
creator: reviewer_zuqb

<content>
summary:
The paper proposes a data genertor based on Bayesian Relation Network and a causal generation mechanism, aiming to using LLMs to simulate users (i.e., generate user profiles / attrubutes) and generate evaluation datasets (i.e., generate lots of user descriptions based on previous sampled profile). Furthermore, the paper evaluate the quality of collected dataset -- MemDaily and provide performance analysis using GLM4-9B.

strengths:
1. The diversity, scalability and reliability of generated datasets can be improved since it mainly based on LLMs to automatically construct the data, and the user descriptions (a.k.a, messages) related to user profile is highly controllable.
2. The dataset considers several practical usage cases, including single-hop QA, multi-hop QA, comparative QA, aggregative QA and  post-processing QA.

weaknesses:
1. lots of statements are not convincing. There are several examples: 1) line 64, the author claims the work is first work to evaluate memory of LLM-based personal assistants in an objective and automatic way. there are many studies evaluation memory usages in objective and automatic ways, such as [1]; 2) line 99, the authors keep emphasising the importance of "objective" evaluation, since they think the human annotator introduce bias. However, my personal concern is LLM also has bias, just like human. In this way, I would not say the objective evaluation is guaranteed.

2. baselines are too weak and experimental results are not convincing. The baselines used in both section 4.1 and 4.2 are too weak, and there are no implemention details. For examples, it is not hard to deign multi-stage but easier prompting strategy to generate user profiles instead of designing such complext sampling mechanisms. Even in this way, the performance gap in table 4 is not significant. The simple baseline leads to better performanc in most metrics. No detailed analysis is provided.

3. the value of dataset is not significant, and the unique advantages compared with existing datasets are not clear. there are several observations: 1) according to table 2, the TPM is around 15 and total messages is around 4000, then for each test instance, if we consider all user message as retrieval pool (note this is also not indicated in the paper), the max token number should be 15*4000 = 60000 token, while most of existing long-term memory benchmark consider much longer context, not even to mention if the pool becomes much smaller if the retrieval pool does not use all messages (so here many detailed statistics are missing). 2) user messages are generated using prompting given detailed user atttributes. Despite it can reduce hallucinations of LLMs, it also poses many constraints of LLMs and make the expression is not natural with real-world interections, such as the user may not explicitly talk about these attributes; 3) according to table 6, the simpliest baseline (RetrMem or FullMem) can achieve almost 80% accuracy, and FullMem can achieve 95%, which further support my claim that the dataset is relatively easy and not to mention the author does not use the existing SOTA model such as gpt4o or others.

[1] Hello Again! LLM-powered Personalized Agent for Long-term Dialogue

questions:
1) See above
2) any prompts or implemention details for baselines and experiments?
3) notions are not clear.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730655199707

thread_type:
official_review

<thread_object>
creator: reviewer_vwqn

<content>
summary:
- The paper proposed MemSim, a simulator for automatically constructing diverse and rational QA pairs based on the Bayesian Relation Network. 
- The paper also constructs a MemDaily Dataset using MemSim to evaluate the memory mechanisms of LLMs.

strengths:
- The paper adopts BRNet to sample user profile graphs and build question-answer pairs according to these graphs based on rules. It reduces the hallucinations compared to methods that construct samples directly.
-  Human evaluation seems to validate the effectiveness of rationality and diversity for generated results from the proposed MemSim.

weaknesses:
- The human evaluation lacks a detailed description. How can an annotator give a diversity score to a sample without comparing all the samples in the produced dataset?
- The tables are hard to understand, no detailed description about the metric abbreviation. For example in Table 6,7.

questions:
- It is kind of confusing that the author mentions Baysian Relation Network. It seems to be a Probabilistic Graphical Model. Why the proof in Sec3.2 can give the conclusion:"we introduce prior knowledge of the specific scenario into the graphical structure and sampling process, which can improve the diversity and scalability of user profiles,"
- Where is the BRNet coming from? How is user profiles generated?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732327063580

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Kindly remind and eagerly await feedback

comment:
Dear Reviewers,

We sincerely appreciate the time and effort dedicated by all reviewers in reviewing our paper.

**We have addressed all concerns one by one and clarified the misunderstandings raised in the reviews. As the discussion deadline approaches, we eagerly await your feedback on our responses.**

We would appreciate the opportunity to address any remaining concerns that you may still have.



Sincerely,

Submission1435 Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors present a framework to generate new questions from user messages to evaluate model memory capability.
The work seems technically sound, with questions about data generation and baselines addressed during rebuttal. The overall novelty and impact of the work however was not very obvious -- it seems like a fairly complex / bespoke approach that wasn't very well explained or fully justified in its complexity and broad applicability.