Title: MemSim: A Bayesian Simulator for Evaluating Memory of LLM-based Personal Assistants
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM-based Agents

Memory

Personal Assistants
Abstract: LLM-based agents have been widely applied as personal assistants, capable of memorizing information from user messages and responding to personal queries. However, there still lacks an objective and automatic evaluation on their memory capability, largely due to the challenges in constructing reliable questions and answers (QAs) according to user messages. In this paper, we propose MemSim, a Bayesian simulator designed to automatically construct reliable QAs from generated user messages, simultaneously keeping their diversity and scalability. Specifically, we introduce the Bayesian Relation Network (BRNet) and a causal generation mechanism to mitigate the impact of LLM hallucinations on factual information, facilitating the automatic creation of an evaluation dataset. Based on MemSim, we generate a dataset in the daily-life scenario, named MemDaily, and conduct extensive experiments to assess the effectiveness of our approach. We also provide a benchmark for evaluating different memory mechanisms in LLM-based agents with the MemDaily dataset.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents MemSim, an automated framework for evaluating the memory capabilities of large language model (LLM)-based personal assistants.  By constructing a Bayesian Relation Network (BRNet), the authors generate logically coherent and diverse user profiles.  These profiles serve as the foundation for a causal generation mechanism that simultaneously produces user messages and corresponding question-answer pairs from shared structured hints, mitigating hallucinations common in LLM-generated content.  Based on this approach, the authors build MemDaily, a comprehensive evaluation dataset that spans various memory task types, and introduce a benchmarking framework to evaluate the memory capability of LLM-based personal assistants.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.The introduction of the Bayesian Relation Network (BRNet) to simulate user profiles and model causal dependencies among attributes through a graph structure enhances the diversity and consistency of user information.\n\n2.The MemDaily dataset constructed based on MemSim covers six common types of memory tasks encountered in real-world scenarios, providing a unified benchmark for subsequent related research. \n\nWeaknesses:\n\n1.While the paper presents a systematic approach to constructing a factual QA dataset, the evaluation scope is limited to explicit and structured factual memory.  It overlooks higher-level memory types, such as dynamically evolving memory representations from long-term interactions, as well as implicit forms of memory like user preferences, intentions, and emotional tendencies.\n\n2.The authors attempt to mitigate hallucination issues in LLMs through a causality-based generation mechanism guided by structured hints.   However, the generation of user messages and questions still heavily relies on the language modeling capabilities of LLMs, and no real user dialogues or log data are used for validation.   As a result, the evaluation may not accurately reflect the memory capabilities of LLMs in open-domain settings.\n\n3.The paper relies heavily on human subjective ratings, which are treated as the primary evaluation metric.   This approach is somewhat unconvincing due to its strong subjectivity and limited reproducibility.   A more robust evaluation strategy would prioritize structural consistency or other quantitative metrics, using human judgment as a supplementary measure.   Besides, the paper should clearly specify the human evaluation process, including the criteria, procedures, and consistency scores\n\nQUESTIONS:\n1.It would be helpful if the authors could elaborate on the motivation for choosing BRNet over other data modeling methods (e.g., knowledge graph-based sampling). The paper would be strengthened by either (1) providing theoretical or empirical evidence that BRNet yields higher diversity or consistency in user profiles, or (2) including a comparative experiment demonstrating that datasets generated with BRNet outperform those generated by alternative methods in terms of QA reliability and coverage.\n\n2.Although the authors designed multiple internal baselines to compare dataset attributes, all of these baselines rely solely on their self-constructed simulation environment.  The lack of horizontal comparison with existing benchmarks—such as the EpisodicQA-generated datasets [1]—limits the persuasiveness of the proposed method.  [1] Huet A, Houidi Z B, Rossi D. Episodic Memories Generation and Evaluation Benchmark for Large Language Models. arXiv preprint arXiv:2501.13121, 2025.\n\n3.It may be beneficial to include datasets that are more reflective of real-world scenarios—such as multi-turn dialogue settings (e.g., a user says “I’m going to see the dentist” today, and three days later mentions “The doctor reminded me to check for cavities”)—to better evaluate whether the method can maintain and retrieve information consistently in context.\n\n4.The current benchmark evaluation (Section 5.2) is conducted solely using the GLM-4-9B model, which may limit the generalizability of the findings. It is recommended to include evaluations using multiple LLMs (e.g., LLaMA) to provide a more comprehensive assessment of the dataset’s robustness and applicability.\n\nLIMITATIONS:\nThe evaluation protocol relies heavily on subjective human ratings and lacks a standardized, reproducible scoring framework, which may undermine the consistency and generalizability of the results.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MemSim, a Bayesian simulator designed to automatically generate reliable datasets for evaluating the memory of LLM-based personal assistants, thereby addressing the high cost of human annotation and the hallucination issues in existing automatic methods. MemSim leverages a Bayesian Relation Network to model user profiles with causal dependencies and a novel generation mechanism to create user messages and question-answer pairs from shared, structured \"hints.\" Using this simulator, the authors created the MemDaily dataset to test various memory mechanisms, revealing that current LLM assistants struggle significantly with complex comparative and aggregative questions, especially when faced with noisy or irrelevant information.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1: The paper addresses a genuine and important problem in LLM agent evaluation. The technical approach is well-motivated and the experimental evaluation is comprehensive, including both dataset quality assessment and memory mechanism benchmarking.\n\n2: The combination of Bayesian networks for user profile generation and causal generation mechanisms is novel for this application domain.\n\nWeakness:\n1: While the authors claim scalability, the approach requires manual design of entity-attribute relationships and causal structures for each domain, which may limit generalizability.\n\n2: Disconnect from Recent Advances: The paper doesn't connect with recent developments in agentic systems like tool use, reflection mechanisms, or planning capabilities that are increasingly important for personal assistants with memories. (e.g. only tests retrieval and basic reasoning, missing evaluation of how memory integrates with other agent capabilities like tool calling or multi-step planning). Also as for memory systems, modern memory systems often incorporate semantic similarity, knowledge graphs, or learned embeddings. The lack of such comparison limits the scope of this paper and I am skeptical about its value to the community given the rapid advancement in modern AI Agents.\n\nQUESTIONS:\n1: Your current evaluation treats memory as static fact retrieval, but real personal assistants need to handle evolving user information, conflicting updates, and temporal dependencies. How would you extend MemDaily to evaluate these dynamic aspects of memory management?\n\n2: You claim the approach is scalable, but BRNet requires manual specification of entities, attributes, and causal relationships for each domain. Can you provide concrete evidence of how easily this transfers to new domains (e.g., professional contexts, healthcare scenarios)? What would be required to adapt MemSim to evaluate memory in coding assistants or research agents?\n\n3: While you position this as an evaluation methodology paper, the core technical contributions appear to be primarily about reliable dataset generation with BRnet sampling and causal strurcual prompting method rather than novel evaluation metrics or frameworks. What makes this fundamentally different from a benchmark paper that happens to introduce a novel dataset generation method?\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMemSim is a Bayesian simulator that automatically generate reliable, diverse, and scalable evaluation datasets MemDaily.  MemDaily assesses the memory capabilities of LLM-based personal assistants. MemSim uses BRNet, which models hierarchical user profiles by sampling entities and their attributes in a causally structured, loop-free graph. A causal generation mechanism uses these profiles to produce user messages and construct QA pairs to reduce hallucination. MemDaily comprises six QA types, reflecting real-world query complexity. Extensive human and automated evaluations demonstrate that MemDaily maintains high accuracy, fluency, and diversity. The authors benchmark various memory mechanisms on this dataset, including full memory, recent memory, and retrieved memory, measuring both effectiveness and efficiency.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1.\tMemSim’s pipeline is automatic, which saves expensive human annotation. It generates QAs from simulated user profiles, which enables large-scale and reproducible memory evaluation.\n2.\tThe proposed method derives user messages and QAs from the same hints sampled from BRNet. This method effectively reduces hallucination as shown in Table 5.\n3.\tThe authors propose a comprehensive QA taxonomy, including the 6 QA types (simple, conditional, comparative, aggregative, post-processing, and noisy). This taxonomy covers a wide range of reasoning scenarios, which provides a better and more detailed evaluation guideline for memory mechanisms.\n4.\tMemDaily dataset has a high diversity in the simulated profiles and messages, as proven by its Shannon–Wiener score. It shows that BRNet’s hierarchical sampling and the causal mechanism are useful in generating varied user profiles.\n5.\tThis papers benchmarks the memory mechanisms in an end-to-end manner, including evaluateing multiple memory strategies on both vanilla and noisy variants of the dataset. The authors conduct extensive experiments and provide clear metrics for effectiveness and efficiency.\n\n\nWeaknesses:\n1.\tThe experiments uses MemDaily to evaluate only GLM-4-9B. It would be better to evaluate models from different families or models of different sizes. A comparison of the results from different models can better showcase the benchmark’s ability to evaluate models.\n2.\tAlthough you mention in check point 14 (Line824 to 838) that you have included full text instructions given to participants, I could not  find any from the paper, including appendix. This brings that problem that your human evaluation to user profile (line236 - 237) and user messages (Line 254 to 255) become very unclear. Human evaluators give scores scale from 1 – 5, but what is a score 1 profile like? What is a score 5 profile like? What aspects should the evaluator look into a profile? Would evaluator A’s score 3 be the same as evaluator B’s score 3? Without the guideline, your human evaluations are very weak.\n3.\tSome generated questions are still very artificial, especially the multi-hop questions and post-processing questions. For instance, in appendix F3 (Line 534 - 535), “where does the person with only a high school education work now?” or (line 540-541),” Which of the following descriptions matches the work location of the person whose birthday is July 15th?” I hardly think a question like this will appear from any real user. There is still a gap in the distribution of actual encountered questions and the generated ones.\n4.\tYour method of controlling dataset quality is through adding more noise from social media. However, since your useful information are model generated, they naturally have a different distribution from the real message from social media. And hence the negative samples inserted are not high-quality negatives. Hence the difference of performance between different difficulties are not very obvious (table 11 – table 22)\n5.\tThere are some further areas of improvement. \nLike a) including a memory editing mechanism in the benchmark through dynamic dialogue\nAnd b) your college and relative entities are a little toyed. (line 518-522, table 23). In real life a person’s relationship is far beyond colleagues and relatives. It would be better to set up the colleges/relatives also as individual entities, and each entity can be a (self), the user entity can be connected to many other entities with more complex relationships.\n\nQUESTIONS:\nsee Weaknesses\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MemSim, a Bayesian simulator that automatically generates diverse and scalable question-answer pairs. These pairs are designed for evaluating the memory of LLM-based personal assistants. The authors highlight the lack of objective and automatic evaluation methods for this task.\n\nMemSim uses a Bayesian Relation Network (BRNet) and a causal generation mechanism. This helps reduce hallucinations from LLMs when creating factual information. Based on MemSim, a new dataset called MemDaily is created, simulating daily-life scenarios. The paper provides a benchmark using MemDaily to evaluate different memory mechanisms in LLM-based agents. The authors state their work is the first to offer an objective and automatic way to evaluate LLM memory.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. Addresses a Key Gap: This paper fills a critical void by introducing an objective and automatic evaluation method for LLM-based personal assistant memory.\n2. Novel, Scalable Data Generation: MemSim, with its BRNet and causal mechanism, offers an innovative approach to automatically and scalably generate reliable evaluation data, mitigating LLM hallucinations.\n3. Valuable Dataset & Open-Source: The new MemDaily dataset is a significant contribution. Its open-source release promotes reproducibility and future research.\n\n### Weaknesses\n\n1. Limited Cross-Dataset Comparison: The paper lacks explicit comparisons with existing public datasets, making it difficult to fully contextualize performance. While the authors note the lack of previous datasets for this specific task, comparison with less related existing datasets would also be valuable.\n2. Insufficient Detail on Data Generation Prompts: The paper does not provide the specific prompts used for data generation, nor does it analyze the sensitivity of data reliability to different LLMs or prompting strategies.\n\nQUESTIONS:\n1.  How do variations in the prompts used for data generation (e.g., user messages, facts, questions) impact the quality and characteristics of the generated dataset? Furthermore, what is the impact of using different base LLMs?\n2.  Could the proposed MemSim methodology be adapted to generate memory-specific evaluation data based on existing datasets, rather than purely synthetic ones?\n3.  In the Bayesian Relation Network (BRNet), is the directed graph structure manually designed (e.g., based on pre-defined rules) or is it automatically learned/generated? If the attribute set is very large and causal relationships are complex, would the graph construction process become challenging?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-08"}