Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale
Authors: Bowen Jiang
Zhuoqun Hao
Young Min Cho
Bryan Li
Yuan Yuan
Sihao Chen
Lyle Ungar
Camillo Jose Taylor
Dan Roth
Venue: COLM 2025 (2025)
Keywords: personalization
long context
memory
conversational chatbot
TL;DR: We introduce the PERSONAMEM benchmark to evaluate state-of-the-art LLMs’ ability to infer a user’s profile and how it evolves, as well as whether LLMs can apply what they learn about the user to provide personalized responses across task scenarios.

Abstract: Large Language Models (LLMs) have emerged as personalized assistants for users across a wide range of tasks – from offering writing support to delivering tailored recommendations or consultations. Over time, the interaction history between a user and an LLM can provide extensive information about an individual’s traits and preferences. However, open questions remain on how well LLMs today can effectively leverage such history to (1) internalize the user’s inherent traits and preferences, (2) track how the user profiling and preferences evolve over time, and (3) generate personalized responses accordingly in new scenarios.

In this work, we introduce the PERSONAMEM benchmark. PERSONAMEM features curated user profiles with over 180 simulated user-LLM interaction histories, each containing up to 60 sessions of multi-turn conversations across 15 real-world tasks that require personalization. Given an in-situ user query at a specific time point, we evaluate LLM chatbots’ ability to identify the most suitable response according to the current state of the user’s profile. We observe that current LLMs still struggle to recognize the dynamic evolution in users’ profiles over time through direct prompting approaches. As a consequence, LLMs often fail to deliver responses that align with users’ current situations and preferences, with frontier models such as GPT-4.5, or Gemini-2.0 achieving only around 50% overall accuracy, suggesting room for improvement. We hope that PERSONAMEM, along with the user profile and conversation simulation pipeline, can facilitate future research in the development of truly user-aware chatbots.

Published as a conference paper at COLM 2025
Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale
Bowen Jiang"; Zhuoqun H Young-Min Cho', Bryan Li", Yuan Yuan', Sihao Chen², Lyle Ungart, Camillo J. Taylor Dan Roth
University of Pennsylvania, Philadelphia, PA1 Microsoft, Redmond, WA2
{bwjlang, zhuoqunh, jcho, bryanli, shaochen@microsoft..c {ungar, cjtaylor, danroth]@upenn.ced
edu
Abstract
Large Language Models (LLMs) have emerged as personalized assistants for users across wide range of tasks from offering writing support delivering tailored recommendations or consultations. Over time, the interaction history between user and an LLM can provide extensive information about an individual's traits and preferences. However, open questions remain how well LLMs today can effectively leverage such history to (1) internalize the user's 's inherent traits and preferences, (2) track how the profiling and preferences evolve over time, and (3) generate personalized responses accordingly in new scenarios.
In this work, we introduce the PERSONAMEM benchmark. PERSON- AMEM features curated user profiles with over 180 simulated user-LLM interaction histories, each containing up to 60 sessions of multi-turn con- versations across 15 real-world tasks that require personalization. Given an in-silu user query, i.e. query issued by the user from the first-person perspective, we evaluate LLM chatbots' ability to identify the most suitable response according to the current state of the user's 's profile. We observe that current LLMs still struggle to recognize the dynamic evolution in users' profiles over time through direct prompting approaches. As a consequence, LLMs often fail to deliver responses that align with users' current situations and preferences, with frontier models such GPT-4.1, 04-mini, GPT-4.5, ol, or Gemini-2.0 achieving only around 50% overall accuracy, suggest- ing room for improvement. We hope that PERSONAMEM, along with the user profile and conversation simulation pipeline, can facilitate future re- search in the development of truly user-aware chatbots. Code and data are http///wweuuuu/eeeeeeeeee
available at
Introduction
In recent years, Large Language Models (LLMs) have rapidly evolved as general task solvers, demonstrating remarkable performance (Srivastava et al., 2023; Zhou et al., 2023; Yue et al., 2024; Rein et al., 2024). Today, many users rely on LLMs as their personalized chatbots or assistants in a wide range of daily tasks - from offering writing support (Mysore et al., 2024; Tian al., 2024) to delivering recommendations (Hua et al., 2023) or consultations (Xie et al., 2024a; Zheng et al., 2024), etc. Personalization in LLMs involves adapting model responses to specific traits, preferences, and historical interactions of each user, moving beyond generic responses more relevant and tailored ones. Since different users have different personas, becomes an emergent need for LLMs to be pluralistic- capable of adapting to different user characteristics across different scenarios (Sorensen et al., 2024; Jiang et al., 2024; Xie et al., 2024b; Kirk et al., 2024), thereby enhancing user experience and engagement.
"Equal contribution
*Fqual advising

---

Published as a conference paper at COLM 2025
User Persona
User Utterances that Implicate Preference Evolution
Session 2 ( (202202/21
Shatic Profile: Nome, Age, Gender, Rocal Idanrify. Occupation and Short Big
User: I am a big fon of ktalian food, especially pizza.
Initial Preference
Dynamic Profile: Pernonal Histery, Perzonality Traits, Preferences, and Other Characterishics
Session 12 62028/6/17
recently found allergic to glafen, I've started exploring Mediterraneon cuisine, specifeally grilled seafood.
Multi-Session User-Chafbot
Preference Update
Interaction History
Earlier
Food Recommendation
Session 20 @2024/11113
User: Can recommend that I could cook tonight?
Travel Planning
Chatbot: Since you're dllergic gluten, would you prefer some Meditermanean seafood, other gluten-fres options?
Preference-Aligned Recommendations
Type: Provide
Food Recommendation
Chatbot: I remember you prefer Italion food. Would you like some posto recipes?
User: I went to Stalion restauront yesterday.
Theropy Consultation
Type: Revisit
Chatbot: Good to heart How's your allergy been? guess they must have gluten-fres pizzo.
Reasons bahind
More
Food Recommendation
Preference Update
Recent
Chatbot: Good heart Hope you had great night there.
Figure Overview of PERSONAMEM benchmark. Each benchmark sample a user persona with static (e.g., demographic info.) and dynamic attributes (e-g,, evolving preferences). Users engage with a chatbot in multi-session interactions across a variety topics such as food recommendation, travel planning, and therapy consultation. As the user's 's preferences evolve over time, the benchmark offers annotated questions assessing whether models can track and incorporate the changes into their responses.
For LLMs deliver personalized responses, a practical challenge lies in the fact that LLMs cannot easily access all the information about a user. This challenge is further amplified by the ever-changing nature of user preferences over time (Radlinski Craswell, 2017; Dean & Morgenstern, 2022). For example, illustrated in Figure 1, a user initially said, "I like pizza", but mentioned in a later session, "I've started exploring gluten-free options," upon discovering gluten allergy. When the user again asks for food recommendations, personalized LLM chatbot should be able to track the change, and provide recommendations according to the user's 's current situation. Current LLM chatbots often fail to recognize and adapt to evolving user personas. This may lead users to perceive these chatbots as less helpful and empathetic, ultimately diminishing satisfaction (Aggarwal et al., 2023; Ait Baha et al., 2023).
In this work, we evaluate LLMs' ability to leverage the past interaction history with a user in order to deliver a personalized response in real time. Recent studies (Lin et al., 2024; Shi et al., 2024; Zhao et al., 2025) have found that user-LLM interactions can be rich (but often implicit) information source on the user's characteristics and preferences. However, remains an open question whether LLMs can effectively use the interaction histories to (1) internalize the user's 's inherent traits and preferences, (2) track how the user's 's characteristics evolve over time, and (3) generate personalized responses accordingly in new scenarios.
To study these questions, we propose the @ PERSONAMEM benchmark, comprising over 180 simulated user-LLM interaction histories with up to 60 multi-turn sessions across 15 personalized task scenarios. Each history built from a detailed user persona whose characteristics evolve over time. Based on the user's 's profile at different points, we simulate task-specific conversations (e.g., travel, therapy, food) and concatenate them in temporal order to capture the user's 's profile evolution throughout the entire interaction history.
With
we evaluate whether state-of-the-art LLMs can infer evolving user profiles and generate personalized responses across task scenarios. To emulate the realistic settings user-LLM interactions, we design 7 types of in-situ user queries (Table 1), where users issue queries to LLMs from first-person perspectives. We evaluate whether LLMs can select the correct response that best aligns with the current state of the user. We find that frontier models such as GPT-4.1, 04-mini, GPT-4.5, ol, or Gemini-2.0-Flass score only around 50% overall accuracy and Llama-4-Maverick slightly lower at 43% using direct prompt approaches. While models perform reasonably well recalling facts and tracking
2

---

Published as a conference paper at COLM 2025
preference changes (60-70% accuracy), they struggle to incorporate users' latest situations into responses (30-50% accuracy). We provide detailed analysis on how factors such as history length, preference positioning, and memory components may impact performance.
To summarize our key contributions and findings:
We propose the PERSON AMEM benchmark and its synthetic dialog generation pipeline for persona-oriented, suutiis and timelined user-chatbot interaction history.
We assess 15 LLMs on 7 types of itt-itu responses aligned with user's 's dynamically changing profile across 15 task scenarios.
queries and evaluate their ability to provide
With PERSONAMEM, we observe that frontier models such as GPT4.1, 04-mini, GPT-4.5, o1, DeepSeek-Re R1, Gemini-2.0, Llama-4, and Claude-3.7 still struggle to be user-aware and deliver personalized responses, especially when the knowledge of the user needs to be applied across new scenarios.
2
PERSONAMEM Benchmark: Overview
We present an overview of the PERSONAMEM benchmark in Figure 1. Each instance in the benchmark dataset features a user profile or persona, which includes basic demographic information (such as name, age, gender, and occupation), as well as dynamic user character- istics such as user traits, preferences, and events happening in the user's ss life. The dynamic user characteristics change over time as different events happen to the user that will lead to changes in users' traits and preferences specific to each task scenario.
At different points in time of a user's profile evolution, the user engages multi-turn conversations with LLM and seeks help or suggestions from LLM on one of the task scenarios. In each task scenario, the user would ask for the LLM's suggestions given the user's need and current situation. The conversation sessions across different tasks are interleaved by the temporal order in which the sessions happen.
To understand how well LLM chatbots can track the evolution in user's profile from the conversation histories, we evaluate LLMs by whether they can provide the most suitable response to in-sifu user queries, where the user issues the query to LLM in tion session from the first-person perspective. Depending on the time of the in-silu query, the expected response from the model will differ. We cast the problem selection, where LLM needs to identify the correct response out of four choices, where the incorrect choices are based on either outdated or irrelevant information with respect to the current state of the user's 's profile.
conversa-
multiple-choie
Types of skills evaluated. To evaluate LLMs' ability (1) memorize the user profile, (2) track how the user profile evolve over time, and (3) generate personalized responses accordingly in new scenarios, we design the following 7 types of in-situ user queries in the PERSONAMEN benchmark. We include examples for each type of user queries in Table 1.
Recall user-shared facts. events, activities, or interests the user has shared in previous interactions, and incorporate the information in its responses.
We evaluate whether personalized chatbot can recall static
2. Suggest new ideas. We evaluate whether chatbot can suggest new items or activities that have not been mentioned in the interaction history, when users explicitly request so, e.g. "suggest new restaurants I haven't 't ordered from before".
3. Acknowledge latest user preferences. We evaluate whether a chatbot can recognize the latest preference expressed by the user in the interaction history.
4. Track full preference evolution. We evaluate whether chatbot keep track of how users' preferences shift by time.
Revisit reasons behind preference updates. We evaluate whether chatbot can recall the reason(s) or event(s) leading to the preference change from a user.
Provide preference-aligneee recommendations. We test whether a chatbot can proac- tively offer new recommendations that aligns with the user's current preferences.
3

---

Published as a conference paper at COLM 2025
7. Generalize to new scenarios. We evaluate whether chatbot can transfer what it learns about the user from other task scenarios to a new task.
Benchmark data statistics. PPERSONAMEM features 20 personas, with over 180 interac- tion histories. Each interaction history contains 10, 20, or 60 sessions, where we dynamically adjust the total length of the history to approximately 32k, 128k, and 1M tokens, respectively, Each session consists of 15-30 conversation turns between a user and an LLM chatbot. The user-LLM conversations span across 15 diverse topics, ranging from therapy and legal advice to recommendations on books, music, movies, and food; personal matters such as family, dating, health, and finance; and practical tasks like travel planning, online shopping, studying tips, and home decoration. In total, the benchmark features around 6k in-situ user query and LLM response pairs across the 7 query types. Detailed dataset breakdown is discussed in Appendix D. The size of our benchmark is not limited by the scalability of the synthetic data pipeline but to make the evaluation cost reasonable.
Query type
Examples of in-situ user queries and chatbot responses
[1]
Recall user-
"User: shared my playlist with my friends and they loved it. (later) User: What are some creative ways to share music? Chatbot: Curating personalized playlists can be fun."
shared facts
[2] Suggest new ideas
"User: ...... (later) User: Suggest new restaurant I haven't ordered before."
[3] Acknowledge latest prefer- ences
"User: am big fan of Italian food. (later) User: I prefer
Medilerranean cuisine now (later) User: 1 went to an Italian restau- rant yesterday Chatbot: Nice to hear! Thought thought you prefer Mediterranean food now."
[4] Track full prefer- evolution
"User: ...... (later) User: I am thinking about attending cooking class. Chatbot: understand your perspective cooking classes has evolved. Initially, you expressed dislike, but later you mentioned your interest in them."
[5] Revisit reasons behind preference updates
"User: am big fan of Itllian food. ..... (later) User: 1 recently found I'm allergic gluten, I've shifted to other dishes. ... (later) User: I went to an Italian restaurant yesterday Chatbot: How's your allergy been? I guess they must have a gluten-free pizza."
[6] Provide prefer- ence aligned recom- mendations
User: I prefer Mediterranean seafood now. ...... (later) User: How can I make impactful lifestyle changes? - Chatbot: How about diving into the Mediterranean diet plan?"
[7] Generalize new scenarios
"User: prefer my yoga class because its fixed weekly schedule. (later) Liser: recently haven't had time to cook. Chatbol: Since you seem to love noutine would you interested in weekly scheduled bento-box box deliveries?"
Table 1: Examples of the 7 types of in-situ user queries and expected chatbot response in the PERSONAMEM benchmark. We highlight words that signal the user's facts or preferences.
Constructing Examples in PERSONAMEM At Scale
We develop modular data curation piweline-powered by GPT-4o (Hurst et al., 2024) to synthesize persona-oriented, multi-session user-model conversations with long context. The pipeline minimizes irrelevant or randomly injected content to better evaluate how effectively LLM chatbots address the challenges outlined in Section 2, while ensuring cost- effectiveness and scalability: generating data for each persona on each conversation topic costs approximately $2, independent of the context window length up to 1M tokens.
Constructe user profile and persona. We sample a set of random personas from Person- aHub (Ge et al., 2024), each comprising about one to three sentences, and augment them with additional demographic information and extended personal details. We also construct timeline and populate it with events that align with the persona. These events serve as
4

---

Published as a conference paper at COLM 2025
Data Curation Steps
Facifi inguanic user-friendly apps
leverage understunding of people
Stop (1) Construct User Persona
Ferfer
Recent Time
Step Get Up Frans and Gentrote General Personal History
ling Exerl
Tirs, Ement X4
Thme, Feant d Preference
Toplc-
New Freferance
Tine, Exent New Profarince Resiers
and Recacts
Time, Sierr hew Preference
Time, Swent
Step Rapond Topic- Speci-eecifc
Tranel Plamning Therspy consullation
Travel Panning
Trave Plasring
Trase Planning Therapy Consulaliin
Personal Histories info Comversation Sessions
Therspy Coroulle
Therapy Censullation
Fred Recannandition
Reconnvnsattonn Si
Recommuraatinn Gife
Food Recirmunataiion
& Giaa
Step (6) contructing Long Contexte via Soadion Concotenatipe
Perfom Sppllgirel wrtting
the
Therspy
Feed Seedis
Troval Seelion
hood
Troual Sestion
Ived Sessiin
Sendien
Seasion
Figure 2: An overview of the persona-oriented multi-session data curation process. We construct user personas, build time-stamped general and topic-specific personal histories, expand them into conversation sessions, and topologically concatenate sessions to create long conversation contexts resultinn scalable generation framework.
the general personal history, such as education, career development, and life experiences, to provide a richer context. The prompts used in the process can found in Appendix G.
Building on the persona and general personal history, we generate one additional topic- specific personal history for each conversation topic. Under each topic, we define set of initial preferences, ensuring no overlap across different topics. Each topic-specific history includes events, timestamps, associated preferences, potential updates those preferences, and the underlying reasons for those changes. This approach ensures coherent progression of user experiences while maintaining a strong connection to their personas.
The structured personal histories also facilitate the curation of question- pairs. We leverage short-form information within these histories to extract ground-truth user profiles and preferences at any specific time, ensuring that the correct answers are both event- and persona-groundee overlook the user's persona or contradict it. Additionally, exclude all questions that the model can answer correctly without seeing any contextual information from the benchmark.
In contrast, distractor options, while generally reasonable, either
Simulate conversation sessions from user profile. We divide the timeline into multiple segments, resulting in segments personal histories that follow a causal, chronological order. Each segment is then expanded into full user-model conversation session, designed to cover all details of the corresponding topic-specific personal history segment, together with additional storytelling context as if the user is talking with chatbot naturally. For example, under the therapy consultation topic, we frame the interaction as a user seeking guidance from AT therapist.
To enhance the quality of the conversations, we incorporate several tricks: (1) Before generating each moer- interaction turn, we prompt GPT-4o to first identify and cite the relevant event from the personal history. These citations serve as internal guidance and are not included in the final evaluation data. (2) Since GPT-4o may miss some events, leading to incomplete preference update sequences, we employ a self-reflection mechanism. We ask GPT-4o review the generated conversation and identify any missing events from the personal history, ensuring better coverage and coherence across the interaction.
Assemble interaction history via session concatenation. Generating large-scale, persona- oriented long-contes conversations can be both cost-efficient and scalable. For each persona, topologically sort conversation sessions based their ending timestamps, and only

---

Published as a conference paper at COLM 2025
need to make sure sessions within the same topic maintain causality. Differrnt numbers of sessions can be concatenated in multiple valid orders. This flexible design allows for multiple valid interleavings of sessions across different topics, meaning we only need to generate sessions themselves-nee nno every entire long-contell comversation from scratch. To further extend context length and simulate more natural interactions, short interactions between sessions where the user asks random knowledge questions or programming helps without indicating any user preferences.
insert limited number of
Human validation on dataset quality. To evaluate the quality of our generated data, we conduct a human study on 90 random query-response pairs from PERSONAMEM, each grounded in user persona, personal histories, and associated utterances in conversation. Three annotators assess each Q&A pair across four dimensions: appropriateness, relevance, correctness, and best response. Judgments were very high for all dimensions 97.8%, 95.6%, 97.8% and 90.0% respectively. Further details are provided in Appendix B.
Experiment
4.1 Evaluation Settings
Given an in-sifu user query and the user's interaction history up to a point in time, we evaluate models' ability to select the most appropriate response according to the current state of the user amonst four different choices. Only one of the choices fits the user's current status, and the other choices contain either irrelevant or outdated facts or preferences from the user. During evaluation, apart from the conversation history, the models have access to the basic demographic information of the user, including name, age, gender identity, racial identity, and occupation. The models do not have direct access to the user's 's other dynamic characteristics and personal history otherwise.
For selecting the most appropriate response, we evaluate models under both discriminative and generative settings. In the discriminative setting, the models are presented with all four response choices denoted with (a), (b), (c) and (d) with random ordering among the choices. The model is asked to output the correct choice along with a brief explanation. In the generative selfing, the models still see one question at a time. We compute the log-sum of token probability of generating each option individually with length normalization, and select the option with the highest probability as the model response. We use the discriminative setting for main evaluation (S 4.2, 4.3, 5 4.4) and adopt the generative setting in & 4.5, as requires access to logits over entire vocabulary during decoding, which is not available from most proprietary models. No LLM judges are involved in the evaluation process.
4.2 Evaluating Language Models in Long-Context Settings
We first evaluate language models in the long-context setting, where the full user-LLM inter- action history is provided input to the models. Due the length of the history, all models here were evaluated zero-shot, without demonstration examples of other histories and user queries. Our evaluation covers 1PT-4.1, 04-mini, 03-mini, GPT-4.5, o1, GPT-4o, GPT- 4o-mini, Gemini-22.0FFlar Gemini22.0Flashh-itte Gemini-1.5-Flan 1-5-ashh, DeepSeek-R1-777 Llama-4-Maverick, Llama-3.1-405B, Claude-3.7SSonnet,, and Claude-3.5-Haiku (OpenAI, 2025a; 2024b; 2025b; 2024a; Hurst et al., 2024; Team et al., 2024; Guo et al., 2025; Grattafiori et al., 2024; Anthropic, 2024) on 128k-token context windows. We also evaluate models that M-vverric,,,,, Demini-2.0-Flaas Gemini-2.0-Flash-
support longer Comini-1.--llshh-oe 1M-token context windows. We report the following findings:
and
GPT-445, GPT-411, and Gemini-1.5 achieve the highest overall performance. Among leading foundation models, GPT-4.5 and Gemini-1.5 outperform others in overall accu- racy. However, their performance still hovers around 52% in a multiple-choice setting, highlighting substantial room for improvement. Notably, reasoning models such as ol, o3-mini, o4-mini, and over non-reasoning models in the personalization tasks evaluate.
do not demonstrate competitive advantages

---

Published as a conference paper at COLM 2025
Batta
Query Type Model
taccing fusl Pertesre Eritdion
647
960
E.0
068
5.66
064
045
622
055
Hicsn Shared Race
1464
5.81
CAS
6-8
8-00
G44
641
043
5.31
D44
066
12
023
640
025
650
8.4
E.00
042
8/8
C.39
640
On tie Nester
0.54
8.00
0.83
038
1.0
031
0.00
6.30
038
0.0
0.30
030
521
0.0
029
030
025
0.99
011
9.50
0.11
0.11
028
0.0
Figure 3: Evaluation results across different models on 7 in-situ query types. We observe models perform reasonably well at recalling user facts and preferences. However, models struggle at providing novel suggestions, or applying users' preferences in new scenarios.
1254
Wike of Seasdara Averall
7
5
3
18
11
12
13
14
25
25
37
15
10
29
0.48
0.51
0.8
0.90
0.42
0.54
5.48
0.89
0.57
GFT-5
0.02
0.4
0.50
0.57
0.55
0.54
0.52
0.50
0.44
0.52
0.45
0.45
0.41
0.52
2.50
0.36
0.40
0.53
0.55
0.48
2.42
GPF41
0.00
0.50
040
0.01
0.59
0.58
0.48
0.68
0.48
0.55
0.12
0.45
0.45
5:44
0.64
0.54
0.61
0.52
0.00
035
0.2
0.55
0.40
0.45
0.48
0.40
0.51
2:50
D.44
0.42
0.41
0.50
C.6
DAR
0.46
0.51
0.51
0.50
242
0.45
0.42
0:00
G
0.00
045
022
0.99
0.42
0.44
C.40
D/46
0.35
0.33
0.42
0.44
D.7
DAL
0.56
0.53
0.53
0.50
2.40
GP%
0.45
0.51
0.6
0.44
0.43
0.47
0.28
0.42
0.49
0.40
08:30
0.38
0.45
0.50
0.30
0.30
0.80
0.54
0.48
0.40
0.72
0.91
0/5
0/40
0.33
0.33
0.27
0.45
2:06
0.39
0.33
0.41
0.37
1:50
0:50
0.54
0.62
0.53
0.50
2.50
0:00
0.29
6.20
0:97
0.27
0.38
0.36
0.41
0.29
0.35
0.30
D46
0.36
0.34
0.37
0.28
0.25
0.25
0.30
0.23
0.32
0.24
0.23
0.36
0.42
0.50
0.44
0.27
6:61
0.40
8.30
0.2
0.27
0.25
0.24
0.88
5.26
6.04
0.30
0.30
0.37
0.26
1.28
6:36
0.30
0.42
0.38
0.27
1.51
0.26
0.72
0.27
0.01
0.25
0.20
0.28
0.21
0.20
2.10
0.15
0.17
0.12
0.23
1.20
0.10
0.29
0.47
0.33
0.77
2.19
Aveuge
0.4
0.40
0.00
0.44
0.41
0.41
0.41
0.40
0.00
0.35
0.35
0.30
0.41
1.50
0.36
0.44
0.00
0.49
0.43
2.07
0.56
0.56
6.9%
0.76
0.75
0.35
0.35
0.34
5.96
1M
Geneenl M-Ptis
5.45
0.64
0.80
0.81
0.55
0.45
0.29
0.42
0.48
0.57
0.35
6:56
2.42
0.00
0.51
0.90
0/22
0.11
0.90
0.09
0.77
0.44
0.31
0.22
0.51
0.37
0.30
0.34
0.32
0.51
0.44
0.00
2.45
2.40
0.29
0.00
0.00
0.30
0.20
0.50
0.37
Anerage
6.57
0.47
0.49
0.45
0.48
0.08
0.51
0.34
0.87
0.51
0:55
8:40
0.02
0:20
0:90
0.00
0.36
0.58
0.69
0.38
Randon
0.29
0.29
0.29
0.29
0.25
0.29
0.55
0.5
025
Figure 4: Model performances by number of sessions elapsed since most recent preferences were mentioned in long context. Top: up to 20 sessions /12kk tokens; Bottom: up to 60 sessions 1M tokens. Long-context retrieval is important for personalization in practice.
LLMs demonstrate reasonably good performance in recalling simple user facts. For tasks involving the retrieval of static user information, such as previously mentioned items, activities, or reasons behind preference changes where the reasons themselves won't change, most LLMs have reasonable chance of succeeding.
Incorporating the latest user preference into responses is more challenging than recalling the change in user profile. We observe that models struggle to incorporate the latest preference or state of the user in responses. Surprisingly, models generally get higher performance when asked to recall how the user preferences evolve over time. We observe that asking the model to iterate through all preference updates may encourage it to think through the preference evolutions, often making the task easier.
Models fall short on generating new ideas or providing suggestions new scenarios. As shown in Figure 3, tasks such as "Suggest New Ideas", "Provide Preference- Rec- ommendations", and "Generalize Reasons to New Scenarios" yield the lowest performance across all models, highlighting the challenge of generating personalized responses in novel contexts taatexts-patticulaaty when identifying new facts.
4.3 Effect from the Position of User Information in Interaction History
To understand how the model performance is affected by the position in which the relevant user facts or preferences appear in the conversation history, we report the model perfor-
7

---

Published as a conference paper at COLM 2025
+005
+2.05
G714s
07140 (aed
Cfidearite
-38.%%
44.0%
+24.0%
+01.%%
0.2%
+22.3%
122.0%
+2..4%
+2.0%
12.0%
+7.00
10.0
Redelt Reakon Proferece Opdotes
Tracong Pdl Preference Eecidion
Provile Preferenct Racemmande meseomnnddattnn
Recial Facts
Acknosledpy Leient Preferncee
Geneediit Nossons Now Sctnaris
Suppect Niwe
Query Type
Figure 5: Performance on different question types for GPT-4o and GPT--o-mini with 32k- token contexts. We compare vanilla models to the ones with MemO and RAG setups.
mance by the session in which the relevant user information appears in the history. The results are shown in Figure 4. Generally, we observe that the model performs better when the relevant information appears in the earler or later sessions of the conversation history. The findings here generally echo previous findings on long-context inputs to models, where context information tends to get "lost in the middle" (Liu et al., 2024; Wu et al., 2024).
4.4 Evaluation with External Memory Modules
We evaluate whether using a retriever identify relevant information in the history will help improve model's performance. We evaluate two external memory approaches-RA. RAG (Lewis et al., 2020) and Mem0 (Mem0, consider only the GPT-4o and GPT-4o-mini models. We show their latency in Appendix E.
vanilla LLMs. For these experiments, we
For RAG, we consider straightforward implementation that retrieves the top five most relevant messages per question using dense BGE-M3 embeddings (Chen et al., 2024). For Mem0 which provides an additional memory layer to LLMs, we iteratively build a memory database using LLM-generated facts over each turn. At inference, retrieve the top 5 relevant facts per question. For efficiency, we use 32k-token contexts for evaluation.
Retriever-based memory module can improve model performance. Overall, external memory modules significantly improve accuracy for both models. Notably, Recall User- Shared Facts and Generalize to New Scenarios benefit the most, highlighting the effectiveness of retrieval in factual tasks. In contrast, Revisit Reasons Behind Preference Updates shows smaller gains. RAG consistently outperforms Mem0 across most question types, although MemO more computational expensive, suggesting that retrieving semantically similar messages is more effective for personalized reasoning.
4.5 Evaluation of Language Models in Generative Settings
In real-world use cases, the chatbots do not have access the potential options of responses during inference. For such reason, we additionally evaluate models on the more realistic generative settings, where the model sees only one option time, and the best response is selected by the joint sequence probability of options from model predictions.
Approaches. Given the user-LLM history and in-situ user query, we compare the joint sequence probabilities by taking the log-sum of the token-level probability of each response option. Specifically, given a conversation history (denoted as C) and the user query (4),
evaluate each candidate response (1.33A) consisting of tokens {x?.
of
total token length I. Due the autoregressive nature of causal language models, the joint log probability for each query-ans pair is computed by summing the conditional log probabilities of each token given its preceding context, formalized as
Tj Σ
log P(r C.q)
log p(x xxx.x...,,xx-1))))
8

---

Published as a conference paper at COLM 2025
K4-0iNAAL Aattettt-ttteeee eaa- 90
3.3-780
3.5-80
41
(9ca)
Evrrape
Euery Rodel
Ploral Reapory
0.00
0.82
0.88
Trocking Rull
077
0.58
0.76
0.53
0.00
0.25
Shaned
Recal
0:44
0.22
0.72
0.18
0.57
0.25
A lireed Recarewnnbiivee
Provide Preferece
0.89
0.29
0.27
0.44
0.58
0.25
User Preference
0.30
0.14
0.24
0.35
0.25
to Niw Scanariss
0.30
0.19
0.32
0.26
Paggest
Figure 6: Generative evaluation 10-session (32k token length) version of PERSONAMEM.
As the method requires logarithmic probability of output tokens over the entire vo- cabulary, which is often not available in proprietary models, we evaluate ween-w models-LLaMA-33-0000 LLaMA-3.1-301 and DeepSeek-Dien Distill-LLaMAASST Due con- straints in computation resources, we only evaluate the models on the 10-session version of the benchmark, which includes around 32k-tokens per session.
Results. As shown in Figure 6, we observe the similar trend to our discriminative evalu- ation results in terms of difficulty by different user query types. Models get reasonably good performance on recalling facts and tracking preference changes, while giving new suggestions and generalizing to new scenarios are still the most challenging types of queries for models. Interestingly, when comparing the same model, specifically I.Lama-3, 1.1-BB- instruct, under discriminative and generative settings, we see the performance is better in the generative setting, potentially suggesting that the model is able to provide a per- sonalized response without seeing all the candidate options in the input. Since we only managed to run evaluation on 32k context length with the generative setting, it remains to be investigated whether results in generative context length and for different models. We also find that model performance declines as users' new requests become more distant from their previously revealed information. Detailed results are provided in Appendix C.
discriminative settings stand for longer
5 Related Work
5.1 Evaluating Long-Context Memory Capabilities of LLMs
feedle-in-the-haystaee tests, which task models to locate specific facts within a given long context, are common method for this evaluation. Prior benchmarks perform tasks from direct information retrieval (Kuratov et al., 2024; Nelson et al., 2024) to question answering and summarization (Xu et al., 2024; Bai et al., 2024; Zhang et al., 2024). A more real-world setting for such evaluation is through dialogue conversations. Earlier benchmarks curated human-human (Xu, 2021) or human-Al interactions Xu et al. (2022), with sessions up to 10K tokens. More recent works have used LLMs to generate much longer sessions of 100k+ tokens long (Maharana et al., 2024; Kim et al., 2024; Castillo-Bolado et al., 2024). More recently, Wu et al. (2024) present LONGMFMEVAL, dialogue benchmark which also considers contexts up to 1M, and uses persona-driven sessions. The major differences are that sessions from PERSONAMEM consider a broader range of topics than just task-oriented ones; and that the evaluation of PERSON AMEM focuses on fine-grained personalization concerns, rather than more general memory abilities.
5.2 Towards Personalization in Large Language Models
As users have diversity of preferences, both at demographic-level (Santurkar et al., 2023) and at an individual-level (Zollo et al., 2024). Personas are short biographies of individ- uals, that capture both levels, and can be generated Researchers have used personas to evaluate how LLMs can adapt to users and environ- ments (Castricato et al., 2024; Tseng et al., 2024). Reliable evaluation of personalization is
masse by LLMs (Ge et al., 2024).

---

Published as a conference paper at COLM 2025
LOCOMO
LongMemEval
PrevEval
PersonaMem
Focused Tasks
Long-term memory
Long-term memory
User
Fine-grained
preferences
personalized
responses
Avg. Single Session Len
477 tokens
3k tokens
No info
6k tokens
Max Context Len
9k tokens
1.5M tokens
100k tokens
1M tokens
Data Sources
MSC & own
ShareGPT & UltraChat & own
LMSYS-Chat-
PersonaHub
1M
own
Query Perspective
third-person
first-person
first-person
first-person
Max Knowledge Updates
No update
update
Multi-Session Reasoning
Yes
Yes
No
Yes
LLMs Evaluated
5
15
Table 2: Comparison of related benchmarks, including LOCOMO (Maharana et al., 2024), LongMemEval (Wu et al., 2024), and PrefEval (Zhao et al., 2025). LOCOMO and Long- MemEval focus general long-term memory tasks. In contrast, PersonaMem centers personalization beyond memory retrieval, with all conversations in our benchmark are built around coherent user persona with evolving preferences, mimicking more realistic user-chatbot conversations. PrefEval, which focuses on personalization too, but by first generating user preferences and then inserting them into randomly sampled contexts.
also key. Many of the aforementioned benchmarks through formulation NLP tasks, and another line of work uses LLMs automatically judge texts along different axes of personal- ization (Dong et al., 2024; Wang et al., 2023). The approach taken by PERSONA A MEM follows the former, as we report performance on question-answering personalization evaluation is by design of the questions and answers, each of which is grounded in specific temporal events, and is generated to adhere to a specific question type.
Importantly though, the
Turning to the dialogue setting, earlier works like LAMP and PERSONALLM consider personalization within a single turn or session (Salemi et al., 2023; Jiang et al., 2023; Kirk et al., 2024). More recently, IMPLEXCONV (Li et al., 2025) focuses reasoning within personalized conversations. PERSONA BENCH (Tan et al., 2025) simulates social interactions among diverse users through numerous but shorter sessions and ac- cess to synthetic private user data. PERSOBENCH (Afzoon et al., 2024) leverages existing persona-aware datasets to evaluate language quality, persona coverage, and consistency. LONGLAMP (Kumar et al., 2024) focuses on generating long-form form texts other than more interactive responses within long context. Zhao al. (2025) introduce PREFEVAL, which evaluates LLMs' preference-following abilities for 20 topics in persona-oriented dialogues of 100k+ tokens. MERSONAMEM, besides the flexible setting of generating numerous M-ttoken contexts efficiently, places greater emphasis on personas as simulated humans in user- model interactions, featuring multiple fine-grained personalization tasks where profiles and preferences evolve through temporally grounded events.
modeling implicit
Conclusion
In this paper, we introduce the 4 PERSONAMEM benchmark, featuring scalable and persona- oriented multi-session user-LLM interaction histories, as well as fine-grained in-situ user query types designed to evaluate LLM capabilities memorizing, tracking, and incor- porating users' dynamic profiles into personalized responses. Through comprehensive assessments of 15 state-of-thh-art LLM models and retrieval-based methods, we highlight current challenges in enabling LLMs to deliver truly personalized conversations with users, especially in novel scenarios and long contexts. We hope that our benchmark opens new av- enues for future exploration and advancement in personalized LLM chatbot development.
10

---

Published as a conference paper at COLM 2025
References
Saleh Afzoon, Usman Naseem, Amin Beheshti, and Zahra Jamali. Persobench: Bench- marking personalized response generation in large language models. arXiv preprint arXit:2410.03196, 2024.
Abhishek Aggarwal, Cheuk Chi Tam, Dezhi Wu, Xiaoming Li, and Shan Qiao. Artificial intelligence-baseed chatbots for promoting health behavioral changes: systematic review. Journal of medical Internet research, 25xe40789, 2023.
Tarek Ait Baha, Mohamed El Hajji, Youssef Es-Saady, and Hammou Fadili. The power of personalization: A systematic review of personality-adaptiie chatbots. SN Computer Science, 4(5))661, 2023.
Anthropic. The claude 3 model family: Opus, sonnet, haiku. https://wwwe cdn. com/ com/de88a9b0111 hctoo0ccbbbb33333bb/b 8857627/Model Caar_Calauuaaaaaaaa Claude 3. 3.pdf, March 2024. Accessed: 2025-04-10.
cw..wwwpp..antropi..... anthropic.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 3119-3137, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 11166633/11/224.adll 172. URL https:://claantholooo or org/o0g// acl-long. 172/-
David Castillo-Bolado, Joseph Davidson, Finlay Gray, and Marek Rosa. Beyond prompts: Dynamic conversational benchmarking of large language models. arxiv preprint ar arXiv:2409.20222, 2024.
Louis Castricato, Nathan Lile, Rafael Rafailov, Jan-Philipp Fränken, and Chelsea Finn. Persona: A reproducible testbed for pluralistic alignment. arXiv preprint fit:2207.17387.
2024.
Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-lingual, multi-functionaliiyy, multi-granularity text embeddings through self-knowledge distillation. ar preprint arXie:2402.032116 2024.
Sarah Dean and Jamie Morgenstern. Preference dynamics under personalized recommenda- tions. In Proceedings of the 23rd ACM Conference on Economics and Computation, pp. 795-816, 2022.
Yijiang River Dong, Tiancheng Hu, and Nigel Collier. Can Ilm be personalized judge? arxiv preprint ar xxXvv2206..16577 2024.
Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. Scaling synthetic data creation with 1,000,000,000 personas. arviv preprint ar Xrii::2406.20094, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. Xiv preprint ar Xii::240..2178. 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning. arviv preprint arXiv:2501. 129444,, 2025.
Kilem Li Gwet. Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1):29-48, 2008
Wenyue Hua, Lei Li, Shuyuan Xu, Li Chen, and Yongfeng Zhang. Tutorial on large language models for recommendation. In Proceedings of the 17th ACM Conference on Recommender Systems, pp. 1281-1283, 2023.
11

---

Published as a conference paper at COLM 2025
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arxio preprint ar arXo::2400.21276 2024.
Bowen Jiang, Yangxinyu Xie, Zhuoqun Hao, Xiaomeng Wang, Tanwi Mallick, Weijie J Su, Camillo J Taylor, and Dan Roth. A peek into token bias: Large language models are not yet genuine reasoners. arXiv preprint ar Xffo:240.111050 2024.
Hang Jiang, Xiajie Zhang, Xubo Cao, Cynthia Breazeal, Deb Roy, and Jad Kabbara. Per- sonallm: Investigating the ability of large language models to express personality traits. arXiv preprint ar arx:::2305.025: 305.0255., 2023.
Jiho Kim, Woosog Chay, Hyeonji Hwang, Dacun Kyung, Hyunseung Chung, Eunbyeol Cho, Yohan Jo, and Edward Choi. Dialsim: A real-time simulator for evaluating long-term dialogue understanding of conversational agents. arXiv preprint arXiv:2406.11111 131444 2024.
Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, et al. The prism alignment project: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models. ar Xrv preprint ar Xio:2404.1160 2024.
Ishita Kumar, Snigdha Viswanathan, Sushrita Yerra, Alireza Salemi, Ryan A Rossi, Franck Dernoncourt, Hanieh Deilamsalehy, Xiang Chen, Ruiyi Zhang, Shubham Agarwal, al. Longlamp: A benchmark for personalized long-form text generation. ar preprint wxXiv:240..11016, 2024.
Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. In search of needles in a 10m haystack: Recurrent memory finds what Iims miss. Xrviv preprint ar arXi::2402.10733 107900 2024.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval- augmented generation for knowledge-intenivve nlp tasks. Advances in neural information processing systems, 33:9459-9474, 2020.
Xintong Li, Jalend Bantupalli, Ria Dharmani, Yuwei Zhang, and Jingbo Shang. Toward multi- session personalized conversation: A large-scale dataset and hierarchical tree framework for implicit reasoning. ar preprint arXiv:2503.07011 2025.
Ying-Chun Lin, Jennifer Neville, Jack Stokes, Longqi Yang, Tara Safavi, Mengting Wan, Scott Counts, Siddharth Suri, Reid Andersen, Xiaofeng Xu, Deepak Gupta, Sujay Kumar Jauhar, Xia Song, Georg Buscher, Saurabh Tiwary, Brent Hecht, and Jaime Teevan. Interpretable user satisfaction estimation for conversatinnal systems with large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11100- 11115, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/ /v1 /2024.acl-loog.g long.598. URL https:// org/ ort.org///222/ 2024. acl-long. 598/.
Nelson F Liu, Kevin Lin, John Hewitt Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024.
Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. Evaluating very long-term conversational memory of 11m agents. arXiv preprint ar X717: 2024.
Mem0. Mem0: An additional memory layer for language models. https://ment ai, 2024. Accessed: 2025-03-27.
Sheshera Mysore, Zhuoran Lu, Mengting Wan, Longqi Yang, Bahareh Sarrafzadeh, Steve Menezes, Tina Baghaee, Emmanuel Barajas Gonzalez, Jennifer Neville, and Tara Safavi. Pearl: Personalizing large language model writing assistants with generation-calibratee
12

---

Published as a conference paper at COLM 2025
retrievers. In Sachin Kumar, Vidhisha Balachandran, Chan Young Park, Weijia Shi, Shirley Anugrah Hayati, Yulia Tsvetkov, Noah Smith, Hannaneh Hajishirzi, Dongyeop Kang, and David Jurgens (eds.), Proceedings of the 1st Workshop Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (CustomNI.P4U)) pp- 198- 198-219, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/ 10.18653/v1120224.. /ttp:///A...uuooompuuuu.... acl aclanthologgy.org org/ 2024. customnlpuu-1.10
URL https://
Elliot Nelson, Georgios Kollias, Payel Das, Subhajit Chaudhury, and Soham Dan. Needle in the haystack for memory based large language models. arXiv preprint 4rX1::2407.01437, 2024.
OpenAL. Openai o1 system card, 2024a. 01 system-cardd 20241 2054 pdf. Accessed: 2025-03-27.
URL https://cdn.openai..co
OpenAL. Openai o3 and 04-mini system card, 2024b. URL http:///openai.comm/meeess
o3- mimi--ysttemmm card/. Accessed: 2025-04-18.
OpenAL. Openai gpt-4.5 system card, 2025a.
URL https://cdn.openai.ccom
00t-4-5-yystem-card-22222
pdf. Accessed: 2025-03-27.
OpenAL. Openai 03-mini system card, 2025b. URL https://cdn.openai.. com/
03-mini 23-mini-system-cant card-m Accessed: 2025-03-27.
Jiaxin Pei, Aparna Ananthasubramariam, Xingyao Wang, Naitian Zhou, Apostolos Dede- loudis, Jackson Sargent, and David Jurgens. Potato: The portable text annotation tool. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 327-337, 2022.
Filip Radlinski and Nick Craswell. A theoretical framework for conversational search. In Proceedings of the 2017 conference on conference human information interaction and retrieval, pp. 117-126, 2017.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google- proof q&a benchmark. In First Conference on Language Modeling, 2024.
Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. Lamp: When large language models meet personalization. arXiv preprint Xoi:::2304411466,, 2023.
Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? In International Conference Machine Learning, pp. 29971-30004. PMLR, 2023.
Ying-Chun Taiwei Shi, Zhuoer Wang, Longqi Yang, Ying Chun Lin, Zexue He, Mengting Wan, Pei Zhou, Sujay Jauhar, Sihao Chen, Shan Xia, et al. Wildfeedback: Aligning Ilms with in-situ user interactions and feedback. Xrviv preprint ar Xix::244 15549, 2024.
Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, et al. Position: a roadmap to pluralistic alignment. In Proceedings of the 41st International Conference on Machine Learning pp. 46280-46302, 2024.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023(5):1-95,, 2023.
Juntao Tan, Liangwei Yang, Zuxin Liu, Zhiwei Liu, Rithesh Murthy, Tulika Manoj Awal- gaonkar, Jianguo Zhang, Weiran Yao, Ming Zhu, Shirley Kokane, et al. Personabench: Evaluating ai models on understanding personal information through accessing (syn- thetic) private user data. arXiv preprint 2rXio:2502.20616 2025.
13

---

Published as a conference paper at COLM 2025
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
Yufei Tian, Tenghao Huang, Miri Liu, Derek Jiang, Alexander Spangher, Muhao Chen, Jonathan May, and Nanyun Peng. Are large language models capable of generating human-level narratives? In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 17659-17681, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. (8.18653/vll 2024. emn.p-main. 998//.
cecillpmaii.9977 9788. URL https: httts::/aallanthollogy.. org/
Yu-Min Tseng, Yu-Chao Huang, Teng-Yun Yun Hsiao, Yu-Ching Hsu, Jia-Yin Foo, Chao- Wei Huang, and Yun-Nung Chen. Two tales of persona in Ilms: A survey of role-playing and personalization. arxiv preprint ar xrXio:2406.0117 2024.
Yaqing Wang, Jiepu Jiang, Mingyang Zhang, Cheng Li, Yi Liang, Qiaozhu Mei, and Michael Bendersky. Automated evaluation of personalized text generation using large language models. arviv preprint arXiv:2310.11593, 2023.
Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, and Dong Yu. Long- memeval: Benchmarking chat assistants on long-term interactive memory. arXiv preprint arXiv:2410.10813, 2024.
Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. Travelplanner: a benchmark for real-world planning with language agents. In Proceedings of the 41st International Conference on Machine Learning, pp.
2024a.
Yangxinyu Xie, Bowen Jiang, Tanwi Mallick, Joshua David Bergerson, John K Hutchi- son, Duane R Verner, Jordan Branham, M Ross Alexander, Robert Ross, Yan Feng, et al. Wildfiregpt: Tailored large language model for wildfire analysis. ar preprin! (rXiv:2402.07877, 2024b.
Xu. Beyond goldfish memory: Long-term open-domain conversation. arXiv preprint arXivv2107.0756: 2021.
Xiaoyue Xu, Qinyuan Ye, and Xiang Ren. Stress-testing long-context language models with lifelong icl and task haystack. arXiv preprint 20X6:22407.16665, 2024.
Xinchao Xu, Zhibin Gou, Wenquan Wu, Zheng-Ya Yu Niu, Hua Wu, Haifeng Wang, and Shihang Wang. Long time no see! open-domain conversation with long-term persona memory. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguistics: ACL 2022, pp. 2639-26500 Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/ 1el.1866/v/20000 v122022 findings-acl.2207 URL http:::///aaaanthhooiii
org/2022 findings-acl. 2c7.0
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9556-9567, 2024.
Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, et al. oo bench: Extending long context evaluation beyond 100k tokens. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15262-15277, 2024.
Siyan Zhao, Mingyi Hong, Yang Liu, Devamanyu Hazarika, and Kaixiang Lin. Do llms recognize your preferences? evaluating personalized preference following in llms. In The thirteenth international conference on learning representations, 2025.
Huaixiu Steven Zheng, Swaroop Mishra, Hugh Zhang, Xinyun Chen, Minmin Chen, Azade Nova, Le Hou, Heng-Tze Cheng, Quoe Le, Ed H Chi, et al. Natural plan: Benchmarking llms natural language planning. Xiv preprint ar arXiv:2406. 044200 2024.
14

---

Published as a conference paper at COLM 2025
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou, Instruction-followwin evaluation for large language models. arXiv preprint ar X'io:2311.079111 2023.
Thomas P Zollo, Andrew Wei Tung Siah, Naimeng Ye, Ang Li, and Hongseok Namkoong. Personalllm: Tailoring llms to individual preferences. arXiv preprint ar XrXi::2409.20296
2024.
15

---

Published as a conference paper at COLM 2025
A Limitations and future work
A.1 Broader context in user privacy concerns
Privacy is a critical aspect of LLM personalization the real world. In our setting, we personalize responses based on only preferences and activities shared by the user in previ- ous user-chatbot interactions, and the model uses this information for its own responses without external sharing. To avoid potential privacy risks associated with real user data, we intentionally propose a synthetic data curation pipeline in this work. This synthetic ap- proach allows researchers in the community to safely explore personalization methods. One possible direction for future work could be designing question-answer pairs that specifically involve sensitive user information.
A.2 More advanced retrieval methods
Our current exploration of retrieval-augmented methods, such RAG and Memo, is intended as proof of concept, as the primary focus of this work on the design and release of the personalization benchmark. We excited to encourage more exploration on state-of-the-art long-context, memory, and retrieval-augmenned generation methods in future work, especially those that preserve and understand the evolution of user personas and reasons behind preference updates, as well enhancing user personalization in new or unseen scenarios.
A.3 Potential artifacts in the synthetic data generation process
To reduce artifacts that might make the benchmark artificially easier, we've taken several steps. For example, we removed question-answer pairs where the correct answer was unintentionally obvious, such as being noticeably longer or sharing identical key words with the questions. We also filtered out queries that an LLM can answer correctly more than once three attempts, without seeing any actual conversation context. Besides, we have included checks in our human evaluations to confirm that the correct answers can indeed be derived from the provided context.
A.4 Potential gaps between evaluations on open-ended generations and multiple choices
In purely open-ended generative settings, personalization can lead to many possible correct answers, depending on how the user persona is used and which related user preference is used. Meanwhile, open-ended evaluations are computationally expensive due to the need for --dM-as-a-Judge for each question-answer pair. As a result, we evaluate generative tasks by computing the joint log-likelihood of each candidate option, without explicitly presenting all four options in the prompt. This approach yields similar patterns with those observed in standard discriminative evaluations in our experiment, while offering more reliable basis for benchmarking performance compared to fully open-ended ones.
16

---

Published as a conference paper at COLM 2025
B Details on Human Evaluation
The purpose of the human evaluation study is to validate the overall quality of the gen- eration process described 3. Note that we are not asking for human performance on the questions, given the intractability of reading the long contexts. Instead, we provide evaluators with the questions and answers, well as the conversations and meta-data that they are grounded in.
We use the potato package (Pei et al., 2022) for implementation the interface. screenshot is shown in Figure 7. For each entry, we ask for True/Falss evaluations on 4 dimensions:
1. Appropriateness: The question is well-formed and corresponds to the type.
2. Relevance: The question relevant to the conversation and persona.
3. Correctness: context.
'Correct Response' is indeed correct, and can be derived from the
4. Best Response:
'Correct_Response' is better than all of the Incorrect,Responsee
We recruited three authors from among the authors of this work. 1 We iterated the annotation instructions and template with active feedback from the annotators, leading to the finalized version.
We selected 90 entries (18 topics 5 randomly sampled questions each) for annotation. To ease annotator mental load, all entries come from single persona. Each entry is annotated times, and we assign the majority class label. Each task took about 1.5 minutes to complete.
For each entry and each dimension, calculate the proportion of 'True', well We calculate inter-rater reliability with Gwet's AC1 (Gwet, 2008). We use this metric as it accounts for the heavy class imbalance towards True. Considering the results, 97.8% of entries were rated as appropriate (AC1=0.928), 95.6% as relevance (AC1=0.899), 97.8% as correct (AC1=0.877), and 90% as being the best response (AC1=0.560), All proportions are over 90%, and agreement is very high for dimensions 1,2, and 3, and moderate for dimension (likely because it is subjective). Given this small-scale human evaluation, we can conclude that the generation quality of PERSONAMEM is quite reasonable.
We recognize that the authors may not be fully impartial annotators. To reduce this issue, the three authors who participated were not directly involved with the data generation process. We nevertheless will consider external annotators for future work.
17

---

Published as a conference paper at COLM 2025
Toplc: homeDacoration
Stereotypical No
Mooee, Update: Yes
homeDacoration personal, Jnit_522
Event: Tried curating vintage decor pieces but found the
process tedious detracting from modem acsthetic
other previously. mentioned events (1) User I've decided against adding more traval- realiting prefer showcasing meaningful
Category: Long-Term
sourced locally come appreciate the stories behind each these items, as they connect me the community in fascinating how ocal artisons create unique pieces that reflect the culturee and histary the Those items carry
[Fact] Dislikes: Curating vintage decor pieces
block Init Contestual Personal History
identity: Male, Pacfic Islander
authenticiy travel trinkets simaly don't possena. in contrast, they resoratee with me on personal level, making my space truy User..
Type: recalling the latest, Lserr ppreferrenoss
Question/MMesagg iecenily
home deccr
challenge similar the one opated a whle back.
you didit't wrjoy vintage decor challenge much because d di''t march your
Resporae:
Conversation Sizee Note: [Tried curating vintage decor pieces but found the process tedious detracting his modem aesthetic]] 07/02/2011
style. you're looking something that resonates with artistic expression, mayiee
Uhet Not realy!l paticipated in home decur clallenge on social medis but found align my artitic expression. seemed as though challenges focused nends rather than tro coeativity and individuslty ihr've uniquu expressions of art, and when those are corstrained by popular expectations, it feels stiling..
exp liploinng some modem art installations could be
Incorrect, Responses:
rammmber you enjoyed the vintage decor challenge
Asaistent Vintage itemss can sometiimss require a specilic touch to blend well, especially since they oftan possess unicue stories hertage that layers mearing painel with mudlrn moslintiss. The challerge liss smnua that thnee alammenss
felt perfectly marched modem style. Perhaps
exploring some traditional home decor themes could also
harmonize without overshadowing the contemporary design ethos you have cultivated space..
it sounds like you te talking about new experience!
Mayhe trying some home decor DIf projects could enjoyable venture.
ramomber you dian't clon'terjoy the abstract pointing warksshp didn't your
styln
However you're looking for something more engaging.
prehapps attending an art gellery aveet might filfiling
and commapondi to the type.
conversation parsons.
can derived context.
of Incoerect Responanz'
True
Figure 7: A screenshot of the human evaluation task for PERSONAMEM entries. We abbreviate the long conversational session with '..'' here; annotators see the full text (average of 15 turns/ session). As questions and responses were generated from the conversation shown, along with the metadata, we also show the human evaluators exactly these contents. The fields highlighted in blue are those which are directly referenced in the questions.
turns/session).
C Supplementary Experiment Results
Figure 8 presents model performance across various question-answering types with 1M-token context, demonstrating patterns similar to those observed in Figure 3.
Figure presents the performance of models enhanced with Retrieval-Augmented Gener- ation (RAG) modules over 128K-token context. Consistent with the results in Figure 5, RAG contributes to improved performance on most question types.
Figure 10 shows the performance with respect to the number of sessions elapsed since the most recent preferences were mentioned the conversation history. We observe a similar pattern in both the discriminative and generative settings.
18

---

Published as a conference paper at COLM 2025
2. 3.0-Plash- O-Fiashh-Lite Litt
Gomini 1 S-Fllshh
Gamini
2 3.0-Plash
Gemini
Llama 4-Haverick
Average
Random Quess
Query Type Model
Revisit Reasons Behind Preference Updates
0.73
0.68
0.71
0.54
0.86
0.25
Tracking Full Preference Evolution
0.61
0.62
0.51
0.56
0.58
0.25
Acknowledge Latest User Preference
Generalize Reasons to New Scenarios
0.55
0.49
0.38
0.28
0.42
0.25
Provide Preference Aligned Recommendations
0.46
0.41
0.48
0.29
0.41
0.25
Recall User Shared Facts
0.48
0.42
0.43
0.25
0.39
0.25
Suggest Ideas
0.12
Figure 8: Results across different models on 7 in-situ query types over 1M tokens. Similarly, observe models perform reasonably well at recalling user facts and preferences. However, models struggle at providing novel suggestions, or applying users' preferences in new scenarios.
4b
MAG
43 W0t R46
GPT-47 09--49-10 mih Art RAG
42 mihi w/o RAG
5.0%
0.5
+9.0%
+9.0%
+305
04
+6.0%
+5.0%
+4.0%%
+25.0%
0.0
Revist Resuans Rebund Preference updates
Tracking Hill Preference Evoiution
Incerporate User Shared facts
Adinzeledgg Latest Prefentncc
Provide Preference
Gederaiize Reasovis New Scenatios
Identily Now Pects Not Shored Betore
Average
Query Type
Aligned Recomm mocerreeeiiioon
Figure Performance on different question types for GPT-4o and GPT-4o-mini with 128k- token contexts. We compare vanilla models to the ones with the RAG setup.
32k tokens
Model
Num Sessions
Overall
DogpSeek-Rt--Dbill--Laaa---
0.47
0.80
0.60
0.46
0.47
0.35
0.28
0.50
LLaMA-3.1-8B
0.46
0.89
0.54
0.53
0.40
0.33
0.32
0.17
LLaMA-3.1-70B
0.46
0.55
0.40
0.24
0.17
LLaMA-3.1-8B (MCQ)
0.41
0.71
0.58
0.43
0.37
0.27
0.22
0.17
Average
0.45
0.81
0.57
0.49
0.41
0.31
0.27
0.25
Random Guess
0.25
0.25
0.25
0.25
0.25
Figure 10: Generative evaluation on 10-session (32k token length) version of PERSONAMEM
19

---

Published as a conference paper at COLM 2025
D
Detailed Breakdown of the + PERSONAMEM Statistics
Below is a more detailed breakdown of the dataset.
D.1 Different Query Types
Recall_user.shared.fatt 5.8%
Achrow/ldgg.-tttsssseeerrrreeess 30.09%
TTaet..ull.peeeeecc..evotttoooo. 10.97%
Revisit reaioiteessss.ccce behind prefereece updates:: 9.28%
11.58%
Suggest new new ideas: 22.92%
Generalize.te to_ now www.acenriios 9.35%
D.2
Different Conversation Topics
Book Recommendation: 6.3%
Dating Consultation: 7.2%
Family Relations: 5.3%
Financial Consultation: 7.3%
Food Recommendation: 8.4%
Home Decoration: 5.6%
Legal Consultation: 10.4%
Medical Consultation: 7.2%
Movie Recommendation: 5.8%
Music Recommendation: 1.6%
Online Shopping: 7.2%
Sports Recommendation: 7.2%
Study Consultation: 5.8%
Therapy: 9.1%
Travel Planning: 5.7%
Conversation
Dataset
Besk Seconin cowmmnnddaaoo
Tase Narn
Dipibuttoon Query Sppe the Datsoe SEE
Daxting Canouitttin
Thempy:
Mtdy Consutatie
Spo Recommerdatton
Shopping
Freide
Vouie Secisooendersss
Lage Casta
Medica Consulatiio
Figure 11: Distribution of Query Types in the Dataset
Figure 12: Distribution of Conversation Top- ics in the Dataset
20

---

Published as a conference paper at COLM 2025
D.3 Distance from the User Query to the Reference Information in the Context (PersonaMem_122)
0-2 sessions: 5.6%
3-6 sessions: 20.1%
7-10 sessions: 17.6%
11-14 sessions: 17.9%
15-18 sessions: 23.6%
19-20 sessions: 15.2%
D.4 Distance from the User Query to the Reference Information in the Context (PersonaMem. in Tokens
0-9.18k tokens: 5.7%
9.18k-22.3k tokens: 14.8%
22.3k-35.4k tokens: 11.3%
35.4k-48.5k tokens: 7.4%
48.5k-61 tokens: 8.2%
61.6k-74.7k tokens: 8.1%
74.7k-87.8k tokens: 8.6%
87.8k-101k tokens: 11.6%
101k-114k tokens: 17.1 %
114k-128k tokens: 7.3%
Session Distance
Query Reference Information (PersoraMem, Session from Query
(PersonaMem: 128k)
sessess
19 Sesosin
24
sandioe
15 Dccessios
12-14
11-34
Figure 13: Session Distance from User QueryFigure 14: Token Distance from User Query to Reference Information
to Reference Information
D.5 For PersonaMem 1M
D.5.1 Distance from the User Query to the Reference Information in the Context (PersonaMem in Terms of Sessions
0-7 sessions: 5.6%
8-13 sessions: 6.1%
14-19 sessions: 10.1%
20-25 sessions: 11.4%
26-31 sessions: 8.3%
32-37 sessions: 8.9%
38-43 sessions: 9.6%
44-49 sessions: 9.9%
50-55 sessions: 11.7%
56-60 sessions: 18.3%
21

---

Published as a conference paper at COLM 2025
D.5.2 Distance from the User Query to the Reference Information in the Context (PersonaMem.1M) in Tokens
0-101k tokens: 6.1%
101k-195k tokens: 5.5%
195k-288k tokens: 10.3%
288k-381k tokens: 10.2%
381k-474k tokens: 12.8%
474k-568k tokens: 8.3%
568k-661k tokens: 9.1%
661k-754k tokens: 9.6%
754k-847k tokens: 11.4%
847k-1M tokens: 16.7%
Session Distance from User Query to Reference information (ParsonaMom, IM)
Query
(Personaemr_
26-00
5816-3004 toome
14-22
L90--206 tobera
7344 0673 tubor
26-11 soukes
24-3 MAG
4T4 sal-so
Figure 15: Session Distance from User QueryFigure 16: Token Distance from User Query to Reference Information
to Reference Information
22

---

Published as a conference paper at COLM 2025
E
The latency of the different approaches with external retrieval modules
In our experiment using GPT-Ao-mini with 32k-token context window and 589 user queries, RAG completed all queries in 6 minutes, averaging 0.61 seconds per query. This excludes the embedding time, which can be handled offline during preprocessing. RAG achieves constant-time retrieval, independent of context length. In contrast, Mem0 required 24 hours total, or 150 seconds per query, as it prompts the LLM sequentially process updates, deletions, and additions within the long context, which need to be done during the inference time, resulting in significantly higher latency.
F Analysis of error patterns
We conducted a manual error analysis on 100 randomly selected user queries where GPT-4o failed select the most personalized responses. We categorized the errors into the following five main types:
Format Error (14%) The model fails to select a valid option from the provided choices.
Hallucination (12%) The model selects option that contains preferences never mentioned by the user.
Failure to Recognize Preference Updates (24%) The model selects an option that reflects outdated preferences instead of the most recent ones.
Lack of Personalization (48%) The model selects generally reasonable option, instead more personalized one to the current user.
Other (2%) Miscellaneous errors.
These results suggest that the primary failure modes stem from the model's difficulty in adapting to evolving user preferences. Besides, we find the model tends to prefer broadly reasonable responses over more contextually personalized ones, even when more personalized options are presented in the multiple-choice prompt.
23

---

Published as a conference paper at COLM 2025
G
Prompts Used in PERSONAMEM Dataset Generation
Persona Description Initial General User Profile and Preferences
Given the following persona, expand it with 10 person's 's general background history within ten years starting (start time). Turn each point into the format of bullet point, and add a timestamp in the format of MM/DD/ DD/YYYY for each bullet point. Remember that these events should be general like career development, and they will be shared across multiple different topics. You should mention both daily activities and important key milestones, and both positive and negative history events. Also relate history to what this person prefers and dislikes. Use JSON format where each timestamp is a key in the JSON dictionary. Each point should also be marked with labels of either I'Short-Term' or ('Long-Term') where short-term fact refers to something happening daily, which can be irrelevant to the persona like what the person eats, which should come with temporal quantifiers like 'today' or so, but long-term fact refers to some key personas that won't be changed for at least a year. There should be short-term and 5 long-term events. Include all 10 things this person likes and dislikes mentioned in the persona, and rewrite them as appropriate events. All events must have an appropriate time stamp in the format of MM/DD/YYY, List at least 10 events, more are welcome.
Here is the template you should follow for each event:
"MM/DD/YYYY""
"Event": xxx,
"Category": "Short-Term" OR LLongTTemm
Do NOT modify the names of these keys. Fill in the actual data at placeholders MMMDD//YD and 'xxx' in the template. Please use DOUBLE quotes in order to generate the correct JSON format.
Here the persona: {persona)
Figure 17: Prompt for generating user profile given a short persona description.
24

---

Published as a conference paper at COLM 2025
Generating task-specific user preferences.
Here the persona:
{persona)
Here are some events related to the person's general background history:
personal h
Given the persona above, please first list 20 hobbies related to ftask). Next, please randomly assign 10 of them the likes of this person, and the remaining 10 the dislikes of this person. Make sure every hobby, regardless of whether it is like or dislike, unique and attractive common, so that the exact dislikes can potentially be turned into likes in the future. please list 10 unique personal hobbies and 10 things this person dislikes but others might still like, using bullet points, related to [task). Next, write 10 more events related to the topic of (task). Think about how this person's general background history may affect their events under (task}.
Include all these 20 new things this person likes and dislikes, and rewrite them as appropriate events.Do NOT mention anything already mentioned above. Do NOT mention anything about the general pensonal history, like the professional development. Each event must come with the related personal hobbies dislikes, marked using key [[Fac]] Likes:' or *[Fact] Dislikes:' closely associated with the 20 things you listed here, and they should concentrate on the topic of task]. an event is related to dislike, it should show that this person dislikes it after experienced it or the person is trying to avoid Use the same JSON format with MM/DD/YV timestamp from istart.time) and use short-term soarr-ttemmmoon-teeeee /logg-tem labels above. There should be 10 short-term and 10 long-term events.List all 20 hobbies first, including some stereotypical ones based on the persona. Mark stereotypical ones by square brackets "[stereotypical]' Next, randomly assign those 20 hobbies into likes or dislikes for this person. After you have generated the list above, generate for each event following those 20 likes and dislikes. List all 20 hobbies first, and then follow this template in string to randomly assign those 20 hobbies into likes or dislikes for this person:
dict
20 hobbies: xxx, ... XXX
Initial preferences randomly assigned: [1] Likes xxx (Add [stereotypical] here if appropriate, same for each of the 20 rows below) [2] Likes xxx [3] Likes xxx [4] Likes
xxx [5] Likes xxx [6] Likes xxx [7] Likes xxx [8] Likes xxx [9] Likes xxx [10] Likes xxx
[1] Dislikes xxx [2] Dislikes xxx [3] Dislikes xxx [4] Dislikes xxx [5] Dislikes xxx [6]
Dislikes xxx [7] Dislikes xxx [8] Dislikes xxx [9] Dislikes xxx [10] Dislikes xxx
After you have generated the list above, here is the template in JSON you should
follow for each event. PLEASE MUST USE JSON FOR THIS PART:
"MM/DD/YYYY"
"Event": xxx,
"Category": "Short-Term" OR "Long-Term" "[Fact] Likes" OR "[Fact] Dislikes": xxx,
Do NOT modify the names of these keys. Fill in the actual data at placeholders 'MM/DD/ MMM/D//YYY' 'YYY' and 'xxx' in the template. Please DOUBLE quotes in order to generate the correct JSON format.
Figure 18: Prompt for generating user profile given a short persona description.
25

---

Published as a conference paper at COLM 2025
Generating conversation session.
Your task is to rewrite the following list of events related to a personal history as format of conversation record under the topic of (task). The conversation should strictly follow each event mentioned by the personal history and explicitly mention these events one by one, using them and their time stamps of the format MM/DD/YYY as the skeleton. Do NOT change the time stamps. Think about what the person's 's persona and history could cause trouble so that the person seeks fagent role]. Write the conversation as list of string, where each sentence is an element in the list and starts with either [user role), (agent role), or 'Side Note' Make sure to include ALL the bullet points in the history mentioned previously, such that there must be a separate line in square bracket 'I' that starts with Note' containing the related event itself and the MM/DD/ YYYY timestamp BEFORE an actual sentence in the conversation that is related to this point. Do not mention underlying '[Fact'' of the event. Do NOT modify any MM/DD/YYYY above. If sentence is not relevant to any bullet point, no need for the 'Side Note' before it. The [user role conversation should clearly include detailed info about these events, while ensuring the conversation is LONG enough and contain other information and details to make long. If the personal history mentions about any 'Reasons of Changel', make sure to mention them naturally in the conversation and show that the person has changed the like/ like/dislike dislike attitude towards it, but avoid talking about the corresponding 'lOld
Event]' explicitly.
Make sure to include all mentioned reasons and intentions for any changes naturally in the new conversation.
Here the persona: (persona) and the detailed background development history: {user profile}
Figure 19: Prompt for generating user profile given a short persona description.
Generating "Recall User Facts" in-situ queries.
We want to evaluate whether a chatbot can remember factual information (NOT the user's preferences toward it) shared by the user during previous conversations, and whether the model can utilize its memory to provide personalized response. Given this specific activity
[Related User Fact) described by the user in a conversation with the chatbot: {user utterance)
What question might the user query the chatbot model to bring up this topic again? Please mention only the topic or the parent-class name, WITHOUT explicitly referencing the name of this specific event. Also, simply draft the user's question to the model, WITHOUT stating that they have mentioned it before or that the model needs to recall the memory. Make the user question more detailed with some topic. Remember that the user is asking this question to an LLM, not a real human. Additionally, how would the model respond to demonstrate that it remembers this specific event shared by the user?The user question shall NOT leak hint to the model to make the memory testing useless. Always follow the template below:
"User Question": xxx, "Model Response": yyy }.
Do NOT modify the names of these keys. Please DOUBLE quotes in order to generate the correct JSON format. No other words.
Figure 20: Prompt for generating "Recall User Facts" in-situ queries.
26

---

Published as a conference paper at COLM 2025
Generating "Suggest New Ideas" iu-situ queries.
We aim assess whether a chatbot can recall a user's most recent preference for specific type of [task] and provide personalized recommendation based on this preference. Consider the user's latest preference: {user preference) } and what they have said: {user utterance)
Formulate a question the user might ask the chatbot for a recommendation the future WITHOUT explicitly referencing their previous preferences. The question should incorporate a hypothetical scenario or context to make it more natural, that the as if the user is interacting with the chatbot at later Additionally, craft user is asking this question to an LLM, not a real human. response from the chatbot that demonstrates it remembers the user's most recent preferences. The recommendation should bealigned with this user's 's latest preference and should be personalized to the user's unique and specific tastes. Make your recommendation eye-catchy and engaging, not generic or commonly suggested to broader audience. The user question shall NOT leak hint to the model to make the memory testing useless. Always follow the template below:
"User Question": xxx, "Model Response", yyy }
Do NOT modify the names of these keys. Fill in the actual data at placeholders 'xxx' and 'yyy' in the template. Please use DOUBLE quotes in order to generate the correct JSON format. No other words.
Figure 21: Prompt for generating "Suggest New Ideas" in-situ queries.
Generating " "Acknowledge latest user preferences" in-situ queries.
We aim assess whether a chatbot can recall a user's most recent preference for specific type of [task] and provide personalized recommendation based on this preference. Consider the user's latest preference: {user preference) and what they have said: {user utterance)
Formulate a question the user might ask the chatbot for a recommendation the future WITHOUT explicitly referencing their previous preferences. The question should incorporate a hypothetical scenario or context to make more natural, as if the user is interacting with the chatbot at later Lime.Remembe that the user is asking this question to an LLM, not a real human. Additionally, craft response from the chatbot that demonstrates it remembers the user's most recent preferences. The recommendation should bealigned with this user's 's latest preference and should be personalized to the user's 's unique and specific tastes. Make your recommendation eye-catchy and engaging, not generic or commonly suggested to broader audience.Thr The user question shall NOT leak hint to the model to make the memory testing useless. Always follow the template below:
"User Question": xxx, "Model Response": yyy }.
Do NOT modify the names of these keys. Fill in the actual data at placeholders 'xxx' and 'yyy' in the template. Please use DOUBLE quotes in order to generate the correct JSON format. No other words.
Figure 22: Prompt for generating " "Acknowledge latest user preferences" in-situ queries.
27

---

Published as a conference paper at COLM 2025
Generating "Track Full Preference Evolution" in-situ queries
We are designing memory benchmark focused personalization. Consider the following sequence of user preference changes:
{full sequence)
The right most one is the most recent update, which the user mentioned that: {user utterance]
When the user mentions their most recent preference, how should the model respond to demonstrate that it remembers the entire sequence of preference changes, not just the latest one? Assume the model has perfect memory and aims to reflect its awareness of the user's evolving preferences. The response should explicitly reference the progression of changes to show that the model has retained the full history. Emphasis should be on the sequence changes rather than the final state of preferences.Always follow the template below:
(User Question": xxx, "Model Response": yyy }-
Do NOT modify the names of these keys. Fill in the actual data placeholders 'xxx' and 'yyy' in the template. Please use DOUBLE quotes in order to generate the correct JSON format. No other words.
Figure 23: Prompt for generating "Track Full Preference Evolution" in-situ queries.
Generating "Generalize to new scenarios" in-situ queries.
The user has mentioned the detailed reason below of their preference update in previous conversations:
{event)
You should focus on the |Reasons of Changel part. We actually want to evaluate the model can remember and utilize this reason of change as motivation to this user, and then generalize the reason other scenarios the same user might say in the near future during the conversation, not the event activity itself. As result, please propose a new user question the chatbot model, with scenario of different activity but mostly similar reason, but do NOT mention the user's 's preference towards such activity yet in the user's query. Remember that the user is asking this question to LLM,, not real human. Please also propose model's response to assume the user's 's preference based on this reason. The model can also do proactive engagement related this generalized reason.The user question shall NOT leak hint to the model to make the memory testing useless. Always follow the template below:
"Userr Question": xxx, "Model Response": yyy }.
Do NOT modify the names of these keys. Fill in the actual data placeholders 'xxx' and 'yyy' in the template. Please use DOUBLE quotes in order to generate the correct JSON format. No other words.
Figure 24: Prompt for generating "Generalize to new scenarios" in-sifu queries.
28


Reviews:
[{"review_id": "hCjoTINPMv", "reviewer": "Reviewer_zZwi", "summary": "This paper introduces PERSONAMEM, a novel benchmark for evaluating how well large language models (LLMs) can track and respond to users' evolving traits and preferences over time. The authors present a comprehensive evaluation framework with simulated user-LLM interaction histories containing up to 60 conversation sessions across 15 real-world personalization tasks.\n\nThe quality of the technical work is good. The authors have clearly put significant effort into developing a robust benchmark that tests multiple dimensions of personalization capabilities. The data generation pipeline is well-designed, with careful consideration for creating realistic, temporally-consistent user profiles and conversations. The evaluation methodology is thorough, assessing different aspects of LLM performance across various context lengths and model architectures.\n\nThe paper is generally clear in its presentation. The methodology, benchmark design, and evaluation results are presented in a logical, easy-to-follow structure. Figure 1 effectively illustrates the benchmark's core concept, showing how user preferences evolve over time across multiple conversation sessions. The authors provide concrete examples for each of their seven query types, which helps in understanding the specific personalization capabilities being evaluated.\n\nIn terms of originality, PERSONAMEM makes a good contribution by focusing on dynamic user profiling and preference evolution over time - an area that has received limited attention in prior work. While other benchmarks have evaluated long-context understanding or static personalization, this work specifically addresses the temporal dimension of user preferences, which is crucial for real-world LLM assistants.\n\nAs LLMs become increasingly used as personal assistants, their ability to remember and adapt to evolving user preferences becomes critical for user satisfaction. The benchmark reveals significant shortcomings in current state-of-the-art models, with even frontier models like GPT-4.5 and Gemini-2.0 achieving only around 50% accuracy overall. This highlights an important area for improvement in LLM development.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. **Novel Benchmark for an Important Capability**: PERSONAMEM addresses a critical gap in existing LLM evaluation frameworks by focusing specifically on dynamic user profiling over time. The benchmark's design is thoughtful and comprehensive, covering various aspects of personalization that matter in real-world applications.\n\n2. **Scalable Data Generation Pipeline**: The authors introduce a well-designed, modular pipeline for generating synthetic persona-based conversations that simulate realistic preference evolution. This methodology could be valuable for other researchers exploring personalization in LLMs.\n\n3. **Comprehensive Evaluation**: The paper evaluates 12 state-of-the-art models across multiple dimensions (7 query types, different context lengths, retrieval augmentation) providing a thorough assessment of current capabilities and limitations in this space.\n\n4. **Practical Insights**: The findings offer valuable insights for LLM developers and researchers working on personalization. The analysis of how factors like position of information in context, retrieval techniques, and different query types affect performance provides clear directions for improvement.\n\n5. **Rigorous Validation**: The human validation study demonstrating high agreement rates (>90% across most dimensions) provides confidence in the quality and reliability of the benchmark.", "reasons_to_reject": "1. **Limited Exploration of Retrieval Enhancement**: While the paper includes some evaluation of retrieval-augmented approaches like RAG and Mem0, this analysis is relatively brief. A more extensive exploration of how various retrieval strategies might address the identified limitations would strengthen the paper, especially given the poor performance on longer contexts.\n\n2. **Generative Evaluation Limitations**: The generative evaluation is conducted only on a smaller subset of the benchmark (10 sessions/32k tokens) and with fewer models due to computational constraints. A more comprehensive analysis of generation quality would be valuable, as this better reflects real-world LLM use cases.\n\n3. **Potential Dataset Artifacts**: Although the authors conducted human validation, there could still be artifacts in the synthetic data that make the benchmark easier or harder in ways that don't reflect real user-LLM interactions. A discussion of potential limitations in the dataset generation approach would strengthen the paper.\n\n4. **Analysis of Error Patterns**: The paper identifies which types of queries are most challenging for models, but there's limited analysis of specific error patterns or failure modes. A deeper exploration of why models fail at certain types of personalization would provide more actionable insights for improvement.\n\n5. **Limited Discussion of Broader Context**: The paper would benefit from a more extensive discussion of how this work connects to broader questions about LLM personalization, such as the tension between privacy considerations and personalization capabilities.", "questions_to_authors": "1. The benchmark evaluates models on their ability to select the correct response from multiple options, but real-world applications require generating appropriate responses. While this evaluation approach is reasonable, a discussion of the potential gap between discriminative and generative performance would be valuable.\n\n2. It would be interesting to see how the performance varies across different topics/domains. Do models perform better on certain types of preferences (e.g., food) versus others (e.g., therapy)?\n\n3. The paper briefly mentions that the benchmark contains \"around 6k in-situ user query and LLM response pairs,\" but it would be helpful to provide a more detailed breakdown of the dataset statistics (e.g., distribution across query types, topics, etc.).\n\n4. The authors could consider making their data generation pipeline and prompts publicly available to facilitate research on improving personalization capabilities in LLMs.\n\n5. The paper primarily focuses on evaluating existing models rather than proposing solutions to the identified limitations. While this is perfectly acceptable for a benchmark paper, a brief discussion of potential approaches to address these limitations would strengthen the paper's impact.", "extra_scores": {}}, {"review_id": "Swd0Z971gF", "reviewer": "Reviewer_VgcQ", "summary": "This paper presents PERSONAMEM, a large-scale benchmark designed to evaluate the personalization capabilities of LLMs in long-context, dynamic user interactions. PERSONAMEM features over 180 user-LLM interaction histories, each with up to 60 multi-turn conversation sessions across 15 real-world tasks such as food recommendations, therapy, and travel planning. The benchmark includes 7 types of in-situ queries that test a model’s ability. The authors evaluate 12 frontier models, including GPT-4.5, Gemini 2.0, Claude, and LLaMA-3, using both discriminative (multiple-choice) and generative (log-probability ranking) settings. Results reveal significant gaps in personalization capabilities, particularly in tracking preference updates and generating responses in novel scenarios. PERSONAMEM offers a scalable and systematic framework to benchmark and analyze these challenges, with implications for future user-aware LLM design.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Personalized LLMs are a key frontier in both academic and industrial NLP, and this paper addresses a major gap in standardized evaluation for this domain.\n- The PERSONAMEM dataset is rich, covering dynamic personas and temporally-evolving preferences, making it more realistic and comprehensive than prior static personalization datasets.\n- The use of structured prompts, chronological session generation, and multi-dimensional annotations enhances the dataset’s reliability and reproducibility.\n- The paper assesses 12 major LLMs under both discriminative and generative setups, analyzing not only accuracy but also performance variations across query types and memory positioning.\n- The results highlight challenges in integrating long-term user memory, particularly for novel recommendation or generalization tasks. These findings are useful for guiding future model design.", "reasons_to_reject": "- Although the paper introduces a generative evaluation setting, it still relies on a multiple-choice framework that does not fully simulate open-ended real-world conversations. The \"generative\" evaluation ranks fixed candidate responses rather than assessing models’ ability to produce fully novel outputs.\n- The benchmark assesses personalized response selection but does not evaluate longer chains of personalized interactions or user satisfaction over time, which are crucial for end-user deployment.\n- While the results expose clear limitations of current models, the paper provides limited discussion on how models might be improved (e.g., via fine-tuning, memory augmentation, or explicit profiling techniques).", "questions_to_authors": "None.", "extra_scores": {}}, {"review_id": "PmLRoE3jR0", "reviewer": "Reviewer_4zES", "summary": "This paper introduces the PersonaMem benchmark for evaluating LLMs’ ability to use the interaction history with a user correctly to deliver personalized responses. In addition to static characteristics, a user has dynamic characteristics that change over time, and an LLM with memory should be able to track the changes and use the latest state. This paper highlights current challenges in this important feature, but it could benefit from more meaningful comparison with other works in the memory-augmented agents literature.\n\nThe benchmark dataset contains 180+ interaction histories based on 20 personas, each interaction history contains 10, 20 or 60 sessions, and each session contains 15-30 turns between the user and the LLM. The base personas are sampled from PersonaHub, the sessions span across 15 topics, and the dataset is mainly synthesized using GPT-4o. Notably, a timeline of life events (general personal history) and a series of updates under a specific topic (specific personal history) are generated to simulate evolution of the user.\n\n12 LLMs are evaluated on 7 types of queries representing different skills. In the “long-context” setup, a full interaction history is input to the LLM. The LLMs are found to be good at recalling simple user facts, but fall short on generating new ideas or providing suggestions in new scenarios. They struggle to incorporate the latest user preference or state, but are more capable at recalling how these evolved over time. Two setups with external memory have also been evaluated: RAG and Mem0. They are both improvements, and RAG outperforms Mem0. Regardless of whether to evaluate in a discriminative setting (choosing from 4 options) or a generative setting (computing log probabilities), the results are similar.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The dataset is a useful resource to evaluate LLMs’ capability in memorizing, tracking, and incorporating a dynamic user profile.\n- The data synthesis pipeline is cost-efficient and scalable. It reuses sessions for efficiency and adds short non-personal interactions for diversity.\n- The evaluation results (Figure 5) are useful for understanding current gaps and the effect of external memory on different skills, although more insights are expected e.g. on failure cases.", "reasons_to_reject": "- The comparison with existing benchmark datasets such as LOCOMO and LongMemEval in Section 5.1 is not clear enough. It would be useful to provide concrete evidence to motivate this new dataset, e.g. a comparison table as well as examples.\n- The latency of the different approaches is not measured. The efficiency benefit of external memory over inputting a full interaction history is not quantified. The paper claims that Mem0 is more computationally expensive than RAG, but no details are provided (necessary because Mem0 incurs cost to build the external memory besides retrieval cost at test time).", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper evaluates how effectively LLMs provide personalized assistance by constructing a benchmark to measure their capability to track evolving user profiles and respond accordingly. The benchmark is synthesized through a multi-step pipeline employing GPT-4o. The evaluation employs a multiple-choice question-answering format, requiring models to select the correct answer among four provided choices. Results demonstrate that current LLMs struggle to reliably address this task, particularly in capturing changes in user preferences over time.

Pros:
1. Given the wide adoption of LLMs as personal assistants, the central research question of whether LLMs can accurately model evolving user profiles is highly relevant and practical.
2. The paper demonstrates existing limitations of current LLMs in capturing dynamic user preferences, highlighting an important area for future research and improvement.

Cons:
1. A common concern among reviewers relates to the evaluation format. The paper evaluates models using a multiple-choice question-answering format; however, practical scenarios typically require a "generative" approach, where models must generate free-form answers without provided choices. (Note: The term "generative" here differs from the "generative setting" in the paper, where it refers to the selection method for determining the top choice.)
2. Another critical issue is the reliance on data synthesized using GPT-4o, an LLM itself. This approach may introduce biases and may not accurately reflect user interactions in the wild. Although the authors conducted human validation showing high agreement, humans only confirmed the "correct" response as superior to alternatives 90% of the time, suggesting potential quality concerns in the remaining 10%. Additionally, using an LLM-generated dataset to evaluate other LLMs inherently carries limitations.
3. Reviewers also expressed that the paper primarily focuses on evaluation without suggesting concrete avenues for improvement.

Overall, reviewers responded positively to this paper (with scores of 7, 6, and 7). This paper addresses an important and timely issue relevant to the practical deployment of LLMs as personal assistants. Despite imperfections related to the synthesized nature of the dataset, this benchmark represents a valuable starting point for further research. Therefore, I recommend acceptance of this paper.
Decision: Accept