CUPID: Evaluating Personalized and Contextualized Alignment of LLMs from Interactions
Authors: Tae Soo Kim
Yoonjoo Lee
Yoonah Park
Jiho Kim
Young-Ho Kim
Juho Kim
Venue: COLM 2025 (2025)
Keywords: LLM
Evaluation
Benchmark
Personalization
Preferences
Interactions
TL;DR: We introduce CUPID, a benchmark that evaluates LLMs capability to infer a user's contextual preferences from prior user-LLM interactions.

Abstract: Personalization of Large Language Models (LLMs) often assumes users hold static preferences that reflect globally in all tasks. In reality, humans hold dynamic preferences that change depending on the context. As users interact with an LLM in various contexts, they naturally reveal their contextual preferences, which a model must infer and apply in future contexts to ensure alignment. To assess this, we introduce 🏹 CUPID, a benchmark of 756 human-curated interaction session histories between users and LLM-based chat assistants. In each interaction session, the user provides a request in a specific context and expresses their preference through multi-turn feedback. Given a new user request and prior interaction sessions, our benchmark assesses whether LLMs can infer the preference relevant to this request and generate a response that satisfies this preference. With CUPID, we evaluated 10 open and proprietary LLMs, revealing that state-of-the-art LLMs struggle to infer preferences from multi-turn interactions and fail to discern what previous context is relevant to a new request—under 50% precision and 65% recall. Our work highlights the need to advance LLM capabilities for more contextually personalized interactions and proposes CUPID as a resource to drive these improvements.

Published as a conference paper at COLM 2025
X CUPID: Evaluating Personalized and Contextualized Alignment of LLMs from Interactions
Tae Soo Kim Yoonjoo Lee Yoonah Park *
Jiho Kim
Young-Ho Kim Juho Kim
KAIST Seoul National University
Q Calvin University
NAVER AI LAB
taesoo. kinbkaisimkkist..o.o kc.
Abstract
Personalization of Large Language Models (LLMs) often assumes users hold static preferences that reflect globally in all tasks. In reality, humans hold dynamic preferences that change depending on the context. As users interact with an LLM in various contexts, they naturally reveal their con- textual preferences, which model must infer and apply in future contexts
to ensure aligmment. To assess this, we introduce X CUPID, bench- mark 756 human-curated interaction session histories between users and LLM-based chat assistants. In each interaction session, the user provides a request in a specific context and expresses their preference through multi- turn feedback. Given a new user request and prior interaction sessions, our benchmark assesses whether LLMs can infer the preference relevant to this request and generate response that satisfies this preference. With CUPID, we evaluated 10 open and proprietary LLMs, revealing that state-of-the-ee LLMs struggle to infer preferences from multi-turn interactions and fail to discern what previous context is relevant to new request-unde 50% precision and 65% recall. Our work highlights the need advance LLM capabilities for more contextually personalized interactions and proposes CUPID as a resource to drive these improvements.
https: thub. wwt.///////o//u////
1 Introduction
Interaction Session
mestommrtiin pesso
Large Language Models (LLMss) have shown remarkable capabilities across various tasks (Nakano al., 2021; Achiam al., 2023), benefiting users through diverse applications and conversational assistants (OpenAI, 2022; An- thropic, 2023; Microsoft, 2023). Aligning these models with human values and preferences is crucial as they are increasingly integrated into user experiences (li et al., 2023). Initial efforts focused on aligning LLMs on broad, general values (e.g., helpfulness, harmlessness, honest) (Bai et al., 2022a;b) or the aggregated preferences of diverse users (Ouyang et al., 2022; Köpf et al., 2024). Recognizing that these approaches overlook users' diverse expecta- tions, more recent work focuses on personalized alignment, where LLMs are aligned with the users' individual preferences (Wu et al., 2024b; Jang et al., 2023; Lee et al., 2024). Despite this progress, existing work largely assumes that users have static preferences (i.e,, preferences are global and do not vary over time) (Wu et al., 2024b; Lee et al., 2024) focus stable characteristics (e.g., sociodemographics) to characterize preferences (Kirk et al., 2024).
Contect
tractiinnalll epethods
Here's revised version of you matheratiion proof: Thwcrere for any
Chen
Commentu Preference
with teams there optimnal
Support aeguments seing
clannical mathods rether
Coul you romorkk the proof avord relying on
Interaction
Hile no witte the methodorig ogy section of out paper with Dr. Park
Hire cra't
Factor
ancouraged
innoustive and creanue amprosches
Support argumente
computational methods.
Interaction Session
Inced help cevelosing a proof stratory for a now theooem discuss Dr. Cheo
What preference will the user have this context?
Figure 1: Example instance in IX CUPID illustrates a user that holds distinct pref- erences in different contexts due to per- sonal experiences, where the preferences are only revealed to the LLM through the user's 's feedback in prior interactions.

---

Published as a conference paper at COLM 2025
In reality, human preferences are context-dependent (Warren et al., 2011; Baker et al., 2009; Malaviya et al., 2024): an individual's intents and expectations shift depending on their situation. For example, in Figure 1, researcher consults an LLM to refine their paper's writing but their preference varies by collaborator: focus on classical methods with Dr. Chen due to prior disagreements while embracing computational methods with Dr. Park who advocated for them. These contextual variability means that models must understand what values and preferences a user holds in distinct contexts. Interactions between user and an LLM may indirectly reveal these shifting preferences as users provide feedback to the model in diverse situations (Wu et al., 2024b; Kim et al., 2024c). However, current LLMs possess the capability to identify a user's 's preference from their feedback, infer the relevant context, and proactively apply this knowledge in future interactions.
unclear whether
In this work, we present 2 CUPID' (Contextual User Preference Inference Dataset), challenging and scalable benchmark that is designed to assess LLMs' ability to infer users' preferences tied to diverse contexts from user-LLM interaction histories. CUPID consists of 756 human-curated interaction histories, where each history presents a series of interaction sessions or dialogues between a simulated user and an LLM. Each interaction session presents task-oriented dialogue where a user asks a request explicitly stating the context, and then gradually reveals their contextual preference through multi-turn feedback to the LLM. Given a new request and history of previous interaction sessions, the benchmark evaluates whether LLMs can (1) infer the user's s preference related to the context of the new request, and (2) generate a response to the request that will satisfy this preference.
To construct CUPID, we designed pipeline that generates diverse and rich interaction sessions. After generating a persona pool, for each persona, the pipeline generates a list of context factors (i.e., significant people, objects, locations, etc. in the persona's s world that influences their expectations) and contextual preference (i.e., value, principle, or criterion associated with each factor). For each persona, the pipeline then generates series of interaction sessions that involve these factors and preferences. We combine the concepts of LLM-as-a-Judge (Zheng al., 2023; Ye et al., 2023) and LLM-simulated users (Wu et al., 2024a;b) to create multi-turn dialogues where simulated user evaluates and provides feedback LLM's responses, gradually disclosing its contextual preferences. All the data was verified with human annotators and ~9% of the data was manually edited.
With CUPID, we assess 10 open and proprietary LLMs. We find that all models struggle to infer users' preferences based on past interaction sessions, with no model exceeding 50% precision and 65% recall. Models failed at recognizing relevant contexts in prior interactions and extracting preferences from multi-turn conversations. We find strong correlation (r=0.764) between performance in inferring preferences and generating responses that satisfy them, suggesting that the capability to infer context-dependent preferences preffeeenn underpins personalization capabilities. Finally, we present finetuned metric, PREFMATCHER-TBB to reduce the cost of evaluation on our benchmark and larger unverified dataset to support training and further research.
X CUPID Benchmark
CUPID evaluates LLMs' ability to infer a user's distinct preference in different contexts from prior interactions between the user and the LLM.
2.1 Definitions
Our benchmark is composed of interaction sessions S (c,, Pi, D;).
Represents an element (e.g., person, location, tool, activity) in a user's 's environment or world, which plays a role in and influences that interaction session. We assume that each session's context is only defined and influenced by single factor.
Context Factor, c;
2 https://huggingface htt.///nww....teaatttttt//////aa
http://hwgggggeeeeooooo///////ne 3 htttss///hggnngiiee..cc//assssbs
2

---

Published as a conference paper at COLM 2025
Contextual Preference, p; Represents value, principle, criterion, requirement, straint that the user holds when the context factor çj is involved in the situation. While prior work focused soocy"odd") food") (Wu et al., 2024b; Zhao et al., 2025), we focus on more complex, abstract, and task-oriented preferences (e.g., "Instructions must break down complex procedures into num- bered micro-steps with verification points after each major secfion " Preferences in our dataset often require multiple rounds of feedback be satisfied.
more simple and concrete lifestyle preferences (e.g. "I cannot eat
- more examples in Table 6.
Dialogue, D; = (M M1,1 ..., 1 1 1 Represents interaction between user and an LLM. The dialogue begins with the user's 's initial request 4; that explicitly states the context factor Gj- In subsequent turns, the user iteratively provides feedback until the model's responses (m1, 112 -), satisfy preference The dialogue length Li is determined when p; is fully satisfied, confirmed by the user's 's final utterance Wi,L.
2.2 Problem Formulation
Each task instance in CUPID consists of a 4-tuple (Mcurrent, Courrenty Pcurrent/ S), where Meurrent represents the user's 's current request to the LLM, Ceurrent and Peurrent represent the context factor and contextual preference in the new request, and S is the history of previous interaction sessions, where at least one S; E S satisfies cj
Ccurrent and p; Peurrnt-
Our benchmark evaluates LLMs on two tasks:
Inference:
Given Meurrent and D (D; S E S), the model should infer the user's contextual preference Peurrent-
Generation:
Given Mcurrent and D = {D; S E S}, the model should generate
response y that will satisfy or align with the user's 's preference Peurrent-
2.3 Metrics
Inference Preference Match Preferences in our benchmark are complex, with each preference entailing multiple fine-grained expectations (i.e,, sub-preferences) An inferred preference should capture all sub-preferences (recall) without including irrelevant ones (precision). To assess this, we design an LLM-based preference matching metric, inspired by work on measuring factual precision through atomic facts (Min et al., 2023) and fine- grained checklist evaluations (Lin et al., 2024; Cook al., 2024). Specifically, we first use an LLM to decompose the ground-truth and inferred preferences, p and p, into atomic checklists Qp = {91 ..., 9n) and Ôp = ($1,.., )n), where each checklist item 9: or 4: assesses a single sub-preference. Then, we employ an LLM with few-shot prompt (Figure 14) to evaluate whether each checklist item matches the other preference:
1, if aligning with p fully covers 9, 0.5, if aligning with p partially covers 9,
MATCH(9.P)
0, otherwise.
We use GPT-4o for decomposing and matching. For matching, we conducted meta- evaluation with 230 human-annotated data points, separate from our dataset. GPT-4o achieved a Krippendorff' s alpha (Krippendorff, 2004) of 0.769 (substantial agreement) with the majority vote of human annotators. To reduce the cost of evaluating on our benchmark, we also present PREFMATCHER-7BB a finetuned model for preference matching that achieves Krippendorff's alpha of 0.748 (substantial agreement). Prompts, human meta-evaluation details, and finetuning details are in Appendix A.1. Finally, we compute: Precision = 10p Eqeo, MATCH P), and the F1 score.
10pl EjeO, MATCH( p), Recall
1
Generation Preference Alignment For the generation task, we evaluate a model's re- sponse r on its alignment with the contextual preference Peurrent using the LLM-as-a-Juute -Judge approach (Zheng et al., 2023). As prior work has shown that LLMs can evaluate other models' responses diverse skills, principles, Ye et al., 2023), we prompt GPT-4o to provide a score ranging from to 10 on the degree to which model response satisfies the ground-trun -ruth contextual preference. As checklists can
criteria (Fu et al., 2023; Kim et al., 2024b;
3

---

Published as a conference paper at COLM 2025
increase the consistency and reliability of LLM evaluations (Lin et al., 2024; Cook et al., 2024), we also provide the automatically decomposed checklist Qp used in preference matching to the LLM judge. Prompt and details are in Appendix A.2.
3 Data Generation Pipeline (Figure 2)
To capture realistic interaction patterns and context-dependent user preferences (Warren et al., 2011), we designed synthetic user interaction sessions and preferences in CUPID with the following desiderata: (1) User-specific and unique: Focus on highly personalized contexts and preferences not inferrable from prior knowledge or commonsense; (2) Indirect and gradually revealed preferences: Preferences expressed through multiple turns of feedback and clarification to mirror real user behavior; (3) Patterns of shifting preferences: Capture how a user's 's preferences may shift over time or even conflict across contexts. Unlike prior work focused lifestyle preferences, we center our benchmark preferences and dialogues, reflecting real-world usage of LLMs (Tamkin et al., 2024). The pipeline uses Claude 3.5 Sonnet, unless noted otherwise. Full details in Appendix B.
task-oriented
Although synthetic data may not fully represent real user behavior and diversity, we opt for synthetic data over human-collected data because it provides: (1) precise control over dataset difficulty, including how preferences are revealed and contexts are repeated; (2)
higher quality by ensuring that preferences are revealed and models can infer them; and (3) enhanced diversity by varying context factor, preference, and task types.
3.1 Generation Process
Persona Pool We aimed to construct context factors and preferences that were specific and unique to a user (Desiderata 1), rather than general factors (e-g., "Fender Stratocaster" vs. "electric guitar") and preferences (e-g., "instructions must include precise hand positioning details" vs. "instructions must be precise"). Existing persona datasets (Ge et al., 2024; Zhou et al., 2023) lacked sufficient detail to generate unique factors and preferences. Thus, we created new persona pool by sampling seed persona descriptions from Ge et al. (2024), combining them with attributes (e-g,, personality, personal values) inspired by Zhou et al. (2023), and using an LLM to expand each into rich persona cessiipiioss-yieldinng 252 distinct personas.
Constructing Contexts To generate diverse yet internally coherent contexts for each persona, we instruct an LLM to first generate a list of 8 context factors and associated essentiall constructing each persona's world. Before each factor-preference pair, we prompt the LLM to first generate a background narrative to develop more specific
Persona Pool
Contexts
Sessions
Dialogue
Simulaced
Use Smumre
Conserue Podeetnce
Toood holo plarnng Canon MP-E
Technical apacinsaionns
65mm lena using Cancn NP
ecleransily deteded befone
Borm
cutiee
creative supppsttios tows are provided
mandittes in action,
Sample
ChiCh
Simalate Diglague
Persona
Persona Profile
De. Sarah Martines
Skoee Life the tha must to saupor led
lesst two pane-mnieeen medd
Session
Indo mo create tecatmottt for
Use touaee
Sceentiic ten eive mese locus on obserust lncil details without dramctio diemonts.
Proforenos
Nature Documentary
Färmaaker (Senion
Gentexts
onstroct
Geneeate
Sessioris
PGS Natura docurentaryy segment
Doll outterly metamorphods
The enould oover
Canibolad Nature
follwei
High Conscerruuseeess
Low Emnovesenn
obowevations deriile without
Unive for
Myttioca cramado mennents.
Session 15 Doument Dest
Drocdrn
Contoll Facter
P69 Koturg occumeetaryy regment
Use Sweeee
hooo me create beatmmnt for
oww betaedo Bin loeding inteactionn, the nna the bustertly's roke the acoeysterr?
Corsooaal Praferanc
or urhooood with dramatic longuage and conelict-drioen alory ras iefile nointaiiing tnchol accuraty.
Sclent tifc
sbout utten shoulinng orieed
mandioss The segment should
vntiaiionn cor prording
Figure 2: Data generation pipeline for X CUPID. For each persona, we construct diverse context factors and preferences, then generate chronologically linked interaction sessions. For each session, we simulate dialogue where the user persona evaluates an AI assistant's responses and provides feedback based on the contextual prefexxeelpeeferennee
4

---

Published as a conference paper at COLM 2025
and unique contexts (Desiderata 1). To increase diversity, we provide the LLM with prede- fined taxonomies for factor types (egg, person, object, location, activity) and preference types lists in Appendix D. To reflect how users may hold contrasting preference in different contexts (Desiderata 3), the LLM also creates factor pairs Ccurrent and Coontrast whose preferences conflict or contradict, Peurrent Pcontrast
(e.g,, creativity, sensitivity, trustworthiness)
Generating Sessions With each persona's list of factor-preference pairs, we prompt an LLM to generate series of chronologically connected interaction sessions, S. Instead of generating each session and its interactions one-by-one, we generate all the sessions first to ensure that they are logically and chronologically coherent. When generating each session S the LLM is instructed to first narrate a task-related situation that involves a context factor, and then generate the initial request Ui, where the user explicitly mentions the factor and seeks an LLM's help with the task. To ensure that the series of sessions reveal specific contexts and preferences (Desiderata 2), we prompt the LLM with a 13-session template: the final session S13 includes Ceurrent; two sessions Su and S, also with Ccurrenti two sessions with Ccontrasti and two sessions Sy and S (y,z < u, v) with Ceurrent but a prior preference Pprior = Peurrent that reflects significant change in the user's 's preference (Desiderata 3).
Dialogue Simulation For each interaction session, simulate multi-turn dialogues by using two distinct LLMs: an user simulator role-playing as the persona and an assistant simulator that responds to the user. Each session begins with the user's request H1,1, which the assistant answers. Adapting the LLLM-ss-a-Judye approach (Zheng et al., 2023), the user simulator evaluates the assistant's response against the contextual preference pz, scoring from 1 to 10, and then generates feedback that indirectly hints at the preference (e.g., noting the issues with the response, rather than explicitly stating the preference). The assistant simulator responds and the dialogue continues, gradually revealing the preference, until the user simulator provides a score of 10 (Desiderata 2). Each dialogue is designed contain sufficient evidence for the underlying preference to be inferrable.
Creating Instances Finally, we construct instances to account for the different patterns in user preferences (Desiderata 3): a user holding conflicting preferences in different contexts, and their preferences changing over time. Specifically, for each series of synthesized sessions, we create three types of data instances:
Consistent: Basic instance where there is at least one S E S such that G; =
Courrent, Pt Peurrent and 1g₃1 # uccureent (i.e., at least one previous interaction session possesses the same context factor and preference as the current request).
Contrastive: Meets the same condition as Consistent, and there is at least one S₁ E S such that C # Courrent and pt Peontrast = Peurrent- Here, prior session includes a context factor with preference that conflicts with the current context's.
Changing: Meets the same condition Consistent, and there is at least one Sm S such that m < k, Cm = Ceurrent, but Pm = Pprior = Peurrent- Here, the user previously held an opposite preference in relation to the same context factor.
Each instance has 9 sessions: prior interaction sessions, and the last session with flcurrent-
3.2 Data Validation
We conducted human validation to ensure that our instances were solvable (i.e., preferences are inferrable from prior interactions), challenging (i.e., preferences not inferrable from current requests), and realistic. For each instance, we validated that relevant preferences (Peurrent, Peontrast, Pprior) were fully expressed in the simulated dialogues, but Peurrent was not expressed in Mewrrent. To reduce annotators' cognitive load, we employed human-Al ap- proach (Lee et al., 2022; 2023) where an LLM first extracts user messages from the dialogues that potentially expressed each preference. Then, two human annotators recruited via Prolific then annotated these messages and we considered that preference was expressed marked by at least one annotator. Then, the authors manually revised these cases (~~9% of all instances). Annotators also rated the realism of the current requests at an average of 4.08 (SD=0.85) out of More details in Appendix E.
5

---

Published as a conference paper at COLM 2025
3.3 Data Statistics
CUPID is composed of 756 instances, with 252 instances for each type: Consistent, Contrastive, and Changing. On average, contextual preferences are 18.4 tokens long and are decomposed into 2.90 checklist items, dialogues have 6.38 turns and are 921.5 tokens long, and prior interaction sessions are 8186.7 tokens long. Figure 7 shows the distribution of types for context factors, contextual preferences, and tasks in our dataset.
4 Experiments
With X CUPID, we evaluate open and proprietary LLMs on inference and generation.
4.1 Experimental Setup
Baseline Models We tested total of 10 state-of-the-art instruction-tuned LLMs and few reasoning models: GPT-4o (Hurst et al., 2024), 03-mini (OpenAI, 2025), Claude 3.7 Sonnet (Anthropic, 2025), Claude 3.5 Sonnet (Anthropic, 2024), Llama 3.1 405B (Grattafiori et al., 2024), Mistral 7B (Jiang et al., 2023), Qwen2.5 72B (Yang et al., 2024), DeepSeek-R1 (Guo et al., 2025), Gemini 2.0 Flash Thinking, and Gemini 2.0 Pro (Google, 2024). Details in Table 8.
Prompts For the inference task, all models were zero-shot prompted (Figure 21) to analyze prior interaction sessions and infer the preference that the user will likely hold in the current request, but is not mentioned in the request. For the generation task, all models were zero-shot prompted (Figure 22) to generate response for the current request by considering the possible preference that the user has based on prior interaction sessions.
Oracle We test an oracle setting where models receive only prior sessions that share the current request's ss contextual preference, isolating the impact of identifying and retrieving relevant prior sessions. For the Generation task, we test an oracle preference setting where models are given the ground-truth preference when generating responses, allowing us to evaluate how their ability to adhere to preferences affects performance.
Interaction Summary Recent LLM-powered AI assistants are equipped with long-term memory allowing them to record, recall, and reason about prior user interactions (Zhong et al., 2024; OpenAT, 2024). To understand how this type of intermediate representation could enable models to infer a user's preferences in different contexts, we design a setting where models are first prompted (Figure 23) summarize each dialogue the prior interaction sessions and, then, perform the tasks with these summaries.
Metrics (Section 2.3) We report precision, recall, and F1-score for the inference task, and alignment score of model responses [1-10] for the generation task.
4.2 Results
In this section, we report performance on inference (Section 4.2.1), generation (Section 4.2.2), and with interaction summaries as an intermediate representation (Section 4.2.3).
4.2.1
Inference Task
Models struggle to infer contextual preferences from interactions. Table 1 shows the performance of all tested models on the inference task-visualized in Figure 8. All models struggled adequately infer the preference relevant to the current user request from the previous interaction sessions- no model surpassed an Fl score of 60% and with most under 50%. Considering how our benchmark was designed to be challenging but not overly difficult (e.g,, each preference is revealed in multiple sessions, only 8 prior sessions per instance), we expect that performance will degrade significantly in more realistic settings.
Model size and reasoning capabilities increases performance. Larger model size lead to better performance with the smallest model, Mistral 7B, showing the lowest performance by notable margin of 7.8 points. Additionally, our results show reasoning models perform the

---

Published as a conference paper at COLM 2025
All Instances
Consistent
Contrastive R
Changing R
Oracle Setting
Model
P
F1
P
R
F1
P
F1
P
F1
R
F1
Mistral 7B
26.9
30.7
28.6
22.3
25.6
23.8
23.0
26.7
24.7
35.4
39.7
37.4
46.8
59.8
52.5
Qwen2.572B
32.8
44.9
37.9
30.0
42.6
35.2
27.5
36.2
31.3
41.0
55.9
47.3
60.4
77.1
67.7
Llama 405B
30.2
45.1
36.2
26.4
39.1
31.5
27.8
41.3
33.2
36.4
55.0
43.8
58.4
77.4
66.5
DeepSeek-R1
42.0
59.5
49.3
442
62.9
51.9
41.3
58.8
48.5
40.6
56.8
47.4
62.8
81.2
70.8
GPT-4o
36.6
52.1
43.0
36.5
50.1
42.2
32.6
46.4
38.3
40.7
59.7
48.4
57.1
77.7
65.8
03-mini
33.6
479
39.5
324
47.
38.6
31,0
44.2
36.5
37.
51.6
43.3
48.9
68.7
57.1
Claude 3.5 Sonnet
40.9
56.9
47.6
40.9
56.2
47.4
38.2
52.5
44.3
43.6
62.0
51.2
6288
81.
71.0
Claude 377 Sonnet
49.1
55.8
52.5
48.3
61.5
46.5
65.9
54.5
84.5
76.3
Gemini 2.0 Flash Thinking
37.8
53.7
44.4
373
53.0
43.8
35.5
49.
41.3
40.6
58.6
48.0
59.5
76.0
66.8
Gemini 2.0 Pro
40.1
63.5
49.2
39.0
63.3
38.3
58.7
43.0
68.5
52
83.1
71.4
Table 1: Precision (P), Recall (R), and F1 score for all models on the inference task in CUPID, averaged across all instances, each instance type, and the oracle setting. Best results in each column are bold-faced and second best results underlined.
best in this task with DeepSeek-R1 and Claude 3.7 Sonnet reaching the highest performance.
This suggests that greater train-time and test-time compute enables models to better reason about user preferences and their relevant contexts from prior interactions. As Claude 3.7 Sonnet performed best in our dataset mostly generated by Claude 3,5 Sonnet, we analyze the potential of self-enhancement bias (Appendix F55) but find evidence of its presence.
Retrieving relevant context can significantly increase performance. In the oracle setting, all models improved by approximately 20-30 points, highlighting how inference perfor- mance is strongly dependent on the capability to retrieve and focus on prior interactions with the same context. However, even with only the relevant sessions, precision across models was still under 70,, implying that models are still inferring less relevant details.
Surprisingly, Changing instances led highest performance. Most models perform worse in Contrastive instances but excel in Changing, contrary to our expectation that models would struggle with evolving preferences. Deeper analysis suggests that models prioritize the most recent sessions. Figure 3 shows that performance improves when relevant sessions (i.e., contain the current preference) appear at the end of the history. In Changing instances, earlier sessions reflect the preference before change and later ones capture the change- meaning that relevant sessions tend to be positioned towards the end.
Error Analysis To understand the errors that models make when inferring preferences, we sampled and qualitatively inspected 50 responses each from DeepSeek-RI and Llama 3.1 405B with Fl scores below 20% (~botttom 25%). Table 2 summarizes the identified error types. We observe that models failed to focus on the relevant contexts, struggled to extract the specific preference from multi-turn interactions, only performed shallow inferences, or hallucinated preferences. The models exhibited distinct error patterns: Llama 3.1 405B produced more shallow inference errors, focusing only on the current request, while DeepSeek-RT showed less shallow errors but more incorrect context errors, reasoning over prior interactions but failing to focus on those with shared context.
Error Type
Description
DeepSeek-R1
Ilama 3.1 450B
Incorrect Context
The model incorrectly infers preferences from other contexts that are not relevant the current request.
86%
40%
Shallow Inference
The model infers preferences explicitly mentioned in the re- quest or that are commonsense for the request, instead of in- ferring from prior interactions.
10%
50%
Vagueness
The model inferred preferences that were relevant but were too broad or vague, lacking the specific details the target preference.
2%
6%
Hallucination
The model inferred preferences without clear evidence or con- text, likely influenced by internal assumptions or biases.
2%
4%
Table 2: Error types identified in preferences inferred by DeepSeek-RI and Llama 3.1 405B, with proportion of errors that each model made for each type.
7

---

Published as a conference paper at COLM 2025
32
doud serri
Audd Screag
72-
Swingg tapp
Svint Abd Thon
searsb
d Aomei
073 75
Mmn glmuhi 4356
50
Figure 3: Mean Fl score across all models against mean position of relevant sessions in histories.
Figure Correlation be- tween F1-score when com- puted with GPT-4o and with PREFMATCHER-7B.
Figure 5: Inference perfor- mance against generation performance, averaged for each model.
Evaluation with PREFMATCHER- shows almost perfect correlation with GPT-4o assess- ments (Figure 4, Table 11). The Pearson correlation for model-wise average performance with GPT-4o and PREFMATCHER-7B was 0.997 (p<0..011. Given this result and the strong correlation between inference and generation (discussed later), we suggest evaluating on CUPID only on the inference task and with PREFMATCHER-7D5 to reduce cost.
4.2.2 Generation Task
Performance in the generation task follows similar trends the inference task. Table shows the models' performance in the generation task-visualized in Figure 8. Similar to the inference task, we observe that model performance is generally low, increases with model size and reasoning capabilities, increases significantly in the oracle setting, and is highest in Changing instances while being generally lower in Contrastive.
Strong correlation between the inference and generation tasks. Figure 5 shows positive correlation between model performance in the inference and generation tasks, The Pearson correlation for model-wise average performance in the tasks was 0.764 (p=0.010), suggesting that the inference task can serve as proxy to evaluate models' ability to generate contextu- ally personalized responses. Interestingly, certain models excelled at one task over the other: Claude 3.7 Sonnet led inference, but was outperformed by DeepSeek-RR R1 in generation.
Generation performance is not tied to response generation capabilities. Almost all of the models show significantly high performance in the oracle preference setting (i.e., generating response given the ground-truth preference). This indicates that the low performance across the models is not due to the models' inability to generate responses that satisfy the preference, but rather due their inability to precisely infer the relevant preference.
4.2.3 With Summaries
Summaries have an equalizing effect across models. Figure 6 compares model perfor- mance with and without summaries results in Table 9 and Table 10. Summaries offer
Model
All Instances
Consistent
Contrastive
Changing
Oracle
Preference
Mistral 7B
4.76
4.20
6.06
6.12
7.36
Qwen2.5 72B
6.14
5.60
3.38
7.45
7.33
8.69
Llama 3.1 405B
4.93
4,43
4.29
6.06
6.96
8.84
DeepSeek-R1
7.68
7,50
7.33
8.21
8.95
9,66
GPT-4o
6.46
5,88
3.75
7.74
7.79
9.20
03-mini
6.76
6,31
6.04
7,93
8.10
9,72
Claude 3.5 Sonnet
3.78
5.25
3.08
7.01
8.05
Claude 3.7 Sonnet
7.16
2,13
6.50
7.84
8.61
9,63
Gemini 2.0 Flash Thinking
6.54
6.03
5.91
7.58
9.43
Gemini 2.0 Pro
6.47
6.29
3.81
7.31
7.81
9.45
Table 3: Preference alignment scores for all models on the generation task in CUPID, mea- sured for all instance types, each instance type, oracle setting, and oracle preference setting.
8

---

Published as a conference paper at COLM 2025
Full
Summmny
Dwend 778
Liane 4050
Canote 3.5 sereer
Crede 3.7 Someet
Think
setea 70
owedd 728
Casate 3.5 Some!
<l
Clanto Soncel
rart Yook
Genmal 0.8 he
Geriel
desiel
Figure 6: Comparison of each model's results for the inference task (left) and generation task (right) with the full prior interaction sessions or summaries of these sessions.
lowering performance for reasoning models -et substantially boosts weaker models, bringing them closer the performance of strong models. This suggests that summaries help weaker models to reason about and extract pref- erences from each session, but may lead to information loss for strong models. Notably, the smallest model, Mistral 7B, shows substantial increase of 13 points in inference, suggesting potential for local and privacy-preserving LLM personalization.
minimal gains for strong
Even with summaries, models show low precision. We observe that the increase in inference performance with the summaries is mostly attributed to recall. This indicates that the summaries are not necessarily helping the models focus on the relevant interaction sessions, but rather it is helping models extract the preferences from all the sessions.
4.2.4 Practical Implications
Our findings suggest three actionable directions for personalized LLMs. First, integrate retrieval techniques that identify prior sessions from users' interaction histories that are contextually relevant to the current request, as oracle results show 20-30 point improvements. Second, when deploying smaller or local LLMs, cache summaries of each interaction session focusing on context and preferences. Finally, prompt or tune models to perform reasoning about users' underlying preferences during multi-turn interactions, rather than simply inferring from surface-level expressions.
Related Work
Personalized LLMs To reduce harms and increase helpfulness, research has explored how to align LLMs with human values (Bai et al., 2022b;a) by training them on general preferences (Bai et al., 2022b;a; Ouyang et al., 2022; Cui et al., 2023; Kim et al., 2023). Recent work has taken more personalized view to alignment, considering individual preferences and values (Sorensen et al., 2024; Tseng et al., 2024). For example, Kirk et al. (2024) collected preference dataset with detailed user information (e-g., demographics, behavior attributes). Other work explored prompting LLMSs with user profiles (Yang et al., 2023; Wang et al., 2024a), interaction logs (Baek al., 2024), or user-written artifacts (Salemi al., 2023; 2024). Alternatively, Jang et al. (2023) and Lee et al. (2024) fine-tuned LLMs on decomposed preferences to produce diverse responses for users. More recently, PREFEVAL (Zhao al., 2025) evaluates whether LLMs can identify user's 's global preferences from long-context conversations. In this work, we take a more nuanced view to LLM alignment by considering that each individual has distinct preference in different contexts.
Interactions with LLMs LLMs possess the capability to interact with human users (Wang et al., 2023). User-LLM interactions can organically reveal details about user intents and preferences (Kim et al., 2024c; Shi et al., 2024), enabling models to align themselves with users. Recent work explores how to extract feedback from user-LLM interactions (Don- Yehiya et al., 2024; Wu et al., 2024b), and how to enhance LLMs' memory to leverage user knowledge from interaction histories (Zhong et al., 2024; Wang et al., 2024b; Zhong et al., 2022; Wu et al., 2024a; Kim et al., 2024a). Other work focused on training LLMs to capture richer details from user interactions by clarifying intents or eliciting information (Qian et al., 2024; Li et al., 2023; Andukuri et al., 2024). In this work, we evaluate LLMs' capabilities to infer the preferences that users hold in different contexts from interaction histories.

---

Published as a conference paper at COLM 2025
6 Conclusion
In this work, we introduce 2 CUPID, a challenging benchmark that assesses LLMs' ability to infer user's contextual preferences from interaction sessions and apply these in future contexts. Our benchmark consists of task-oriented dialogues where users gradually and indirectly reveal their preferences through feedback, assesses performance in inference and generation tasks, and employs LLM-based evaluation methods-inclun LLM metric. Experiments with 10 state-of-the-ar LLMs demonstrate that LLMs struggle to both extract users' preferences from interactions and to discern which preferences are relevant to which contexts -raeeettt-reciiiion under 50% and recall under 65% for all models. While explicit memory (i.e., interaction session summaries) benefits smaller models, they have minimal or even negative impact on larger models unddersornng the need for further improvements. CUPID highlights LLMs' current limitations in supporting personalized interactions, and serves as a valuable resource for advancing conversational Al systems for interactive and contextual personalization. In practice we recommend evaluating models on the inference task of CUPID by using PREFMATCHER-7B.
fine-tuned
7 Ethics Statement
In this paper, we introduce CUPID, benchmark designed to evaluate LLMs'' ability to infer user's contextual preferences from interaction sessions. Since the data in our benchmark is newly created, conducted rigorous human validation process. Specifically, during the human validation that assessed that whether the data instances were solvable, human annotators were also asked to flag any possible data points that appeared harmful or offensive. Our human data validation protocol has been determined exempt by the IRB of the author's institution. Additionally, the authors manually inspected all of the data points (i.e., context factors and contextual preferences) to identify and filter out any that appeared offensive, harmful, or unethical. We conducted all experiments using either publicly available models or through documented commercial API access. We have detailed the API versions and configuration of all of the evaluated LLMs for reproducibility. We acknowledge that LLMs may inherit biases from their training data, potentially leading to our dataset incorporating these same biases and not being fully reflective of real users. To mitigate this issue, we aimed to increase the diversity of the user personas, context factors, contextual preferences, and conversation topics. Namely, we created taxonomies for each of these components to guide the generations to reflect diverse types for each of these components. Finally, we acknowledge personalized conversational AI systems can raise concerns related to user's privacy as user information is stored, recalled, and analyzed. It is crucial that such Al systems implement mechanisms to anonymize data, provide mechanisms for users control what data stored, or support local, on-device storing and processing to ensure that user information is never transmitted externally. To promote reproducibility and advance research in this field, we have made our benchmark dataset, code, and model publicly available.
Acknowledgments
This work was supported by NAVER- Hypercreative AI Center and by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea governmen((MSIT) (No. R5-2024-00443251, Accurate and Safe Multimodal, Multilingual Personalized Al Tutors). We would like to express our gratitude to Hyunwoo Kim for his invaluable discussion and insights. We would also like to thank Yuewen Yang for their help during early stages of this project. We also thank all of our Prolific participants that helped us validate and construct our dataset.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.
10

---

Published as a conference paper at COLM 2025
Gpt-4 technical report. arXiv preprint arXio:2333.06674,222
Chinmaya Andukuri, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah D Good- man. Star-gate: Teaching language models to ask clarifying questions. arviv preprint urXio:2403.9115: 2024.
Anthropic. Introducing claude, 2023. URL https://ww.anthropitc.ccm///s/ introducing-claudes Accessed: March 19, 2025.
Anthropic. Introducing claude 3.5 sonnet, 2024. URL https://ww..antthppiccccc//s/ss
claude-3- 3-5-sonnet. Accessed: March 19, 2025.
Anthropic. Claude 3.7 sonnet and claude code, 2025. URL https://ww.anthhopic.ccom news/claude-3-1 s-nnoneet. Accessed: March 19, 2025.
Jinheon Baek, Nirupama Chandrasekaran, Silviu Cucerzan, Allen Herring, and Sujay Kumar Jauhar. Knowledge-augmenter large language models for personalized contextual query suggestion. In Proceedings of the ACM on Web Conference 2024, pp. 3355-3366, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. ar preprint 1XXi:2204.05862 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitu- tional ai: Harmlessness from ai feedback. arXiv preprint arXi::2212.08073, 2022b.
Chris L Baker, Rebecca Saxe, and Joshua Tenenbaum. Action understanding as inverse planning. Cognition, 113(3):329-344 2009.
Jan Cieciuch and Eldad Davidov. A comparison of the invariance properties of the pvq-40 and the pvq-21 to measure human values across german and polish samples. In Survey Research Methods, volume 6, pp. 37-48, 2012.
Jonathan Cook, Tim Rocktäschel, Jakob Foerster, Dennis Aumiller, and Alex Wang. Ticking all the boxes: Generated checklists improve Ilm evaluation and generation. arXiv preprint ar arXio:2410.03603 2024.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high- quality feedback. 2023.
Shachar Don-Yehiya, Leshem Choshen, and Omri Abend. Learning from naturally occurring feedback. Xrv preprint ar XrXic:2407.10944 2024.
Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire. ar Xiv preprint ar arXto:2302.04166 2023.
Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. Scaling synthetic data creation with 1,000,000,000 personas. arXiv preprint ar Xii::2406.20094 2024.
Lewis Goldberg. The development of markers for the big- live factor structure. Psychological assessment, 4(1):26, 1992.
Google. Introducing gemini 2.0: our new ai model for the agen- tic era, 2024. google-geminic wtte--eeeiiiiiiaitteeeeeeee-eeee
URL https://blog, wwwggee/eechooooy/geeeooooooeeee hwwpllogggggggleedeee legooglleedeeeiindl
Accessed: March 19, 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinay Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint ar arxiv:2407.2111 217833, 2024.
11

---

Published as a conference paper at COLM 2025
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Iims via reinforcement learning. Xrviv preprint ar arXiv:2501.12948) 2025.
Katherine Hamilton, Shin-I Shih, and Susan Mohammed. The development and validation of the rational and intuitive decision styles scale. Journal of personality assessment, 98(5): 523-535, 2016.
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arviv preprint arXi::240.221276 2024.
Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Personalized large language model alignment via post-hoc parameter merging. arXiv preprint ar Xrri:22310.111 11564, 2023.
Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. Ai aligmment: A comprehensive survey. aiv preprint arXiv:2310.19852 2023.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL wwtparwwwi..ogorr....///////e 03662
Jiho Kim, Woosog Chay, Hyeonji Hwang, Daeun Kyung, Hyunseung Chung, Eunbyeol Cho, Yohan Jo, and Edward Choi. Dialsim: real-time simulator for evaluating long-term dialogue understanding of conversational agents. arviv preprint ar Xrxx:2446.13144 2024a.
Sungdong Kim, Sanghwan Bae, Jamin Shin, Soyoung Kang, Donghyun Kwak, Kang Min Yoo, and Minjoon Seo. Aligning large language models through synthetic feedback. ar arxiv preprint ar Xit:2305.13735, 2023.
Tae Soo Kim, Yoonjoo Lee, Jamin Shin, Young-Ho Kim, and Juho Kim. Evallm: Interactive evaluation of large language model prompts on user-defined criteria. In Proceedings of the CHI Conference on Human Factors in Computing Systemss, pp. 1-21, 2024b.
Yoonsu Kim, Jueon Lee, Seoyoung Kim, Jaehyuk Park, and Juho Kim. Understanding users' dissatisfaction with chatgpt responses: Types, resolving tactics, and the effect of knowledge level. In Proceedings of the 29th International Conference on Intelligent User Interfaces, pp. 385-404-4 2024c.
Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Michael Bean, Katerina Margatina, Rafael Mosquera, Juan Manuel Ciro, Max Bartolo, Adina Williams, He He, et al. The prism alignment dataset: What participatory, representative and individu- alised human feedback reveals about the subjective and multicultural alignment of large language models. In The Thirty-eight Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, et al. Openas- sistant crrsations-democratinn Information Processing Systems, 36, 2024.
large language model alignment. Advances in Neural
Klaus Krippendorff. Reliability in content analysis: Some common misconceptions and recommendations. Human communication research, 30(3):411-433 2004.
Seongyun Lee, Sue Hyun Park, Seungone Kim, and Minjoon Seo. Aligning to thousands of preferences via system message generalization. ar preprint arXio:2405.17997, 2024.
12

---

Published as a conference paper at COLM 2025
Yoonjoo Lee, John Joon Young Chung, Tae Soo Kim,Jean Y Song, and Juho Kim. Prompti- verse: Scalable generation of scaffolding prompts through human-ai hybrid knowledge graph annotation. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pp. 1-18, 2022.
Yoonjoo Lee, Kyungjae Lee, Sunghyun Park, Dasol Hwang, Jaehyeon Kim, Hong-in Lee, and Moontae Lee. Qasa: advanced question answering on scientific articles. In International Conference on Machine Learning, pp. 19006611052. PMLR, 2023.
Belinda Z Li, Alex Tamkin, Noah Goodman, and Jacob Andreas. Eliciting human preferences with language models. Xiv preprint ar Xit::2300.11589 2023.
Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmark- ing lims with challenging tasks from real users in the wild. arviv preprint arXtt:2406.04770, 2024.
Chaitanya Malaviya, Joseph Chee Chang, Dan Roth, Mohit Iyyer, Mark Yatskar, and Kyle
Lo. Contextualized evaluations: Judging language model responses underspecified queries. Xiv preprint ar arXio:2411.0723/ 2024.
Microsoft. panion, 2023.
Announcing microsoft copilot, your everyday ai com-
URL https://blogs: https://blogss.cccoooft... hww.croootccoombllg///2222 comm/logg/223/09/22
oowp 19, 2025.
Accessed: March
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit lyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generationn. Xiv preprint arXiv:2305.14251, 2023.
Rejichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser- assisted question-answering with human feedback. ar preprint ar Xix::2112200933,, 2021.
OpenAL. Chatgpt, 2022. URL https: // http://aaaa....ai....ooo aeeenai.co/cceee com/chat. Accessed: March 19, 2025.
OpenAl. Memory and new controls for chatgpt, 2024. URL //opp::///ppaaiii. http:////ppaai..mm//neee// memorry-and--ee--controls cor-chhtgp//
Accessed: September 15, 2024.
OpenAL. Openai 03-mini, 2025. URL cessed: March 19, 2025.
com/ http://paanae.mmmmmme//////eaaaa 03-min// Ac-
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-277444 2022.
Cheng Qian, Bingxiang He, Zhong Zhuang, Jia Deng, Yujia Qin, Xin Cong, Zhong Zhang, Jie Zhou, Yankai Lin, Zhiyuan Liu, et al. Tell me more! towards implicit user intention understanding of language model driven agents. arviv preprint ar XrXi:2240..09205 2024.
Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. Lamp: When large language models meet personalization. Xrv preprint ar arXiv:2304.11406 2023.
Alireza Salemi, Surya Kallumadi, and Hamed Zamani. Optimization methods for personal- izing large language models through retrieval augmentation. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 752-762, 2024.
Taiwei Shi, Zhuoer Wang, Longqi Yang, Ying-Chur Lin, Zexue He, Mengting Wan, Pei Zhou, Sujay Jauhar, Sihao Chen, Shan Xia, et al. Wildfeedback: Aligning llms with in-situ user interactions and feedback. ar arviv preprint ar Xiv:2408. 103.008...5555 15549, 2024.
13

---

Published as a conference paper at COLM 2025
Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, et al. A roadmap to pluralistic aligmment. arXio preprint ar X50:24402.00070, 2024.
Alex Tamkin, Miles McCain, Kunal Handa, Esin Durmus, Liane Lovitt, Ankur Rathi, Saffron Huang, Alfred Mountfield, Jerry Hong, Stuart Ritchie, et al. Clio: Privacy-preserving insights into real-world ai use. arviv preprint ar arXr:24412.13677 2024.
Yu-Min Iseng, Yu-Chao Huang, Teng-Yun Yun Hsiao, Yu-Ching Hsu, Jia-Yin Foo, Chao-Wei Huang, and Yun-Nung Chen. Two tales of persona in llms: A survey of role-playing and personalization. arXiv preprint Xio:2406.001177 2024.
Tiannan Wang, Meiling Tao, Ruoyu Fang, Huilin Wang, Shuai Wang, Yuchen Eleanor Jiang, and Wangchunshu Zhou. Ai persona: Towards life-long personalization of Ilms. Xiv preprint Xix::2412.111103 2024a.
Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Augmenting language models with long-tern memory. Advances in Neural Information Processing Systems, 36, 2024b.
Zekun Wang, Ge Zhang, Kexin Yang, Ning Shi, Wangchunshu Zhou, Shaochun Hao, Guangzheng Xiong, Yizhi Li, Mong Yuan Sim, Xiuying Chen, et al. Interactive natu- ral language processing. arviv preprint ar Xiv:22305. 13246, 2023.
Caleb Warren, A Peter McGraw, and Leaf Van Boven. Values and preferences: defining preference construction. Wiley Interdisciplinary Reviews: Cognititee Science, 2(2):193-2055
2011.
Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, and Dong Yu. Long- memeval: Benchmarking chat assistants on long-term interactive memory. arXiv preprint #rXiv:240.100113, 2024a.
Shujin Wu, May Fung, Cheng Qian, Jeonghwan Kim, Dilek Hakkani-Tur, and Heng Ji. Aligning Iims with individual preferences via interaction. arXiv preprint arXiv:2410.0366 00,06422, 2024b.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. arXiv preprint 2IXiv:442.151115 2024.
Fan Yang, Zheng Chen, Ziyan Jiang, Eunah Cho, Xiaojiang Huang, and Yanbin Lu. Palr: Personalization aware Ilms for recommendation. arXiv preprint arXin:2305.07622, 2023.
Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. Flask: Fine-grained language model evaluation based on alignment skill sets. ar Xiv preprint ar Xrv::23307 109228 2023.
Marc Zao-Sanders. How people are really using genai. Harvard Business Review, 2024.
Siyan Zhao, Mingyi Hong, Yang Liu, Devamanyu Hazarika, and Kaixiang Lin. Do lims recognize your preferences? evaluating personalized preference following in llms. Xiv preprint ar Xxxi:2502.095977 2025.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging lim-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595- 46623, 2023.
Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. Memorybank: Enhanc- ing large language models with long-term memory. In Proceedings of the ÁAAI Conference on Artificial Intelligence, volume 38, pp. 19724-19731, 2024.
Zexuan Zhong, Tao Lei, and Danqi Chen. Training language models with memory augmen- tation. ar Xiv preprint Xr::::220 12674, 2022.
14

---

Published as a conference paper at COLM 2025
Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis- Philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, et al. Sotopia: Interactive evaluation for social intelligence in language agents. ari preprint xfxo:2230.11667, 2023.
A Metrics
A.1 Preference Match
To measure the similarity between inferred and ground-truth preferences, we use two LLM prompts:
preference decomposer, which decomposes preferences into multiple checklist questions (Figure 15), and preference-checkist matcher, which assesses how much each question in checklist matches a preference (Figure 14). In our experiments, we use gpt-40-2024-11 through the OpenAI API with a temperature of for both steps.
Meta-Evaluation for Preference Match Using our pipeline, we created an additional 240 preference-checkliss pairs. Then, we recruited annotators through Prolific to rate the match between preference and the entries in the checklist for each pair. In total, we recruited 90 annotators, where all passed a qualification task, and each annotator inspected a total of preference-checklist pairs. Annotators spent average of around 10 minutes in the task and were compensated E2.25 for their time. In our interface (Figure 12), annotators were presented with preference and each item from a checklist side-by-side, and were asked to rate whether each item was "Fully Covered", "Partially Covered" or "Not Covered" by the preference. Through this, we obtained annotations per preference-checklist pairs. pereetseeeeeeeeeeeeeeeeeeeeee
The agreement between annotators resulted in Krippendorff' 's alpha of 0.467 (moderate agreement). Due to the moderate agreement, for each checklist item a pair, we found the rating (i.e., "not covered", "partially covered", or "fully covered") with the majority vote and this was set as the rating for that checklist item. We removed any checklist items without a majority vote, where 2 checklist-preference pairs were removed entirely all the checklist items had no majority vote. This resulted in 238 preference-checkis pairs, where 8 pairs were used few-shot examples in the prompt and the remaining 230 pairs were used for the meta-evaluation. Each checklist in a pair had an average of 3.31 checklist items, leading to total of 762 match ratings.
Finetuning Details for PREEMATCHER-7B To finetune PREFMATCHER-77, we first syn- thesized separate dataset of 64 additional user personas and their interaction sessions with our pipeline. For each persona, we created 10 data instances: Consistent, Contrastive, Changing, and separate instances for each session that does not include the current pref- erence in the other three instances. We then selected three models that reflected a range of performance in the inference task (i.e., and DeepSeek-RI) and evaluated them on the created instances using our pipeline. This resulted in 2 matched preference-checkkist pairs for each model and instance (i.e., predicted preference matched with the ground-trute checklist, and ground-trufh preference matched with the predicted checklist) total 6 per instance and total of 4,224 samples, which use for finetuning. We take Twwen..-7b-Inssrruc as the base model and finetune it through QLoRA with the hyperparameters reported in Table 4.
Qwen2.5-72B-BB Insstruct
A.2 Preference Alignment
Hyperparameter
To evaluate the alignment of generated responses with the ground-truth preference, we use the LLM-as-a-fudge method (Zheng et al., 2023) with gpt-40-2024-11- and temperature of 0. When evaluating, the LLM receives the user's 's request, the model's 's response, the ground-truth preference, and decomposed
Epochs
deridevice
size
Gradkent accumulation steps
Learning role
x 10
Weight deaay
x 10
Optimizer
AdamW (orch)
LR scheduler type
Casine with warmup
Number warmup stops
100
LoKA Rank
LORA Alpis
the checklist that the preference into. We adapt the prompt from Cook et al. (2024) that evaluates LLM outputs with checklists, which
Dropout
Tanget Modules
Qroi, proj, Output proj Trie
Apply LoKA MLP
Table 4: Summary of LoRA fine- tuning hyperparameters.
15

---

Published as a conference paper at COLM 2025
demonstrated higher consistency with human pref- erences. Full prompt in Figure 16.
B Supplementary Details for Data Generation Pipeline
We mostly employed claude- 1orode---5-ae 3-5-sonnet-2222 202411 022 through the Amazon Bedrock API.
B.1 Generating User Personas
To generate k personas, our pipeline first constructs k persona templates consisting of the following com- ponents: seed persona description, career level, personality traits, personal values, and decision- making style. Each template is constructed by bal- anced random sampling of values for each of com- ponent. For the seed descriptions, we sample per- sona descriptions from PersonaHuub (Ge et al., 2024), where each description is a sentence that describes the persona's 's occupation, and their responsibilities or interests. We noticed that PersonaHub's personas skewed heavily towards subset of occupations. To diversify our persona pool, we first collected all the tokens from the persona descriptions in the dataset and sorted them based on frequency. Then, we man- ually selected tokens that represented distinct occu- pations, which resulted in 115 highly-comme cupation tokens. To sample seed descriptions, we first sample distinct occupation tokens from this list, and then select random descriptions that contain each sampled token.
Category
Possible Values
Career
Entry-level. Beginner/Noon Novice
Level
Mid Mid-level/ Intermediate Associate
Senior-level/ Advanced/ Expert
Personality
Openness Experience (High/Low)
Traits
Conscientiousees. (High/ Low)
Agreeableness (Highh [ow]
Neuroticism (High/1oow)
Personal
Self-Direction
Values
Achievement
Universalism
Power
Stimulation
Benevolence
Security
Tradition
Conformity
Hedonism
Decision- Making Style
Analytical
Conceptual
oc-
Behavioral
Table 5: User persona profile trait cat- egories and their possible values.
Inspired by Zhou et al. (2023), we expand each template with additional persona charac- teristics and behavioral traits. Specifically, each template includes attributes across four categories: career level, two traits from the Big Five personality traits (Goldberg, 1992), two values from the Schwartz's 's basic human values (Cieciuch & Davidov, 2012), and decision-man makiin style (Hamilton et al., 2016). The detailed traits and their possible values are summarized in Table 5. For set of templates, we prompt Laude-3-5-sonne- -etpnneettt22222222 20241022 (Figure 13) with temperature of 0.7 to generate 6-sentence persona description for each template, which enriches the persona with name and a narrative of their needs, challenges, background, typical behaviors, etc.
B.2 Generating Context Factors
Given persona description, our pipeline generates a list of 8 context factors and associated preferences for that persona by prompting Laade-3-5-sommee s0neet-22222 022 (Figure 17) with temperature of 0.7. Áll of these factors are generated at once to guide the LLM to generate a list of mutually consistent and coherent context factors. To generate each factor, we instruct the LLM to first generate background story in the persona's life to elaborate persona's 's broader world and guide it to generate more unique and specific factors. Then, the LLM selects a type for the factor from a pre-defined list and a set of task types that could be influenced by the factor- lists for these types in Appendix D. After naming the context factor, the LLM selects preference type and generates a preference relevant to this context factor and preference type. While we could have guaranteed an equal distribution for factor, task, and preference types by sampling and pre-selecting these for the LLM, we observed that allowing the LLM to select these types led to more coherent factors and preferences, and increased the coherency between them. In the same prompt, the LLM is also instructed to ensure that the last two context factors are contrastive; the context factors of the same type and similar, but their contextual preferences should contradict with each other and be mutually exclusive.
the
16

---

Published as a conference paper at COLM 2025
B.3 Generating Interaction Sessions
Given persona description and a list of context factor-preference pairs, our pipeline gener- ates a series of 13 unique interaction sessions by prompting --ouud-----oonn--200- (Figure 18) with temperature of 0.7. Specifically, the LLM is prompted to generate a series of chronologically connected situations where the user is performing specific task and decides to seek the help of the AT assistant. For each scenario or session, the LLM selects a context factor and its preference from the given list, selects a task type that the user will perform in the session, generates a story that describes the situation, and then generates the request that the user would ask to the Al assistant in this situation. We noticed that the LLM frequently mentioned or implied the preference in the user's 's request, which would reveal the preferences without requiring inference from the interactions. To avoid mentioning the preference in the request, the LLM was instructed to first generate a version of the request that includes the preference and then generate a version without it. Considering that real-world users frequently provided additional context or resources in their requests when performing tasks (e.g,, documents, code, data), we also instruct the Al assistant to separately generate these resources if necessary in a session.
To construct the various types of instances in our dataset (i.e., Consistent, Contrastive, and Changing), the LLM is provided with a template for the sequence of interaction sessions that specifies which context factors should be included in which sessions. Specifically, we first randomly select one factor from the contrastive pair to be the current farren while the other factor in the pair is considered to be the confrastive factor. Then, the template is constructed by first randomly selecting 4 positions within the sequence (excluding the final position) and specifying that each of these sessions should include the current fictor-ensuuing that the current factor appears in random positions in the series of sessions. Among these positions with the current factor, the template also specifies that the first two positions should include the original preference related to the current factor, but the last two should share a new contextual preference, changed preference, that contrasts or conflicts with the original preference. This reflects a significant change in the preference related to the current factor. From the remaining positions but excluding the final position, two positions are randomly selected and the template specifies that these should include the contrastive factor and its related preference. Finally, the template specifies that the last position should include the current factor.
B.4 Simulating User-LLM Dialogues
For each interaction session, our pipeline then creates user-LLM dialogue by simulating a dialogue between an LLM simulating the user persona and an LLM simulating an Al assistant. For the user simulator, we employ got-4o-2024-11-
with temperature of 0.3 as the user simulator (prompt in Figure 19). At each turn in the dialogue, the user simulator is provided with the persona's ' profile description, the context factor and contextual preference for that interaction session, the checklist for that preference, and all the messages up to that turn between the user simulator and the assistant simulator. The user simulator is prompted to evaluate the assistant's most recent response on each of the checklist items and to provide score, ranging from 1 to 10, to the response. To ensure that each message only partially implies the preference, we instruct the user simulator to select a subset of checklist items to reference or imply in their message. Based on this selection, the user simulator is then prompted to generate its message to the assistant simulator while ensuring that its concise and indirect (e.g., instead of stating the preference, indicating part of the assistant's response that misaligned with the preference). When the evaluation score is a 10, the user simulator is instructed to end the conversation but it is still instructed to generate final positive message to the Al assistant. While this may differ from realistic user-LLM dialogues where the LLM's message is always the last, we observed that this was a necessary to ensure that all aspects of preference were mentioned at least once in each dialogue.
For the assistant simulator, we employ Llama-3. of 0.7. We used a smaller and relatively weaker model for the assistant simulator expected that this model would require more turns to fully satisfy the user simulator's 's feedback-leadingg to more opportunities for the user simulator to implicitly reveal the
with temperature
17

---

Published as a conference paper at COLM 2025
relevant contextual preference. For the assistant simulator, only provide simple system prompt that instructs it to generate relatively concise responses to ensure that the overall dialogues are not excessively lengthy: "You are conversational assistant. Limit t your answers to 6 sentences."
B.5 Creating the Instances
Our dataset consists of three types of instances: Consistent, Contrastive, and Changing. For each persona, we generate 13 interaction sessions and we construct these instances by removing specific sessions from this list. Specifically:
Consistent: To create the Consistent instance, we remove the two interaction sessions that include the current factor but with the changed preference. We additionally remove the two interaction sessions that include the contrastive factor. As a result, the instance contains prior interaction sessions, where include the current factor with its original preference, and the final session that also includes the current factor.
Contrastive: To create the Contrastive instance, we again remove the two interaction sessions that include the current factor but with the changed preference. We addition- ally select and remove two random sessions that do not include the current factor or the contrastive factor. As a result, the instance contains prior interaction sessions, where two include the current factor with its original preference and two include the contrastive factor, and the final session that also includes the current factor.
Changing: To create the Changing instance, we remove the two interaction sessions that include the contrastive factor. We additionally select and remove two random sessions that do not include the current factor. As result, the instance contains 8 prior interaction sessions, where two include the current factor with its original preference and two include the current factor but with the changed preference, and the final session that also includes the current factor.
C Data Examples
Table presents couple of example user persona profiles, context factors, and associated contextual preferences from our dataset.
D Types for Context Factors, Preferences, and Tasks
We define taxonomies context factor, preference, and task types. Summarized in Table 7.
Context Factors For context factors, we manually annotated a sample of 256 random requests from WildBench (Lin al., 2024) as these represent realistic requests that human users ask to LLMs. This resulted in taxonomy of 8 context factor types.
Preference Types For preference types, we adapted the multiple preference facets de- scribed by Lee et al. (2024). Specifically, we adapted the sub-dimensions under the "style", "informativeness"" and "harmlessness" dimensions. We did not include the "background knowledge" dimension the sub-dimensions were less explicit about how the user's preferences would change. This resulted in 16 preference types.
Task Types To guide the LLM to consider diverse tasks that user may perform with the help of other LLMs, we also created a taxonomy task types. We referenced prior work that analyzed the most common tasks where real-world users are employing LLMs (Tamkin et al., 2024; Sao-Sanders, 2024). We combined the list of tasks described in these work by removing duplicates and combining similar or related tasks. This resulted in 12 task types.
18

---

Published as a conference paper at COLM 2025
Persona Descriptions
Context Factors
Contextual Preferences
Sarah Chen, 24, she/ her, recently started pursuing amateur astronomy while working as a junior accounlant, spending her evenings studying the night sky. She yearss master the traditional constellations and their mythologies, seeking validation from more experiencvd astronomers in her local club, though she often doubts [..]
Celestron NexStar 6SF Telescope
Instructions must include explicit warnings about what actions to avoid, with detailed explanations of polential consequences.
Astrophotography
Each step in the process must be broken down into micro-steps with specific validation checkpoints throughout.
Sessions
Family Stargazing Night
Explanations must incorporate narrative elements and personal connections while maintaining scien- tific accuracy.
Marco Santos, 24, he/ he/him, /him, is an entry-level cartographer a geographical research institute in Barcelona, bringing fresh energy to traditional mapping techniques. He seeks to make his mark by creating innovative visualization me thoodd Catalonia's 's diverse landscapes and dreams of developing an [...]
Dr. Elena Martí
Every proposed innovation must be justified with at least three established cartographic principles from standard textbooks.
Catalonian ronmental Agency
Envi-
All documentation must follow structured format with numbered sections and subsections, maintain- ing formal technical language throughout.
Barcelona Hub
Tech
Documentation should use bullet points and infor- mal language, focusing on quick capture of essential information only.
Marcus Chen, 34, he/him /him, is a high school chemistry teacher who brings energetic presence to his classroom. He strives to make science accessible and exciting for his students by drawing unexpected parallels between sports and chemical reactions. constantly soeking ways to help
Chemistry Lab Stor- age Room
Storage and inventory documentation must specify exact shelf locations and quantities with zero ambi- guity or room for interpretation.
Regular Chemistry Class Slides
Slides must prioritize visual demonstrations and an- imations while including hidden detailed text notes that can be revealed when needed for absent stu- dents.
athletes in the school's 's teams understand how science impacts [...]
AP
Chemistry
Slides must prioritize comprehensive written expla- nations while relegating visual elements to small supporting diagrams in corners.
Class Slides
Marcus Thompson, 32, heeh,,, is an established freelance photographer specializing in landscape and cultural photography. He thrives chasing the perfect shot, often embarking on spontaneous road trips and scaling challenging terrains to capture unique perspectives that will stand out in the competitive photography market [...]
Instagram Photog- raphy Community
Content should emphasize personal creative vision while acknowledging audience preferences as sec- ondary considerations.
Adobe Lightroom
Software guidance should focus workarounds and customizations that prioritize creative freedom over organizational efficiency.
Capture One Pro
Software guidance should emphasize mastering com- plex features that enable precise control over the cre- ative process.
Richard Blackwood, 58, he/him is a senior-level philatelist who curates one of the largest private collections of military-themee stamps in Europe. His meticulous approach to organizing and documenting collection has earned him recugnition in philatelic sucieties, where he regularly presents his research on rare military [...]
Auction Protocol
Bidding
Bidding analysis must establish three distinct price thresholds with specific justification for each based on historical auction data.
Monthly Philatelic Society tions
Complex technical concepts must be introduced through a three-step process: historical context, vi- sual example, and technical explanation.
Presenta-
Microscopic
Au- Pro-
Authentication must document every visible detail regardless time required, creating comprehensive photographic evidence of all features.
thentication
cess
Table 6: Examples from our dataset: shortened user persona descriptions, context factors, and associated contextual preferences.
E Data Validation
Human Validation Our dataset consists of 252 personas, where a series of interaction sessions was synthesized for each persona. We performed human validation on each of these 252 interaction session series. For each series, we created four validation tasks: three tasks to validate that each instance-relevant preference (i.e., Peurrent/ Pcontrasts Pprior) is expressed in the prior interaction sessions, and one task to validate that Peurrent is not expressed in the current request, Meurrent- This resulted in a total of 1008 tasks, where each task was performed by two annotators. We recruited a total of 252 human annotators, where each annotator passed a qualification task and performed 16 tasks. On average, annotators spent approximately 30 minutes completing all the tasks and were compensated with f4.50. Using our annotation interface (Figure 11), the annotators were presented with each entry from a preference's s checklist, and either list of user messages or only the user's 's request.
19

---

Published as a conference paper at COLM 2025
Dimension
Types
Context Factors
Person/ Groop, Organization/ Inntintttiiititiiinnooooooo Object/ Artifact, Content/ Tool/Technoloee Location /Paac,, Process/ Activity, Event/Time
Preference Types
Style: Formality, Clarity, Conciseness, Vividness, Format, Tone Informatioeness: Relevance, Depth, Creativity, Efficiency, Practicality Harmlessness: Accuracy, Morality, Safety, Sensitivity, Trustworthiness
Task Types
Information seeking, Learning, Reasoning, Planning, Content creating,
Communicating, Writing & editing, Coding & debugging, Problem solving, Data analysing, Advise seeking, Brainstorming
Table 7: Summary of Taxonomies for Context Factors, Contextual Preference, and Tasks.
Conteatuua Preference
Fncteaity
Conten CIaTr
Cadyy
Cuality
13:25
BON
7.2%
21.3
11
UeceterPace
TAYA
Oiyanial ached bolin
Det's enant
Figure 7: Distribution of the types of context factors, contextual preferences, and tasks included in our dataset.
Annotators were asked to mark whether the messages request expressed hinted at the checklist entry or not. We asked for annotations on each checklist entry-rather than on the overall preference-to validate that each aspect of preference was expressed (or not) in the relevant messages. We then manually revised any messages were at least one annotator marked that any checklist entry was not expressed in the messages (or was expressed in the user's 's current request). Additionally, for the task where annotators validated the current request Meurrent, we also asked annotators rate how realistic that request was. Specifically, annotators were asked to consider whether it is realistic for a user to ask this request to an Al assistant (e.g., ChatGPT, Gemini, Perplexity, Claude, DeepSeek, etc.) by providing rating between 1 (not realistic at all) to 5 (extremely realistic).
Message Extraction To reduce the cognitive load on the annotators and the task time, we used an LLM to first extract the message fragments relevant to a preference from the prior interaction sessions. Specifically, for each preference, we first prompted gpt-40-2824-11-28 with the preference's checklist and the relevant dialogues, messages that could hint or express each entry in the preference. Prompt is in Figure 20.
asked it to extract the
F Additional Experiment Details and Results
Full details for models evaluated in our experiments are shown in Table 8.
E1
Visualization of Results
Figure 8 presents visualizations of the results for all the evaluated models in the Inference and Generation tasks.
E2
Summary Results
Table 9 and Table 10 presents the full results for the inference task with the summaries. Most open-source models appear to present significant increase in performance with the
20

---

Published as a conference paper at COLM 2025
summaries. Interestingly, the performance of certain models decreases slightly with the summaries, which appear to mostly be the models with reasoning capabilities. Additionally, similar trend in performance regarding the instance types, where, in general, performance is highest in the Changing instances and lowest in the Contrastive instances reflecting that models may also be focusing on the most recent summaries as they did with the interaction sessions. Interestingly, we observe that the summaries help each model in different task. Specifically, we observe that Mistral 7B, Qwen2.5 72B, and Llama 3.1 405B see significant increases in inference performance with the summaries, but not in the generation task. While Claude 3.5 Sonnet and Gemini 2.0 Pro see minimal gains in the inference task with summaries, but their performance in the generation task increases significantly.
Model
Version
API
Additional Parametees
CPT 4o 03-mnini
2024 08-07
OpenAl
jemperature-0
2025-01-31
OpenAl
(" temperature not supported.)
temperature-1 only supported setting.)
Claude 3.7 Sonnet
20230219
Amazon Bedrock
thinking- (type: "enabled", budget.token
Claude 3,5 Sonnet
20240620
Amazon Bedrock
temperature=0
Laama 3.1 405B
Instruct Turbo
Together Al Together
temperature-0
Mistral
temperature-0
Qwen2.5 72B
Instruct Turbo
Together Al Tagether Al
temperature=0
DeepSeek-RI R1
temperature-0
Comini 2.0 Flash Thinking
exp-01-22 exp-022-05
Vertex Al
lemperature-Q
Gemini
temperature=0
Table 8: Full details for models tested in experiments.
182
Oxen2.5 720
Usno 4058
DeepScok
70
Swen2.5
Liama 4050
DespSesiifi
c3-mini
Oode:
Docc:
Figure 8: (Top) Average F1, precision, and recall for all models in the inference task all instances. (Bottom) Average alignment score for all models in the generation task averaged across all instances, for the oracle setting, and for the oracle preference setting.
All Instances
Consistent
Contrastive
Changing R
Model
P
F1
P
R
F1
P
R
F1
P
F1
Mistral
36.0
49.8
41.8
35.1
49.1
40.9
33.4
44.2
38.0
39.8
56.3
46.6
Qwen2.5 72B
37.2
55.3
44.5
37.7
5622
45.1
37.3
52.8
43.7
36.7
56.8
44.6
Llama 3.1 405B 405B
35.4
54.7
43.0
36.4
56.9
44.4
34.4
51.5
41.2
35.3
55.7
43.2
DeepSeek-R1
40.1
60.0
48.0
41.1
61.9
19.4
41.3
48.3
37.8
59,9
46.3
GPT-4o
37.7
58.5
45.9
37.1
56.2
44.7
36.6
56.2
44.3
39.6
63.1
48.6
03-mini
34.7
50.7
41.2
35.4
52.0
42.1
33.2
49.4
39,8
35.4
50,8
41.7
Claude 3.5 Sonnet
42.1
61.9
50.1
43.0
63.9
51.4
40.5
56.8
47.3
42.8
65.0
51.6
Claude 3.7 Sonnet
45.4
53.3
45.6
64.0
53.3
52.6
45.6
66.1
54.0
Gemini 2.0 Flash Thinking
37.3
53,9
44.1
36.4
52.7
43.0
37.1 1
52.4
43,
38.6
56.7
45.9
Gemini 2.0 Pro
37.1
62.1
46.4
35.1
62.5
45.0
33.8
55.0
41.9
42.5
68.9
52.5
Table 9; Precision (P), Recall (R), and Fl score for all models on the inference task with sum- maries. Best results in each column are bold-faced and second best results underlined.
21

---

Published as a conference paper at COLM 2025
Vatrral 78
50
Swer2 728
Gwan2.5
Laama 1:1 405e
DospSeak-R2
DeepSock
GPT-4
53-mn:
abmth
Cluuee 3-5 Sonnet
Claude sonnet
Cleuce 3.7 Sonnez
Claude
10
Gem 2.0 Hash Think
Gamint Fissh
Gem 2.0 Pr
Gemint 2.0
10000
20303
30365
10000
20060
10300
00003
Figure 9: Average result for each model in the inference task (left) and generation task (right) according to the maximum token length of the instances.
F33 Inference with PREEMATCHER-7B
Figure 4 shows the full results in the inference task when assessed with PREFMATCHER-7B. As shown, inference performance when assessed with PREFMATCHER-78 and with GITT-4o generally only show marginal differences, without one of the models skewing towards providing higher or lower scores.
F4 Performance against Length of Prior Interaction Sessions
Figure 9 shows the average results of each model in the inference and generation task depending on the token length of the previous interaction sessions the instances. Specifi- cally, we divided all the instance data into equally-sized groups (bins). Each bin contains instances with lengths larger than the maximum length in the previous bin, but less than the minimum length in the next bin. Then, averaged model performance for all data in each bin. This allows us to observe that each bin represents model performance at distinct range of lengths. As
observe general trend where model performance in both tasks decreases with the length of the prior interaction sessions-which coincides with findings in similar work (Wu et al., 2024a; Zhao al., 2025). This suggests that, with even longer interaction sessions, models will struggle to infer users' personalized and contextual preferences from interactions.
from the results,
E5 Analyzing Possible Self-Enhancement Bias
As CUPID context factor-preference were generated by Claude 3.5 Sonnet, Claude 3.7 Sonnet's s superior inference performance raises the possibility that implicit biases trained into these models were reflected in the data. To test this, we synthesized small datasets using three high-performing models-Claude 3.7 Sonnet, GPT-4o, and DeepSeek-Rl-base on the 64 persona profiles our benchmark that produced the lowest average performance across all models. For each persona and model, we used our synthesis pipeline to generate Consistent, Contrastive, and Changing instances Instances-leading to small datasets of 192 instances
Judge Scores Contrastive
Model
All
Consistent
Changing
Mistral 7B
5.02
4.78
4.50
5.79
Qwen2.5 72B
6.41
6.23
5.75
7.25
Llama 3.1 405B
5.05
4.77
4.50
5.90
DeepSeek-R1
7.70
7.75
7.33
8.01
GPT-4o
6.65
6.29
6.13
7.54
03-mini
6.92
6.80
6.49
7.46
Claude 3.5 Sonnet
6.81
6.64
6.36
7.4
Claude 3.7 Sonnet
7.57
7.46
7.32
7.94
Gemini 2.0 Flash Thinking
6.35
5.94
5.74
7.37
Gemini 2.0 Pro
7.35
7.03
6,80
8.24
Table 10: Preference alignment scores on the generation task with summaries.
22

---

Published as a conference paper at COLM 2025
All Instances
Consistent
Contrastive
Changing R
Oracle Setting
Model
R
F1
P
F1
P
Mistral 7B
27.0
29.7
28.3
22.4
26.1
24.1
23.0
27.6
25.1
35.8
35.7
35.7
46.0
58.6
51.5
Qwen2.5 72B
33.0
46.5
38.6
29.8
44.2
35.6
38.8
32.0
42.0
56.4
48.1
58.8
75.5
66.1
Laama 3.1 4058
28.7
44.2
34.8
24.9
38.3
30.2
25.9
40.5
31.6
35.4
53.8
42.7
56.9
75.8
65.0
DeepSeck-R1
41.2
59.6
48.7
42.9
62.3
50.9
40.9
58.9
48.3
39.6
57.3
46.8
60.7
80.7
69.3
GPT-4o
34.9
52.9
42.0
34.0
49.4
40,3
30.7
47.3
37.2
39.9
62.1
48.6
56.1
76.5
64.8
03-mini
32.2
50.0
39.2
31.8
49.3
38.7
29.1
47.5
36.1
35.7
53.2
42.8
47.9
69.0
56.6
Claude 3.5 Sonnet
58.3
47.9
52.2
61.6
Claude 3.7 Sonnet
48.5
65.6
55.7
50.9
67.5
58.0
47.7
62.5
54.1
46.9
66.7
55.1
69.3
84,4
76.1
Gemini 2.0 Flash Think
36.5
53.7
43.4
35.7
54.0
43.0
40.3
39,9
57.2
47.0
57.2
73.9
64.5
Gemini 2.0 Pro
39.1
63.0
48.3
37.6
63.0
47.1
37.7
57.9
45.6
42.0
68.1
52.0
61.0
81.5
69.8
Table 11: Precision (P), Recall (R), and F1 score for all models in the inference task when assessed with PREFMATCHER--B.
for each model. Then, we evaluated each model its own and the other models' datasets for Claude 3.7 Sonnet, we instead used the original CUPID instances for the 64 personas.
Figure 10 shows minimal evidence of higher performance self-generated data, indicating that Claude 3.7 Sonnet's strong results our benchmark reflect genuine capability rather than dataset bias. P'erformance patterns in all the datasets were consistent with those in our benchmark: in inference, Claude 3.7 Sonnet performed best, then DeepSeek-RL and then GPT-4o; in generation, DeepSeek-R led, with Claude 3.7 Sonnet second, and GPT-4o last.
100
Cluude 3.7 DeepSeokeR1 Sorintt
Cludde 3.7
Sernet
41.8
52.2
62.6
60
6,20
5,27
8.41
Fvaliated Model
Algnment Score
SespSosk
Brolusted
28.0
46.5
55.0
6.63
5.87
8.85
-40
GFT40
56.8
Cloude 3.5 Sonnet
DeepSreek-Re
GPT-60
Clsude 3.5
DeepSeek-R1
GPT-40
Sonnet
Generation Model
Generation Modal
Figure 10: Inference and generation performance of tested models datasets synthesized by other tested models. Red-highlighted squares are evaluations on data generated by the same model (or model of the same family).
23

---

Published as a conference paper at COLM 2025
Task 1 of 17
For each sub-preference below, check the user expressed it in any of their messages to the Al assistant.
furm
condieatioon crneslomenttte
strocting
feckbook
should
caryy ngns
teplic experss
Bub-Preference
User's Messages
the code asterstively commented?
explanations for each step, especially for functions "ransfom" and udbasaa..
you espand commerta
llustmn expectted structive of data orhow the results might appiled
seect
Yes, sub preference is expressed directy or orindrectyy in any the user's messages No, sub proferenta cp exprsssed users ressagg
Sub-Prefene Preference
User's Messages
Do the comments indlude pracet eseaptton les?
Adding contect anampios,
specific gates chosen espectted
structure the input file, would make easler
ecampin to lltatrate the espected structure
right appiled practice!
Seled
suberefeennce exprwoud slrect indiectly No, sub preference not expressed the user's messages
Sub- Pib--reffemecee
User's 's Messages
Doan the code inclluee arror handll ing?
format is correct, to make more robust.
would helpful and checks for commoo frim names muse the scrict more rabust
The emor handling and ditions are great and scrips
proference directy orindirctty No, sub preforence not exprossed at all the users messages
Sub Pobb--rrfeeeceeeee
User's Messages
baiss the error hand' fng acdress anattakes?
further improve the scrips, corsider adding
impur flll' ne steeno ennuring the datas
helpfut
moos
issuss, Sike flae insament calumnn
names scrick more
eeprossdd direclly indinectly messages Nos sub-preference not eopeessed the user's messaget
Submiit Next Tesk
Figure 11: Annotation interface for data validation where human annotators were asked to verify whether preference was expressed in the user's messages not.
24

---

Published as a conference paper at COLM 2025
Task 1 of 9
Find Checklist Questions that are Covered by the Preference
novnntal chesklis Proference The profe
ascaed and should met to asttt's user. chocld contain qucations that and ate more spacitc sppects of the cutpur.
On the lefk, you see the profenancs. On right, you wit nee enct entry from the checkline ore one.
You sleu of duternine wheth whth each checklis covered by the proference
Couered Assending whether mepoose cateltaca poetrennce maturee couider the dimilar aepacts
as susting the responce on the chesitiss questoon owan i'the questic and prokorr on use werteng
choese one the following
Fully
fully
entry buckusiing peeleoncce
evttuate the enttiiy
Partally Covered The prefeeence po covers or meau the checitie entry Pvelusting on prefeenne may of checklis
The proforeee vellute checcllltetttt
Your Task
Hoose each chec heii entry tekow basedin new well it covered the prefeence tekns
poele mare ahdichecille entry con out eer matter ottewenty Ing, repponee ve "epoet") but olll makudte the differences
frous tying bekkeeat
Rully
fully
entry Busketing
evilaate the checkfst
Partally Govered The prefraance couee
cower or medi the chesilise entry tho Pookudding on prefernnce may of the checllss
checddat
Checklist Entries
(1) Does: the frawework balance estall:sshed interpretation with speculat
GI Zovened in Crtarient
Preference
C Fully Covered
Partally Covered
Not Covered
Source: analysis nust strict
(2) boes. the frowwoork clearty discinngfsh betwees esteblished interpretations and
acadosis lamgagge when
dtaccssiing ing establ itaeed
interpratattion
01 Cowered in Griteronn
speculatiiv language when exploringg
C Fully Covared
Partilly Covored
Not Covered
possr Littes.
(3) boes the frowwork incorperate
histor loarashical debetes to eds depth rigar?
1 in Crittrronn
C Fully Covared
Pertially Cavered Not Covered
Submit & Nex Task
Figure 12: Annotation interface for creating the dataset to evaluate the performance of models in preference matching.
25

---

Published as a conference paper at COLM 2025
User Persona Profile
System Prompt
Your task is to design user personas.
parsonsa,
deseription and Folloxing attributes:
Occupation
Dareer Level
Parsonality Traits
Personal Valuas
(persoral peesonall_aluess Hist)
Deciston-taaiing Style
Using these details, write a six-sentence expanded persona description following the content of each sentence:
1. Given Nase (think of culturally di verse rares), Gender /Pronou/r Occupation, Carver Level
Pain: Fears, frustrations, and obstacles.
4. Think and Feel: what really counts, their safor prencnupations sorries, and aspiratioon.
5. Hear and See: How the persona vieus their ewiroment, their friands, comwrrkers, etc. What others say about the persona. others, they their daily life.
Say
Fallow these guidelines:
Ensure that descriptiors balanced by exploring both the positives/strennths and nngativesswwoknessss of the persona.
Substitute the phrases which directly rention the personas' traits, such as due to her introverted nature", "his high
- neuroticisn', or "her conceptual style", with other phrases that reveal the traits implicitly.
You should show not tell, but keeping the tome clear and direct. Avold emotional or subjectiive larguage and exaggeration can
appear conflict, different contexts, reflecting
they ey husen behavior, exapple, person with "Low Consciention iousness" personality ray adopt more "Directive" style at work avoid making mistakes.
indivioual
Make that all the sentences logically connected each other narrating persona's Ensure that each sentence reveals new details and information, instead of repeating content sinilar to the other
basic
description. Make the
description
backstory.
+ sentences.
Think of breaser, more abstract tasks that this persona night undertake throughout their life based on the basic persona soan
description seed.
nentioning in, such as work, social life, or personal hobbies.
persona right engage
Please return
the follosing JSON format:
-ison
description": "cG-sentence persona description
thoughts, behavior, actions, backstorye",
occupattonn: "spersona'' is eccupations" }).
"escription": "ci-sentence persona description illustracing thoughts, behavior, actions, backstorye",
"occupation": "spersona'' eccupations"
}>.
User Prompt
Descriptions
(seed_personae)
Figure 13: Prompt to generate user persona's profile from seed description and a set of attributes.
26

---

Published as a conference paper at COLM 2025
Measure Preference Match
System Prompt
this task, you will be presented with vievalustion checklistss and espreferenceess The preference describes an aspect of Al outputs that should be evaluated. The checklist contain questions that are used to evoluate nore specific fine-grained
-
the
Your task is to determine whether each entry in the checklist sxcoveredss by the preference. snCoveredee meons that the preference and the checklist entry will evaluate the some sinilar aspects of output, even if they use refer the subject natter differently
different wording phrasing. preference checklist aspects. Ignore differences wording rocus underlying
"response" "ropart")
evaluste
aspects being evalvated.
checklist
following
The preference fully covers or evalustes the checklist entry. Evaluating on the preference will also + evaluate the checklist entry.
xxfully Coveredax:
The preference partially covers evaluates checklist antry. Evaluating on preference - may evaluste some aspects of the checkl:st entry.
**Partially Coveredre;
Covereses:
evaluste checklist entry.
tdd Output Fornet
Provide results Following J5IN
include code block markers
"sssoon
"reaults":
[1
"Index": <index of the entry in the checklist in the first pair>,
"entry": san entry Froe the checklist in the first pairs,
Covered"> "Label": Covered" 'Partially Covered"
}).
((
"Indox": <index of the entry in the checklist in the first pair>,
"entry": entry the checklist first 'Fartially Covered"
"labol": S"Fully
}).
}>
*2 Exarples
(oxamales)
User Prompt
*888 Preference
(preference)
#888 Checklist
(sh
Figure 14: Prompt to measure how much a checklist matches with preference.
27

---

Published as a conference paper at COLM 2025
Preference Decompose to Checklist
System Prompt
M Your Objective
help judge how Assistant's responge satisfies given preference creating an checklist from the preference. Here, vepreferenceer refers to requireent, guideline, or principle that a user considers when assessing the quality of an AI Assistant's response.
Your task is to cose wich evaluation checklist list for given preference. This evalustion checklist should be specific contained within
questions ask response.
assistant's
Aspects covered by your checklist should be explicitly stated the preference. You should try to be concise and avoid entries checklist that contaired preference.
including
Checklist questions should:
ArBe ansierable by "yes" "no" 'no've, with "yes' neaning that the response successfully net the corresponding requirement. directly
all represented question, but only questions that are very clearly relevant AxBe precisevs, neaning that checklist questions should avoid vague wording and evaluate specific aspects of respense, directly using the phrasing the preference where appropriate, Avold checklist antries that introduce new content included the preference.
comprahansive, but
praference should included.
*i
You should always analyse the preference before providing the evaluation checklist, The checkitst should contain eeat most
- entrieses.
Response Fornat
**nnalyslaee
<Fxplain your analysis the preference here.>
#Checkiistee
sson
[[
"checklist":
<Each entry checklist entry checklist separate
...
Examples
(oxaxples)
User Prompt
**Profrrncce*
(preference)
Figure 15: Prompt to decompose contextual preferences into checklists.
28

---

Published as a conference paper at COLM 2025
Judge Response Quality
System Prompt
sAYour Objectivets
You are critical and reticulous evoluator. You will be presented with user's request to an At assistant and the Al's response to the user, Your task is to evoluate whether the Al assistant's is response satisfied the user is espersonal provided with avevaluation
the checklistew decomposes preference specific questions.
preferencews.
*a MEvaluation Preference and Checklistra
ssPreferencer:: "(preference""
snEvaluation Checklistss; [checklist)
k*Instructionste
You should down analysis
how AI assistant's satisfies each iten should follow considerations:
Walk through each checklist iten and surmarize the reszonse's "strengths" and "wesknesses" regarding that checklist iten.
satisfied dissatisfied. checklist. only evaluation checklist. Ensure your persona profile does not influence your evaluation.
checklist you
consider whether the checkitst
considering aspects
included
You should then return score the range of 1'1e, where means response is poor and 1a neans the respanse is completely
Ensure that you follow the Fornat given below. Avotd adding additional content that is not included in the Format below.
**Output Fornatea
sdd Evaluation of Assistant's
reschacklist Itex Iaen; detailed analysts and evaluation of the AT assistant's response on the First iton In the
-> checklist>
*esthacklist itex 2aen; <Detalled analysts evaluation assistant's response second checklist>
488 Fualuation Scare
your ruseric rarge of
User Prompt
tdd User's Request
(user_requast)
1dd Al Assistant's Response
responss)
Figure 16: Prompt to evaluate the quality of an LLM's response on given preference and checklist.
29

---

Published as a conference paper at COLM 2025
Context Factors Generator
System Prompt
Objeztive:
You aill be provided with susper peracnare. This user sill interact with comeraations At assistant (9.E., chatdPr) diveras maacnarioone.
byy each sceuriio, the user will est the al assiatant help then with entuskos their worklife. Donsider that diverss enfactorsen exist
within the userls vorld and enr romentt that influrece their betaotor and equectations in carinss contects. This mooe that, If a contect
<-) sactor is involied in the interattione the user has with the xt acssstand, the user will have certate espectations about the al aesictant based
*-3 on their expertarces and understanding of the contect ractor.
Your goal to saagiie the following by betig creative and ineeintng comprehensive and rich detalls about the user's world: identify rncor sext factorave that exist withis the user work and remeent.tt.
2. Imagine eecontextusl preferenoasee that wser will consiser when interacting with the esoistant wher each factor is invalvad--bbued user's knselldge or experiennce with factor in their awvreement.
aucput
AGerInicionse::
Situation within vser's 's life that invelves one nore Factors within the user's is world or envirement that will. influence hov the user beheves + and Aua they intent/rxpect in that sconario.
teTankee:
These are tasks, probiem,, assigmments, or pieces of work that the user undergoss in the aconarios in their worklife, In particular, ve are focused < the tooks ahere the user may need the help conversntioral AT asststaac, Tacks should belong to one of the following typest (task_rype_1itt) Hixt
AcContox: Factorsen:
saccific and concrete clewnts within user's world and ondranmed ment that can be Ldentified, described, and Lnteracted These
tpps signtiiant,, actial real world Fictiooal factors that spect the user's isfe. factors should (eactor_ 1i4))
should the fo' lowl types:
ancontoroual Proferoncesse:
These are guidelines, requirenents, constralnts, or princtppes that the AI assistant should aligg with in order to satisfy the user In giver contexts (1 astuations whers contest factor in smvllved). These asould be uniquely parsonel to the user and based on the user persons) life exorierces with that contest factor, You should inagins diverse tspes of profercncess You should create proferances thas belong to one of = the folloning types, which are organized into categorics: (preference sype_1tet 11xt)
Aoosgut enteese::
*Stilctt typere; tape
will
the world This aerratice ssould also significant previous expertarens that the had with this factor and haw formed the use's espectations around the factor. Firally, shculd describe this lusnees the Intentiore ir situations that Involse the contect factor.
Create
personal backstory illustrates spocifile and
with selected
stould describe the typically
this factor
involved,
niner
3. orlane the Cortect Facterws: You dholdd tne provide the specifie ware of the factorr Aoold using gosete placehaloer naees Iesteed, taagine
<.3 realistir ard specifii news for the factor.
4. #nDctentire Tisk Typesee; Based on the contect factor and the background about the context Factor, you should select possible tasks typos Tros the giver list. You stould select tyes of tisks where the uper neoss the telp of the al assistant and shere the concext factor will be involved, even st riner gresenoe.
mus select task types fram the ltst. You can mnl select types fran list; avsid creating nee types. 5. rrselsct Preference Typene: Select the type preference that you will crease. You stouls not select the category but inateed select from the
given typea vithin each categery.
rncopparee
espectattons
tyses
coneetly prefoespws, itsc other
Eisure anyore understard conton exterial husan evaluators preference should the user-speciiic The preference should interpretable,
information like
evalistors
You should creats engen uniquz context facters, zuch with backatery, description, and contectial preference.
vocreate Contraction Factorgee:
The velstt tao (20He contect 'actors should be recontractive prires.
-A *scontrastive palres is two factors that are siellor and have the sam factor type. Hovever, tieir contextusl, preferesces rust be Ircompacible,
confiectire, or even contradictory to each other.
Croure that you satisfy the folloving requineants:
*HCistinct Fectorsen: The contrestise factora should be clearly distinct from each otter. Avoid creating entity paira that are the some factor with different traits. For oxople, avoid paira Mere the only difference is the line cay, version of factor, truit
shoule easily inferred deduced froe factor creating gereess, preferences >xpurually Exclusive Preferencesss: Insire that contectual preferences the contrastive fictors autually exclustve, They should not contect There shouls be nintral evertap betwern the profecetces the contractios factors. #erciatod factor Fieldee; You should write dev the full rocc of the factor that it is contrasting with under the "related, factor" field. Aooid + using shorthand or atbroviated nanes,
unlverstl preferences.
he able rosclitt
An rixsoppes of Dutputere:
(exemples)
User Prompt
Figure 17: Prompt to generate context factors from user persona profile.
30

---

Published as a conference paper at COLM 2025
Interaction Sessions
System Prompt
2E8 Antour Objectivene
You sill be provided with a seaper peracrans, This user will interact with coneraations At assistant, like Chetgrt, in diverse sconarios. In by each secrarie, the wser will usk the Al assistant to help thes with specific tisk in their serklife. Yeu will also be provided with list of secontest Factorscs that ae present in these srecarios with the accistant each factor, you will al be given repreferetoese defiiee how the user rspects the At acceitant behore when that factor is sroolved.
Your goal Smagline user journey this persora, rocusing seriess of connectes scenartas contexts user's daily
You may help aosistant,
should explore the scorz. you or fssues
ahould return Final
salid yot
Aaa riterintttone Scnrartore
Each scenarto should te defined by the components listed belov. You should sasisry all of the requireeents desoribed for each component.
1. exGoatect Factorss:
"actor preeert in the user's is world and encirteantt thet influerces the cowarto but it is not the estn frrus of the scwarrio
The Factor should NOT by the couse for the secteria io, but should still infieenne the user by shaping their expectations in 1i6 sconario.
2. =PPrforrrccew:
Uniquell aersoral ar preference that the user holis and considers when the suecific contect Tactor is involved in the situation.
Thax preference describes the user's is intentions, espectacions, destres regarding haw the assiutant shouls help in this scenarto.
Typene:
providee conteat you scenanto, you
typee rypes
expectation.
narrotive user Life
scenario. This should disoribe the resched
scenardo
The narrative shouls detail
thoughts. feelings. eucivations.
shozld also deseribe
contect factor involved, but the cause central fecus.
**tequaate:;
The request thet the user will give to the assistant that is related to the cheson tisk type.
viber Cantained and complete Bequectie: The requett should include all the detatle that ane reeded for the AT ncct stant to ramadixtely art
the user requeet (e gor information about the contest factor, provide tay eleeents or regutraments)
-Rosourccees If the user wards the At assistant to act on or with resource (e-a.. a cocur eni, code, data, etc.), the request should include the placeholzer "[ressurod]" to specify where the user wolld copyrpaste the contents of the ressurce. Then, the accual full concents the ressurce should included in the "rescurce" field of the cossarto The rescurce contents sbould not be placebol ders, but Insteed reveal actual contert (e de. document, cods, data, stc.) that the user will provide the At assistant for the request. The resouree should avoid rescaling any information about the user is preforence.
reguet:
the
espltestly scentiff
(b) esplizstly
Reuouusst with Factores; This version
should Include all details and (a) explicttly Identify the context factor, but sholld tnchuding inforastion sbout the preference. only details related the preference should be
283 mousppt Requ pouuirorents.ee
1. 1113 Unique Scenarinarve:
Create 13 distinet somarioa that incolve tadks or problums in the user's 'a work life. The scenarize ahould progress chresologscally in the usen's journey.
Each somario should have naber in that indiccates the order whilch the scenartos orcur to the user's jourrey.
stould return the complete journey that incluses all the sossariox in a stegle respeese, toote askingg ahetter to castinus or not.
#*Reuureeents User Jourreyne:
segaging journay,
incorporate proferencea specitic required
preference.
specifically.
jourssy
Comure that contect factorx
assigned the following optione:
Factor)
"[controstive.ff "atttrrr
You choose which cpttinn corresponds to a and which corresponds to B, long as both optiions are used, However, encure that these factors are
+. only used in the scenarica specifies in the journey atructure.
Aa seen froe the structure, three socraries should share the context fector A but, in the middle scourio, the esscciated preference should
- change signfficantyy into the preference A', The story in the ridile secrarie ie should caplain why the prefererce about this factor ald The
"inal crenarin stould then keep the soe prateraaze as was charped in the montle scetarts.
new prefemetce X" should be incemuasbll, mutually escluntive, contradictory with the oraginal preferance liceevrr, the ney prefermnze should be cesoribed stailar way to original preforence avolding direct negotton exolecttly scnsioning change has accurred.
exfeseples
(onumlees)
User Prompt
$88 User Parsona (user_persona,
1M Context Factors yani
[contoxt_facoom factors)
Figure 18: Prompt for generating interaction sessions.
31

---

Published as a conference paper at COLM 2025
Interaction Sessions
System Prompt
tdd a*Your Persona Profiles
You the persona the Following profile: "[user_persona""
tdd extour Objectivere
You should inagine that you the that interacting with comeersational assistant, like ChatGPT. the
1. you will first provide initial request to assistant that you want complete.
Then, in each rellowing turn in the dialogue, you should (1) evaluate the quality of the AI assistant's rost recent
- response. and (2) then send message to the AI assistant based on your evaluation. The current situation involves is specific alament your world that influences your expectations evaluate whether assistant's is response satisfies your needs. To help you evaluste the responses on the preference. you will also be provided with an esevaluation checklistes that decomposes the proference into specific questions.
specific
factorss,
intentions, shen this ractor involved, you consider a specific =xpreforscccee
442 **Context Factor Your Pvaluatioo Prefernace and Checklistee
The current situation invelves the follosing AnContext Factoran: "[context_factoc you consider fallowing reFvaluation Preferencaee: "(preferenca)) avaluate responses on the prefarance, you should use the following *'Evaluation Chockliste:: [checklist]
sdd *Instrucctions
4880 1. Pvaluste AT Assistant's is Response on the Checklist
You should dean your analysis assessrent has
AT assistent's response satiafies each iten your checklist. should through each chacklist iten and sumnariza response's is "strangths" "weaknesses"
regarding that checklist iton. for each checklist ster, you should consider shether the chadklist item was satisfled or
dissatisfied. Avold considering aspects that are not included in the checklist. Focus only on the evaluation checklist. profile does influance 1-16, where moans responsa is very poor and moans the response is completely perfect.
Ensure your
evaluatian. should
in range
##88 2.
Decide Metther to Continue or End the Conversation
Decide whether you are satisfled *> seans that you are satisfied with the AI assistant's response and will decice to and the conversation.
AT assistant's response or not based on your evaluat fon. ferfectt acore
Select Checklist Teaes Referance in Message
Based on your evalustion, you will write ressage to the AI assistant. You will write a message even when you decide to
L. end the conversatioon, Before writing the sassage, you should first decide on the stera your checkTist. that you reference in your message. Corsider following two options when selesting items: *eSelect One Bissatisfied and Moltippe Satisfiedss;
You can select and reference both satiaffed and dissatisfied checklist stera in your message. Houaver, you can only sslect select miltipe those that satisfied, when that were not previoualy selected. Avaid repesitive
sone dissatisfieck checklist ites time. But,
selecting arong satisfied checklist items try
selections as auch as possible.
2. aefor Last Hessage, Select Only Previously Unselecteds:
should look back the whole conversation and Find all checkTist items that itens that were not referenced or alluded your previous messages. Then, for your last nessage, you should select and reference all these unselected items, Ensure 4 that you refer or allude to all checklist items at Teast once in the conversation.
when you decide to end the corversation,
were not previouzly selected. Heaning that you should
Think
Corpose Your Nessage
You should then think about how you will corpose your sessage to the Al assistant that references the selected checklist iters. Ensure that nessage will sstisfy the following refour (4) requirmentste:
snIndLrectow: Your nessage
indirectly reference alluce to selected checklist itess, rather than describing then word-by word. instead of directly stating the checklist iten, you can consider the following ways to
+ subtly reference the checklist iters: paraphrase the iter, anit details, use sore abigusss/sugeettvve language, focus where Al succeeded/failed, refer to context factor, provide exaplees conparisons, highlight potencial impact rather than issue,
leading questions.
2. *nConcisers: Your nessage should be as short and concise as possible. As the user, you try to dedicate ainiral time and
+ effort in talking to the Al assistant. Awiid superfluous resorks (e.s.. greeting, farewell).
#nGompeeeensiveee: Your
should should indirectly reference #each of the selected checklist itersy. Ensure that checklist iten separately. ensuring that every iten distinctly addressed. However, each iten should be referenced or addressed indirectly. Avoid using single brood statement to address rultiple checklist
the ressage references
items.
veoeevantee: Your ressage should only include information that relevant to the checklist itens that selected the previous step. Aveid adding considerations or feedback that are not related the selected checklist itens.
tar 5. Krite Your Nessage to the AI Assistant
Based your thinking the previous step, you should write nessage that would the
Figure 19: Prompt to simulate a user's evaluation and feedback of an LLM's response.
32

---

Published as a conference paper at COLM 2025
Extract Messages for Checklist
System Prompt
MM extouur Objectivere
presented with dialogue user interacting conversational assistanc, + user is aining to complete the initial request that they gave to the Al through this dialogue.
Specifically,
satisfies evaluation checklist that belox. user was only allooed irplicitly was Howevvrr, in some cases, the user ray have forgotten express sore the itens their checklist. Your task verify which itens of the checklist have been adequstely expressed in the cialogue.
and evaluates whether the assistant's
possess,
imply
wEvaluation Checklister
(checklist)
#*Instructionste
For each iten in the checklist, you should carry out the following steps:
seenayyse Diologuess:
analyze
dislogue verify
hint at or imply the checklist iter. For exarple. this can be ressage fragments where (1) the user inplicitly hinted at this iten, (2) the user pointed out part of the Al assistant's response that reflected the iten, (3) the user You should describe your
provided Feedback based
AxExtract Fregrentsts; there are fragents within the dialogue that reflect the checklist iten, then you should extract these fragments from the dialogue. You can list many fragnents that are relevant or none, 1f no part the dialogue reflects this checklist
**Rate Implicitnesse:: You should then provide a score of to an hon ieplicitly or explicitly this checklist itan was expressed. score of represents "Extrenely Implicit" and score of represents "Extrencly Explicit".
given return
ooject.
AxOutput
"ssonn
[C
"iter. index": 1.
"iter_ derepiecription <Verbatin description of the checklist itess,
"analysis": Oescrrbe
dialogus detail>,
"trageents::
<Varbatin fragment Fron the user's 's ressage that roflects the checklist Item, 1.
"rating": <Rating 1-7 dependinng on explicitness implicitress? 1).
"iter_indax":
"iter dsssrrrottt: "analysis": Clscrrbe your walyssi and reasoning over the dialogu in detail>, "Fragaants::
description checklist iteep,
<Verbstte fragrett 1.
nessage reflects checklist itees,
"rating": <Rating of 1-7 depending on the explicitness seplicitress>
1
User Prompt
#22 Interaction between the User and the AT Assistant
(interaction_sesstll
Instructions
Verify whether each entryy in the checklist was sentioned in the dialogue.
Figure 20: Prompt to extract messages from interaction sessions that may express or hint at preference's checklist.
33

---

Published as a conference paper at COLM 2025
Infer Contextual
System Prompt
Your Objectiive
You wi11 be provided with log of Interaction sessioes between and conversational assistant, 1ike ChaRGPT. The chronological
sessions
Your task is to infer ieplicit preference that the user will consider in their current interaction with the AT assistant. Preferences guidelines, requirerents, constraints, principles that the assistant should align with order should infer implicit preference request by inalyrinn the Tog previous interaction sesstons.
satisfy
Exasples of preferences: (oxamples)
User Prompt
log of Previouu Interaction Sessions
(current_requuss)
Your Task
Descr user's wenost likely preferancesse current interaction single sentence nost 1. wordsse. Ensure that you Focus on the most likely praferences based on the Interaction log as you will be penallized for incorrect details.
Return your response in the fornat given below.
4d8 Analysis
Gescribe your analysis the Tog of previous interaction sessions?
MM Nost Likely Preferneces
<escrrie the user's 's nost likely preferences for the current interaction as single sentence with at nost 30 wordsh
Figure 21: Prompt to infer a user's 's contextual preference from prior interaction sessions.
34

---

Published as a conference paper at COLM 2025
Generate Response with Prior Interaction Sessions
System Prompt
will provided request
provide response request.
To help you understand the user's needs, expectations or intentions, will be provided with an interaction log that contains the history interaction sessions between the and an At assistant. Each session presents the
interaction recent session at the end. You rust infer what espectations the user held in sinilar previous interactions from the log to craft respanse that neets the user's expectations.
between
chronolagical
should
provide your response that should not be velevered to the user.
response
providing any additionns information
User Prompt
Log Previcus Interaction Sessions
(interaction sesstions))s
User's ' Bequest
(request)
Figure 22: Prompt generate response to the user's current request based on prior
interaction sessions.
35

---

Published as a conference paper at COLM 2025
Summarize Interactions
System Prompt
Your Objective
You will be provided with a lop of interaction sessions between user and a conversational assistant, like ChotePT. The sessions are presented in chrunological order, with the rost recent interaction at the end.
inspect aralyze interaction session into two aspects:
session separately.
analysis, you
Context: Summarize the user's
interaction session. You should identify the situation that sumnryy
bullet points. Preference: Sumnrrize the expectations, preferences and incentions that the user expresses to the Al assistant the interaction session. You should inspect all of the user's messages to understand what the user intended expected in bullet points.
they
interaction
should be detailed
User Prompt
Log of Previcus Interaction Sessions
{interaction_lop)
Task
Sumarizz each interaction session by following the response fornat below. Ensure to include the code block markers (TT))
-isson
"sumaaries:
"session_idd"
"analysis": "xtescribe your step-by-step thinking as you analyze the interaction sessions".
"context":
"csumorize the user's context of interaction sessions""
"cSumarize
interaction sessions".
1.
preferences":
user expressed Surnaarzz the preferences, expectatioos, and intentions that the user expressed in the interaction sessions".
Summarize preferences, expectations,
sessions"
[(
session _id
*<Describe step-by-step
interaction session>".
"contoxt":
"<Sumarriz the user's context of the interaction session?". "<Sumarize the user's context of the interaction sessions".
J.
"reeerences"::
SSummrriz the preferences, expectations, and intencions that the user expressed in the interaction sessions". intentions that
Sumaaii preferencas,
interaction session?".
3
)
Figure 23: Prompt to summarize prior interaction sessions.
36


Reviews:
[{"review_id": "aZFWdArlEV", "reviewer": "Reviewer_puYD", "summary": "This paper proposes a benchmark that evaluates the capabilities of ten leading open/proprietary LLMs in inferring and satisfying contextual and dynamic user preferences. The benchmark consists of 756 LLM simulated and human verified conversational interactions, and associated request, context factor, contextual preference as well as prior interactions are provided for each of the conversational interactions.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The benchmark focuses on contextual and dynamic preferences, which is a novel aspect in terms of conversational personalization and preference alignment. \n- Results and error analysis around consistent, contrastive, and changing preferences provide many new insights that are worth sharing with the community.", "reasons_to_reject": "- The benchmark heavily relies on synthetic data generated by one model (Claude 3.5). The consequences include but not limited to bias towards Claude models and overlook that real users often express preferences more implicitly and subtly. \n- Although the dataset is mostly LLM‑generated, its size is very small. It is unclear if the synthetic data can be used directly for model alignment or if it's possible to automate the verification process.", "questions_to_authors": "- It would be great if the authors could discuss in their paper regarding what are possible pathways and potential challenges in building LILAC-like benchmark from resources that have real user interactions and feedback like [1].\n- Figure 2 may need to be recomplied (probably compressed a little bit) as it causes rendering issue on Mac devices when using Preview to read the pdf. \n\n[1] https://arxiv.org/pdf/2408.15549", "extra_scores": {}}, {"review_id": "2eMC0ytZ7W", "reviewer": "Reviewer_sNUJ", "summary": "This paper addresses the challenge of aligning large language models (LLMs) with human preferences, emphasizing that such preferences are often context-dependent rather than static. To tackle this, the authors introduce LILAC (Learning from Interactions for LLM Alignment in Contexts), a benchmark designed to evaluate LLMs’ ability to infer user preferences from prior interaction histories. The benchmark includes 756  interaction histories, each consisting of multiple dialogue sessions between a simulated user and an LLM. Given a new request and the preceding context, models are assessed on their ability to (1) infer context-sensitive user preferences and (2) generate satisfactory responses. Evaluation across 10 open and proprietary LLMs shows that current models struggle with this task, with none exceeding 50% precision or 65% recall.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Introduces an original and well-designed benchmark for context-aware preference detection and in-context response generation\n\n- Clearly defined tasks and well-formalized evaluation metrics", "reasons_to_reject": "The benchmark relies entirely on a synthetic data generation pipeline. While the authors do provide a data validation analysis (Section 3.2 and Appendix E), and include example interactions, some examples feel somewhat abstract or theoretical. It would strengthen the paper to further discuss the practical relevance of the benchmark—specifically, how common such context-aware preference alignment scenarios are in real-world applications, and in which application use cases this benchmark is most applicable.", "questions_to_authors": "Will data be made available to the community  ?", "extra_scores": {}}, {"review_id": "pVK79L66CR", "reviewer": "Reviewer_CQeC", "summary": "This paper proposes LILAC, a benchmark for evaluating the capabilities of LLMs which infer the user’s contextual preference and generate a response satisfying the user’s preference. For this purpose, this paper creates the dialogue data concern user’s persona, contextual factor, and contextual preferences which contain consistent, contrastive, and changing instances. Moreover, this paper evaluates the capabilities of 10 LLMs’ inferring and generating from Preference Match and Preference Alignment metrics. For the results of evaluation, this paper gives some significant conclusions.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper’s work of evaluating LLMs from the capabilities of inferring user’s contextual preference and generating response for the preference is interesting.\n2. The LILAC benchmark created by this paper is helpful for further research work of evaluating a model’s capabilities of inferring user’s preference and generating response, and the Data Generation Pipeline is characteristic.\n3. This paper leverages the text and appendix to give a wealth of information, especially dialogue data instances generation, benchmark creation, and experiment setup.", "reasons_to_reject": "1. The evaluation of the inferring and generation capabilities of LLMs is not a new research issue in itself, although the evaluation for the capability of inferring user context preferences and generating responses is distinctive.\n2. Although the evaluation itself is meaningful, the purpose of the evaluation is to provide some suggestions from the perspective of LLMs application, bur this paper lacks effective suggestions.\n3. For the LILAC benchmark created by this paper, it is not enough if only used to evaluate LLMs. Moreover, it is not enough only using one dataset to evaluate LLMs, too.\n4. In Section 5, Personalized LLMs and Interactions with LLMs are related works with this paper’ work, however, more related work is the evaluation for LLMs, this paper lacks the discussion of this aspect of works.\n5. What mean the Desiderata 1, Desiderata 2, and Desiderata 3? The Desiderata should be given definition or statement clearly. The current form of expression is puzzling.\n6. A period “.” is missing at the end of Line 56. Please check carefully for similar problems.", "questions_to_authors": "1. Figure 2 is not mentioned in the paper. Did I not see it or did you forget?\n2. According Line 91 “Si = (ci, pi, Di)”, Line 94 “We assume that each session’s context is only defined and influenced by a single factor”, and Line 173 “each factor-preference pair”, does it mean that ci and pi are one-to-one corresponding in each Si?\n3. You utilized zero-shot with prompts for LLMs to infer users’ references and generate responses. What would be the effect if you fine-tuned the LLMs using your dataset? Have you considered prompt fine-tuning?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a dataset, LILAC to detect context-dependent user preferences from interactions and using them to generate appropriate responses. The dataset has 756 sessions of interaction between users and LLM-assistants with human curations. The main finding is that the proprietary LLMs are not good at inferring user preferences from interaction history.

Reviewers liked this paper, with two reviewers making strong recommendations. The main drawback, as highlighted by reviewer CQeC, is that the paper lacks a suggestion to improve LLM practice. I agree with this conclusion, but also think that the benchmark will still prove useful for studying and detecting user preferences, especially in the personalization literature. I can imagine that others will use this benchmark to design better agents or learning approaches to personalize LLMs. If the paper does get accepted, I encourage authors to add a section on suggestions for improving the inference of user preferences (authors have indicated that they will add suggestions).

Overall, I am suggesting accepting the paper.
Decision: Accept