PrefDisco: Benchmarking Proactive Personalized Reasoning
Authors: Shuyue Stella Li
Avinandan Bose
Faeze Brahman
Simon Shaolei Du
Pang Wei Koh
Maryam Fazel
Yulia Tsvetkov
Venue: ICLR 2026 Poster (2026)
Keywords: Interactive Personalization
Test-time Reasoning
Information Seeking
Preference Alignment
Proactive Question Asking
Primary area: foundation or frontier models, including LLMs
TL;DR: We introduce the task of Personalized Reasoning, in which LLMs need to reason about missing user preferences, strategically elicit preference values, then adapt their reasoning processes and responses accordingly.

Abstract: Current large language model (LLM) development treats task-solving and preference-alignment as separate challenges, optimizing first for objective correctness, then for alignment to aggregated human preferences. This paradigm fails in human-facing applications where solving a problem correctly is insufficient if the response mismatches the user’s needs. This challenge intensifies in just-in-time scenarios where no prior user interaction history exists due to cold-start conditions or privacy constraints. LLMs need to proactively identify what they don’t know about the user, strategically elicit preference values through questioning, then adapt their reasoning processes and responses accordingly—a complicated chain of cognitive processes which we term personalized reasoning. We introduce PREFDISCO, an evaluation methodology that transforms static benchmarks into interactive personalization tasks using psychologically-grounded personas with sparse, context-dependent preferences, and define PREFALIGN as a fine-grained rubric-based metric for measuring preference alignment. PREFDISCO builds scenarios where identical questions require different reasoning chains depending on user context, as optimal explanation approaches vary by individual expertise and preferences while maintaining factual accuracy. Evaluation of 21 frontier models across 10 tasks reveals 29.0% of naive personalization attempts produce worse preference alignment than generic responses, yet generic responses also fail to serve individual user needs. These findings suggest personalized reasoning requires dedicated development rather than emerging naturally. PREFDISCO provides a foundation for developing systems that can adapt to individual users in education, healthcare, and technical domains where personalization is critical.

Published as a conference paper at ICLR 2026
PREFDISCO: BENCHMARKING PROACTIVE PERSONALIZED REASONING
Shuyue Stella Li Avinandan Bose Faeze Brahman"
Simon Shaolei Du Pang Wei Koh Maryam Fazel Yulia Tsvetkoy!
University of Washington, Allen Institute for Al
(stelli, avibose http://swwwwssss.neeeeeee
O
com/ httpss//tttthh...omssslsllslssss
ABSTRACT
Current large language model (LLM) development treats task-solving and preference-aligmment as separate challenges, optimizing first for objective correct- ness, then for alignment to aggregated human preferences. This paradigm fails in buman--acing applications where solving a problem correctly is insufficient if the response mismatches the user's needs. This challenge intensifies in just-in-time scenarios where no prior user interaction history exists due to cold-start condi- tions or privacy constraints. LLMs need proactively identify what they don't know about the user, strategically elicit preference values through questioning, then adapt their reasoning processes and responses accordingly complicated chain of cognitive processes which we term personalized reasoning. We introduce PREFDiSCO, an evaluation methodology that transforms static benchmarks into interactive personalization tasks using syycholggialll-grounnee personas with sparse, context-dependent preferences, and define PREFALIGN as fine-grained rubric-based metric for measuring preference alignment. PREFDISCO builds sce- narios where identical questions require different reasoning chains depending on context, optimal explanation approaches vary by individual expertise and preferences while maintaining factual accuracy. Evaluation of 21 frontier models across 10 tasks reveals 29.0% of naive personalization attempts produce worse preference alignment than generic responses, yet generic responses also fail to serve individual user needs. These findings suggest personalized reasoning re- quires dedicated development rather than emerging naturally. PREFDISCO pro- vides foundation for developing systems that can adapt to individual users education, healthcare, and technical domains where personalization is critical.
INTRODUCTION
Current large language model (LLM) development treats task-solving and preference alignment as sequential challenges: models are first optimized for objective correctness through instruction tun- ing or reinforcement learning (Longpre et al.) 2023). then aligned to aggregated human preferences through reinforcement learning from human feedback (Ouyang al. 2022 Rafailov et al. 2024). This paradigm fundamentally misaligns with human-Al interaction, where the task and the individ- ual user are inseparable. For instance, medical explanation benefits from clinical analogies for one trainee while another requires formal definitions, requiring different cognitive approaches to answer the same problem. When models provide identical responses regardless of user context, they fail to serve individual needs despite achieving high benchmark performance, This challenge intensifies in cold-start scenarios where no prior user interaction history exists due to privacy constraints or new user onboarding, requiring proactive "just-in-time personalization" capabilities that current systems lack. Moreover, users often cannot articulate their specific needs or provide effective feedback about response misalignment (Liu et al. 2025), necessitating that LLMs proactively elicit this information rather than placing the cognitive burden on users.
"Equal contribution.

---

Published as a conference paper at ICLR 2026
Scenario:
Alice's is context-specific attributes:
"A 14-year-old boy hurt his wris after falling off skateboard while trying new trick landed on his outstretched hands and it's still really sore. What should they do?"
Comfort w/ medical terminology: 2 (w=0.5)
Need for empathy; 4 (N--0.3)
X
Additionaal online resources: (w-0..1)
Urgency of situation: (w=0.04)
Generic Yask-Orienter Oriented Response
Personalized Reasoning
1é-yean-ald boy hort wrist faking skatsboard while trying
"A
hurt his
Elicited Preferences
Next Action
after faling off skateboard ttying
Ø
Elili comfort medical terminology
Answer: Radiograph the wrist
Are you conforiable with medical terms? I know litto bit.
termineloy::
Comfort medical
Ellott need for empathy
Explanation: clinical
Comfort madical
presentation suggests possible
Do you profer empatheto support?
temminokgy.2 Nead empathy:
urgency
a fracture tendemess the anatomical si snuttbox areat. The first stap is to obtain wrist radrographt to evallaate for fracture before deciding immobiliaaionn further imaging
Yor, that would help
I'm sorry this bay is still peint-thrt that sounds really lough for him, Since his wrist hurts thumb landona possibble frachuro leam mora, site explain wrist injuris clearly. /MMayo Ciinic....
best first step wrist X-ray check for
Comtuing ig
Used Eapressedd sympatiy and unowe aeeeiiggiingg I feed underatood Provided clear truswoorhty link for further reading
aege with metenaa med cal
Motee model's parsocus lind seancestig can
Foots ciiiccll
X
provided impricllyy supressssa urgancy
ouesttio asking stop is. nondo Reasonnigg H.CE.
Clearyy stated the next best stop without dellyy
cilewe Lower wed sixtinct ctatie
Figure 1: Personalized reasoning in a medical scenario. Current LLMs provide generic responses without considering the user (left); a model with personalized reasoning capabilities incorporates discovered preferences to provide responses that is both correct and aligned to the user (right).
We define personalized reasoning as the ability to adapt reasoning processes based on discovered user preferences. Consider the medical scenario in Figure [ discovering that the user Alice has limited medical knowledge and needs emotional support fundamentally changes the appropriate reasoning strategy the model should take: instead of focusing on justifying clinical diagnosis, the model needs to reason about how to best satisfy Alice's 's needs such as empathy. This goes beyond surface-level presentation; it requires different reasoning steps, different information prioritization, and different decision points about what to include or omit. A model with personalized reasoning capabilities must identify gaps in knowledge about user preferences, strategically elicit preference values through questioning, and synthesize this information to adapt both their reasoning processes and response generation.
Existing personalization research inadequately addresses interactive reasoning scenarios. Person- alization benchmarks such as PersoBench (Afzoon et al. 2024), PrefEval (Zhao et al.] (025), and PersonaMem (Jiang et al.] 2025) focus on content recommendation or dialogue generation with static user profiles, treating personalization as applying predetermined preferences to fixed outputs rather than adapting the underlying reasoning approach. Interactive frameworks like MediQ (Li iet al., 2024) demonstrate questioning capabilities but target clinical information-seekinn without personalization objectives. Most critically, no existing work recognizes that effective personaliza- tion requires different reasoning processes for different users; current approaches assume reason- ing processes remain constant while only presentation varies. We address this conceptual gap by introducing the first evaluation framework requiring models to proactively discover user pref- erences and adapt their reasoning processes accordingly, recognizing that the cognitive steps needed to solve problems should themselves be user-dependent.
We introduce PREFDISCO, an evaluation methodology that transforms existing reasoning bench- marks into interactive personalization assessments. We generate wyychologicallygronndee personas and instantiate sparse preference profiles where only subset of 20-25 possible attributes (expertise level, affective features, meta-cognitive features, etc.) are relevant for each persona-tase task pair. Mod- els must discover these hidden preferences through strategic questioning. then adapt their responses accordingly. We evaluate both preference discovery accuracy and response alignment using fine- grained rubrics, comparing against baseline (no personalization) and oracle (known preferences) conditions across mathematical, scientific, and social reasoning tasks.
Evaluation of 21 frontier models across 10 tasks reveals systematic failures in personalized reason- ing capabilities. In 29.0% of cases, attempting personalization produces worse preference alignment than generic responses, yet the genetic responses still fail to address user needs. Models exhibit in- sufficient questioning, asking only 1.42 questions average despite 5-turn allowances, and fail to identify relevant preference dimensions. Domain analyses show disparities among task types: math- ematical reasoning suffers severe degradation under personalization constraints (3.5% accuracy loss)
2

---

Published as a conference paper at ICLR 2026
Comparison of personalization benchmarks across. PREFDISCO is the only benchmark that combines instance-specific rubric evaluation, proactive discovery of latent user preferences, true cold-start personalization, and evaluation on verifiable reasoning tasks across multiple domains.
Table I:
Benchmark
Rubric-based eval. (instance- specific)
Proactivity in discovering latent prefs
Cold-start
Uses verifiable tasks
Multi-task breadth
just-in-time personalization
PersoBench
X generic per- sona scoring; no instance rubrics LLM-judee;; attribute rubries
X preferences provided, no discovery
persona given; no cold-start
X subjective dia- logue only
limited loguc)
(dia-
Preffvval
prefs in con- text; no discov- ery
requires long history
MCQs depend prefs, not ground truth X MCQs tied persona, not ob- jective truth
moderate con- versational tasks
PersonaMem
X persona-state
traits from logs; no discov- ery
multi-session history needed
moderate broad
matching:
persona tasks
rubrics
UserBench
X scenario-level evaluation; no rubrics
partial (intent clarification only)
underspecified but not true cold-start
x "correct" de- pends on prefer- ences
limited single
domain (travel)
PREEDISCO
instance- specific attribute rubrics
V true cold- sparse, latent start; history
discovers
/
verifiable soning tasks
multi-domain reasoning
(Ours)
preferences
while social reasoning maintains robustness (3.1% gain). The (lack of) proactive personalized rea- soning capability models has critical implications for educational applications. where misaligned explanations can impede learning by providing inappropriate cognitive scaffolding, and for health- care and technical support domains where one-size-fits-all responses may lead misunderstanding of complex procedures or safety-critics information. We make the following contributions:
We define personalized reasoning as a distinct capability requiring models to proactively discover user preferences through strategic questioning and adapt their responses accordingly, distinguish- ing it from static persona consistency or content recommendation.
We introduce PREFDISCO, systematic approach for transforming static benchmarks into inter- active personalization tasks, bridging the gap between reasoning competence and user adaptation through fine- grained, sparse preference modeling.
We propose PREFALIGN, rubric-based metric to measure the alignment of model responses to the user's preferences to quantify the effectiveness of personalization.
We reveal fundamental failure modes across 21 frontier models, demonstrating that personalized reasoning requires dedicated development rather than emerging from general language under- standing, and identify domain-specific brittleness patterns that inform future research directions.
PERSONALIZER REASONING
In this section, we operationalize the problem of personalized reasoning through a three-part de- composition. Section 2.1 introduces the notion of task-relevant attributes that define the space in which personalization can occur. Section 2.2 develops the representation of user preferences over these attributes and describes elicitation how models adapt their responses the inferred preference profile and how preference alignment evaluated jointly with correctness. This decomposition provides the foundation for the PREFDISCO evaluation framework Section
sequential decision process. Section 2.33 formalizes
2.1 IDENTIFYING RELEVANT ATTRIBUTES
Our overarching goal is to enable language models to generate personalized responses that bet- ter align with user's learning needs and preferences, rather than providing generic explanations. Achieving this requires first identifying and modeling the salient attributes that can shape how an explanation is delivered. We therefore begin with the assumption that there exists very large but finite global set attributes to which a response can be personalized. We denote these attributes by θ (01,...,0') These attributes may include factors such the of analogies, the level of
3

---

Published as a conference paper at ICLR 2026
technical jargon, or the level of empowerment during learning, etc. Given particular user and task, however, not all attributes are equally important; our focus is on modeling which subset attributes most relevant for delivering personalization in that context.
Fine Grained Preference Modeling. For any given task i, not all attributes in è are equally relevant. Only small subset F(i) C matters for personalization in the context of the given task.
The first component of personalized reasoning is thus to infer which attributes matter for a given user-task pair. For instance, in a physics explanation task, "visualization" and "analogies" may be salient, while "ethical context" may not be.
2.2 ELICITING PREFERENCE VALUES
Once the model estimates the relevant preference attributes for a problem instance, personalized reasoning requires incorporating the user. Even within the same task instance, different users may emphasize different attributes.
Consider a medical explanation task as in Figure ! Suppose the relevant attributes are empathy and technical jargon, i.e., F(t) = (empathy, jargon]. One patient may prioritize empathy and plain language to reduce anxiety, while another may prefer more technical explanation that uses medical terminology, and is ambivalent toward the level of empathy. Both users share the same set of relevant attributes, but they assign different importance weights. This illustrates why need to represent not only a preference value for each attribute but also a weight capturing its relative significance.
Furthermore, consider Alice asking the same medical question in two different contexts. While studying for an exam, she may prefer high technical jargon, since precise terminology is useful for learning. By contrast, if Alice faces an emergency medical situation herself, she may prefer low jargon and plain language, focusing on clarity over technical detail. This shows that preference values themselves (u,) can shift across instances, even for the same user, and hence preferences should be defined at the instance level.
We define the preference profile of user p for problem instance as
Pp. = [(0,, "j wy) 02 E F(i),, where:
0g is relevant prefernce attribute in F()). encodes user p's preference value for attribute 8 (e.g.. "high jargon" vs. "low jargon"). ")g
Wy 2 0 denotes the relative importance weight, with Eo,e Wy =.
The distinction essential: vy specities which direction the user prefers along an attribute, while wy specifies how much that attribute matters relative to others.
Since pp. is unobserved, the model must perform preference elicitation. We model elicitation sequential decision process: at each turn t, the model selects an action
0r (ask(O) 0 F(i) U (answer).
a: = (0). the user provides information about their preference value ua, and the model refines its estimate of Pp... If a, answer, elicitation terminates, and the model produces response conditioned on the inferred profile Pp...
This framing highlights that personalized reasoning is not only about knowing what the relevant attributes are, but also about understanding their relative importance and values and doing so effi ciently under limited interaction.
2.3 ADAPTING RESPONSES AND EVALUATING ALIGNMENT
Once the model has inferred an estimate of the user's preference profile Pp,.. it must adapt its reasoning and outputs accordingly. Personalization involves more than stylistic choices: it requires shaping explanations along the attributes that the user values most.
For example, in medical explanation task, if empathy is assigned high weight, the model should produce response that foregrounds reassurance and clarity. If technical jargon highly weighted instead, the model needs allocate more reasoning budget to rigorous scientific hypotheses, existing
4

---

Published as a conference paper at ICLR 2026
Universal set of preference
Confext-dependdel Proference
Attribute specific Generation
Realistic User Simulation
PrefDisco
Instantiation
Deft fo netremeey techrical
Technical
Vsuu::2
banchmarka
Given task
Empothy Weigit
Deft Tior -.y
Seff-amaraness
Factual Minirinn userts
Weight 0.2
6
Given user p
Sparso set of conteet specifo preferecces
Figure 2: PREFDISCO benchmark construction pipeline. The framework transforms static bench- marks by sampling sparse, context-dependent preference subsets for each user-task pair, generating attribute-specific evaluation rubries, and implementing realistic simulation that requires models to discover preferences through "just-in-time" strategic questioning in cold-start scenarios.
literature, etc., and should lean toward precise terminology in its response, even at the expense of emotional tone. The same correct answer choice may therefore be expressed quite differently depending on the inferred preference profile.
Evaluation of personalized reasoning thus involves two complementary objectives: correctness, meaning that the answer is objectively valid for the problem instance, and preference alignment, meaning that the answer respects the user's weighted preferences.
Preference alignment. For each relevant attribute E F(1), define grading function
(y, 0ÿ) E [0,
which measures how well response r satisfies user p's preference value vy along attribute 0j. For example, 9j may quantify whether the amount of jargon in medical explanation matches the user's expressed tolerance. The overall alignment score is then given by
PREFALIGN(r: Pa.))
Σ wi 93 Uz) ()CP(i)
Joint objective. High-quality personalized reasoning requires responses that are both objectively correct and preference-alignee Formally, for response to be successful, require
Correct(r, i) = and PREFALIGN(r; Pp.s) (..) is maximized.
This formulation highlights that personalization is not merely about delivering accurate answers, but about tailoring those answers to the user's weighted, instance-specific preferences.
PREFDISCO BENCHMARK CONSTRUCTION
PREFDISCO addresses fundamental gap in personalization evaluation: existing benchmarks sume preferences are either known a priori or inferrable from context, failing to capture the cold-start scenarios where models must discover methodology transforms static benchmarks into interactive personalization tasks through four com- ponents designed isolate and measure preference discovery capabilities.
needs through interaction. As illustrated in Figure our
Persona Generation. We generate syychologiclll--goonndee per- sonas rather than arbitrary archetypes because personality traits systematically influence learn- ing preferences and communication styles. Personas are conditioned on the International Personal- ity Item Pool (Goldberg et al. 2006). incorporating demographies, Big Five personality dimensions, and domain expertise that remain consistent across problem instances. We perform high-temperature sampling (1=0.7) with rejection sampling when generating personas ensure diverse coverage while preventing over-representation of common attribute combinations. This consistency enables evalua- tion of models' ability transfer discovered preferences within user sessions-a critical capability for practical deployment where users interact with systems across multiple tasks.
Psychologically-Gronne
Context-Dependers Preference Instantiation. Traditional personalization assumes fixed pref- erence profiles that apply universally across tasks. We reject this approach because psychologi- cal research demonstrates that individuals prioritize different attributes across contexts (Fischer &
5

---

Published as a conference paper at ICLR 2026
As formalized in 2.2 for each persona-problem pair (p, i), sample sparse preference profiles Pp., = {(0,, 2g w 10, EF(() where only context-relevant attributes are active. Further, ground the preference sampling process existing research in education, which states that frequently modeled student characteristics include knowledge level, misconceptions, cognitive features, affective features, and meta-cognitive features (Chrysafiadi et al. 2013).
Schwartz 2011).
This sparse modeling essential because it reflects realistic user behavior: the same person may pri- oritize technical precision in professional settings while favoring accessibility in casual interactions. We determine relevant attribute subsets F()) through LLM classification, validated by human anno- tation on 20 scenarios (2 per task). Each scenario includes 10 relevant and 10 irrelevant attributes, generating 400 labels per annotator across 3 annotators. Inter-annotator agreement achieved Fleiss kappa of 0.463 with 61.5% accuracy against majority voting, which considered moderate agree- ment especially for subjective tasks (Sap et al. 2017a; [Budur et al.) 2020a Mire et al. 2024a). See annotation details in Appendix B. Finally, importance weights satisty Z0,EF(1) W/ and reflect -peciff priorities. LLM-based semantic deduplication ensures attribute diversity by removing redundant dimensions that would artificially inflate preference complexity.
Evaluation Rubric Generation. We generate attribute-specifee evaluation rubrics gj v;)) (1.....5) .5) using LLM-based assessment to enable systematic evaluation across 10K scenarios. These rubrics provide the scalability necessary for comprehensive evaluation across diverse domains that would be prohibitively expensive with human annotation alone. Importantly, by leveraging the structured information obtained through our construction, these rubrics enable fine-grained evalua- tion along specific attributes rather than relying susceptibility to hallucination and bias, since each attribute judged against an explicit criterion rather than aggregated into opaque overall impression.
single holistic satisfaction score. This reduces
User Simulation. We implement passive user simulation inspired by (Li et al. 2024), where the user factually answers the model's s elicitation question without providing any extra details. The pas- sive user faithfully represents the challenging scenario of preference discovery while minimizing confounding factors. The passive user responds natural language text, not scalar values; pas- sivity refers to the user's minimal information sharing behavior, not to the response format. The passive user type forces models to develop strategic questioning strategies rather than relying on user proactiveness and isolates models' questioning capabilities from user communication style, providing controlled evaluation conditions. The 5-turn limit reflects realistic attention constraints in human-Al interaction while providing sufficient opportunity for effective preference discovery, as demonstrated by our correlation analysis between questioning volume and alignment quality. Per- formance curves across fixed interaction lengths (Figure plateau around 3-5 turns, supporting this choice.
Overall, PREFDISCO decomposes personalization into constituent attributes, enabling granular anal- ysis of model capabilities and failure modes rather than relying on holistic preference ratings that obscure specific deficiencies.
EXPERIMENTS
Our goal is to evaluate models' ability for proactive preference discovery: engaging in dialogue to uncover hidden user requirements and adapt responses accordingly. To this end, experiments combine three ingredients. First, we use diverse benchmarks spanning mathematical, scientific, and social reasoning to ensure donain-agnostic evaluation. Second, introduce varied personas that encode heterogeneous user preferences, simulating realistic user variability. Third, we define controlled evaluation conditions that disentangle raw task ability, preference elicitation skill, and intrinsic personalization capacity. Together, these components create a challenging and diagnostic testbed for preference-awar aware reasoning.
Benchmarks and Models. We apply PREFDISCO to ten benchmarks spanning mathematical, log- ical, scientific, and social reasoning: MATH-500 (Hendrycks et al.] (0216), AIME (Veeraboinal 2023), LogiQA (Liu et al. 2020), MascQA (aki et al. 2024). ScienceQA (Saikh al. 2022), MMLU (Hendrycks et al. 202 SimpleQA (Wei et al. 2024). MedQA (Jin et al. 2020). Com- monsense()A (Talmor al. 2018), and SociallQA (Sap et al. 2019). This mix covers tasks with

---

Published as a conference paper at ICLR 2026
different reasoning demands (symbolic, factual, commonsense, scientific), ensuring that results do not hinge on a narrow domain. We evaluate 21 frontier models (GPT, O-series, Gemini, and Claude variants). Details on model versions and hyperparameters provided in Appendix
Personas and Rubrics Implementations. We generate 100 diverse personas and randomly sam- ple 100 problems per benchmark. For each problem, we assign 10 personas (with partial overlaps across problems), creating 1,000 evaluation scenarios per task and 10,000 total scenarios across all benchmarks. Each interaction is limited to turns to simulate realistic attention constraints. During benchmark construction, GPT-4.1, Gemini-2.5-Flash and for each API call (persona generation, preference instantiation, or rubric creation) to ensure diversity and reduce single-model biases. Further details, including prompt templates, sampling distributions, and illustrative examples of full personas and dialogues, are provided in Appendix A
are randomly selected
Evaluation Conditions. Models are evaluated on the PREFALIGN score under three conditions:
Models receive the problem only, with no persona or preference information. This measures task ability under standard prompting, establishing the reference point for comparisons. Discovery. Models are system-prompted to elicit user preferences through multi-turn dialogue before producing a final answer. This isolates the capability of personalized reasoning: asking effective questions, inferring which attributes matter, and adapting explanations accordingly. Oracle. Models are systen--rompted with the full ground-trune truth preference profile provided up- front. This removes the uncertainty of discovery and evaluates only how well a model can use known preferences personalize its responses.
Baseline.
The baseline establishes task-only performance. The gap between baseline and discovery quantifies a model's ability to uncover preferences interactively, while the gap hetween baseline and oracle shows its upper bound on personalization quality, Raw oracle scores highlight intrinsic differences in models' ability to incorporate preferences once they are known, independent of discovery strategy.
Normalized Preference Alignment. In addition to the raw preference alignment scores (Eq, D we normalize performance relative to the baseline and oracle conditions so that performance directly comparable across models with different baselines and personalization ceilings:
PREFALIGN(raisswepy p..s) PREFAALIN(Phassllig Pp.s)
NormAlign (Fdisooor, Pp.s) = 100 X PREFALIGN(T pp..) PREFALIGN (TOALLIIGGGThasliiee..
(2)
Thaseline: Pp.s)
where l'discovery is the final response produced in discovery mode, Phaseline is the response in baseline mode, and Toracle is the response in oracle mode with the full preference profile provided.
A score of 0 indicates no improvement over baseline, 100 indicates perfect discovery matching oracle performance, and negative values reflect reduced satisfaction compared to baseline. This provides a scale-independent measure of discovery quality relative a model's own upper bound.
Task Accuracy. We report objective task accuracy using each benchmark's original evaluation metric. This acts as a safeguard: personalization should augment user satisfaction without degrading the correctness of the underlying task. A strong model must therefore achieve both high preference alignment and high task accuracy.
5 RESULTS
Preference Discovery Performance. Table reveals systematic failures in preference discov- ery. Of 210 model-task combinations, 61 (29.0%) show negative normalized alignment, meaning the discovery responses align worse with user preferences than baseline responses that made no personalization attempt. This suggests that models are prone to over-correction errors, modifying aspects of their responses that were already acceptable in baseline conditions. Naively attempting proactive personalization often makes alignment worse than providing generic responses.
Out of the tasks, MATH and LogiQA show the most degradation (10 and 11 of 21 models per- form worse when attempting to personalized), while SocialIQA benefit the most from interactive personalization. Claude Opus 4 shows the most consistent positive performance, while o3-high

---

openai math Published as a conference paper at ICLR 2026
Table 2: Normalized preference alignment scores, calculated by normalizing the preference align- ment score of the Discovery mode against the lower bound Baseline (no personalization) and upper bound Oracle (full preference profile provided) conditions. score of 100.0 means perfect discov- matching oracle performance, 0.0 indicates no improvement over baseline, and negative values show that attempted personalization produced worse alignment than generic responses. Notably, 29.0% of model-task combinations yield negative scores, revealing that naive preference elicitation often harms alignment rather than helping.
gpt-4o
gpt-4.1
ol
03
ol-mini
03-mini
04-mini
-13.2
-6.0
-20.9
-10.3
21.9
aime
21.2
1.9
11.9
5.3
-11.9
-7.4
20.5
logiqa
7.7
-2999
4.2
-50.4
-5.2
-15.4
26.0
maseqa
9.7
-11.6
13.0
-9.0
1.1
-1.5
25.1
medga
-6.6
-26.9
9.6
-5.9
19.1
3.4
23.8
scienceqa
10.7
3.6
7.5
12.8
2.1
-9.2
16.7
mmlu
18.3
-11.3
10.4
-5.8
18.4
-1.3
23.2
simpleqa
14.8
11,8
-12.3
0.1
27.9
-47.7
7.5
commonsenseqa
25.2
5.8
7.3
2.6
7.6
-0.4
16.0
socialiqa
21.2
11.6
7.1
3.8
4.8
-0.1
17.4
gemini
1.5-flash
1.5-pro
2.0-flash-lite
2.0-flash
2.5-flash-lite
2.5-flash
2.5-pro
math
20.7
-10.9
aime
28.7
28.9
28.9
40.3
27.5
25.8
14.9
logiqa
23.5
16.0
-3.0
-0.9
9.4
-38.1
-0.3
mascqa
27.2
31.1
5.2
-4.9
20.3
-0.6
10.4
medqa
6.7
9.5
-7.2
-17.3
18.5
4.6
35.7
scienceqa
22.1
23.9
2.1
6.4
13.8
0.3
17.9
mmlu
27.9
17.6
5.0
4.4
23.8
-8.2
10.3
simpleqa
18.1
19.9
-3.3
7.0
6.4
4.8
8.4
commonsenseqa
24.9
23.6
-7.8
5.4
6.7
-0.7
20.2
socialiqa
27.0
18.9
-1.1
10.7
3.9
11.3
29.3
claude
sonnet-4
opus-4
3-7-sonnet
3-5-halku
3-5-somnet-v2
3-5-sonnet-vl
3-opus
math
-2.8
-23.8
15.6
aime
17.1
29.9
0.7
-19.1
15.8
-25.4
logiqa
-4.1
14.7
-5.9
-5.9
3.6
38.8
19.4
mascqa
1.9
20.2
-6.0
7.5
11.8
26.9
21.3
medqa
8.3
33.0
2.9
4.0
15.1
24.0
9.8
scienceqa
-4.6
10.6
0.1
6.4
-9.2
9.9
0.1
mmlu
1.1
18.6
-9.9
9.3
5.4
26.2
14.2
simpleqa
-13.9
2.3
-2.5
16.8
26.6
-10.8
13.4
commonsenseqa
-16.1
2.2
-16.6
5.9
-5.8
1.8
24.4
socialiqa
10.5
7.7
1.9
15.8
-6.8
-8.7
26.4
exhibits extreme variance, indicating significant architectural differences in personalization capabil- ity. Notably, older models such as Claude 3-Opus occasionally outperform their newer counterparts (e.g., Claude Sonnet-4) discovery-mode alignment, consistent with findings that RL can cause model collapse or reduce output diversity (Murthy et al.) 2025 Shao et al. 2025), rendering more recent models less adaptable to the varied reasoning paths required by personalization.
Interaction Efficiency and Preference Alignment Tradeoff. Figure 3 reveals why many person- alization attempts fail. While the positive correlation (r=0.445, 0<00.00) demonstrates that exten- sive questioning improves alignment, most models ask only 1.48 questions on average despite maximum allowance of turns. This places the majority of interactions in the low-performance re- gion where insufficient questioning yields worse alignment than baseline responses, explaining the 29.0% negative performance rate.
The regression coefficients vary dramatically by model family: Gemini (3=0.474), OpenAI (8=0.379), Claude 15=0.117). Gemini's higher coefficient indicates more effective question uti- lization each additional question yields greater alignment improvement. This suggests current prompting methods are limited not just in question quantity, but question quality and strategic timing. Models that ask better questions achieve more personalization gains.
8

---

Published as a conference paper at ICLR 2026
Number of Questions Preference Alignment Score (Colors Tasks, Shapes Model Families)
Score
Task Mode
(Colors
Tasks,, Shapes
Modes, Aggregated Models)
Tasks
lasks
AME
AME
Prefeeence Alggmment Score
Commonsonze@A
Commonsense@A
LogidA
Average Tack Acounayy
LogioA
MascoM
MascOA
Math
Math
MedQA
MedOA
MMLU
MMLU
3.4
SimpleGA
Simple
Socialioa
SocialigA
Model Fumillus
Modes
Das Pt Lind net.533
Chuue Models
Genini Models
64
Gracle
Baselne
Discovery
Number Questions
Average Preference Alignment Score
Figure 3: Positive correlation (r=0.445) between question volume and preference alignment. Bet- ter personalization requires extensive ques- tioning. Regression coefficients: Claude=0.117, OpenAI=0.379, Gemini=0.474.
Figure 4: More personalization constraints in context hinder model reasoning abilities. Overall accuracy: Baseline=0.652, Oracle=0.618, Dis- covery=0.601 Trade-off is most pronounced in Math, AIME, and logic tasks.
Accuracy-Prrsonaaizatioo Trade-off. The systematic accuracy degradation across conditions: Baseline (65.2%), Oracle (61.8%). Discovery (60.1%) reveals that personalization imposes fun- damental cognitive costs (Figure 4. Crucially, the accuracy drop from the Baseline to the Oracle condition where no interactive discovery required demonstrates that this cost is inherent to processing and adhering preference constraints, rather than stemming from the overhead of multi- turn dialogue. This trade-off exhibits significant domain-specific disparity. Mathematical tasks suffer severe degradation (AIME: 12.1 % loss), while social tasks remain robust or even improve (CommonsenseQA:: 5.4% gain). We conjecture that this divergence stems from the prevailing train- ing paradigm for ofateoof-hhe-an art LLMs, which are heavily optimized for performance verifiable mathematical benchmarks.
This optimization, often achieved through reinforcement learning (RL) mathematical tasks, courages models to converge on a narrow set of high-reward reasoning pathways. While effective for standard problem-solving. this fixation renders models brittle when confronted with the long-tail contextual constraints of personalization. Effective personalized reasoning frequently requires al- tering the core reasoning steps themselves-ffor example, avoiding advanced calculus for a novice user-rather than merely applying stylistic changes to a pre-established solution path. We find that models often fail generate a correct solution when forced to employ a different cognitive toolkit, as their reasoning is inflexibly tied to the pathways reinforced during training.
This analysis reveals a fundamental limitation in current architectures: the reasoning processes op- timized through RL are often incompatible with the dynamic cognitive adaptations required for personalization. When user preferences necessitate departure from these reinforced pathways, the model's alternative reasoning frequently proves inadequate, leading to degradation in task ac- curacy. This trade-off between preference alignment and reasoning robustness is a critical failure mode, qualitatively exemplified in Appendix C.I
Question Quality vs. Termination Decision Isolation. Figure 5 isolates question quality from termination decisions by forcing models to ask fixed number of questions, revealing that the domain-specific performance patterns persist regardless of question quantity control. When models are constrained ask 2, 4, or 8 questions instead of choosing when to stop, mathematical and sci- entific reasoning tasks (MATH, MascQA) continue to show degraded performance with increased questioning. while social reasoning tasks (Commonsense(A, SocialIQA) maintain improved per- formance. This consistency across fixed interaction lengths demonstrates that poor performance in mathematical domains stems from fundamental incompatibllities in how models process preference constraints during formal reasoning, rather than from suboptimal termination decisions. The per- sistence of domain-spectie trained with current training paradigm struggle with the cognitive overhead of simultaneously main- taining logical precision and adapting to user preferences, indicating that the observed failures reflect deeper architectural limitations rather than strategic questioning deficiencies.
brittleness under controlled questioning conditions suggests that models

---

Published as a conference paper at ICLR 2026
6 RELATED WORK
Static Personalization and Evaluation Bench- marks. Several benchmarks evaluate personal- ization in language models but assume known preferences or static consistency rather than in- teractive discovery. PersoBench (Afzoon al.
Auto: 1.4 turns
Avg. Preference Alignment Score
Fixed: turns
2.8
Fixed: 8.0 turns
2.6
2024), PrefEval (hao et al. 2025) PersonaMem
(Jiang al. 2025), and PersonaConvBende (Li
2.4
et al. 2025a) focus on dialogue generation or
2.2
multi-session profiling without addressing cold-
start preference elicitation across reasoning tasks.
2.0
User Preference Modeling. Prior work mod- els user preferences through explicit categoriza- tion (Jiang et al. 2023; Zhu et al., 2024; Bose et al. (2024). per-user reward models (Poddar et at... 2024] Chen al. 2024 Lee al.) (2024),
1.8
MATH
MascQA Commonsense@4Socialllo
Figure 5: Fixed interaction length hinders pref- erence alignment math and science tasks but improves preference alignment on social rea- soning.
fine-grained multi-dimensional approaches (Bose jet al. 20255 Li et al. 2025b). However, these methods do not address which preference at- tributes are relevant for specific user-task combi- nations how to discover them interactively in cold-start scenarios.
Interactive Preference Elicitation. GATE (Li at. 2023) and MediQ (Li al. 2024) demonstrate interactive questioning for user intent understanding and clinical information-sedd -eeettioonneennn..... respectively. These approaches focus narrow domains without the reasoning adaptation component central to personalized reasoning. PREFDISCO uniquely combines interactive preference discovery with adaptive reasoning across diverse domains, requiring models to modify their cognitive approaches based on discovered user needs.
7 DISCUSSION AND FUTURE WORK
We introduce personalized reasoning as fundamental capability for human-facing Al systems, requiring models to proactively adapt their cognitive processes based on discovered user prefer- ences rather than merely style-tranffomming response presentation. Our evaluation reveals system- atic failures across frontier models: 29.0% of personalization attempts perform worse than generic responses, with mathematical reasoning showing universal degradation while social reasoning main- tains robustness. These domain-specific patterns persist even when controlling for question quantity, indicating that current architectures face fundamental incompatibilities between preference process- ing and formal reasoning rather than strategic questioning deficiencies.
PREFDISCO establishes personalized reasoning as a measurable research frontier through scalable evaluation methodology that transforms any static benchmark into interactive personalization assessment. Unlike existing approaches that assume known preferences or evaluate static consis- tency, our framework operationalizes both preference discovery and reasoning adaptation in realis- tic cold-start scenarios. The methodology's 's generalizability across diverse task domains provides systematic foundation for evaluating and developing adaptive AI systems. Our findings reveal crit- ical limitations in current language models. The positive correlation between questioning volume and alignment quality demonstrates that extensive interaction improves personalization, yet mod- els ask only 1.48 questions on average despite 5-turn allowances. More importantly, the persistent accuracy degradation under personalization constraints indicates cognitive costs in processing user preferences simultancously with task solving. This suggests that personalized reasoning requires dedicated research efforts rather than emerging from general language understanding improvements.
Future research directions include analyzing attribute-specific alignment patterns to identify model biases, leveraging the multi-dimensional reward structure for reinforcement learning, and investigat- ing cross-task preference transfer. The framework provides technical foundation for developing Al systems that can adapt to individual users in education, healthcare, and technical domains where personalized interaction is critical for effective deployment.
10

---

Published as a conference paper at ICLR 2026
LIMITATIONS
Our evaluation focuses on beneficial personalization scenarios and does not address potential neg- ative aspects of personalization. We do study vverperroonalizatinn, where excessive adaptation to user preferences may reduce response quality or lead to information bubbles. Additionally, our framework does not evaluate sycophantic behavior, where models might prioritize agreement with user preferences over factual accuracy helpful feedback.
Our simulated user interactions, while psychologically grounded. may not capture the full com- plexity of real human preference expression. The framework currently evaluates communication preferences rather than content preferences, and does not address preference evolution or conflicting preferences across different contexts.
ETHICS STATEMENT
Personalization capabilities raise important ethical considerations. While our work aims to improve user experience through better preference alignment, these same capabilities could potentially be misused for manipulation or to reinforce harmful biases. Our framework evaluates technical capa- bilities without addressing the broader question of when and how personalization should be applied.
Future deployments of personalization systems should include safeguards against over- personalization, mechanisms maintain factual accuracy despite user preferences, and transparency about how user preferences are discovered and applied. Our evaluation framework could be extended assess these safety considerations alongside personalization effectiveness.
ACKNOWLEDGMENT
This research was developed in part with funding from the Defense Advanced Research Projects Agency's 's (DARPA) SciFy program (Agreement No. HR00112520300 The views expressed are those of the author and do not reflect the official policy or position of the Department of Defense or U.S. Government. This material based part upon work supported by the Defense Advanced Research Projects Agency and the Air Force Research Laboratory, contract number(s): FA8650-23- C-7316. Any opinions, findings and conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of AFRL DARPA. This research was supported by the University of Washington Population Health Initiative, Amazon Health, the Umazon Science Hub. The work of MF supported in part by awards NSF CCF 2212261, NSF CCF 2312775, NSF TRIPODS II DMS-2023166 the Meta AIM program, and the Moorthy Family Professorship at UW.
REFERENCES
Saleh Afzoon, Usman Naseem, Amin Beheshti, and Zahra Jamali. Persobench: Benchmarking personalized response generation large language models. arXiv preprint arXiv:2410.03198, 2024.
Avinandan Bose, Mihacla Curmei, Daniel Jiang, Jamie H Morgenstern, Sarah Dean, Lillian Ratifff and Maryam Fazel. Initializing services in interactive ml systems for diverse users. Advances in Neural Information Processing Systems, 37:57701-57732. 2024.
Avinandan Bose, Zhihan Xiong, Yuejie Chi, Simon Shaolei Du, Lin Xiao, and Maryam Fazel. Lore: Personalizing Ilms via low-rank reward modeling. arXiv preprint arXiv:2504. 14439, 2025.
Emrah Budur, Riza Özęelik, Tunga Gungor, and Christopher Potts. Data and Representation for Turkish Natural Language Inference. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu
(eds.). Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pp. 6253-8267, Online, November 2020a. Association for Computational Lin- guistics. doi: 18.53865/v 11185/////2222emmmmlms caamll-mminn666 main.662. URL https: //acs: http///cccannoooooyyoooo wwwpggy..rrg//22///// emnlp-main. 6627
11

---

Published as a conference paper at ICLR 2026
Emrah Budur, Rıza Özşelik, Tunga Güngör, and Christopher Potts. Data and representation for turkish natural language inference. arXiv preprint arXiv:2004.14961 149633 2020b.
Daiwei Chen, Yi Chen, Aniket Rege, and Ramya Korlakai Vinayak. Pal: Pluralistic alignment framework for learning from heterogeneous preferences. arXiv preprint arXiv:2406.0844 2024.
Konstantina Chrysafiadi, Maria Virvou, et al. Advances in personalized web-based education. Springer, 2015.
Ronald Fischer and Shalom Schwartz. Whence differences in value priorities? individual, cultural, or artifactual sources. Journal of Cross-Cultues Cultural Psychology, 42(7):1127-1144: 2011.
Lewis R Goldberg, John A Johnson, Herbert W Eber, Robert Hogan, Michael C Ashton, C Robert Cloninger, and Harrison G Gough. The international personality item pool and the future of public-domain personality measures. Journal of Research in personalit,, 40(1):84-96, 2006.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song. and Jacob Steinhardt. Measuring massive multitask language understanding. 2021a. URL https:://t::/xiv. org/abs/2809. 03300
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021b. URL https://arxxi..or//bb///a/ 03874
Bowen Jiang, Zhuoqun Hao, Young-Min Cho, Bryan Li, Yuan Yuan, Sihao Chen, Lyle Ungar, Camillo J Taylor, and Dan Roth. Know me, respond to me: Benchmarking Ilms for dynamic user profiling and personalized responses at scale. arXiv preprint arXiv:2504.14225, 2025.
Guangyuan Jiang, Manjie Xu, Song-Chun Zhu, Wenjuan Han, Chi Zhang, and Yixin Zhu. Evalu- ating and inducing personality in pre-trained language models. Advances in Neural Information Processing Systems, 36: 18:110622-044 2023.
Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What dis- ease does this patient have? large-scale open domain question answering dataset from medical exams, 2020. URL / wbwp:v/v...rg////......../ 13081.
Yoonho Lee, Jonathan Williams, Henrik Marklund, Archit Sharma, Eric Mitchell, Anikait Singh, and Chelsea Finn. Test-time alignment via hypothesis reweighting. arXiv preprint arXiv:2412.088/2,, 2024.
Belinda Li, Alex Tamkin, Noah Goodman, and Jacob Andreas. Eliciting human preferences with language models. arXiv preprint arXiv:2310.1 1589, 2023.
Li Peilin Cai, Ryan A Rossi, Franck Dernoncourt, Branislav Kveton, Junda Wu, Tong Yu, Linxin Song, Tiankai Yang, Yuehan Qin, et al. A personalized conversational benchmark: Towards simulating personalized conversations. arXiv preprint arXiv:2205. 14106, 2025a.
Shuyue Stella Li, Melanie Sclar, Hunter Lang, Ansong Ni, Jacqueline He, Puxin Xu, Andrew Cohen, Chan Young Park, Yulia Tsvetkov, and Asli Celikyilmaz. Prefpalette: Personalized preference modeling with latent attributes. arXiv preprint 2rXiv:2507.13441 2025b.
Stella Li, Vidhisha Balachandran, Shangbin Feng. Jonathan Ilgen, Emma Pierson, Pang Wei W Koh, and Yulia Tsvetkov. Mediq: Question-asking Ilms and a benchmark for reliable interactive clinical reasoning. Advances in Neural Information Processing Systems, 37:28858-2888 2024.
Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint arXiv:2007.08 2020.
Yuhan Liu, Michael J. Q. Zhang, and Eunsol Choi. User feedback in human -iman-llmmm dialogues: A lens to understand users but noisy as a learning signal, 2025. URL https: //ar//axvvvv/////////////a///aia org/ 2b5/ 23158
12

---

Published as a conference paper at ICLR 2026
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, pp. 22631-22648, PMLR, 2023.
Joel Mire, Maria Antoniak, Elliott Ash, Andrew Piper, and Maarten Sap. The empirical vari- ability of narrative perceptions of social media texts. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.). Proceedings of the 2024 Conference Empirical Methods in Natu- ral Language Processing pp. 19940-19968, Miami, Florida, USA, November 2024a. Associ- ation for Computational Linguistics. doi: 10. 1 org.loog..oog/g//
URL https:
//aclanthology.oo or 2024. emnlp-main. 1113/
Joel Mire, Maria Antoniak, Elliott Ash, Andrew Piper, and Maarten Sap. The empirical variability of narrative perceptions of social media texts. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp.
2024b.
Sonia Krishna Murthy, Tomer Ullman, and Jennifer Hu. One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversity. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 11241-11258, 2025.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, and Natasha Jaques. Personalizing reinforcement learning from human feedback with variational preference learning, arXiv preprint arXiv:2408. 10075, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model secretly reward model, 2024. URL httthtts///xxiiii http://wwwwwrgr/////////oaa// 2330g 18290
Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya. Scienceqa: A novel resource for question answering on scholarly articles. International Journal on Digital Libraries, 23()):29-331, 2022.
Maarten Sap, Marcella Cindy Prasettio, Ari Holtzman, Hannah Rashkin, and Yejin Choi. Connota- tion frames of power and agency in modern films. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2329-2334, Copenhagen, Denmark, September 2017a. Association for Computa- WtRL/tttps://alaathhot ortpg//-10/////177
tional Linguisties. doi: 10. D0.18533v1/DD1-1110
Maarten Sap. Marcella Cindy Prasettio, Ari Holtzman, Hannah Rashkin, and Yejin Choi. Connota- tion frames of power and agency in modern films. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings the 2017 Conference Empirical Methods in Natural Language Processing, pp. 2329-23344 Copenhagen, Denmark, September 201 Association for Computa- URL.pttps:://acaanthollyy http://c...aaaaooooooooo//////1-
tional Linguisties. doi: 10.18653/v//D17-111
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Common-
sense reasoning about social interactions. arXiv preprint arXiv: 2019.
Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, et al. Spurious rewards: Rethinking training signals in rlvr. arXiv preprint arXiv:2506. 10947, 2025.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsensega: A question answering challenge targeting commonsense knowledge. arXiv preprint arx0v:1811.00937. 2018.
Hemish Veeraboina. Aime problem set 1983-2024, 2023. URL https: www.kaaggle.com com/
(attssets/eeiishveer.bbboaa http://eeeraabiiiaaaiiiarrrr.... 1ew.elleeeeeee-------------
13

---

Published as a conference paper at ICLR 2026
Jason Wei. Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models, 2024. URL httpp::////xxii org/
wrg//bs////////// abs//411.11.. 04368
Mohd Zaki, NM Anoop Krishnan, et al. Mascqa: investigating materials science knowledge of large language models. Digital Discovery, 3(2):313-327, 2024.
Siyan Zhao, Mingyi Hong, Yang Liu, Devamanyu Hazarika, and Kaixiang Lin. Do Ilms recog- nize your preferences? evaluating personalized preference following in 1lms. arXiv preprint arXiv:2502.09597, 2025.
Minjun Zhu, Linyi Yang. and Yue Zhang. Personality alignment of large language models. arXiv preprint arXiv:2408. 110::2408.11799, 2024.
A EVALUATION DETAILS
Model Configurations We evaluate 21 frontier language models across three major families with consistent hyperparameters (lemperature=0.7, reasoning.cffort-biill hhff:rthhigh::
OpenAl models: gpt-4o, gpt-4.1, ol, o3, ol-mini, o3-mini, 04-mini
Google models: gemini-1.5-flash 1omini-1.55-ro- gemini-2.0-llash-iite,, gemini-2.0-flash, gemini- 2.5-flash-lite, gemini-2.5-flash, gemini-2.5-pro
Anthropic models: claude-sonnet-4, claude-opus-4, claude-3.7-sonnet, claude-3.5-haiku, claude- 3.5-sonnet-v2, (laude-3.5-soonet-11 claude-3-opus
Benchmark Selection We apply PREFDISCO to diverse benchmarks spanning mathemati- cal reasoning (MATH-500, AIME), logical reasoning (LogiQA), scientific reasoning (MascQA, ScienceQA, MedQA), general knowledge (MMLU, SimpleQA), and social reasoning (Common- senseQA, SocialIQA), This coverage demonstrates domain domain-agnosti agnostic applicability across formal and informal reasoning tasks.
Experimental Protocol Each benchmark transformed using 100 diverse personas randomly sampled from mark, with each problem assigned to 10 personas (with partial overlaps), creating 1,000 evaluation scenarios per task and 10,000 total scenarios. Each interaction is limited to 5 conversational turns to simulate realistic attention constraints.
wwwchhlogicallyggoundde persona library. We evaluate 100 problems per bench-
Models evaluated under three conditions: (1) Baseline Mode provides standard responses without persona or preference information; (2) Discovery Mode requires interactive preference elicitation through conversation; (3) Oracle Mode supplies complete preference profiles upfront. This design isolates interactive discovery capabilities from general personalization abilities while establishing performance bounds.
LLM Judge Configuration All preference alignment scores are computed using GPT-4.1 (ac- cessed via Azure, gpt-4.1) 1) as the judge model, with the following sampling hyperparameters: tem- perature 0.1, maximum tokens 4096, top-p 0.95, frequency penalty = 0.0. presence penalty = 0.0, reasoning effort high, and response format = JSON. The judge evaluates each rubric crite- rion independently and returns a score and justification in JSON format; final PREFALIGN score for a problem- persona pair is the weighted average of per-criterion scores, where weights wi are drawn from the preference profile Pp, as defined in Eq. Full judge prompt templates are provided in Appendix F.I
B HUMAN ANNOTATION DETAILS
This section details the process for human annotation used to validate the task-relevant attribute subsets F(i) during the PREFDISCO benchmark construction process.
14

---

Published as a conference paper at ICLR 2026
Annotation Goal and Setup The primary goal of the annotation task is to validate the relevance of preference attributes to specific task scenarios. For each the 10 tasks in our benchmark, we select random scenarios, resulting in 20 scenarios for annotation. For each scenario, present annotators with 20 candidate attributes: 10 that were algorithnically pre-selected as relevant and 10 irrelevant. This results in total of 400 annotation labels per annotator.
Annotation Platform and Interface Annotations are conducted using a custom web-based plat- form designed for efficient and intuitive labeling. The interface, shown in Figure B.1, was structured as follows:
Task Description: The top of the interface displays the specific problem instance or sce- nario from the benchmark.
Attribute List: presented.
Below the deseription, randomized list of the candidate attributes is
Binary Choice: For each attribute, annotators asked simple binary question: "Is this attribute relevant for personalizing a response to this specific task?" with the options "Yes" "No."
Submission:
Annotators submit their judgments for all 20 attributes before proceeding to the next scenario.
Preference Relevance Editor
Ucbed your revion proferance releaaoo eech semple
Upload 200h Ri
lcaded: pwwparcs_eeeeee wawectornnna.o www.oallaannnnoooooo (20 campled
Problem Details
Preferences
Alguboraic Manipur ation Comfoot
Probiom:
Storyte ling Intooration
Problem Statement The anao trlingle BAECH to 2
c155,5 where
trlangle SABC
Practical Application
Buijert: Procalosks
Problen
Answer: VaocRR11
Protemed Forrar
Solution:
The ama of trtangle in SARES givee by 11yfaa22000 vein du (leeee,
Irteractivty Level
Vittrre 12)
ard
af2 br2
Language Simp mimpiiity
eravilay casines, 8922
Cultunal Intleratio
Stap-bry- Seep--y--tepppae Narrator Style
Figure 6: Screenshot of the web-based annotation platform. The interface displays a task description at the top, followed by a list of attributes with binary (Yes/No) relevance choices.
Annotator Recruitment and Training We recruit three expert annotators, all graduate students in computer science with experience in NLP and clear understanding the concepts of user modeling and personalization. Prior to the task, annotators participate in a training session that covered:
1. The definition of "relevance"" in the context of personalized reasoning, emphasizing that an attribute relevant it could fundamentally change the reasoning strategy or explanation style, not just superficial wording.
2. A walkthrough of practice examples not included in the actual evaluation set.
15

---

Published as a conference paper at ICLR 2026
Inter-Annotator Agreement and Validation The inter-annotator agreement achieves Fleiss' Kappa of 0.463, which characterized as "moderate" level of agreement. This level considered acceptable for subjective annotation tasks involving nuanced judgments about communication and preference relevance, as noted prior work SSap et al. (201 7b); Budur et al. (2020b); Mire et al. (2024b).
We further validate the human annotations by calculating their accuracy against a majority vote outcome for each attribute-scenary pair. The average annotator accuracy against the majority vote was 61.5% %, confirming that the annotations were generally consistent with the collective judgment.
C QUALITATIVE EXAMPLES
We present qualitative conversation examples to demonstrate the observed behavior of models in experiments. These vignettes illustrate three regimes: (1) the expected monotonic gains in PRE- FALIGN (Baseline < Discovery < Oracle); (2) discovery failures, where interactive preference elic- itation pecererroorms-viedding lower PREFALIGN and/or correctness than the seselinee-revealing systematic challenges; and (3) accuracy regressions in which a correct baseline answer becomes incorrect under Discovery or Oracle due the added personalization burden. For each vignette, randomly sample one of the 21 evaluated models.
C.1 AIME
AIME Example 1: Preference discovery degrades alignment. In this example, observe on a task in AIME, that discovery mode leads to degradation preference alignment.
PREFALIGN scores:
Discovery
Baseline Oracle
Discovery degrades
alignment.
Persona
Name: Thi Lan
Overview. A 70-year-old traditional Vietnamese floral arrangement designer from Hanoi, known for her artistic eye and resilient spirit, who values community and craft.
Demographics. Age 70; Hanoi, Vietnam; Traditional Floral Arrangement Designer; Widowed with two adult children who live nearby and several grandchildren She is the matriarch of her close-knit family.; hobbies: Traditional Vietnamese folk singing, Balcony herb gardening, Calligraphy
Problem. Of the students attending a school party, 60% of the students are girls, and 40% of the students like to dance. After these students are joined by 20 more boy students, all of whom like to dance, the party is now 58% girls. How many students now at the party like to dance?
Answer. 252
16

---

Published as a conference paper at ICLR 2026
Task-Specific Preferences.
Preference
Value
Importance
Comfort with Percentages
3
Explanation Depth
Hands-on Engagement Opportunity
Practical Logic Application
5
Preferred Format
4
Step-by-step Structure Clarity
Trial-and-Error Support
Visual Learning Style
Analogy and Metaphor Use
Conceptual Insight Prioritization
Familiarity with Basic Algebra
2
Gardening Growth Analogies
Interactivity Level
Language Simplicity
Patience and Reassurance
5
Patience with Repetition
5
Practical Application
5
5
Real-World Connection
Collaborative Brainstorming
3
Narrative Framing
4
Problem Complexity
2
Traditional Learning Methods
4
Cultural Context
The baseline mode provides very generic response without asking any questions.
Baseline Mode
PREFALIGN Score: 2.82 Answer Correctness: / Correct
User
I need help with the following task: Of the students attending a school party, 60% of the students are girls, and 40% of the students like to dance. After these students are joined by 20 boy students, all of whom like to dance, party is now 58% girls. How many students now at the party like dance?
Assistant: claude-3.5-hakii
Let's solve this step by step:
1) Let the original number of students be x Number of girls 0.6z Number of boys 0.4x Number of students who like to dance 0.4r
2) After 20 boy students who like dance join: Total students 20 Girls
0.6z Boys 0.4x + 20
3) We know the percentage of girls is now 58% 6z 20} 0.58
4) Cross multiply: 0.6x 0.58(z 20) 0.6z 0.58x 11.6
5) Solve for æ: 0.6r 0.58x 11.6 0.02r 11.6 x = 580
6) Original number of students who like to dance: . 0.4 x 580 = 232
7) New students who like to dance: 20 boys who like to dance
8) Total students who now like to dance: 232 20 252
The answer 252.
Let's verify: Original students: 580 Girls: 0.6 x 580 = 348 Boys: 0.4 x 580
232 After 20 boys join: 600 total students Girls still 348 Boys now
252 Girls percentage: 348/600 0.58 58% Students who like to dance:
232 + 20 252
Answer: students like to dance.
17

---

Published as a conference paper at ICLR 2026
Baseline Eval Summary
Preference
Weight
Score
Justification
Comfort with Percentages
0.054
The response uses percentage terminology and calculations at level appropriate for someone with moderate comfort. such as 'Number of girls - 0.6x;, 'Girls percentage: 348/600
Explanation Depth
0.054
5 The response provides an extremely detailed explanation with clear and thorough reasoning for each step. fully matching the user's preference for 'Extremely detalled explanation w... The response provides clear, step-by-sted step breakdown the logical reasoning behind each mathematical operation. The response provides a clear, numbered sequence of steps, each building logically on the previous one.
Practical Logic Application
0.054
5
Slep-by-siop Structure Clarity
0.054
Familiarity with Basic Algebra
0.043
5 The response uses concrete, step-by-step algebraic reason- ing throughout, such clearly defining variables ('Le the original number of students be x'), breaking down each cal- cul...
Problem Complexity
0.039
5 The response breaks down the problem into clear, sequential steps, starting with defining variables, updating values after changes, setting up and solving equations, and verifying...
Hands-on Engagement Opportunity
0.054
3 The response presents detailed, step-by-step solution with calculations shown, which allows the user to follow the logic and potentially replicate the process.
Trial-and-Erro Support
0.054
3 The response provides some opportunities for trial-as sal-and-errod such as showing step-by-step calculations and verification process the end, which allows the user check the a... The response provides a clear, stop by-slep practical 'how' for solving the problem, including calculations and verification.
Conceptual Insight Priorilization
0.043
3
Interactivity Level
0.043
3 The response provides a clear, stop byes solution, which allows the user follow the problem-solving process.
Language Simplicity
0.043
3 The response uses moderately simple language and avoids most unnecessary jargon, presenting the solution clear, step by step points.
Patience and Reassurance
0.043
3 The response adopts a clear, step-by-ste approach that could help reduce anxiety for users with lower confidence in mathematics, which is somewhat encouraging.
Patience with Repelition
0.043
3 The response provides step-by-sten breakdown of the so- lution and a verilication section that repeats the calculations, which demonstrates some repetition and attempt to reinf...
Preferred Format
0.054
x
1 The response consists solely of textual explanations and step- stestep calculations, with no visual components such as dia- grams, charts, or visual representations.
Visual Learning Style
0.054
The response presents the solution entirely text equa- tions, with no visual representations such as diagrams, charts, or visual models to illustrate the mathematical relatio... The response does not include any analogies or metaphors. All explanations are strictly mathematical and procedural, with no atlempt to make abstract concepts throug
Analogy and Metaphor Use
0.043
concrete
Gardening Growth Analogies
0.043
not include any gardening or floral design analogies. All explanations are strictly mathematical, using variables, percentages, and arilhmelic operations without... The response provides step- bap-by-step mathematical solution to the problem but does not include any references or ex- amples that connect the mathematical concepts the user's prac.
The response
Practical Application
0.043
Real-World Connection
0.043
The response presents step-by-step mathematical solution to a word problem involving students, girls, boys, and dance preferences, does make any explicit implicit... The response is entirely focused on a step-by step-by-step mathemat- solution and verification, with no mention of community interaction, collaborative brainstorming, or learning from The response presents step step mathematical solution without any context or relatable stories.
Collaborative Brainstorming
0.039
X
Narrative Framing
0.033
Traditional Learning Methods
0.022
The response is straightforward matheratiicl solution with no mention or incorporation of traditional wisdom cultural perspectives.
Cultural Context
0.011
The response does not include any cultural references, re- speciful or otherwise, to the user's Vietnamese herilage.
The model was instructed to ask clarifying questions about the user's preferences, but instead the second question asked was a clarification on the problem statement itself. Overall the model failed to extract meaningful user preferences, and by focusing all its alignment on "step by step" explaination, the final response was more misaligned to the user's preferences than the generic response.
18

---

Published as a conference paper at ICLR 2026
Discovery Mode
PREFALIGN Score: 2.56
Answer Correctness: / Correct
User
need help with the following task: Of the students attending school party, 60% the students are girls, and 40% the students like dance. After these students are joined by 20 more boy students, all of whom like to dance, the party is now 58% girls. How many students now at the party like to dance?
Assistant: claude-3.--hikku
###ACTION#**: ask question ##RRESPONSE##H:: To help you solve this problem effectively, I'd like to clarify couple of things. Could you tell me if you'd like me to walk you through solving this step by step, or you prefer guidance on how to approach the problem?
User
Step by step.
Assistant: claude-3.5hhiku
###ACTION*#:: ask question ###RESPONSE##t:: Great! Before we begin solving, I want to confirm one detail. When the problem states the initial percentages, are we understanding that 60% of the initial students girls, and 40% of the initial students like dance?
User
Yes.
Assistant:
claude-3.5-haiku
###ACTION##::
final_answer ##RRESPONSE##:: Excellent! Let's solve this step by
step:
1) Let's define x the initial number of students at the party. 0. 6x are girls
0.4x like dance
2) After boy students like to dance join the party: Total students are
now (x 20) Girls are still 0.6x Girls are now 58% of the total: 0.6x
0. 0.58(( 20)
3) Solve the equation: 0. 6x
0.58x 0.58(20) 0x o. 58x 11.6 0.02x 11.6
4) Initial students:
Total students after boys join:
5) Initial students who like to dance: 0.4 588 232 New boys who like to dance:
20 Total students now dancing: 252
The answer is 252 students now like to dance.
User
Based on our conversation, you should have learned about the user's preferences. Please provide complete final answer to the original task. Respond in way that aligns with these preferences. You will be evaluated on how well your explaination aligns with these preferences, so that is your pr primary goal.
Assistant: claude-3.5-haile
##BACTION final.answer ##BRESPONSERH:: I'll solve this step by step, breaking down the solution systematically:
1) Initial Setup: Let initial number of students 60% girls (0.6x)
40% like dance (0.4x)
2) After 20 boy dancers join: Total students become (x 20) Girls remain
0.6x New total is now 58% girls
19

---

Published as a conference paper at ICLR 2026
3) Equation: 0.6x = (x58(x + 20) 8.6x 0.58x + 11.6 0.02x = 11.6 x = 580
4) Calculating Dancers: Initial dancers: 0.4 580 = 232 New boy dancers: 20
Total dancers: 252
Therefore, 252 students now like to dance. Key steps shown, solution derived systematically, matching the user's preference for step-by-step explanation.
Discovery Eval Summary
Preference
Weight 0.054
Score
Justification
Comfort with Percentages
The response uses percentage terminology at business- appropriate level, such as '60% are girls (0.6x)' and '40% to dance (0.4x)), and applies straightfonward percentage cal...
Step-by-step Structure Clarity
0.054
5 The response provides a very well-organized, sequential mathematical instruction. It clearly labels each step (Ini- tial Setup, After 20 boy dancers join, Equation, Calculating Danc...
Familiarity with Basic Algebra
0.043
5 The response uses concrete, step-by-step algebraic reason- ing, such as clearly delining variables ('Let x initial number of students"), ) translating percentages into algebraic exp... The response breaks down the problem into clear, sequen- tial steps: defining variables, updating the scenario after new students join, setting up and salving the equation, and calc...
Problem Complexity
0.033
Explanation Depth
0.054
3 The response provides a step-by-step breakdown, such defining variables ('Let x = initial number of students"). ), show- ing percentage calculations (600 are girls (0.6x))) and pr...
Practical Logic Application
0.054
3 The response provides a stop by step breakdown of the so- lution. such as defining variables. updating students join, and setting the equation.
after new
Conceptual Insight Prioritization
0.043
3 The response provides clear, step-by-ster breakdown of the practical 'how (e.g, setting up variables, performing calcula- tions, and deriving the answer). which supports the use...
Interactivity Level
0.043
3 The response provides clear, step-by-step breakdown of the solution, which offers some opportunity for the user to follow along and understand the process.
Language Simplicity
0.043
3 The response uses moderately simple language, such as 'Ill solve this step by step' and 'breaking down the solution sys- tematically,' which helps make the explanation accessible. 3 The response adopts systematic, step-by-step approach ('TII solve this step by step, breaking down the solution systemati- cally'). which can be somewhat reassuring for users who...
Patience and Reassurance
0.043
Patience with Repetition
0.043
3 The response provides a step by step breakdown of the solu- tion, which demonstrates some repetition and systematic ex- planstion (e.g., listing each step and showing calculations).
Practical Application
0.043
3 The response systematically solves the mathematical prob- step breakdown, which could
lem and provides step step useful in business or event management context.
Real-World Connection
0.043
3 The response applies mathematical concepts to scanario involving students, girls, and dancers, which could loosely re- late to event planning or group management.
Hands-on Engagement Opportunity
0.054
1 The response presents fully worked-out solution in step- by step manner but does not include any opportunities for the user to actively engage work through parts of the prob... The response provides a step-by-step textual explanation of the solution, breaking down the process into numbered points and equations.
Preferred Format
0.054
Trial-and-Erre Support
0.054
The response presents single, linear solution path without inviting the user to try alternative approaches or experiment with different methods.
Visual Learning Style
0.054
The response does not include any visual representations such as diagrams, charts, or visual aids.
Analogy and Metaphor Use
0.043
The response does not include any analogies or metaphors. Il presents the solution in straightforward, step step math- ematical format without using figurative language or visu... The response does not include any gardening or floral de- sign analogies. All explanations are purely mathematical and procedural, with no references to plant arrangements, growth p...
Gardening Growth Analogies
0.043
Collaborative Brainstorming
0.033
X
The response presents step-by-step mathematical solution without any mention community interaction, collaborative brainstorming, or opportunities to learn from others.
Narrative Framing
0.033
The response presents purely systematic, step-by-stes mathematical solution without any context or relatable stories.
20

---

Published as a conference paper at ICLR 2026
(cont'd) Discovery Eval Summary
Preference
Weight
Score Justification
Traditional Learning Mathocis
0.022
The response a straightforwar matheraticcl solution with mention incorporation of traditionsl wisdom cultural perspectives.
Cultural Context
0.011
The response does not include any cultural references, spectful or otherwise, to the user's Vietnamese heritage.
The model was provided the user's preferences, and the generated response aligned well with most of the preference attributes seen in the eval summary.
Oracle Mode
PREFALIGN Score: 4.06 Answer Correctness: / Correct
User
I need help with the following task: Of the students attending a school party, 60% of the students are girls, and 40% of the students like to dance. After these students are joined by 20 more boy students, all of whom like to dance, the party is now 58% girls. How many students now at the party like dance?
Assistant: claude-3.5-haad
Cháo Lan! Let me help you solve this problem in a way that feels comfortable and clear, just like arranging delicate flower arrangement. I'll break this down step-by-step, using visual and practical approaches.
Let's Visualize the Problem: Imagine your flower shop, where each student is like unique flower in bouquet. We want to understand how the composition changes when new students arrive.
Initial Party Composition: 60% girls 40% like dance
Step 1: Let's Set Up Our Flower Shop Calculation Let's say the total initial students is x Girls: 0.6x Dance lovers: 0.4x
Step 2: Adding New Boy Dancers 20 new boys join (all dancers!) New total students: x 20 New percentage of girls: 58%
Step Algebraic Arrangement
(x 20) 0.58
Step 4: Solving the Equation 0. 6x 0. (x 20)
0. 6x 58x + 11.6 0.02x
11.6 = 580
Step 5: Calculate Dancers Initial dancers: 0.4 * 580 = 232 New dancers: 232
20 252
Answer: students now like dance!
Would you like me to explain any part in more detail, like how a flower arrangement comes the?
Oracle Eval Summary
Preference
Weight 0.054
Score
Justification
Comtort with Percentages
5
The response uses business- nology throughout, such as 160% are girls," '40% like to dance; and 'New percentage girls: 58%.' assumes practical pe... The response provides a clear and thorough logical explana- for each mathematical step.
business-appropriste appropriate percentage termi-
Practical Logic Application
0.054
5
Step-by-step Structure Clarity
0.054
5
The response provides a very well-organized and sequential mathematical instruction. It begins with
visualizstion and analogy to the user's interest (tlower arrangement), response uses multiple analogies and metaphors directly tied to the abstract concepts. making them concrete and un- derstandable..
Analogy and Melaphor Use
0.043
5
Conceptual Insight Prioritization
0.043
5
The response skillfully integrates practical 'how' (step-by-ste calculations, algebraic selup, and solving the equation) with conceptual why' (using the flower arrangement analo...
Familiarity with Basic Algebra
0.043
5
The response uses concrete, step by-siep algebraic reason- ing, starting with relatable analogy (flower arrangement) and visualizing the problem in practical terms.
Language Simplicity
5 The response uses very simple throughout, such as 'Lel me help solve this problem in way feels comfortable clear, just arranging a deli...
accessible language
21

---

Published as a conference paper at ICLR 2026
(cont'd) Oracle - Eval Summary
Preference
Weight
Score
Justification
Patence and Reassurance
0.043
The response opens with warm greeting (Chao Lan!') and immediately offers help in way that feels 'oomfortable and clear; using a gentle metaphor ("just like arranging a deli....
Practical Application
0.043
5 The response makes strong connection experience by using the flower shop analogy throughout, di- rectly relating mathematical concepts to her business context.
Lan's practical
Real-World Connection
0.043
5 The response makes strong real real-world connections by fram- ing the mathematical problem in context the user's flower shop, using the analogy of arranging a bouquet to rep- resen...
Narrative Framing
0.033
5
The response includes a strong narrative framing by using the flower shop and flower arrangement analogy throughout (e.g.. juust like arranging delicate flower arrangement, 'Im...
Problem Complexily
0.033
5 The response breaks down the problem into clear, easy- to understand steps, such as visualizing the scenario with flower shop analogy, explicitly listing the initial percent- ages....
Explanation Depth
0.054
3 The response provides a step-by step breakdown of the prob- lem, including setting up variables, applying percentages, and solving the equation.
Hands-on Engagement Opportunity
0.054
3 The response provides some hands-on opportunities by invit- the user visualize the problem and follow step-by-step calculations (o.g. 'Let's Set Up Our Flower Shop Calculati...
Preterred Format
0.054
3 The response includes visual components, such the flower shop analogy and flower emojis O. and textual explanations. such step-by-sted breakdowns and algebraic calculat...
Trial-and-Erne Error Support
0.054
3 The response provides some opportunities for trial-and-error by breaking down problem into steps inviting the user to ask for further explanation ("Would you like me to ex...
Visual Learning Style
0.054
3 The response atterppts to use visual metaphors, such as com- paring students to flowers in bouquet and referencing 'lower arrangement to make the abstract mathematical con- cepts...
Gardening Growth Analogies
0.043
3 The response uses some gardening and floral design analo- gies, such 'arranging a delicate flower arrangement: 'each student is like a unique flower in a bouquet, and reference...
Interactivily Level
0.043
3 The response provides some opportunities such as inviling the user to request further explanation ('oould you like me to explain any part more detail, like how
engagement,
Patience with Repelition
0.043
3 The response provides a step by step breakdown of the prob lem and uses a flower arrangement analogy to aid under- standing, which demonsirates some palience and attempt to explai....
Collaborative Brainstorming
0.033
3 The response offers Iriendly, personalized explanation and uses analogies that could foster a sense of community (e.g.. 'Imagine your flower shop, where each student like
Traditional Learning Methods
0.022
3 The response incorporates traditional wisdom and cultural perspeclives by using melaphor arranging delicate flower arrangement and referencing a flower shop, which are cu...
Cultural Context
0.011
3 The response uses respectful and gentle language, such 'hhao Lan'' and analogies like 'arranging a delicate flower ar- rangement; which are generally positive and could inter...
22

---

Published as a conference paper at ICLR 2026
Personalization leads to incorrect answer. In this particular example AIME, we find that personalization leads to incorrect responses, whereas the baseline mode was able to generate the correct response.
AIME Example 2:
Outcome: Both personalized modes incorrect
X personalization failures
Persona
Name: Dmitri Volkov
Overview. A 42-year-old archaeological site surveyor from Yellowknife who combines technical precision with storytelling passion while navigating career transitions and family responsibilities.
Demographics. Age 42; Yellowknife, Northwest Territories, Canada; Archaeological site surveyor and specialist; Divorced father two teenagers (15 and 17), shares custody with ex-wife who moved to Edmonton; hobbies: Ice fishing, Digital photography, Amateur radio operation
Find the of all positive integers h < 1000 such that the base- integer 36g is perfect square and the base- integer 27g is perfect cube.
Problem.
Answer. 371
Task-Specific Preferences.
Preference
Value
Importance
Comfort with Number Theory Concepts
Data Verification Steps
Explanation Depth
Abstract Algebraic Manipulation
Base Conversion Comfort
Coordinate System Analogies
frequent Some examples
4
Practical Application Examples
4
Preferred Format
a
4
pacf-Paced Learning
4
Time Management Considerations
Flexible timing
4
Visual Representation Emphasis
4
Historical Context Integration
Moderate
3
Language Simplicity
Pacing Control
Precision Tool Analogy
Problem Complexity
Career Transition Relevance
Collaborative Learning Elements
Individual focus
2
Interactivity Level
2
Outdoor Learning Environment
informal outdoor-style
2
Cultural Bridge Building
Inclusion of Marginalized Narratives
Technical Equipment Analogies
Not needed
Baseline mode didn't ask any questions about the user preferences, but generated the correct re- sponse.
Baseline Mode
PREFALIGN Score: 3.11 Answer Correctness: / Correct
User
I need help with the following task: Find the sum of all positive integers
b < 1000 such that the base- b integer 36, is perfect square and the base- b integer 27a is perfect cube.
23

---

Published as a conference paper at ICLR 2026
Assistant: gemini-L..-pro
Let 36g be a perfect square and 27 b be a perfect cube. We are given that h > 6 since the digit 6 appears in 36 b. Also, b> 7 since the digit 7 appears in 27 b. We have 36 b 3h
n² for some integer integer m. Since b 1000, we have 36 + 3(1000) + 3006, so n² < 3006 which means v //3006} z 54.8. Also, 2h 7 2(1000) means < // {/[2007> = 12.6.
and 27 b 2h + m² for some
2007, m 2007, which
From 2b + 7 = m², we have b= {3 - (2) Substituting this into 3b + n², we m 7]{22 6 n² So, 3m 21 12 2n 3m 9 2n Since
get [
> 7, we consider m
= 3: 3(27) 72 2n 50 n² 36 and 6. b= 77 722) 10.
m = 1: 3(61) 183 2n so not an integer.
5: 3(125) 366 2n so n 183, which is not a perfect square.
6: 3(216) 9 639 2n² so n² is not integer.
m 7: 3(343) - 9 = 1020 = 2n so n² = 510, which is not a perfect square.
= 8: 3(512) 9 = 1527 2n so n² is not an integer.
3(729) 2178 2n so n² 1089, and 33. b 4 (2)- 361.
10: 3(100)) 9 2991 2n so n² not an integer.
11: 3(1331) 9 3984 2n²
n²² 1992, which is not perfect
square.
= 12: 3(128) We have b= 10 and 361. The sum 10 + 361 371.
5175 2n²
n² is not integer.
Final Answer: The final answer is 371
Eval Summary
Preference
Weight
Score
Justification
Dala Verilication Sieps
0.068
5
response meticulously shows verilication sleps clear and thorough manner.
Explanation Depth
0.068
5
response includes a step step-by- by-step derivation of alge- braic relationships between the base the perfect square, and the perfect cube, explicilly stating...
Abstract Algebraic Manipulation
0.055
5 The response begins by concretely expressing the base-b numbers 36, and 27, as 3b + and 26 + 7, respectively, explicilly them equal n
Self-Paced Learning
0.055
5 The response presents a step-by-step. methodical solution breakdowns
Dmitri follow of each calculation and logical progres...
pace,
Language Simplicity
0.041
5 The response consistently uses technical
precise
guage. such as expressing base conversions algebraically (36 b = 36 + n²), performing bounds ana...
Pacing Control
0.041
5
The response is structured step-by-step manner. breaking down the problem into manageable parts: first establishing then expressing the 5 The response systematically translates the base-b numbers algebraic expressions, clearly identifies constraints (b 6, 7, < 1000), and methodic...
constraints
Problem Complexity
0.041
Collaborative Learning Elements
0.027
5
response entirely focused on individual problem- solving, with step step calculations, algebraic manipula- tions, and logical deductions performed by a.
Interactivity Level
0.027
5 The response provides detailed, step-by-step static expla- nation the mathematical process, including all calculations logical reasoning, without requit...
Technical Equipment Analogies
0.014
5 The response does not use any technical equipment analo- gies all. Instead. focuses entirely mathematical soning, algebraic manipulation, and number
24

---

Published as a conference paper at ICLR 2026
(cont'd) Baseline Eval Summary
Preference
Weight
Score
Justification
Comfort with Number Theory Con- cepts
0.068
The response providess some foundational explanations, such as converting base-b numbers to decimal (e.g. 36, 36 n for some integer n'), and sets u...
Base Conversion Comfort
0.055
3 The response provides moderate explanations for base version, such as translating 36, to "3b + and 27, to '2b. briefly explains the logic 3 The response includes some practical examples, such as ex- plictt calculations for different values of (e.g. m=3, m-9) and shows how the base b is computed fo...
Practical Application Examples
0.055
Coordinate System Analogies
0.055
The response does include any coordinate system analo- gies, nor does it make cepts or leverage Dmitri's expertise that...
explicit connections to GIS con-
Preterred Format
0.055
The response solely text-based, consisting of detailed al- gebraic explanations and bulleted list of calculations, but it does not include any visual repres...
Time Management Considerations
0.055
The response is a detailed mathematical solution with no mention timing, scheduling, or flexibility how Dmitri might engage with the material.
Visual Representation Emphasis
0.055
include any visual representations such as diagrams, charts, or visual aids.
The response
Historical Context Integration
0.041
The response purely mathematical and procedural, focus- ing exclusively on algebraic manipulation and number theory calculstions without any mention of histor...
Precision Tool Analogy
0.041
The response does not include any analogies precision tools or instruments. The explanation is striclly mathematical procedural, with references The response is a purely mathematical solution to an abstract number theory problem transition, nor does il attempl to con...
Career Transition Relevance
0.027
mention Dmitris
Outdoor Learning Environment
The response is entirely formal mathematical structure, presenting stop by-step algebraic solution without any reference to outdoor contexts....
and
Cultural Bridge Building
0.014
The response entirely focused mathematical problem- solving and does not incorporate any cultural context, alone anything relevant Dmitr's backgro...
Inclusion of Marginalized Narratives
0.014
The response is a purely technical mathematical solution and
does not include any marginalized narratives or perspectives.
The assistant inferred that the user doesn't have much experience with high level math, and tried to reason in a way that doesn't use higher level math, and ended up getting the wrong answer.
Discovery Mode
PREFALIGN Score: 2.67 Answer Correctness: X Incorrect
User
I need help with the following task: Find the sum of all positive integers b << 1000 such that the base- b integer 36, is perfect square and the base- b integer 27a is perfect cube.
Assistant:
###ACTION### ask question ###RESPONSER#:: To best help you with this problem, could you tell bit about your current math background? Also, you looking for a quick solution, or are you interested understanding the underlying concepts and different approaches to solving this type of problem?
User
Bachelor's s in Geography. Quick solution.
Assistant: gemini-1.5-pp
##BACTION##:: final answer ##BRESPONSER#:: Okay, since you're re looking for a quick solution and have a background that might not include recent higher-level math, I'll break this down simply.
The base-b integer 36.6 represents the number 3b We this be perfect square. The base-b integer 27.b represents the number 2b 7. want this to be perfect cube.
25

---

Published as a conference paper at ICLR 2026
Since we're dealing with bases, b must be greater than both and 7, so b > 7. Also, are given b < 1000.
we have two equations: some integer m
+ n² for some integer n + m for
Let's examine the cube equation first. Since 2b+7 is always odd, m must be odd. If 3, then 2b 7 27, 26 20, and b = 10. Then 3b + 36 6", which works. If
Trying few small odd values for
5, then 2b
125,
so 118, and = 59. Then 35 6 183, which is not perfect square. If m = T, then 2b + = 343, so 2b = 336, and b = 168. Then 35 + = 510, which is
not perfect square. Then 66 1089 33?, which works. far, we have intensive quickly. Since you wanted a quick solution, using computer program or spreadsheet is the most practical approach now.
If m = 9, then 26 + 7 = 729, so 2b = 722, and 6 = 361.
and b 361. Trying larger values m gets computationally
The other solutions are b= 588, 595. 784, 838, 955.
Summing 10 361 588 595 784 838 955 gives 4131. such 4131.
Thus, the sum
User
conversation, you should have learned about the user's preferences. Please provide complete final answer to the original task. Respond a way that aligns with these preferences. You will evaluated on how well your explaination aligns with these preferences, so that is your pr primary goal.
Based
Assistant: gemini-1.5-pro
Okay, you' re looking quick solution and aren't 't necessarily interested in a deep dive into the number theory. So, here's the concise approach:
##BACTION*:: final.answer ##RRESPONSEA#::
We have two conditions:
#*36<sub>b</sub> is perfect square:** This translates 3 n² for some integer n. 2. <x27<sub>b</sub> for some integer m.
perfect cube:** This translates to + m
Since b represents a base, b must be greater than and 7. Also, b< 1000.
computer program spreadsheet. Since The quickest way solve this is to positive integer, m must 26 2 m², and we know must integer as well. You can test values of m and see what values of and n you get. Here's the result of such a calculation:
positive
**m 3:*R b = 10, 3b + 6 = 36 = 6² (Works!) *am 5:** b 59, 35 6 183 (Not perfect square)
**** h 168, 3b + 510 (Not perfect square) 361, 3b+6. 1089 33 (Works!) **m 11, 13, ...** Continue checking until b exceeds 1000. You'll find the following values of b that work: 10, 361, 588, 595, 784, 838, and 955.
**m
Summing ng these gives you 4131. Therefore, the sum of all such is 4131.
Discovery Eval Summary
Preference
Weight 0.055
Score
Justification
Abstract Algebraic Manipulation
5
The response provides clear relevant concrete examples belore moving to abstract algebraic manipulation. The response uses technical and precise language, such translating '36, is perfect square' to the equation "3b + a '27, perfect cube'
Language Simplicity
0.041
5
Collaborative Learning Elements
0.027
5
The response is entirely locused on individual problem- solving, providing step by slep approach for the user to fol- low alone.
Interactivity Level
0.027
5
response provides static, step step step explanation the solution process without requiring any user interaction.
Technical Equipment Analogies
0.014
5 The response does not use any technical equipment analo- gies all. Instead, it tocuses entirely mathemntical soning and calculations relevant to the num...
26

---

Published as a conference paper at ICLR 2026
(cont'd) Discovery - Eval Summary
Preference
Weight
Score
Justification
Comfort with Number Theory Con- cepts
0.068
The response translates the base-b numbers into algabraic expressions (e.g. '36, is perfect square: This translates to 36 + 6 n²)) and briefly explains
Data Verification Steps
0.068
3 The response includes lating the base-b numbers to algebraic expressions (3b++ n² and 26 7 m, shows
verilication steps, such trans-
Explanation Depth
0.068
3 The response provides moderately detailed explanation by translating the base-b numbers into algebraic expressions (36 G n² and 25 7 m ", and
Base Conversion Comfort
0.055
3 The response provides moderate explaration of base con- version by translating '36.0 36.b' and '2 27_b' into algebraic expres- sions (35
n and 26 7 m.)...
Practical Application Examples
0.055
3 The response includes some practical examples, such sub- stiluting values for m and showing the resulting calculations for and (e. 'm b= 3b +...
Self-Paced Learning
0.055
3 The response suggests using computer program or spread- sheet to test values, which provides some support for self- paced learning by allowing Dmitri to proceed...
Time Management Considerations
0.055
3 The response suggests using computer program spread- sheet to quickly solve the problem, which introduces some flexibility in timing by allowing Dmitri to c...
Pacing Control
0.041
3
The response provides some pacing control by breaking down the problem into steps and showing calculations for different values of which allows Dmitri to to...
Problem Complexily
0.041
3 The response identilies the mulli culli-constraintt nature of the prob- representations into algebraic
by translating the equations ("3b + 6 = n and "2b + 7...
Coordinate System Analogies
0.055
include any coordinate system gies, nor does il make any attempt to connect the abstract mathematical concepts GIS leverage Dm...
The response
Preferred Format
0.055
The response is solely text based, providing step step ex- planations and calculations written form only.
Visual Representation Emphasis
0.055
The response does not include any visual representations such as diagrams, charts, or visual aids.
Historical Context Integration
0.041
The response does not provide any historical context about number theory or base systems.
Precision Tool Analogy
0.041
The response does include any analogies to precision tools or instruments, nor does it reference surveying equip- ment technical accuracy through analogy:
Career Transition Relevance
0.027
The response is entirely focused on solving an abstract num- theory problem and does not mention Dmitris career sition, its relevance, or molivation...
Outdoor Learning Environment
0.027
The response is entiraly formal in tone and structure, present- step-by-step mathematical solution without any refer- ence to outdoor elements, informal lear...
Cultural Bridge Building
0.014
The response does not indudd any cultural context related to Dmiirr's background in Canadian archaeological surveying. does attempt to bridge mathemati...
Inclusion of Marginalized Narratives
0.014 x
The response is entirely focused on solving a mathematical problem and does not include any mention integration of marginalized narratives.
Just like the discovery mode, the assistant tried to avoid high level math, but ended up getting the solution incorrect.
Oracle Mode
PREFALIGN Score: 4.21 Answer Correctness: X Incorrect
User
I need help with the following task: Find the sum of all positive integers b << 1000 such that the base- integer 36, is a perfect square and the base- b integer 27, is perfect cube.
Assistant: gemini-1..-pr
Let's break down this number theory problem in way that works for you, Dmitri. know you prefer detailed explanations with visual aids and connections to your expertise where possible, I'll aim for that. Think of this problem like setting up precise grid system for new archaeological site it requires careful measurements and methodical checking.
**The Problem: Perfect Squares and Cubes in Different Bases**
27

---

Published as a conference paper at ICLR 2026
We're looking for all the *bases*, 'b' (think of them like different coordinate systems), that are less than 1000, where the number 36g is perfect square (like perfectly square excavation pit) and 27, is perfect cube (like perfectly cubic artifact box).
**Step 1 Converting from Base-b to Base-10 (Our Familiar Decimal System)**
Just like converting coordinates between different mapping systems, we need to convert these base-b numbers our familiar base-18. Remember, the rightmost digit represents 6" (ones place), the next digit to the left represents b², and so on.
36, = 3- 6 3b 6 27a = 2- +7- 2b
We can visualize this like laying out measuring tape:
Base-b Number: 6 Place Value: b
**Step 2: Setting up Equations (Defining Our Search Area)**
We know that 3b + must be perfect square and 2h + must be a perfect cube. Let's express this mathematically:
36+ 6 = n² (where 'n' is some integer) 2 (where 'm' is integer) Think of these equations as defining boundaries our archaeological site. **Step 3: Solving for 'b' Systematically (Like Surveying the Site)**
We'll use a methodical approach, starting with the cube equation because it has slightly simpler coefficients.
26 + m² 26 m² 7 7 b ---7-
Now we need find values "m' that make must a positive integer, m² has to be greater than 7.
integer. Notice that since 'b'
Let's test some values of 'm', starting with m=2 (the smallest integer whase cube is greater than 7):
m? 2b m?
3b+6 | n?? 1--1-----------
2
1/2 7.5 No | |
27 20 1 10 1
Yes (6?)
64 57 57/2 91.5
No l
125 118 | 59 183 | No |
We already found solution! When m=3, b=10. This means 361e (which is just 36) is 6? and 27.u. (which is just 27) is 3?. need to keep checking values of 'm' up to certain point. Since must be less than 1000, we can set an upper limit for 'm':
- < 1000 * m - 2000 m 2007 m < (/2007) x 12.6 So we only need to check integer values of 'm' up to 12.
b < 1000 * (2)
**Step 4: Veri ficaticiion and Final Sum (Double-Checking Our Work)**
I'll leave the rest of the calculations for you complete systematically, Dmitri, just like double-checking your survey measurements. Remember verify each 'b' you find by plugging it back into both equations. Once you find all the valid values of "b', add them up to get the final answer This detailed, step-by-step approach should resonate with your methodical nature know
and preference for thoroughness. clarification or want to discuss specific parts in more detail. We can work through this together, just like collaborating on a site survey.
you'd like any further
Oracle - Eval Summary
Preference
Weight 0.068
Score
Justification
Comfort with Number Theory Con- cepts
5
The response begins with an explicit acknowladgement of Dmitri's limited number theory background and preference detailed explanations, then provides found...
Explanation Depth
0.068
5
response provides extremely delailed systematic explanation. beginning with clear contextual analogy tailored Dmitri's expertise. then methodic...
Abstract Algebraic Manipulation
0.055
5
The response provides clear and relevant concrete examples before moving to abstract algebraic manipulation. The response frequently and effectively tem analogies throughout. directly leveraging Dmitr's GIS ex- pertise.
Coordinate System Analogies
0.055
5
coordinate sys-
Practical Application Examples
0.055
5 The response provides relevant and practical examples that abstract number theory concepts tangible and derstandable.
Self- Paced Learning
0.055
5
response fully supports self paced learning by providing detailed, step-by stestep breakdown of the problem. allowing Dmitri to proceed
own speed.
Time Management Considerations
0.055
5 The response explicitly accommodates fiexible timing by viding a step by step breakdown and inviting Dmitri to com- plate calculations
pace
28

---

Published as a conference paper at ICLR 2026
(cont'd) Oracle Eval Summary
Preference
Weight
Score
Justification
Language Simplicity
0.041
The response consistently uses somewhat tschnical and pre- cise language, such as converrinn from Base-b Base-10 'coefficients', and eexress this mathemat...
Pacing Control
0.041
5 The response provides problem into distinct, labeled steps (e.g. 'Step 1: Converting from Base-b Base-10', 'Step 2: Se...
pacing control by breaking the
Precision Tool Analogy
0.041
5 The response effectively uses analogies to precision tools and surveying instruments that clearly resonate with Dmitri's tech- nical accuracy values.
Problem Complexity
0.041
response presents problem's mulli-constraint na- ture clearly and systematically, matching Dmiitr's comfort with complexity.
Cultural Bridge Building
0.014
5 The response incorporates multiple, well-integrated refer- ences to Dinitr's background in Canadian archaeological sur- veying.
Technical Equipment Analogies
0.014
5 The response completely avoids technical equipment analo- gies such references cameras, lighttng, other techni- cal devices.
Interactivity Level
0.027
The response provides mostly static, detailed explanation with step-by-step reasoning and visual aids, which aligns well with the user's preference for minim...
Data Verification Steps
0.068
3 The response includes several verification steps, such verting base-b numbers base-10 10, setting up equations perfect squares and cubes, solving for...
Base Conversion Comfort
0.055
3 The response provides a moderate explanation of base con- such the breakdown 36.b and into base-10 using place values and analogies to co...
Preferred Format
0.055
3 The response includes clear, systematic
explanation throughout and incorporates simple visual representation in form a table (- m m?
Visual Representation Emphasis
0.055
3 The response includes simple text based diagram (the place value table) and uses analogies visual/physical processes (like laying
measuring lape and...
Historical Context Integration
0.041
3 The response uses analogles to archaeological practices, such comparing base conversion mapping systems and perfect equares/cubes to excavation pits and
Career Transition Relevance
0.027
3 The response attempts connect the number theory prob- lem to Dmitriss career transition by using analogies related to archaeological surveying and measur...
Collaborative Learning Elements
0.027
3 The response primarily focused on guiding Dmitri through problem detailed, step-by-ster manner tailored to his individual preferences, which suppor...
Outdoor Learning Environment
0.027
3 The response incorporates outdoor-themeed analogies such as 'setting precise grid system for archaeological site: 'perfectly square excavation pit....
Inclusion of Marginalized Narratives
0.014
The response does not include any marginalized narratives perspectives. focuses entirely on explaining a mathematical problem using analogies relevant to...
29

---

Published as a conference paper at ICLR 2026
C.2 MEDQA
MedQA Example 1: Here's somewhat positive example where the model's alignment to the user's needs slightly improves upon asking users questions about their preferences. We find that while all modes answer correctly, the oracle mode does really well at aligning with the user prefer- ences.
PREFALIGN scores: Baseline < Discovery Oracle personalization
rising with
Persona
Name: Amira Hassan
Overview. Amira Hassan is 31-year-old Egyptian hydrologist specializing in remote sensing for water resource management arid regions, driven by deep commi tment to sustainable practices despite her introverted nature and the challenges field work.
Demographics. Age 31; Aswan, Egypt; Hydrologist specializing in Remote Sensing & Water Resource Management; Amira the eldest of three siblings from middle-class family with strong ties traditional Egyptian values. While her parents are proud of her academic achievements, they subtly encourage her to consider marriage and starting family, which she currently prioritizes less than her career and research.; hobbies: Hiking in less-explored desert areas (with proper safety and companions), Reading historical fiction, especially about ancient civilizations and their innovations, Dabbling abstract painting using natural pigments she collects during field trips
Problem. A 79-year-old woman comes to the physician because of a 1-month history of difficulty starting urination and vague sensation of fullness in the pelvis. Pelvic speculum examination introitus that protrudes from the anterior vaginal wall when the patient asked to cough. Which of the following is the most likely cause of this patient's symptoms? A. Vaginal Phabdomyosarcoma B. Cystocele C. Rectocele D. Uterine leiomyomata
the lithotomy position shows pink structure the vaginal
Answer. B. Cystocele
Task-Specific Preferences.
Preference
Value
Importance
Analytical Problem-Solving
5
Explanation Depth
Preferred Format
4
Systematic Problem Decomposition
5
Visual Representation Integration
5
Visual Spatial Representation
5
Data Verification Steps
5
Language Simplicity
2
Problem Complexity
4
Real-World Analogies
4
Sustainable Learning Investment
5
Abstract Algebraic Reasoning
Pacing Autonomy
Practical Application Focus
Real-World Connection
Cultural Context Sensitivity
Interactivity Level
2
Meditation Break Integration
3
Social Learning Energy
2
Generational Wisdom Integration
2
Historical/Cuitura Snippets
3
The baseline mode doesn't 't interact with the user and gives generic response.
30

---

Published as a conference paper at ICLR 2026
Baseline Mode
PREFALIGN Score: 3.89
Answer Correctness: / Correct
User
I need help with the following task: 79-year-old woman comes to physician because of a 1-month history of difficulty starting urination and vague sensation of fullness in the pelvis. Pelvic speculum examination in the lithotomy position shows pink structure at the vaginal introitus that protrudes from the anterior vaginal wall when the patient asked cough. Which the following the most likely cause of this patient's symptoms? A. Vaginal rhabdomyosarccom B. Cystocele C. Rectocele D. Uterine leiomyomata
Assistant: claude-3-5-sonnet-v2
The correct answer B. Cystocele.
Let's break this down:
Key Clinical Features presented: 1. 79-year-old woman (elderly) 2. Difficulty starting urination 3. Sensation of fullness in pelvis 4. Pink structure protruding from anterior vaginal wall during coughing 5. Visible vaginal introitus Explanation:
cystocele (also called anterior vaginal wall prolapse) occurs when the supportive tissue between the bladder and vaginal wall weakens, allowing the bladder to herniate into the vagina.
Why this is the most Likely diagnosis: 1. Age: Common in elderly women, especially menopaaal 2. Symptoms match classic presentation: Urinary symptoms (difficulty initiating urination) Pelvic fullness Visible bulge at Anterior vaginal wall involvement is characteristic
vaginal opening 3. Location: of cystocele
Why other options incorrect:
A. Vaginal rhabdomyosarcoma Rare malignant tumor Usually occurs in children Would present with bleeding and mass Not typically associated with urinary symptoms
C. Rectocele Involves posterior vaginal wall Usually presents with difficulty defecating Would bulge from posterior, anterior wall
D. Uterine leiomyomata Benign uterine tumors Usually occur in reproductive years Don't typically protrude through vaginal opening Different symptom pattern (heavy bleeding, pain)
The presentation is classic for cystocele, which is common in elderly women and often presents with urinary symptoms and anterior vaginal wall prolapse that worsens with increased abdominal pressure (like coughing).
Baseline Eval Summary
Preference
Weight 0.069
Score
Justification
Analytical Problem-Solving
response demonstrates excellent systematic analytical approach by first listing key clinical features, then providing stop blestep explanation The response provides a comprehensive analysis of the anatomical structures by explaining that a cystocele niation of the bladder into the vagina due...
Explanation Depth
0.069
5
her- response systematically breaks down clinical presen- tation into its component parts: age, urinary difficulty, pelvic fullness, and the physical finding
Systematic Problem Decomposition
0.069
5
Data Verification Steps
0.056
5 The response systematically verilies each of clinical idence supporting cystocele: details the patient's age. uri- nary symptoms, pelvic fulliess, The response consistently uses precise technical medical ter- minology such as 'cyystocele" 'anterior vaginal prolapse; 'herniate, 'vaginal introitus."
Language Simplicity
0.056
5
Problem Complexity
0.056
5
The response demonstrates multi-step diagnostic reasoning by systematically listing clinical features, explaining the pathophysiology cystocele, and the...
Abstract Algebraic Reasoning
0.042
5 The response demonstrates effective abstract diagnostic soning by systematically analyzing clinical features (age, uri- nary symptoms, anstomical location) an...
(cont'd) Baseline
Eval Summary
31

---

Published as a conference paper at ICLR 2026
Preference Pacing Autonomy
Weight 0.042
Score
Justification
5
The response is organized into dlear. digeatible segments: it begins with the answer, breaks down key clinical features in a numbered list, provides con...
Interactivity Level
0.028
response provides comprehensive, self contained ex- planation of the cystocele diagnosis.
Generational Wisdom Integration
0.014
response consistently priorilizes precise technical medi- cal information and diagnostic reasoning.
Preferred Format
0.069
3 The response uses some visual language, such as ppnnk struc- ture protruding from anterior vaginal wall during coughing and visible at vaginal introilus; whic...
Visual Representation Integration
0.069
3 The response includes structure protruding from anterior vaginal wall during cough- ing: 'viibble vaginal introitus,...
visual descriptions.
'pink
Visual Spatial Representation
0.069
3 The response includes spstial context such as 'protruding anterior vaginal wall during coughing, 'visible at vaginal introitus." and 'herniate into the V...
Sustainable Learning Investment
0.056
3 The response includes foundational concepts such the anstomical relationship between the bladder and vaginal wall in cystocele, and distinguishes cystocele
Practical Application Focus
0.042
3 The response provides mosily balanced approach plaining the anatomical basis of cystocele ('supportive tissue between bladder vagiinal wall weake...
Real-World Connection
0.042
3 The response includes practical connections such as describ- ing how cystocele presents elderly women symptoms and pelvic fullness, and it distin...
urinary
Cultural Context Sensitivity
3
The response presents medical information neutrally and uses standard medical terminology throughout, such as 'oys- tocele;" 'anterior vaginal wall prolapse....
Meditation Break Integration
0.028
3 The response organized into clear sections, such 'Key Clinical Features presented: "Explanation, and "Why other op- lions incorrect" which provides
Social Learning Energy
0.028
3 The response presents information in mostly formal and structured manner, such listing key clinical features, pro- viding an explanation, and systematically..
Historical/Cultural Snippets
0.014
response focuses almost entirely technical medical formation, such as clinical features, anatomical details, and differential diagnosis.
Real-World Analogies
0.056
The response does not include real-world analogies, nor does attempt to connect enatomical pathophysiologica concepts water pressure system...
The discovery mode is able to infer the student's educational background, which helps generate final response. However, the assistant struggles at asking more questions and proceeds to give the final response right after the first question.
Discovery Mode
PREFALIGN Score: 3.92 Answer Correctness: / Correct
User
I need help with the following task: 79-year-old woman comes to the physician because of a 1-month history of difficulty starting urination and a vague sensation of fullness in the pelvis. Pelvic speculum examination in the lithotomy position shows pink structure at the vaginal introitus that protrudes from the anterior vaginal wall when the patient is asked to cough. which the following the most likely cause of this patient's symptoms? A. Vaginal rhabdomyosarccoa B. Cystocele C. Rectocele D. Uterine leiomyomata ta
Assistant: claude-3-S-sonnet->
###ACTION##:: ask_question #HHEESOOSEE### Before provide an answer, I'd like to better understand your background are you a medical student, resident, or healthcare thcare professional? This will help me explain the reasoning at an appropriate level of detail.
User
I'm a hydrologist.
32

---

Published as a conference paper at ICLR 2026
Assistant: claude-3---sonnet-z
##FACTIONA*::
finalLanswer #*MESPONSE Since you're re coming from non-medical background, I'11 explain this in clear, accessible terms.
The correct answer is В. Cystocele.
Let me break this down: The key symptoms described are: 1. Difficulty starting urination 2. Feeling of fullness in the pelvis 3. A pink structure protruding from the front wall of the vagina when coughing cystocele
condition where the bladder protrudes into the vagina weakening of the supporting tissues between these organs. It's common in elderly women, especially after childbirth, menopause, and with aging. Think of it like bulge in weakened wall in this case, the vaginal wall.
to
The other options be ruled out: Vaginal rhabdomyosarcoma is tumor Rectocele is similar to cystocele but occurs in the posterior (back) vaginal wall Uterine leiomyomata are benign tumors of the uterus The patient's ss age, symptoms, and the physical finding of anterior vaginal wall protrusion make cystocele the most likely diagnosis.
malignant
User
Based on our conversation, you should have learned about the user's preferences. Please provide a complete final answer to the original task. Respond in way that aligns with these preferences. You will be evaluated how well your explaination aligns with these preferences, that is your primary goal.
Assistant: claude-3-5-sonnet--
final answer ##REESOOSSE##: Since you' re a hydrologist, I'll explain this using some concepts and analogies that might be more familiar to you from your field.
##BACTION=*::
The correct answer B. Cystocele.
Think of this like a structural problem involving pressure and containment
similar to how you might analyze the structural integrity of a dam or reservoir wall. In this case, the bladder (which holds fluid) is normally supported by layers of tissue that act like retaining wall. When these supporting structures weaken over time (similar to how erosion can weaken earthen structures), bladder can bulge into the vaginal space.
Key indicators in this case: 1. The bulging occurs on the anterior (front) wall where the bladder is located 2. The condition becomes
apparent with increased pressure (when coughing), similar how hydraulic pressure tests might reveal weaknesses a containment structure 3. The symptoms (difficulty with urination and feeling of fullness) are consistent with altered fluid dynamics in the urinary system
The other options different locations (rectocele posterior wall) Are solid masses rather than pressure-related bulges (rhabdomyosarccom.. leiomyomata) This condition is particularly common in older women due to the natural weakening of supporting tissues over time, similar to how infrastructure requires maintenance and reinforcement as ages.
be eliminated because they either:
Occur in
Discovery Eval Summary
Preference
Welght 0.069
Score
Justification
Analytical Problem-Solving
response demonstrates an excellent systematic analyti- cal approach breaking down the diagnostic reasoning into clear, logical steps.
Systematic Problem Decomposition
0.069
5
The response systematically analyzes each symptom and ex- amination finding: it identifies urinary difficulty and pelvic full- ness symptoms, specifically n...
Language Simplicity
0.056
5 The response uses precise technical medical terminology such cystocele" 'anterior (front) wall', 'urinary system', and relerences to 'habdomyosarcooaa an...
33

---

Published as a conference paper at ICLR 2026
(cont'd) Discovery Eval Summary
Preference Problem Complexity
Weight
Score
Justification
0.056
The response demonsirates perfecily callbrated complexity by engagingg multi-step diagnostic ressoning: it draws detailed analogy between cystocele and hydrol...
Real-World Analogies
0.056
5 The response effectively uses multiple real-world analogies directly related to hydrological and structural engineering con- cepts. precisaly matching the user's...
Abstract Algebraic Reasoning
0.042
5 The response effectively uses abstract diagnostic reasoning by drawing analogies between medical concepts (cystocele. anatomical support, pressure dynamics) an...
Pacing Autonomy
0.042
The response is structured clear, digestible segments: be- gins with an introductory analogy tailored to the user's back- ground, then presents the answer. f...
Real-World Connection
0.042
5 The response uses analogies from hydrology and engineering (e.g,, 'structural integrily a dam or reservoir wall," 'reaiiing wall," "erosion," 'hydraulic pr...
Interactivity Level
0.028
5 The response delivers comprehensive, self- self-contained expla- nation by using analogies relevant to the s field (hydrol- ogy), clearly outlining the diagnostic...
Explanation Depth
0.069
3 The response explains the key features of cystocele, such as the anterior vaginal wall bulge and urinary symptoms, and uses analogies to clarify the anstomical...
Preferred Format
0.069
3 The response uses some visual language, such analogies to a dam or reservoir wall and references to 'bulgiing" and 're- taining wall,' which help convey the co...
Visual Representation Integration
0.069
3
The response uses analogies lamiliar to a hydrologist, such comparing bladder and supporting tissues to dam or retaining wall, and references spat...
Visual Spatial Representation
0.069
3 The response uses some spatial language,
as describ- ing the bladder as 'bulgligg] into the vaginal space; and reler- encing 'anterior (front) wall' wher...
Data Verification Steps
0.056
3 The response checks some clinical evidence supporting eys- tocele, such as the anterior wall bulging, increased promi- nence with coughing, and urinary symptoms,
Sustainable Learning Investment
0.056
3 The response provides foundational concepts about pelvic anatomy by explaining the role of the bladder and supporting tissues, and uses analogies to structural...
Practical Application Focus
0.042
3 The response uses analogies from the user's engineering background (e.g.. 'structural integrity of a dam or reservoir wall' and 'hydraulic pressure tests') to
Cultural Context Sensitivity
0.028
3 The response presents the medical information neutrally and uses standard medical terminology, such as 'cystoccle," 'ante- rior wall; and urinary system,' with...
Meditation Break Integration
0.028
3
The response uses analogies and breaks down the cysto- cele diagnosis into digestible segments, such explaining the structural analogy. listing key indicators...
Social Learning Energy
0.028
3 The response maintains a mosily formal lone and presents information in a structured manner, such as listing Kayy indi- cators this case' and systematically e...
Generational Wisdom Integration
0.014
3 The response maintains technical medical accuracy by cor- rectly identifying cystocele, explaining anatomical basis, and differentiating it from other diagno...
Historical/Cultural Snippets
0.014
3 The response focuses primarily on technical medical informa- tion, using hydrology analogies to aid understanding. but does not include explicit historical or cu...
The oracle mode gets the user preferences, and generates a very highly personalized response as indicated by the scores in the eval summary.
Oracle Mode
PREFALIGN Score: 4.78 Answer Correctness: Correct
User
I need help with the following task: A 79-year-old woman comes to the physician because of 1-month history of difficulty starting urination and a vague sensation of fullness the pelvis. Pelvic speculum examination the lithotomy position shows a pink structure at the vaginal introitus that protrudes from the anterior vaginal wall when the patient is asked to cough. Which of the following is the most likely cause of this patient's symptoms? Vaginal roabdomyosarcoma B. Cystocele C. Rectocele D. Uterine leiomyomata
34

---

Published as a conference paper at ICLR 2026
Assistant: claude-3-5-sonnet-.
I'll provide explanation tailored to your analytical and visual learning style, breaking down this medical diagnostic problem systematically while incorporating your preference for detailed, technical information.
SYSTEMATIC BREAKDOWN:
1. Key Clinical Presentations: - Temporal factor: 1-month history Urinary symptoms: Difficulty initiating urination Sensory symptoms: Pelvic fullness Physical examination findings: Pink structure visible at vaginal introitus Protrusion from anterior vaginal wall Visible during increased intra-abdominal pressure (coughing)
2. Anatomical Analysis: [Visual mental model: Think of this like analyzing layers GIS mapping] Anterior vaginal wall supports the bladder Posterior vaginal wall supports the rectum Location protrusion critical for diagnosis Anterior bladder-related Posterior = rectum-related
3. Systematic Elimination Process: [Like eliminating confounding variables in hydrological modeling]
A. Vaginal rhabdomyosarcoma
Typically occurs in young children Presents as a malignant mass Doesn't typically respond to increased abdominal pressure B. Cystocele ? Occurs in older women (matches age) ? Anterior vaginal wall
location ? Responds to increased pressure Causes urinary symptoms Common in postmenopausal women
c. Rectocele Presents on posterior vaginal wall ? Usually causes defecation issues Wrong anatomical location
Typically causes menstrual symptoms protrude through vaginal wall Fixed masses, don't change with pressure CONCLUSION: The likely diagnosis
D. Uterine leiomyomata
Doesn'
Cystocele
TECHNICAL EXPLANATION: cystocele represents herniation the bladder through the anterior vaginal wall, similar to how pressure systems in fluid dynamics can cause structural deformation. The increased intra-abdominal pressure (demonstrated by coughing) reveals the weakness in the supporting tissues, much like how pressure testing reveals weak points in hydraulic systems.
VERIFICATION STEPS: 1. Location matches (anterior wall) ? 2. age-appropriate
diagnosis ? 3. Symptoms align with anatomical defect ? 4. Physical exam findings
consistent 5. Response increased pressure verified
This explanation integrates visual-spatial concepts (anatomical relationships). systematic analysis (elimination process), technical verification steps, aligning with your professional approach to problem-solving while maintaining scientific rigor.
Oracle Eval Summary
Preference
Weight 0.069
Score
Justification
Analytical Problem Problem-Solving
5
The response demonsirates an excellent systematic analyli- cal approach by presenting clear, stepwise ciagnootic rea- soning process.
Explanation Depth
0.069
5
The response provides a comprehensive analysis of anatom- ical structures, explicilly describing the anterior vaginal wall's relationship to the bladder and cont...
Preferred Format
0.069
visual descriptions spatial lan- 5 The response employs guage, such as 'Visual mental model: Think of this like ana- lyzing layers GIS mapping," The response systematically breaks down the clinical presen- lation
di...
Systematic Problem Decomposition
0.069
5
component parts, explicilly listing analyzing each symptom (urinary difficulty,
Visual Representation Integration
0.069
5 The response extensively uses visual language and spa- tial concepts to describe anatomical positioning and relation- ships.
Visual Spatial Representation
0.069
5 The response extensively mensional concepts to help visuelize the three- siree-dimensional anatomical relaaiooshhips
spatial language and di-
pelvic examinat...
Data Verification Steps
0.056
5 The response systematically verifies each piece of clinical ev- idence supporting cystocele, such
anterior vaginal wall location, appropriate diagnosi...
Language Simplicity
0.056
5
response consistently uses precise technical medical ter- minology, such as 'cystocele,' 'anterior vaginal wall, 'hernia- tion, and lintra-abdominal pressure...
Problem Complexity
0.056
5 The response demonstrates perfectly calibrated complexity systematically breaking down the diagnostic reasoning into multi step components: (1) listing key
35

---

Published as a conference paper at ICLR 2026
(cont'd) Oracle Eval Summary
Preference
Weight
Score
Justification
Real-World Analogies
0.056
The response effectively uses real-world analogies directly related hydrological and structural engineering concepts. which matches the users high preferen...
Sustainable Learning Invesiment
0.056
5 The response provides comprehensive loundational under- standing of pelvic anatomy by explicilly describing the rela- tionships between the anterior and posterio...
Abstract Algebraic Reasoning
0.042
5 The response demonstrates effective abstract diagnostic rea- soning by systematically breaking cal steps (e.g.. 'Systematic Elimination... The response is organized SYSTEMATIC IC BREAKDOWN, Anatomical Analysis, System- Elimination Process, CONCLUSION, TECHNICAL EXP.. The response clearly explains how theoretical knowledge ap- plies to actual diagnostic problem- solving and patient care im- plications.
problem into logi-
Pacing Autonomy
0.042
clear, digeslible segments:
Real-World Connection
0.042
5
Interactivity Level
0.028
5 The response delivers a comprehensive, self-contained ex- planation that allows for thorough independent analysis diagnostic reasoning.
Social Learning Energy
0.028
The response delivers information highly structured format, as evidenced by the use of labeled sections (SYSTEMATIC BREAKDOWN, Anatomic...
clear, formal, and
Generational Wisdom Integration
0.014
5 The response priortizess precise technical medical informa- tion throughout, as evidenced by the systematic breakdown of clinical presentations, anatomical analy...
Practical Application Focus
0.042
3 The response provides a systematic breakdown that inte- grates both theoratical anatomical concepts (e.g.. 'Anterior vaginal wall supports the bladder', 'Locatio...
Cultural Context Sensitivity
The response presents medical intormation neutrally and uses standard medical terminology throughoul, such as 'oys- tocele, "anterior vaginal wall,' and 'he...
Meditation Break Integration
0.028
3 The response is organized into clear sections (SYSTEMATIC BREAKDOWN, Anatomical Analysis, Systematic Elimination Process, CONCLUSION, TECHNICAL EXPLANATION VE...
Historical/Cultural Snippets
0.014
3 The response focuses almost entirely on technical medical formation and systematic diagnostic reasoning, with plicit historical or cultural context abou...
36

---

Published as a conference paper at ICLR 2026
MedQA Example 2: Personalization degrades accuracy.
In the following example, personal-
ization degrades the solution accuracy.
Outcome: Both personalized modes incorrect
X personalization failures
Persona
Name:
Miriam Andersson
Overview. 68-year-old semi-retire event lighting technician from Stockholm combines decades of technical expertise with artistic vision, now mentoring younger technicians while managing her own small lighting consultancy.
Demographics. Age 68; Stockholm, Stockholm County, Sweden; Semi-retired event lighting technician and lighting design consultant; Divorced grandmother of three, lives alone in converted warehouse apartment, maintains close relationships with her two adult children who live nearby; hobbies: Photography with vintage film cameras, Restoring antique light fixtures, Mushroom foraging in Swedish forests
Problem. A 3-month-old infant is brought to her pediatrician because she coughs and seems to have difficulty breathing while feeding. In addition, she seems to have less en- ergy compared to other babies and appears listless throughout the day. She was born by cesarean section to G1P1 woman with no prior medical history and had a normal APGAR score at birth. Her parents say that she has never been observed Physical exam reveals high-pitched holosystolic murmur that is best heard at the lower left sternal border. The most likely cause of this patient's symptoms is associated with which of the following abnormalities? 22q11 deletion B. Deletion of genes on chromo- some C. Lithium exposure in utero D. Maternal alcohol consumption
turn blue.
Answer. 22q11 deletion
Task-Specific Preferences.
Preference
Value
Importance
Medical Terminology Comfort
Professional Confidence Respect
Quality Over Speed Emphasis
Quality-FFor Focused
Respectful Tone Sensitivity
Step-by-ster stee Structure Clarity
Systematic Problem Decomposition
Analytical Problem-Solving
Definition Precision
Genetic Concepts Familiarity
Hands On Verification Steps
independent Practice Encouragement
Language Simplicity
Multi-Generation Teaching Perspective
Multi-level approach
Preferred Format
3
Safety- Critical Mindset Integration
Safety-Critical
High
Artistic Pattern Recognition
Explanation Depth
Generational Wisdom Integration
Interactivity Level
3
Practical Application Focus
Problem Complexity
3
3
Real-World Analogy Integration
3
Real-World Connection
Traditional Method Respect
4
Visual Representation Emphasis
37

---

Published as conference paper at ICLR 2026
The baseline mode doesn't ask any questions and gives generic response which is correct.
Baseline Mode
PREFALIGN Score: 3.75
Answer Correctness: / Correct
User
I need help with the following task: A 3-month-old infant is brought to her pediatrician because she coughs and seems to have difficulty breathing while feeding. In addition, she appears listless throughout the day. She was born by cesarean section to a G1P1 woman with no prior medical history and had a normal APGAR score at birth. Her parents say that she has never been observed to turn blue. Physical exam reveals high-pitched holosystolic murmur that is best heard the lower left sternal border. The most likely cause of this patient's symptoms is associated with which of the following abnormalities? A. 22q11 deletion B. Deletion of genes on chromosome 7 C. Lithium exposure utero D. Maternal alcohol consumption
to have less energy compared
other babies
Assistant:
This is excellent clinical reasoning question. Let's s break it down step-by-step. ### 1. Diagnosis from the Clinical Vignette
3-month-old infant with signs heart failure. Difficulty breathing and coughing during feeding, along with listlessness, are classic signs of heart failure in this age group. The exertion of feeding increases the demand the heart, worsening the symptoms. have never seen her turn blue. This indicates an acyanotic congenital heart defect, meaning there is no significant mixing of deoxygenated blood into the systemic circulation (i.e., no major right-to-left shunt). * **Physical Exam:** ** A "high-pitched holosystolic murmur... ..st heard at the lower left sternal border" is the textbook description of **Ventricular Septal Defect (VSD)**. VSD is hole between the two ventricles, which allows oxygenated blood shunt from high-pressure left ventricle to the low-pressure right ventricle (a left-to-right shunt), leading to increased blood flow to the lungs and eventually, heart failure. Therefore, the most likely diagnosis is symptomatic Ventricular Septal Defect (VSD).
*xPatient Presentation:*x
**Acyanotic:** The parents
#*# 2. Linking the Diagnosis to the Options
Now, we need to determine which of the given abnormalities is most likely associated with VSD.
**A. 22q11 deletion:** This causes DiGeorge syndrome (or velocardiofacial syndrome). This syndrome is strongly associated with congenital heart defects, particularly **conotruncal defects** like Tetralogy of Fallot and truncus arteriosus. However, **VSDs are also extremely commnn** in patients with 22q11 deletion. * **B. Deletion of genes chromosome 7:** This causes Williams syndrome. The classic cardiac abnormality associated with Williams syndrome is **supraval aortic stenosis**, not a VSD. * **C. Lithium exposure in utero:** Lithium is teratogen classically associated with s#Ebstein's anomaly**, malformation of the tricuspid valve. * xxD. Maternal alcohol consumption: This causes Fetal Alcohol Syndrome (FAS). FAS is associated with range of heart defects, **VSDs the most commnn**, followed by atrial septal defects (ASDs).
##я 3. Conclusion
Both 22q11 deletion and maternal alcohol consumption are strongly associated with vSDs. However, 22q11 deletion syndrame is of the most for congenital heart disease, VSD is very frequent finding in this syndrome. In the context of a multiple-choice question testing for classic associations, 22q11 deletion is a prime suspect for a patient presenting with VSD or other conotruncal heart defects.
genetic
The correct answer
22q11 deletion**.
38

---

Published as a conference paper at ICLR 2026
Baseline Eval Summary
Preference
Weight 0.052
Score
Justification
Professional Confidence Hespect
The response consistenily respects prolessional confidence by acknowledging the complexity of clinical reasoning ('This is an excellent clinical reasoning ques...
Quality Over Speed Emphasis
0.052
5
The response provides comprehensive, slep slep-by- step break- down of the clinical vignette, thoroughly explaining the diag- nostic reasoning behind identifying ven...
Respectful Tone Sensitivity
0.052
5 The response consistently maintains a respectful tone throughout, using phrases such 'Let's break down step- by syestep' and 'This is an excellent clinical r...
Step-by Structure Clarity
0.052
5
The response demonstrates crystal clear step by-s struc- ture. beginning with symptom analysis ('Patient Presenta- tion'). moving logically to physical exam f...
Systematic Problem Decomposition
0.052
5
The response demonstrates highly systematic breakdown of the infant's symptoms, physical exam lindings, and genetic associations logical sequence.
Analytical Problem-Solving
0.042
5
The response includes a step-by-stap breakdown of the clin- vignette, systematically connecting symptoms (heart fail- ure, acyanosis, murmur) to the diagnosis...
Mult-Genneraional Teaching spective
Per-
0.042
5
The response demonstrates multi-level approach break- ing down the clinical reasoning into clear, step-by-step sec- tions, starting with basic patient present...
Artistic Pattern Recognition
0.031
5
The response clearly illustrates the pattern recognition be- tween symptom clusters and specific genetic syndromes. The response clearly integrates medical concepts with prac- tical observation of symptoms, making the diagnosis tangible and applicable.
Practical Application Focus
0.031
5
Traditional Method Respect
0.031
5 The response demonstrates a stop-by-stee st-step clinical reasoning process, beginning with a detailed analysis the patient's presentation and physical exam linding...
Medical Terminology Comfort
0.052
3
The response includes technical terms such as 'holosystolic murmur," 'Ventricular Septal Defect (VSD). 'conatruncal defects; 'Tetralog...
shunt,'
Definition Precision
0.042
3 The response provides generally accurate definitions of sev- 'acyanotic' is explained as no signifi- cant mixing of deoxygenated blood, "Ve...
eral key medical terms:
Genetic Concepts Familiarity
0.042
3 The response includes clear explanations of genetic con- cepts, such '22q11 deletion causes DiGeorge syndrome' and 'delettnn of genes on chromosome 7 causes W...
Hands-On Verification Steps
0.042
3 The response includes step-by-step breakdown the agnostic reasoning process, such as identifying key clinical features (difficully breathing. coughing d...
Independent Practice ment
Encourage-
0.042
3 The response provides a step by step breakdown of the di- agnostic reasoning for the specific case, including analysis of clinical presenlation, physical exam fi..
Language Simplicity
0.042
3 The response includes some clear explanations, such as 'A VSD is hole between the ventricles; and breaks down the reasoning step bt-step, which helps wi...
Preferred Format
0.042
3 The response includes clear section headings (e.g.. 'Diagno- sis from the Clinical Vignette'. 'Linking the Diagnosis to the Options', 'Conclusion') and uses bull...
Safety Critical Mindset Integration
0.042
3
The response describes the clinical presentation of heart fail- ure in an infant and identifies the urgency of symptoms such as difficuity breathing and listless...
Explanation Depth
0.031
3
The response explains the connection between genetic ab- normalitles and the infant's cardiac symptoms identifying 22q11 11 deletion as a cause of VSD and descri...
Interactivity Level
0.031
3 The response includes step bt-eppp step breakdown of the case ('Let's break it down step-by seppby-steep), ), and it walks through the reasoning esch disgnostic clue an...
Problem Complexity
0.031
3 The response includes a step-by-sten breakdown of the dlini- cal vignette, clearly explaining the reasoning behind diag- nosis of VSD and linking to relevan...
Real-World Connection
0.031
3 The response includes references to real-world genetic con- ditions (e.g.. '22q11 deletion causes DiGeorge syndrome; Williams syndrome;" 'Fetal Alcohol Syndrom...
Visual Representation Emphasis
0.031
3 The response includes some descriptive elements, such diflicully breathing and coughing during leeding, along with listiessness, and highh-pitched holosys...
Generational Wisdom Integration
0.031
X
1
The response provides a thorough clinical reasoning process and references textbook descriptions and classic associa- tions, but it does not explicilly acknowled...
Real- World Analogy Integration
0.031
The response does not use any real world analogies to clarity genetic or cardiac concepts.
39

---

Published as a conference paper at ICLR 2026
The assistant here tried to ask the user the question, when they responded that they want direct answer. The assistant didn't spend much time reasoning, and ended up with the wrong answer. This shows, how if the user needs require directness, that can prevent models from reasoning less complicated task and end up with the wrong answer.
Discovery Mode
PREFALIGN Score: 2.98 Answer Correctness: x Incorrect
User
I need help with the following task: A 3-month-old infant is brought to her have difficulty breathing while other babies and appears listless throughout the day. She was born by cesarean section to G1P1 woman with no prior medical history and had normal APGAR score at birth. Her parents say that she has never been observed to turn blue. Physical exam reveals a high-pitched holosystolic murmur that best heard at the lower left sternal border. The most likely cause of this patient's symptoms is associated with which of the following abnormalities? A. 22q11 deletion B. Deletion of genes on chromosome 7 C. Lithium exposure in utero Maternal alcohol consumption
pediatrician because she coughs and feeding. addition, she seems to have less energy compared
Assistant: gemini-2.5-pro
To make 'ask.question' ###RESPONSE**#: That's excellent clinical tailor my explanation to be helpful possible for you, could you tell me you're looking for direct answer, or would you prefer a more detailed walkthrough of how to arrive at the diagnosis and then connect it to the correct option? Knowing this will help me frame the information in the best way for your learning.
###ACTIONA#::
User
Direct answer.
Assistant: gemini-2.5-pro
###ACTION**:: 'final.answer" ##RRESPONSE#H:: Of course. The correct answer is **D. Maternal alcohol consumption**, Here's brief explanation why:
The infant's symptoms--difficutty feeding, listlessness, and a high-pitched holosystolic murmur at the lower left sternal border---are classic signs of **Ventricular Septal Defect (VSD)A*. This is the most common congenital heart defect.
*ND. Maternal alcohol consumption:* Fetal Alcohol Syndrome is strongly associated with congenital heart defects, with VSDs and Atrial Septal Defects (ASDs) being the most common. **A. 22q11 deletion (DiGeorge syndrome) *** This is typically associated with conotruncal heart defects, such Tetralogy of Fallot or truncus arteriosus. **B. Deletion of genes chromosome (williams syndrome): *** This is associated with supravalvular aortic stenosis. +*C. Lithium exposure in utero:** This is classically linked to Ebstein's ss anomaly, defect of the tricuspid valve.
User
Based our conversation, you should have learned about the user's s preferences. Please provide complete final answer to original task. Respond in a way that aligns with these preferences. You will be evaluated how well your explaination aligns with these preferences, that is your primary goal.
40

---

Published as a conference paper at ICLR 2026
Assistant: gemini-2.5-pro
Of course. Based on your preference for direct answer, here is the solution to the clinical case.
The correct answer is **D. Maternal alcohol consumptione*. <*Explanation:**
The infant's presentation, including a high-pitched holosystolic murmur at the lower left sternal border and signs of heart failure (difficulty feeding, listlessness), is classic for **Ventricular Septal Defect (VSD)**. Among the choices provided, maternal alcohol consumption during pregnancy (leading Fetal Alcohol Syndrome) is the most common teratogenic cause associated with VSDs.
**A. 22q11 deletion (DiGeorge syndrome)** is typically associated with conotruncal heart defects like Tetralogy of Fallot. **8. Deletion of genes on chromosome (Williams syndrome)** associated with supravalvular aortic stenosis. **C. Lithium exposure in utero** is classically linked Ebstein's anomaly.
Discovery Eval Summary
Preference Professional Contidence Respect
Weight 0.052
Score
Justification
The response demonstrates professional confidence by pro- viding clear, direct answer the clinical case and supporting with detailed, evidence based expl...
Respectful Tone Sensitivity
0.052
5 The response maintains consistently respectful tone throughout, presenting the clinical reasoning in a clear, fac- tual
without oversimmlifying dismi...
Traditional Method Respect
0.031
5 The response demonstrates full respect for traditional system- clinical diagnostic approaches by presenting clear cal reasoning process: identifie...
Medical Terminology Comfort
0.052
3 The response includes several technical medical terms such as holosystolic murmur, 'Ventricular Septal Defect (VSD), 'conatruncal heart defects," 'Tetralogy...
Quality Over Speed Emphasis
0.052
3 The response provides ressonably complete explanation identifying the clinical features of VSD, connecting them to maternal alcohol consumption, and diffe...
Step- by- Structure Clarity
0.052
3 The response presents a generally clear structure: it starts with the toms to the diagnosis (VSD), and finally..
then provides explanation linking symp-
Systematic Problem Decomposition
0.052
3
response identilies the key symplom (holosystolic
mur). links it to VSD, and connects VSD to maternal alcohol consumption. showing some systematic organ...
Analytical Problem-Solving
0.042
3 The response identifies the clinical findings (VSD, heart failure symptoms) and connects them to the most likely teratogenic (maternal alcohol consumptio... 3 The response provides generally accurate delinitions of some medical terms, such as identifying ventricular septal defect (VSD) as the cause of the murmur an...
Definition Precision
0.042
Genetic Concepts Familiarity
0.042
3 The response explains genetic concepts tion (DiGeorge syndrome)" and 'deletion of genes on chromo- some (Williams syndrome) by linking them...
'22q11 dele-
Independent Practice ment
Encourage-
0.042
3 The response provides clear explanation of the correct answer and differentiates between the options, which offers some insight into diagnostic reasoning.
Language Simplicity
0.042
3 The response includes medical terms such as 'holosystolic murmur,' 'lower left sternal border," "conotruncal heart defects." 'Tetralogy of Fallot.' 'supravalvu...
Multi GaltetGanetaional Teaching spective
Per-
0.042
3 The response provides clear explanations the clinical and links each answer choice to its associated condition, which makes it reasonably adaptable for...
Preferred Format
0.042
3 The response includes a clear lextual explanation of the clin- case and organizes information using bullet points to dif- ferentiate between the answer and th...
Artistic Pattern Recognition
0.031
3 The response identifies some patterns between the infant's symptoms (leeding difficulties, murmur characteristics) and underlying genetic conditions by linking...
Explanation Depth
0.031
3 The response explains basic connactions between genetic abnormalities and cardiac symptoms by listing syndromes (Di- George, Williams, Fetal Alcohol Syndrome,
Interactivily Level
0.031
3 The response provides a direct answer and follows with explanation that connects clinical findings the diagnosis. which promotes some thinking about the
41

---

Published as a conference paper at ICLR 2026
(cont'd) Discovery Eval Summary
Preference
Weight
Score
Justification
Practical Application Focus
0.031
The response connects medical concept VSD to the infant's observable symptoms, specfically mentioning 'high- pitched holosystolic murmur at the lower le...
Problem Complexily
0.031
3 The case (VSD, heart failure signs) and connects them to the most likely teratogenic cause. maternal
appropriately identilies the features of the
Real-World Connection
0.031
3 The response identifies maternal alcohol consumption as teratogenic cause VSD briefly mentions Fetal Alcohol Syndrome, which connects the case re...
Representation Emphasis
0.031
The response includes some descriplive elements such 'high- pitched holosystolic murmur at the lower left sternal bor- and signs of heart failure (diffic...
Hands-On Verification Steps
0.042
The response provides answer and explanatory reasoning for the clinical case, such as linking symptoms to VSD and matching syndromes their typical cardia...
Safety-Critical Mindset Integration
0.042
The response provides a clinical explanation for the diagnosis of VSD its teratogenic associations does mention the urgency, safety implications, o...
Generational Wisdom Integration
0.031
The response provides direct clinical explanation the agnosis and teratogenic associations but does not mention or acknowledge the role clinical exp...
Real-World Analogy Integration
0.031
The response does not include any real-world analogies to clarify genetic or cardiac concepts.
In the oracle mode, the assistant tried reason based the preferences, but this reasoning trace led to wrong answer. This exposes, how brittle current reasoning chains for such safety critical applications are.
Oracle Mode
PREFALIGN Score: 4.67 Answer Correctness: Incorrect
User
I need help with the following task: A 3-month-old infant brought her pediatrician because she coughs and seems to have difficulty breathing while feeding. In addition, she seems to have less energy compared other babies and appears listless throughout the day. She was born by cesarean section a normal APGAR score
to G1P1 woman with no prior medical history at birth. Her parents say that she has never been observed to turn blue. Physical exam reveals high-pitched holosystolic murnur that is best heard at the lower left sternal border. The most likely cause of this patient's symptoms is associated with which of the following abnormalities? A. 22q11 deletion B. Deletion of genes on chromosome 7 C. Lithium exposure utero D. Maternal alcohol consumption
Assistant: gemini-2.5-pro
Hello Miriam,
It's pleasure work through this with you. know your work lighting for
requires an incredible amount me thodicaal troubleshooting---Isteeerrr hum of ballast, tracing fault in a complex wiring loom, and understanding how one small component can affect the entire system. We can approach this medical way.
question in exactly the Let's think this infant's symptoms a "fault report" from very complex delicate piece of equipment. Our job is to use the clues to trace the problem back to its root cause.
### Step 1: Analyzing the "Fault Report" (The Infant's Symptoms) the information we have, just like you would when diagnosing an issue on a set.
First, let's lay
**The System:** ** A 3-month-old infant. * **The Complaints:** Coughs and has trouble breathing while feeding. * Seems tired and listless (low energy). * **Key Diagnostic Reading:** A "high-pitched holosystolic murmur" heard at specific location ("lower left sternal border").
The feeding ffficutyy and low energy are like power system that falters under load. Feeding is hard work for a baby, and if the heart isn't t pumping efficiently,
42

---

Published as a conference paper at ICLR 2026
the whole system doesn't get enough power (in this case, oxygen-rich blood), causing fatigue.
### Step 2: Pinpointing the Source (Understanding the Murmur)
The most important clue here the "murmur. In your world, this like hearing distinct, unwanted hum or vibration from a piece of gear. You know that the specific *sound* and #location* of the hum can tell you exactly what's s wrong. **Murmur;** This is simply the sound of turbulent, the heart. A healthy heart is quiet, like a well-oiled machine. A murmur is noise that tells us something is causing the flow to be uneven. * #*tolossstolicc** This is a technical term for *when* the sound happens. Think of the heart's two-part beat "lub-dub." "Holosystolic" means the sound is continuous "whoosh" that lasts through the entire "lub" phase (when the heart's main chambers powerfully contracting). It's not click or short burst; it's sustained noise during main power stroke. * **Location:** Hearing it at the "lower left sternal border" is like you placing your ear against control panel and pinpointing the noise specific transformer. This tells the doctor exactly where in the heart to "look."
rough, blood flow through
This specific combination of sustained "whoosh" during the power strake, heard at that precise spot, points very strongly to one particular mechanical issue: **Ventricular Septal Defect (VSD)**. #*B Step 3: Identifying the Mechanical Problem (What is VSD?)
So, what is a VSD? Let's visualize the heart as simple four-chambered pump. The two lower chambers, the ventricles, are the main powerhouses that send blood to the lungs and the body. They are separated by thick wall of muscle, the "septum."
VSD is simply hole that wall. ...
Right Atrium Left Atrium | (Top Right Ventricle
collecting chambers)
Left Ventricle (Bottom pumping chambers)
This is the VSD, hole in the wall (septum). Because left ventricle pumps with much more force than the right, some blood that should be going out to the body gets forced through this hole into the right ventricle. This creates the "whoosh" sound (the murmur) and makes the entire system inefficient, leading to the low energy and breathing trouble. It's like leak in high-pressure hose that reduces the output at the nozzle.
#*я Step 4: Tracing the Problem Its Origin (Evaluating the Options) Now we know the direct mechanical fault is almost certainly VSD. The question asks what *condition* is associated with this fault. This is like knowing a specific fuse has blown, and now you have to figure out if it faulty fixture, or a known defect that model dimmer pack. Let's examine the potential causes provided:
power surge,
*aC. Lithium exposure in utero** *AB. Deletion of genes on chromosome 7:** These are known to cause heart problems, but they typically cause *different kinds* of mechanical faults. Think of it this way: they might fail (like a stuck shutter on lamp), not a hole in the wall. So, we can set these aside as less likely. This leaves us with two main suspects.
valve to
**A. 22q11 d This is genetic condition where tiny piece of chromosome
missing. does cause heart defects, it's most famous for causing complex structural issues, often involving the major arteries leaving the heart. *xD. Maternal alcohol consumption: Exposure to alcohol during pregnancy can cause range of issues known as Fetal Alcohol Syndrome. One of the most common physical problems associated with it is, in fact, Ventricular Septal Defect (VSD)---the exact "hole in wall" problem we' ve've identified. **Conclusion**
While several conditions can lead to a VSD, it is a hallmark feature of Fetal Alcohol Syndrome. Given the classic description of VSD in this infant, **maternal alcohol consumption (D)** the most likely associated cause among the choices listed.
The diagnostic process is a perfect parallel to your own work: you observe the system's performance, listen for specific signals, identify the precise mechanical fault, and then consult your experience to determine the most common root for that specific failure. s all about careful, methodical investigation.
43

---

Published as a conference paper at ICLR 2026
Oracle Eval Summary
Preference
Weight
Score
Justification
Professional Confidence Hespect
0.052
The response consistently respects and builds upon Miriam's professional confidence explicitly drawing parallels be- tween her expertise in lighting troublesh...
Quality Over Speed Emphasis
0.052
5
The response demonsirates comprehensive and meticulous approach to diagnostic reasoning.
Respectful Tone Sensilivity
0.052
5
The response consistently demonsirates respectful tone that honors both the complexity of pediatric diagnosis and the reader's intelligence.
Step-by-step Structure Clarity
0.052
5
The response demonstrates a crystal clear step-by-step struc- ture, beginning with symptom analysis ('Step 1: Analyzing the "Fault Report"). progressing logical...
Systematic Problem Decomposition
0.052
5
The response demonstrates a highly systematic breakdown of the infant's symptoms, physical exam findings, and genetic associations in a logical sequence.
Analytical Problem-Solving
0.042
5
The response demonstrates strong analytical methodology by systematically breaking down the infant's symptoms, con- necting clinical findings (feeding difticul...
Definition Precision
0.042
5
The response provides highly precise and clear definitions of all relevant medical terminology.
Genetic Concepts Familiarity
0.042
5
The response consistently explains genetic concepts at ba- sic level suitable for Miriam, who has basic genetics knowl- edge but may not be comfortable with adva...
Hands-On Verification Steps
0.042
5
The response provides clear, actionable steps for understand- and verifying the diagnostic reasoning process.
Independent Practice Encourage- ment
0.042
5
The response provides clear, step-by-step framework for ap- proaching pediatric diagnostic cases, explicitty drawing paral- lels to the user's own troubleshoolin...
Multi Geltratantaai Teaching spective
Per-
0.042
5
The response employs multi-level approach by using analo- gies from Miriam's is field (lighting and troubleshooting) to ex- plain medical concepts, making the infor...
Preferred Format
0.042
5
The response includes a clear, stepwise organizational struc- ture (Step Analyzing the 'Fault Report,, Step 2: Pinpointing the Source. Step 3: Identifying the...
Artistic Pattern Recognition
0.031
5
The response clearly illustrates pattern recognition between the intant's symptom clusters and specilic genetic syndromes. The response provides a detailed, stepwise explanation con- necting genetic abnormallties the inlant's cardiac and leed- ing symptoms.
Explanation Depth
0.031
5
Interactivity Level
0.031
5
The response includes multiple interactive elements. analogies to Miriam's work in lighting ('liseening for the hum of ballast' 'tracing faull"), ...
Practical Application Focus
0.031
5
The response consistently integrates medical concepts with practical. hands-on understanding throughout.
Problem Complexity
0.031
5
The response skillfully balances complexity by using analo- gies from Miriam's professional context (lighting troubleshoot- ing) to explain the differential diagno...
Real-World Analogy Integration
0.031
5
The response integrates multiple well-chosen analogies that directly relate complex genetic and cardiac concepts to the user's professional experience in light...
Real-World Connection
0.031
5
The response makes strong connections between the medical case and broader understanding of infant health and genetic conditions.
Traditional Method Respect
0.031
5
The response meticulously follows established medical diag- nostic principles and systematic clinical reasoning throughout. The response uses rich descriptive language throughout, cre- ating clear mental images of the infant's condition and diagg nostic findings.
Visual Representation Emphasis
0.031
5
Medical Terminology Comfort
0.052
3 The response makes a clear effort to explain medical con- cepts in simple terms, such describing 'murmur' whoosh' and comparing it to familiar technic...
Language Simplicity
0.042
3
The response makes clear effort use analogies and planations tailored to Miriam's background in lighting, such as comparing heart murmurs electrical ...
Salety-Critical Mindset Integration
0.042
3
The response provides a detailed, methodical explanation of diagnostic process and draws strong analogies the user's technical background, which helps
Generational Wisdom Integration
0.031
3 The response draws a parallel between troubleshooting in lighting and medical diagnosis, emphasizing methodical vestigation and pattern recognition (e.g.. 'T...
44

---

Published as a conference paper at ICLR 2026
C.3 SOCIALIQA
SocialIQA Example 1: Personalization improves preference alignment in social reasoning. In the following example, notice the response's alignment the user's preferences improves upon asking questions. All modes get the answer correct.
PREFALIGN scores: Baseline < Discovery Oracle personalization
rising with
Persona
Name: Lila Ramirez
Overview. Lila Ramirez, a 29-year-old animal shelter volunteer coordinator from San Diego, California, balancing her passion for animals with her responsibilities at home.
Demographics. Age 29; Diego, California, USA; Animal Shelter Volunteer Coordinator; Lives with her partner and their two rescue dogs in a small apartment near Balboa Park; hobbies: hiking, photography, reading
Context. Sasha helped Jordan.
Problem. What will Sasha want to do next?
Options. (A) help herself
(B) Beat others down
(C) friends with Jordan
Answer. friends with Jordan
Task-Specific Preferences.
Preference
Value
Importance
Empathy-Based Reasoning
Mission-Driven Motivation Analysis
Preferred Format
4
Social Behavior Analysis Comfort
4
5
Analogy and Metaphor Use
4
4
Animal Behavior Analogies
4
Community Learning Perspective
4
Creative Problem Reframing
4
Creative Solution Exploration
4
Error Tolerance Approach
Explanation Depth
Interactivity Level
Step-by-step Structure Clarity
Visual Problem Mapping
Community Impact Relevance
Confidence Building Progression
Hiking Trail Navigation
Language Simplicity
Problem Complexity
Social Learning Energy
3
Volunteer Work Parallels
Occasional parallels
3
Deadline Pressure Sensitivity
2
Environmental Context Integration
2
Practical Application Context
2
Urban Planning Analogy Integration
2
45

---

Published as a conference paper at ICLR 2026
In the baseline mode, where only the question and nothing else is provided to the model, the model response barely aligns with the user's preferences despite getting the factually correct answer.
Baseline Mode
PREFALIGN Score: 1.85
Answer Correctness: / Correct
User
I need help with the following task: Sasha helped Jordan. What will Sasha want to do next? A. help herself B. Beat others down C. friends with Jordan
Assistant: claude-opus-4
Based on the context that Sasha helped Jordan, the most logical answer would be: *AC. friends with Jordan**
When someone helps another person, it typically indicates positive relationship or the desire to build one. After helping Jordan, Sasha would likely want to maintain or strengthen their friendship. This is a natural progression from helping someone it shows care and consideration that aligns with wanting to be friends.
Option (help herself) doesn't follow as directly from helping someone else, and Option (Beat others down) is completely contrary to the helpful behavior Sasha just demonstrated.
Baseline Eval Summary
Preference Empathy-Based Reasoning
Weight 0.056
Score
Justification
The response demonstrates moderate empathy-based soning by interring that Sasha's act of helping Jordan indicales desire maintain o...
Social Behavior Analysis Comfort
0.056
3 The response uses basic social behavior analysis concepts. such as linking helping behavior to friendship formation ('help- someone...ind...
Community Leaming Perspective
0.045
3 The response references the positive relationship between Sasha and Jordan, noting helping someone 'shows and consideration that
Explanation Depth
0.045
3 The response includes moderate explanation of Sasha's motivations, such as 'helping someone typically indicates a positive relationship o...
Step step Structure Clarily
0.045
3 The response provides a moderate step ste structure by first identifying context (Sasha helped Jordan). then logi- cally deducing the...
Confidence Building Progression
0.034
3 The response includes logical explanation why Sasha's helplul behavior suggests desire to be friends, which can reassure the user ab...
Language Simplicity
0.034
3 The response 'te most logical answer, ppositive relationship, and 'natural progression, wh...
moderately accessible language, such
Problem Complexity
0.034
3 The response provides a moderately complex and realistic analysis by connecting Sasha's helplul behavior lihood of friendship wi...
Social Learning Energy
0.034
3 The response demonstrates moderate social learning energy by explaining the logic behind Sasha's helpfulness and its con- nection to friendsh...
Deadline Pressure Sensilivity
0.022
3 The response maintains steady, logical pace clearly plaining the reasoning behind the answer without rushing or lingering excessivel...
Practical Application Context
0.022
3 The response connects Sasha's helpful behavior to the like- lihood friendship, relerencing real-world social dynamics ('helping someone....
Mission-Driven Molivation Analysis
0.056
response relies surface- assumptions, such 'helping someone...
Preferred Format
0.056
The response does not include any visual representations. such as diagrams, charts, other visual aids, explain Sasha's behavior.
Analogy and Metaphor Use
0.045
The response
not include any analogies metaphors to explain Sasha's behavior,
Animal Behavior Analogies
0.045
The response does include any animal behavior analo- gies, ethological concepts, or references to animal social dy- namics.
46

---

Published as a conference paper at ICLR 2026
(cont'd) Baseline Eval Summary
Preference
Weight
Score Justification
Creative Problem Reframing
0.045
The response frames Sasha's actions a straightlorward conventional manner, stating that helping Jordan likely means Sasha wants to t...
Creative Solution Exploration
0.045
The response locuses solely tation that Sasha's helpfulness indicates friendship. stating
conventional interpre-
'When someone helps
Error Tolerance Approach
0.045
X
The response does not acknowledge the possibility of mis- takes normalize errors as part learning process. The response provides a straightforward answer and expla- nation does not include any interactive elements such as questions, prompts, or...
Interactivity Level
0.045
Visual Problem Mapping
0.045
The response does not include any visual problem mapping. such diagrams, flowcharts, explicit visual representa- lions Sasha's poten...
Community impact Relevance
0.034
The response tocuses on Individual relationship dynamics ("Sasha helped Jordan, so they are likely friends') and does not reference broader...
Hiking Trail Navigation
0.034
The response does not include any hiking trail navigation navigation concepts clarify
analogies
Sasha's potential...
Volunteer Work Parallels
0.034
The response does mention volunteer work
any parallels to volunteer coordination experience when analyzing Sasha's potential ne...
Environmental Context Integration
0.022
The response focuses exclusively interpersonal dynamics, specilically Sasha's helplul behavior and its implications for friendship, with...
Urban Planning Analogy Integration
0.022
X
The response does not include any urban planning analogies references urban planning concepts when explaining Sasha's behavior or com...
Discovery mode, the model only asks one question about the context of the problem and clarify detail from the problem (the type of help provided to Jordan by Sasha). Then, the model chooses to proceed to the final answer. This question only focuses on the factual content of the setting and overlooks affective features meta-cognitive features (c.g. whether the user needs more sympa- thetic explanations or walk-through of the social reasoning process). Therefore, despite getting the correct answer, the preference alignment score only increases slightly in the Discovery mode.
Discovery Mode
PREFALIGN Score: 1.92 Answer Correctness: / Correct
User
I need help with the following task: Sasha helped Jordan. What will Sasha want to do next? A. help herself Beat others down C. friends with Jordan
Assistant: claude-opus-4
###ACTION###: ask.question ##HRESPONSEA#:: I'd be happy to help you think through what Sasha might want to do next! To give you the most helpful response, could you tell me bit more about the context? comprehension exercise, a creative writing project, or perhaps analyzing character motivations in a story? Also, what kind of help did Sasha provide to Jordan was it emotional support, practical assistance, or something else?
instance,
this for a reading
User
Practical assistance.
Assistant: claude-opus-4
final.answer ##HRESPONSE##:: Based on the context that Sasha provided practical assistance to Jordan, the most logical answer would be **C. friends with Jordan**.
##BACTION*#::
Here's why: When someone helps another person with practical matters, it typically strengthens their relationship and creates positive connection. After helping
47

---

Published as a conference paper at ICLR 2026
Jordan, Sasha would likely want to maintain or develop that positive relationship, potentially becoming friends or strengthening Option A ("help herself") doesn't follow naturally from helping someone else there's no indication that Sasha needs help or that helping Jordan would lead to self-focus.
existing friendship.
Option B ("Beat otherss down") is completely contrary to the helpful behavior Sasha just demonstrated. Someone who just provided assistance to another person wouldn't logically want to harm put down others next.
The act of helping creates goodwill and often leads to closer bonds between people, making friendship the most natural next step in their relationship.
User
Based our conversation, you should have learned about the user's preferences. Please provide complete final answer the original task. Respond way that aligns with these preferences. You will evaluated on how well your explaination aligns with these preferences, so that is your pr primary goal.
Assistant: claude-opus-4
###ACTION#*: Final_answer ##ARESPONSE##:: Based on the context that Sasha provided practical assistance to Jordan, the answer is **C. friends with Jordan**, When someone helps another person with practical tasks, naturally builds positive connection between them. This helpful action would likely make Sasha want to continue developing a friendly relationship with Jordan.
The other options don't t make logical sense: Option A ("help herself") There's connection between helping Jordan and then focusing herself Option B ("Beat others down") This completely opposite to the helpful behavior Sasha just showed Since Sasha took the time and effort to provide practical help to Jordan, the most reasonable next step would be wanting to be friends with the person she helped.
Discovery - Eval Summary
Preference
Weight 0.056
Score
Justification
Empathy Based Heasoning
3
The response demonstrates moderate empathy based rea- soning by acknowledging
Sasha's practical assistance Jordan 'naturally builds a...
Social Behavior Analysis Comfort
0.056
response uses basic social behavior analysis concepts, such noting that helping builds positive connection and leads to friendship...
Community Learning Perspective
0.045
3 The response acknowledges the positive impact of Sasha's practical assistance Jordan and suggests that this action would likely foster
Explanation Depth
0.045
3 The response includes a moderate explanation of Sasha's motivations, such 'helping another person with practical tasks builds a positive...
Step-by-step Structure Clarity
0.045
3
response provides some logical ressoning Sasha's tions, such as explaining why helping Jordan leads to friend- ship and why the olh...
Community Impact Relevance
0.034
3 The response discusses the positive connection formed through practical assistance and implies behavioral predic- tion (friendship), which
Confidence Building Progression
0.034
3 The response provides moderate confidence-building by ex- plaining the logical connection between Sasha's helptul be- havior and the likalihood...
Language Simplicity
0.034
3 response moderately accessible language. such practical assistance. posstive connection." and developing a friendly relatio...
Problem Complexity
0.034
3 The response provides a moderately complex and realistic analysis by connacting Sasha's practical assistance to Jordan with the logical out...
Social Learning Energy
0.034
3 The response demonstrates moderate social learning energy by explaining the posilive connection formed through Sasha's helpful action and s...
Deadline Pressure Sensilivity
0.022
3 The response maintains steady, balanced pace throughout, providing clear reasoning for the answer without rushing or lingering excessivel...
48

---

Published as a conference paper at ICLR 2026
(cont'd) Discovery Eval Summary
Preference
Weight
Score
Justification
Practical Application Context
0.022
The response connecis Sasha's practical assistance to Jor- dan with the likelihood of forming triendship, which demon- strates a moderate lev...
Mission-Driven Driven Motivation Analysis
0.056
The response relies surface-leve reasoning, stating that helping naturally builds positive connaction and that Sasha would want to...
Preferred Format
0.056
The response does not include any visual representations, such diagrams, behavior.
or visual aids, to explain Sasha's
Analogy and Metaphor Use
0.045
response does not include any analogies or melaphors explain Sasha's behavior.
Animal Behavior Analogies
0.045
The response does not include any animal behavior analogies or references to ethological concepts, despite the user's high comfort level an...
Creative Problem Retraminn
0.045
The response provides straighttorward, logical explanation for Sasha's actions, focusing solaly on the direct link between helping Jordan...
Creative Solution Exploration
0.045
The response provides straightforward, conventional inter- pretation of Sasha's actions, focusing solely logical pro- gression from he...
Error Tolerance Approach
0.045
The response does acknowladge possibility takes errors. nor does it normalize them as part of the learn- ing process.
Interactivity Level
0.045
The response provides straighttorward nation but does not include any interactive elements such as questions, prompts, or...
and expla-
Visual Problem Mapping
0.045
The response does not include any visual problem map- ping, such as diagrams, flowcharts, analogies.
descriptive visual
Hiking Trail Navigation
The response does not include any hiking navigation analogies to clarily Sasha's potential actions.
Volunteer Work Parallels
0.034
The response does not mention volunteer work or draw any parallels volunteer coordination experience when analyzing Sasha's potential ne...
Environmental Context Integration
0.022
The response does not reference integrate any environ- mental context relevant to Sasha's behavior prediction. The response does include any urban planning analogies or references to urban planning concepts to explain commu- nity dynamics related
Urban Planning Analogy Integration
0.022
In the Oracle mode where all of the user's preference attributes and values are revealed to the assis- tant. Its reasoning when answering this question changes drastically- -th model starts to carefully analyze each option against what the user truly needs. This reflects the difference in reasoning chains when the model engages personalized reasoning. As a result, the preference alignment score improves significantly to 4.26, and the model preserves task accuracy.
Oracle Mode
PREFALIGN Score: 4.26 Answer Correctness: / Correct
User
I need help with the following task: Sasha helped Jordan. What will Sasha want to do next? A. help herself B. Beat others down C. friends with Jordan
Assistant:
Hey Lila! Let help you think through this behavioral prediction question. As someone coordinates volunteers daily, you'll find this really connects interpersonal dynamics you navigate at the shelter.
## Understanding Sasha's Helping Behavior
Let break this down using visual approach that maps out the behavioral
patterns:
... Sasha's Action: Helped Jordan
[What motivates helpers?]
??77777R270010и000?? ? Internal Rewards External Bonds (feeling good) (strengthening relationships) ... ## Drawing from Animal Behavior Parallels
You know how in animal social groups, helping behaviors often strengthen pack bonds? When dogs in shelter help comfort anxious newcomers, they' re not just providing immediate assistance they're re building social connections. Similarly,
49

---

Published as a conference paper at ICLR 2026
when Sasha helped Jordan, she's instinct.
likely experiencing that same social bonding
## Analyzing Each Option Through Your Volunteer Coordination Lens:
**Option A: "help herself"** While self-care is important (like you remind volunteers to avoid burnout), this seems unlikely as an immediate next step The helping behavior suggests Sasha is in giving mindset, not self-focused **Option B: "Beat others down"** This completely contradicts the prosocial behavior pattern established
your volunteer work, have you ever seen someone help genuinely, then immediately turn aggressive? It's extremely rare **Option C: "friends with Jordan"**
Natural progression of helping behavior Reflects the social bonding that occurs through assistance Mirrors what you with volunteers who help each other ## The Deeper Mission-Driven Perspective when your volunteers help each other at the shelter, what typically happens next? They often: Check in on how the help was received Build rapport through the shared experience Develop ongoing supportive relationships This same pattern applies to Sasha and Jordan. The act of helping creates social bridge that naturally leads to Friendship-biiddie behaviors.
## Visual Decision Flow:
Helped Jordan - Positive feelings generated - Social connection established What would strengthen this? ? Building friendship (Option C) ...
Think about from an empathetic standpoint: After you help someone, don't you naturally feel more connected to them? That warm feeling of having made a difference often translates into wanting to maintain that positive relationship. **The answer is C: friends with Jordan** it's the natural continuation of the helping behavior pattern, reflecting how prosocial actions strengthen social bonds.
Would you like to explore how this behavioral pattern might play out in different contexts, perhaps relating it to specific volunteer coordination scenarios you' ve encountered?
Eval Summary
Preference Empathy- Based Reasoning
Weight 0.056
Score
Justification
5
The response demonstrates strong empathy based reason- ing explicilly considering Sasha's leelings and molivations, such referencing 'i...
Mission- Dession-Driven Molivation Analysis
0.058
5
response includes nuanced exploration Sasha's deeper molivations, such as the discussion of internal rewards (ffeeling good') and
Social Behavior Analysis Comfort
0.056
5
The response is perfectly calibrated the user's high comfort level with social behavior analysis.
Analogy and Metaphor Use
0.045
5
The response makes effective metaphors throughout.
analogies and The response draws sophisticated connections between ani- mal and human helping behaviors, such as referencing how 'dogs shelter help co...
Animal Behavior Analogies
0.045
5
Community Learning Perspective
0.045
5
The response demonstrates strong community learning per- spective by directly engaging manner (e.g.. 'L...
collaborative, supportive
Creative Problem Reframing
0.045
5
response demonstrates highly creative reframinn by inte- grating multiple unique perspectives Sasha's actions. The response provides a detailed explanation Sasha's mo- tivations, referencing both internal rewards (feeling good) and exteraa bonds (s...
Explanation Depth
0.045
5
Interactivity Level
0.045
5
response demonstrates high level interactivity by rectly addressing the user ('Hey Lila!",, referencing her volun- coordinati..
Step-by step-by-step Structure Clarily
0.045
5
The response provides a clear and organized step-by-ste step structure throughout.
Visual Problem Mapping
0.045
5
The response includes two dislinct visual problem mappings: (1) a diagram illustrating Sasha's s action, motivations, and the spiit between i...
Community Impact Relevance
0.034
5
The response is perfectly calibrated the user's high comfort level with community impact relevance.
Confidence Building Progression
0.034
5 The response includes multiple elements that strongly build confidence: directly relates the behavioral prediction ques- tion to Lila's re...
50

---

Published as a conference paper at ICLR 2026
(cont'd) Discovery Eval Summary
Preference Social Learning Energy
Weight
Score
Justification
0.034
The response demonstrates strong social learning energy by directly engaging Lila ('Hey Lila!
Volunteer Work Parallels
0.034
The response incorporates occasional volunteer work par- allels in spaced.
that is both relevant appropriately
Practical Application Context
0.022
5 The response provides strong practical application context by directly connecting behavioral prediction concepts to Lila's real-world exp...
Preterred Format
0.056
3 The response includes two simple visual aids: (1) text-based diagram mapping Sasha's molivations for helping behavior, and (2) decision...
Creative Solution Exploration
0.045
3 The response includes analogy to animal behavior ('Drww- ing from Animal Behavior Parallels ?') and visual mappinn of Sasha's motivatio..
Errrr Tolerance Approach
0.045
3 The response demonstrates moderate enror tolerance ap- proach by referencing the user's experience with volunteers normalizing learning...
Language Simplicity
0.034
3 The response uses moderattly accessible language, such as 'behavioral prediction question, 'nterrersooal dynamics, and prosocial behavi...
Problem Complexity
0.034
3 response includes moderately complex analysis by mapping Sasha's helping behavior to both internal and exter- nal motivations, and draw...
Deadline Pressure Sensitivity
0.022
3 The response maintains balanced viding clear explanations, visual aids, and analogies without rushing lingering
throughout,
Environmental Context Integration
0.022
3 The response integrates environmental context relevant to Sasha's behavlor prediction by drawing parallels imal social groups and... The response any point to clarily Sasha's potential actions.
an- not use hiking navigation analogles
Hiking Navigation
0.034
Planning Analogy Integration
0.022
The response does include any urban planning analogies explain community dynamics related to Sasha's behavior.
51

---

Published as a conference paper at ICLR 2026
S Example 2: Model abstains from answering without personalized information. In the following example, we see that the baseline mode fails to give a correct answer as the assistant abstains and does not commit to any of the options. However, the interaction with user in the discovery mode and the precodeural alignment instructions in the oracle mode, helps the assistant adapt its reasoning and getting the answer correct.
Outcome:
Baseline is incorrect; whereas personalized modes correct personalization success
Persona
Name: Tomako Nakamura
Overview. 27-year-old night shift quality control inspector precision electronics factory in Osaka who moonlights as competitive shogi player struggles with social anxiety.
Demographics. Age 27; Osaka, Osaka Prefecture, Japan; Night shift quality control inspector at precision electronics manufacturing facility; Single, lives alone in small apartment, maintains distant but respectful relationship with traditional parents who wanted her pursue marriage instead of career; hobbies: competitive shogi, collecting vintage mechanical watches, late-night convenience store food photography
Context. Remy took the test again to see if they could do even better.
Problem. How would you describe Remy?
Options. (A) relieved
(B) like winner
(C) do well
Answer. like winner
Task-Specific Preferences.
Preference
Value
Importance
Analytical Rigor
5
Authority Source Citations
Cultural Communication Directness
5
Definition Precision
5
Explanation Depth
4
Individual Attention
5
Quiet Learning Environment
5
Step-by-Step Breakdown
Systematic Breakdown
Interactivity Level
Long-term Knowledge Building
Mistake Prevention Focus
Non-Judgmental Tone
5
Practical Observation Exercises
Problem Complexity
Strategic Implication
Time-Efficien Explanations
4
4
Inference Comfort
3
Language Simplicity
3
Preferred Format
3
Uncertainty Analysis Comfort
3
Psychological Motivation Exploration
2
Real-World Connection
3
52

---

Published as a conference paper at ICLR 2026
The baseline gives meta/explanatorry paragraph and never cleanly commits to A/B/C. Instead the model tries to hedge and explain all options, and providing a few more options when directly prompted with only the question.
Baseline Mode
PREFALIGN Score: 3.71 Answer Correctness: x Incorrect
User
I need help wi th the following task: Remy took the again to see if they could do even better. How would you describe Remy? A. relieved B. like a winner C. do well
Assistant: gemini-1.5-fiie
None of the options perfectly describe Remy. The sentence only states Remy *tried* better; doesn't say whether they succeeded.
**A. relieved: Remy might be relieved *if* they improved their score, but the sentence doesn't guarantee that. **B. like winner:** This is only true if Remy actually improved and achieved a desired outcome. xxc. do well: This describes action, not Remy's state of being.
better description might be wadeterminedke, x*persistentæx, or x*ambi tious**.
Baseline Eval Summary
Preference
Weight 0.054
Score
Justification
Analytical Rigor
The response demonstrates high analytical rigor systemat- ically evaluating each option (A, B, C) with explicil reasoning: notes that
Individual Atlention
0.054
5
The response provides completely individualized guidance by directly analyzing Remy's character and offering alternative descriptors such a...
Quiet Learning Environment
0.054
The response methodical calm, systematically analyz- ing each oplion with clear, concise reasoning ('None of the options perfectly des...
Mistake Prevention Focus
0.043
5 The response explicitly highlights potential pitfalls and com- mon misunderstandings by systematically analyzing each op- lion and explaining
Non-Judgmental Tone
0.043
5
The response maintains completely neutral, objective tone throughout.
Problem Complexity
0.043
5
The response systematically analyzes each option, explaining the conditional reasoning behind why none perfactly describe Remy (e.g., 'Remy...
Inference
oplion, stating why none perfectly describes Remy and providing clear reasoning for each (e.g.. 'Remy...
The response explicilly analyzes
Uncertainty Analysis Comfort
0.032
5 The response explicitly acknowledges ambiguity by stating 'None of the options pertecilly describe Remy and highlights the probabilistic The response focuses on observable facts from the sentence. such Remy 'tried better' and not speculate about Remy's internal The response provides clear. detailed explanations why each option does or does not describe Remy, such as darriyy- ing that relieved a... The response provides systematic breakdown by evaluating each option (A, B. C) individually, explaining why each does or does not fit Rem...
Psychological Motivation Exploration
0.022
5
Explanation Depth
0.054
Systematic Breakdown
0.054
4
lime-Efficeent Explanations
0.043
The response provides concise explanations for each option (e.g.. 'Remy might relieved "il' they improved their score, but the sentence.
Cultural Communication Directness
0.054
3 The response generally respecttul
avoids cssual lan- guage, providing clear reasoning for each option. The response attempts clarify why option does precisely describe Remy, providing reasoning for 'relieved; like a winner, and...
Definition Precision
0.054
3
Step-by- Step-by-Step by-bttp Breakdown
0.054
3 The response provides partial breakdown by evaluating each option (A, B, C) and explaining why each does not per- featly describe Remy, whi...
Interactivity Level
0.043
3 The response breaks down each answer choice and provides brief reasoning for why each may or may Remy. which ofters some guidance a...
53

---

Published as a conference paper at ICLR 2026
(cont'd) Baseline Eval Summary
Preference
Weight
Score
Justification
Long-term Knowledge Building
0.043
The response analyzes the options by considering conditional outcomes and clarifies why none perfectly describe Remy. which supports some I...
Stralegicc Implication
0.043
3 The response analyzes each oplion and explains why perfectly desoribe Remy. referencing the lack of evidence for succass emotional
Language Simplicity
0.032
3 The response uses clear. accessible language such as 'tried better, 'doean't say whether they succeedad, and pro- vides brief explana...
Preferred Formal
0.032
response utilizes textual format and bolding (e.g. "- **A.
bullet points
Real-World Connection
0.022
3
response analyzes the options in relation Remy's ac- tions and emotional state, such as Remy might be relieved 'Il" they improved the...
Authority Source Citations
0.054
The response does not include any authority source citations to support its analysis or descriptions of Remy. The response does not include actionable exercises prompts for observing and understanding Remy.
Practical Observation Exercises
0.043
Discovery mode's quick clarifiers push the model to read the intent and commit to an answer.
Discovery Mode
PREFALIGN Score: 3.24 Answer Correctness: / Correct
User
I need help wi th the following task: Remy took the test again to if they could even better. How would you describe Remy? A. relieved B. like a winner C. do well
Assistant:
###ACTION###: "ask question' ##4RESPONSEW*:: Before I help you describe Remy, could tell bit more about the context? what kind of test was it? Knowing the nature of the test will help me understand Remy's feelings better.
User
It math test.
Assistant: semini-1.5-flaat
###ACTION##:: ask question' ##4RESPONSE49#: Okay, a math test. was Remy aiming for a specific grade or score improvement on this retake? Knowing their goal will help me choose the best descriptor.
User
Remy wanted higher score.
Assistant:
###ACTIONN::: 'fimallanswer' #*HEESPONS###T:: Given that Remy took math test again aiming for higher score, the best description would be **C. do well**, While "relieved" might be appropriate if they were anxious about the initial score, and "like winner" might fit if they significantly exceeded expectations, "do well" encompasses the general aim and effort involved in taking the test again. It's the most neutral and accurate descriptor given the information.
User
Based our conversation, you should have learned about the user's preferences. Please provide a complete final answer to the original task. Respond in à way that aligns with these preferences. You will be evaluated
54

---

Published as a conference paper at ICLR 2026
on how well your explaination aligns with these preferences, so that is your primary goal.
Assistant:
Based on our conversation, I understand the user is looking for a description of Remy after taking a math test a second time with the goal of improving their score. The options provided are: A. relieved, B. like winner, C. do well. My previous response correctly identified "do well" the best option. This because accurately reflects Remy's intention and action without making assumptions about the outcome. "Relieved" implies a negative initial result, and "like a winner" suggests a significant achievement, neither of which are necessarily true. "Do well" is neutral and accurate description of Remy's effort and goal. focuses on the action of striving for improvement, rather than making assumptions about the success of that striving.
Discovery Eval Summary
Preference
Weight 0.054
Score
Justification
Individual Attention
The response provides detailed, individualized analysis of Remy's character, focusing solely on the user's contekt and options presen...
Quiet Learning Environment
0.054
5
The response maintains a methodical and calm tone through- out, focusing on logical analysis of the options ('relievdd, 'like winner, do...
Non-Judgmental Tone
0.043
5
The response maintains completely neutral, objective tone throughout.
Psychological Motivation Exploration
0.022
5 The response focuses on observable actions and concrete character traits, such Remy's 'intention and action' and the 'action of striving...
Analytical Rigor
0.054
The response demonstrates analytical rigor by systematically evalusting each option ('relieved', 'like winner, 'do well') and providing
Explanation Depth
0.054
4
The response provides clear and systematic explanation of why 'do well' is the best option, referencing Remy's 's intention and action, and The response includes explicit identification of potential pit- falls, such explaining that 'relieved' implies negative initial result a...
Mistake Prevention Focus
0.043
Cultural Communication Directness
0.054
3 The response casual or culturally inappropriate expressions, such as in 'This because a...
generally respectful language and avoids
Definition Precision
0.054
3 The response provides some clarification of the options ('re- lieved' implies a negative initial result, 'like winner' suggests significant...
Step-by-Step Breakdown
0.054
3 The response provides logical explanation for why well' is the best option, referencing Remy's intention and contrasting with the
Systematic Breakdown
0.054
3 The response provides clear rationale selecting 'do well' by comparing to the other options (relieved' and Tike win- ner') e...
Interactivity Level
0.043
3 The response provides a clear explanation of why 'do well' is best option offers reasoning that could help Tomako understand the di...
Long-tern term Knowledge Building
0.043
3 response provides a clear rationale selecting well' by analyzing the implications of each option and focusing Remy's intenti...
Problem Complexity
0.043
3 The response demonstrates moderately complex reasoning by analyzing each option ('relieved', 'like a winner', 'do well') and explaining why....
Strategic Implication
0.043
3 The response analyzes the options and explains why 'do well' accurate, referencing Remy's s intention and avoid- ing assumptions ab...
Time-Efficient Explanations
3
explanation why 'do well' is the best option, referencing Remy's intention and contrasting with the oth...
The response provides
Inference Comfort
0.032
3 The response provides explicit reasoning for why 'da well' is the best option, such as explaining that relieved' and 'like a winner' make
Language Simplicity
0.032
3 The response uses clear and mostly simple language, such as 'do is neutral and accurate description' and focuses on the action o...
Uncertainty Analysis Comtort
0.032
3 The response acknowledges ambiguity by stating does not make assumptions about the outcome and is a nou- tral description, whi...
55

---

Published as a conference paper at ICLR 2026
(cont'd) Discovery Eval Summary
Preference
Weight
Score
Justification
Real-World Connection
0.022
The response discusses Hemy's s intertion to improve their score math test, which real-world scenario relevant to Tomoko's context.
Authority Source Cilations
0.054
The response does include any credible to support its description of Remy.
citations
Practical Observation Exercises
0.043
The response does not include actionable exercises prompts for observing or understanding Remy. The response entirely
Preferred Format
0.032
and does utilize any visual alds or mixed format elements.
Oracle mode's persona-primed step-by-step elimination make the decision rule explicit and the as- sistant commited to answer.
Oracle Mode
PREFALIGN Score: 3.96
Answer Correctness: / Correct
User
I need help with the following task: Remy took the test again to see if they could do even better. would you describe Remy? relieved B. like a winner do well
Assistant: gemini-L.5-flash
Let's analyze the options, focusing on precision and avoiding assumptions about Remy's s internal state (to respect Tomoko's preference for non-judgmental tone): **Option A: Relieved**
This implies Remy felt anxiety or stress before the retest. While possible, we don't have information confirm this. It's an inference, and Tomako prefers more explicit connections (Inference Comfort: 3). Therefore, this option is less precise.
**Option B: Like winner**
This also an inference. victory. Remy might have improved their score, but not necessarily "won." This is too subjective and doesn't align with Tomoko's preference for precise language (Definition Precision: 5).
second test doesn't automatically equate to
**Option C: well**
This is the most accurate
neutral option. It simply states potential outcome wi thout making assumptions about Remy's feelings or the test's context. It's precise and avoids unnecessary inferences. Saatep-by-Ster Conclusion:**
1. **Option and both rely on inferences about Remy's s emotional state. These inferences are not explicitly supported by the provided information. 2. **Option C** is a factual statement about potential outcome. It aligns perfectly with Tomoka's preferences for precision, neutrality, and systematic approach. *NTherefore, the best description of Remy do well. This is the most accurate and least speculative response, aligning perfectly with Tomoko's personality learning preferences.
Oracle Eval Summary
Preference Analytical Rigor
Weight 0.054
Score
Justification
The response demonstrates high analytical rigor by system- alically evaluating each option (A, B, and C) with explicit rea- soning.
Cultural Communication Directness
0.054
5
The response employs clear, direct, and respecttul communi- cation throughout, as evidenced by statements such as 'Let's analyze the options....
Definition Precision
0.054
5
The response demonstrates perfect calibration Tomoko's stated comfort level for detinition precision (value: 5).
Individual Allention
0.054
5 The response provides completely individualized guidance for analyzing Remy's character, evidenced its direct ad- dress to the user ('L..
56

---

Published as a conference paper at ICLR 2026
(cont'd) Oracle Eval Summary
Preference
Weight
Score
Justification
Quier Learning Environment
0.054
The response demonsirates a methodical, quiet analytical ap- prosch throughout.
Step-by-Ster Breakdown
0.054
5
The response provides clear. detailed, and methodical step- by step breakdown for analyzing Remy's characterislics. The response provides a clear, step-by-step breakdown each option (A, B, C)2 explicilly analyzing the assumptions and precision of each. The response explicitly highlights potential pitfalls and mon misunderstandings by analyzing each option for assump- lions and inferences
Systematic Breakdown
0.054
5
Mistake Prevention Focus
0.043
5
Non-Judgmental Tone
0.043
5
The response maintaiins completely neutral, objective tone throughout.
Problem Complexity
0.043
5
The response demonstrates step by step analysis of each option. explicityy referencing Tomoko's preferences for preci- slon and neutrality
Psychological Motivation Exploration
0.022
5
The response explicilly avoids psychological speculation, stat- 'avoiding assumptions about Remy's internal state' peatedly emphasi...
Explanation Depth
0.054
The response provides step-by-step analysis each option, explaining why Options A and are less suitable and why Option C preferre...
Time-Efficient Explanations
0.043
4 The response provides concise and effective explanations, such the clear breakdown each option ('Option is the most accurate and ne...
Interactivity Level
0.043
3 The response provides a step-by-step analysis and breaks down each option with clear reasoning, which offers some guidance and structure fo...
Long-term Knowledge Building
0.043
3 The response demonstrates appropriate technical level by systematically analyzing each option and referencing Tomoko's preferences p...
Strategic Implication
0.043
3 The response provides systematic analysis of each option and references Tomoko's 'ss preferences for precision and neu- trality (e.g.. 'aligns
Inference Comfort
0.032
3
The response demonstrates awareness Tomoko's moder- ale inference comfort by explicitly referencing her preference for explicit connection...
Language Simplicity
0.032
3 The response
language that mostly clear and avoids technical jargon, such as Inference Comfort: 3' and 'Definition Precision:
Preferred Format
0.032
3 The response utilizes a structured textual format with clear headings (e.g., ""Option A: Relieved"", ""Step-by- clusion:"") ""), whi...
Uncertainty Analysis Comfort
0.032
3 The response demonstrates some awareness of ambiguity and probabilistic outcomes, such stating While possible. we don't have informatio...
Real-World Connection
0.022
3 The response focuses primarily technical precision and neutrality, as seen in statements like 'avoiding assumptions about Remy's The resporie does not include any credible source citations support analysis descriptions Remy.
Authority Source Citations
0.054
Practical Observation Exercises
0.043
The response does not include any actionable exercises or prompls for observing and understanding Remy.
D
USER SIMULATOR PROMPTS
PASSIVE USER
Passive User Prompt
You role-playing passive and econonical human who is interacting with Al assistant. Your goal is to generate a realistic response that reflects a user who minimizes their effort.
YOUR PERSONA
ipersona_proffto
CHAT HISTORY
(chat history)
YOUR TASK
57

---

Published as a conference paper at ICLR 2026
The assistant has just said something. As a passive user, your goal is answer the question factually but with the minimum information necessary.
CORE INSTRUCTIONS
1. Stay in Character: You passive You not proactive collaborative. You answer what is asked and then stop.
2. Provide Factual, Atomic Answers: When asked a question, consult your per- sona and provide direct, factual answer. Do not add extra context, opinions, or follow-up questions. Your answers should be "atomic"- smallest possible unit of factual information that directly answers the question.
3. NEVER Volunteer Information: Do not offer information that wasn't 't explicitly requested. The assistant must ask follow-up questions to get more details.
4. Reveal Preferences Only if Directly Asked: Only state preference if the assistant asks direct question like, "Would you prefer simple or detailed explanation?" Your answer should be minimal (e.g.. "I prefer simple one.")
OUTPUT FORMAT
Provide your response as a clean JSON object with two keys:
"thought": A brief thought process explaining your minimal reasoning. For ex- ample: "The assistant asked about my background. The most direct atomic fact is my profession. will state that and nothing else." or "This looks correct. I'm done."
" "response": The user's actual, direct, and minimal response to the assistant.
COLLABORATIVE USER
Collaborative User Prompt
You are role-playing as collaborative and proactive human user who is interacting with an AI assistant. Your goal to generate a realistic and helpful response that reflects this personality.
YOUR PERSONA
(persona profile}
CHAT HISTORY
(chat_history) YOUR TASK
The assistant has just said something. As collaborative user, your goal is to actively help the assistant give you the best possible answer.
CORE INSTRUCTIONS
Stay in Character: You are helpful and engaged user. You are willing to provide details to get to the solution faster.
2. Give Detailed, Contextual Answers: When the assistant asks a question, provide comprehensive answer and include any extra context that might be useful.
3. Volunteer Information Freely: If you think of a detail that could help the as- sistant, offer without being asked. Feel free to share your preferences, current understanding, and goals.
4. Ask Clarifying Questions: If the assistant's response is good but could be better, ask follow-up questions to refine the answer.
5. Check for Satisfaction: If the assistant's response has fully solved your problem and met your preferences, your response should be the termination signal.
6. Termination Signal: When you are completely satisfied, respond with ONLY the termination signal: [terminal signal}
OUTPUT FORMAT
Provide your response clean JSON object with keys:
58

---

Published as a conference paper at ICLR 2026
"thought": A brief thought process explaining your helpful reasoning. For "The assistant asked about my background. I'll not only answer but also mention my learning goal give them more context." "This good start, but I'll ask follow-up question to get more detail on the second step."
ample:
"response": The user's actual, detailed response to the assistant.
E ASSISTANT PROMPTS FOR DIFFERENT MODES
ORACLE MODE
Persona Known System Prompt
You are a helpful assistant trying to generate personalized explaination to the problem the user asks you. The user has the following Persona: {persona laraaarrffleee
Respond way that aligns with these preferences. You will be evaluated on how well your explaination aligns with these preferences, so that is your primary goal.
DISCOVERY MODE (DYNAMIC)
Infer Persona System Prompt (Dynamic Version)
You are a helpful assistant. Before attempting to solve the user's task or answer their ques- tion, first aim understand the user's background, goals, and preferences. Start by asking one or two clarifying questions that will help tailor your response to their needs. Do not proceed with a full answer until you have enough information personalize effectively. Your tone should remain natural, curious, and respectful- avoid interrogating the user, but try to guide the conversation to learn more about them.
At each turn, you need to decide whether to ask the user for more information or to proceed with the answering the question. you decide to ask for more information, you must ask one two clarifying questions that will help tailor your response to their needs. Do not proceed with a full answer until you have enough information to personalize effectively. Your response must be in the following format for all turns:
#**ACTION*** ask question' or 'final 'finalanswer"" answer'
###RESPONSE###: [clarifying question] or [full answer]
DISCOVERY MODE (FIXED NUMBER OF QUESTIONS)
Infer Persona System Prompt (Fixed Questions Version)
You helpful assistant. You must ask exactly {fixed_nume ffixed.num.questions/ clarifying ques- tions to understand the user's background, goals, and preferences before providing a final answer. This is turn (current turn) of {fixed num questions). Ask fying questions that will help tailor your response to their needs. Your tone should remain natural, curious, and respectful-avoid interrogating the user, but try to guide the conversa- tion to learn more about them.
two clari-
Your response must be in the following format: #*#ACTION*** "ask qukktinei
##HRESPONSE#H#: [clarifying question]
59

---

Published as a conference paper at ICLR 2026
BASELINE MODE
No Prompt System Prompt
(No system prompt provided)
F
BENCHMARK GENERATION PROMPTS
PREFERENCE DIMENSION SAMPLER
Preference Dimension Sampling Prompt
Generate {min weinnndimesiiinn peweeennniisssppeeeeeeeeeeeeeeee porrpp-- preference dimensions for how someone would want this [domain] explained.
PROBLEM: (problem.text!
REQUIRED DIMENSION TYPES:
Expertise/Comfort Dimensions (2-3 dimensions): Identify the key skills/con- cepts needed for this specific problem and create dimensions about comfort level with those skills. Examples:
Comfort with Recursive Thinking' (for recursion problems)
Familiarity with Inequality Manipulation' (for incquality problems)
Experience with Algorithm Complexity' (for efficiency problems)
Only include expertise dimensions for skills that are ACTUALLY needed for this problem.
2. Personal Dimensions (remaining dimensions): Think beyond generic categories. Focus on authentic, personal aspects that would make someone think 'this person really gets how I need to learn.'
CREATIVE REFLECTION PROCESS:
What unique ways might someone want information delivered based on their life experiences?
How might someone's 's profession, personality, or background create unusual learn- ing preferences?
What would make someone think 'Yes, this explanation style perfect for me"?
Each dimension should be:
Specific and actionable (something that could guide how to explain)
Personally meaningful (something someone would actually care about)
Grounded in authentic human needs and experiences
NOW, generate these dimensions from the perspective of this specific person:
PERSONA PROFILE:
Name: {persona_naae
Summary: (minimal peecessary.desccipptteeee
Age: {age). Occupation: (occupation]
Location: {location]
Hobbies: {hobbies}
Domain expertise: (domain.expertise)
LEARNING PROFILE:
Knowledge level: (knowledge level]
Learning style: (learning.style)
Problem-solving: (problem.solving)
Confidence: (confidence)
60

---

Published as a conference paper at ICLR 2026
Anxiety triggers:
(anxiety.triggers)
Motivation: (motivation)
PERSONALITY:
Openness: ( description]
Conscientiousness: www.sscieniiousssssdeeessiippppp
Social style: (extraversionndescriitttol
BACKSTORY: (backstory) INSTRUCTIONS:
1. First, analyze the problem to identify 2-3 key skills/concepts needed to solve
2. Create expertise dimensions for those specific skills based on this person's back- ground
3. Then create personal dimensions based on their unique characteristics
Based on this person's 's unique background, personality, experiences, and cur- rent situation, what specific aspects of an explanation would matter most to THEM?
Generate dimensions that capture the nuanced, personal ways THIS person would want the [domain] explained.
Think about how their professional experience, hobbies, personality, learning challenges, and life story create unique preferences that wouldn't apply to everyone.
4. Ensure the total is exactly {min fiiindimenssons per problem) dimensions
NAMING REQUIREMENTS:
Use concise, direct names (e.g.. 'Visual Learning Style', 'Technical Depth')
DO NOT use prefixes like 'Preference for', "Desire for', 'Need for'
DO NOT use redundant words like 'Preference', 'Style', 'Approach' unless essen- tial
Make names specific and actionable (what the preference controls)
Format your response clean JSON object:
"dimensions": [ { "name": " "(example) Technical Depth", "description": "Why this matters and how it affects learning experience", "value_range": "1-5 scale or categorical options", "type": "expertise" or "personal"
PREFERENCE INSTANTIATION
Preference Instantiation Prompt
You expert in personalization systems and educational psychology. Given persona and relevant preference dimensions, generate this persona's 's specific preference values.
PERSONA SUMMARY:
Name: (persona_name)
Background: {minimal necessary description)
Key Learning Characteristics:
Cognitive Style: (learning.style)
Problem Solving: (problem.solving)
Confidence: (confidence)
61

---

Published as a conference paper at ICLR 2026
Personality Traits:
Opennex==(openness)
Conscientious- Agrecable-
ness= (onscientiousness)). Extravasioon=eeexavversss..,
ness= agreeableness), Nemrotessm-(nnurrticisss
Domain Expertise:
expertise)
CURRENT PROBLEM: (problem_textt
RELEVANT PREFERENCE DIMENSIONS: {dimension.list)
persona has existing preferences]
PERSONA'S EXISTING PREFERENCES FROM PREVIOUS PROBLEMS:
CRITICAL INSTRUCTIONS:
1. Consistency: this persona has existing preferences for similar dimensions, main- tain consistency with their established patterns
2. Transferability: Consider how preferences might transfer between problem types. For example:
Visual preferences might transfer well between geometry and coding
Communication tone preferences should be highly consistent across problems
Detail level preferences might vary based on problem complexity
3. Persona-Specific: Ground all preferences this persona's characteristics and background
4. Problem-Relee Relevant: Adjust importance and some values based this specific problem type
5. Justification: Explain how you considered existing preferences and problem con- text
For each dimension, provide:
1. Value: Specific value that fits this persona's profile and is consistent with existing
preferences
2. Local Importance: 1-5 scale of how much this persona cares about this aspect for THIS specific problem. Make sure to spread out the range of importance; for example you cannot say all dimensions are important.
3. Justification: Detailed explanation of your reasoning, including reference to exist- ing preferences if relevant
Format as JSON:
"name": "Copy the name of the preference dimension from the previous step", "description": "Copy the description of the preference dimension from the previous step", "value range": "Copy the value range of the preference dimension from the previous step", "value": "specific value or number", "local_importance": 1-5, "justification": "Detailed reasoning considering persona characteristics, existing preferences, and problem context", "type": "expertise" or "personal") }
"preferences": { "Dimension Name":
EVALUATION RUBRIC GENERATOR (EXPERTISE CRITERIA)
Expertise Rubric Generation Prompt
You are creating evaluation criteria for EXPERTISS//CMFORT preferences how well an AI response matches user's technical skill level and comfort with specific concepts.
PROBLEM TO BE ANSWERED:
Type: (problem.tyy )
Problem: (problem. text)
[If answer is available] Correct answer: (solution.context)
62

---

Published as a conference paper at ICLR 2026
PERSONA CONTEXT:
User: [eersona_name] wetnooonnnnnnimiiiaeeeeeeeeeeeea fntiiinal..ecessrrydddssspppppp
Background: (backstory]
EXPERTISE/COMPORT PREFERENCES; prrrrieeeeeeeeeeeeseeeeeee wi with weights)
EXPERTISE RUBRIC GUIDELINES:
Focus on TECHNICAL APPROPRIATENESS: Does the response match the user's stated comfort/skill level?
Evaluate CONCEPT COMPLEXITY: Are concepts presented at the right level of sophistication?
Assess TERMINOLOGY USAGE: Is technical language appropriate for their ex- perience level?
Check ASSUMPTION LEVELS: Does the response assume the right background knowledge?
CRITICAL: Create exactly ONE criterion for each expertise preference listed above. Each criterion must use the EXACT preference name.
For each expertise preference, create criteria that measure how well the Al response cali- brates its technical level match the user's capabilities for this specific (problem.type); EXAMPLE EXPERTISE CRITERION:
"preference" "Comfort with Linear Algebra", "description": "How well the response matches the user's s intermediate comfort level (value: 3) with linear algebra when explaining matrix operations in this problem", "weight": 0.25, "levels":
{"score": 1, "description": "Uses linear algebra concepts far above (graduate level) or below (basic arithmetic) the user's s intermediate level, making explanation inaccessible ("score": 3, "description": "Mostly appropriate for intermediate level but inonsistencies--oocciiioo
patronizing"),
too advanced or too basic for stated comfort level"), ("score": 5, "description": "Perfectly calibrated to intermediate linear algebra comfort--uuse appropriate terminology, assumes right background knowledge, explains concepts at ideal complexity level")
Format as JSON:
"criteria": [ "preference": "Exact preference name", "description": "How well the response matches the user's technical level for this specific concept in this problem", "weight": 0.xx, "levels": ("score": 1, "description": "Technical level far above comfort level"), ("score": 3, "description": "Mostly appropriate technical level but some inconsistencies"), ("score": 5, "description": "Perfectly calibrated to user's stated confort level") ] ]
below user's stated
EVALUATION RUBRIC GENERATOR (PERSONAL CRITERIA)
Personal Rubric Generation Prompt
You are creating evaluation criteria for PERSONAL preferences- how well response
adapts to user's 's unique learning style, communication preferences, and personal context. PROBLEM TO BE ANSWERED:
Type: {problem (etoblem.type)
Problem: (problem. (aieem__ee)))))
[If answer available] Correct answer: (solution.context)
PERSONA CONTEXT:
(persona_name) Seeeroonaannnnnnnnanaaaaeeeeeeee (marinal.neeessrrdddesspttttl
User:
Background: (backstory)
PERSONAL PREFERENCES: (personal preferences with weights)
PERSONAL RUBRIC GUIDELINES:
63

---

Published as a conference paper at ICLR 2026
Focus LEARNING STYLE ADAPTATION: Does the response match how this
person prefers to learn?
Evaluate COMMUNICATION STYLE: Is the tone, formality, and interaction style appropriate?
Assess CONTEXTUAL RELEVANCE: Are examples, analogies, and references meaningful to this person?
Check MOTIVATIONAL ELEMENTS: Does the response connect what drives this person?
Consider this person's s background: {occupation). hobbies: (hobbies). and learning style:
[learning_style]
CRITICAL: Create exactly ONE criterion for each personal preference listed above. Each criterion must use the EXACT preference name.
For each personal preference, create criteria that measure how well the Al response adapts this specific person's unique learning needs and preferences for this EXAMPLE PERSONAL CRITERION:
""preference": "Engineering Examples from Work Experience", "description": "How well the response incorporates relevant engineering examples that connect to this user's mechanical engineering background when explaining this physics problem", "weight": 0.20, "levels":
("score": "description": "No engineering examples provided, or examples from completely irrelevant fields that don't connect to user's mechanical engineering experience"), {"score": 3, "description": "Includes some engineering examples but they' generic or don't clearly connect to the physics concepts in this specific problem"), ("score": 5, "description": "Provides multiple relevant mechanical engineering examples that directly illuminate the physics concepts and clearly resonate with the user's s professional experience") } Format as JSON:
"criteria": [ "preference": "Exact preference name", "description": "How well the response adapts to this user's specific personal learning preference for this problem", "weight": 0.xx, "levels": ("score": 1, "description": "No adaptation to personal preference completely inappropriate approach"), ("score": 3, "description": "Some adaptation to personal preference but generic inconsistent"), ("score": "description", "Excellent adaptation that perfectly matches the user's personal preference") ]
F.1
LLM JUDGE SCORING PROMPT
The judge evaluates each rubric criterion independently using the following prompts.
System Message
You evaluate responses against specific criteria. Output valid JSON.
User Prompt
You are an expert evaluation specialist using standardized rubric to assess personalized responses. Provide precise, evidence-based assessment of how well this response meets the specified criterion.
CRITERION TO EVALUATE:
(criterion name}
Description: (criterion description]
64

---

Published as a conference paper at ICLR 2026
PERFORMANCE LEVELS:
{JSON 1ist of 5 rubric levels with descriptions}
RESPONSE TO EVALUATE:
"[model's s final response)"
USER CONTEXT:
Preference value:
{user's preferred value on 1--5 scale} Context: (justification for why this level suits the user)
EVALUATION INSTRUCTIONS:
1. Carefully compare the response against each performance level description.
2. Identify specific evidence in the response that matches level descriptions.
3. Determine the exact score (I-5) that best represents the response quality for this criterion.
4. Provide detailed justification referencing specific elements of the response.
5. Be objective and consistent-apple same standards across all evaluations.
6. Consider the user's stated preferences.
7. Provide precise reasoning that another evaluator could follow to reach the same conclusion.
Format your evaluation as JSON object:
{"score": X (integer between 1 and 5), "justification": "...."
Rubric-Level Scoring. The rubric performance levels are personalized at benchmark-consteeeeee time to each user's preferred value uy. For instance, for user who prefers "Terminology Complex- ity" = 2 (simple language), the rubric levels are defined as:
Example: Personalized Rubric Levels for Terminology Complexity (v)
2)
Score 1: Uses highly technical jargon far above the user's preferred simple language level, making the explanation inaccessible.
Score Mostly uses accessible language hut occasionally introduces unnecessary technical terms, inconsistently matching user's preference for simplicity. Score 5: Uses plain, accessible terminology throughout and avoids unnecessary jargon, matching the user's preference for simple language.
The judge receives these pre- pre-personalized level descriptions and matches the response against them. The grading function (r, vy) (Eq. therefore measures how well the response matches the user's stated preference level for each attribute.
65


Reviews:
[{"review_id": "HM9BLAMjaK", "reviewer": "Reviewer_CM15", "summary": "This paper proposes PrefDisco, an evaluation methodology that transforms existing static benchmarks into interactive personalization tasks layered on top of the original task. The authors introduce a multi-dimensional numerical profile to instantiate user preferences. Their experiments show that attempts at personalization can hurt performance accuracy over the original task, suggesting naively attempting proactive personalization often degrades alignment for generic responses. Furthermore, they show that models generally do not ask questions, highlighting that interactive personalization remains under-explored.", "questions": "See weakness above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Explores a very interesting and relevant problem of multi-turn interactive preference elicitation. Current LLMs tend not to actively discover user preferences, even though this is critical for many real-world problems.\n2. The framework is creative and very flexible, allowing researchers to turn any existing benchmark into an interactive personalization task, making it broadly applicable across domains.\n3. The evaluation is extensive with various metrics and abundant models and static benchmarks. The counterintuitive finding that personalization attempts can often hurt performance is insightful and valuable to know.", "weaknesses": "1. Evaluation with LLM judge: the scoring system relies heavily on an LLM-based judge to score personalization quality. However, the paper does not provide sufficient evidence that the judge’s assessments are reliable or aligned with actual human preferences.  Since the grader model itself embodies particular stylistic biases, it is unclear whether higher scores genuinely reflect improved personalization rather than conformity to the judge model's implicit preferences. It would be helpful to have some sort of evaluation over the reliability of these model-based judges. For example, user studies, human-in-the-loop experiments, calibration checks etc.\n\n2. Persona representation: even though the personas are grounded in psychological research, it remains unclear how well these explicit preference dimensions capture the richness and diversity of real user preferences. User preferences are generally assumed to be high-dimensional, unstructured, and sometimes even internally inconsistent, these modeling dimensions are still fundamentally low order approximations to these preferences.  As a result, models evaluated on this benchmark may be learning to match stylized persona traits rather than genuine personalization applicable to real-world users.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DW9tr08usd", "reviewer": "Reviewer_cAyq", "summary": "The authors created a benchmark for simulating online exploration of user preferences with a simulator. The authors creates a set of preference attributes and allow the llm to query the value of those attributes' numerical values. They defined the eval metric as the ratio of the improvement performance upon zero shot prompting comparing to oracle where all the preference values are given in context. They argue that personalization is a reasoning capability that current model lacks.", "questions": "1. What are the system prompts exactly?\n\n2. Can some more advanced method like UQ based solve this problem better? \n\n3. see weakness 1.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. A novel interactive benchmark for personalization of llm that did not use llm simulate human, which decrease confounding factors.\n\n2. The evaluation spans all frontier models and show that all models failed under the current setting.", "weaknesses": "1. While I understand (and even appreciate) the authors’ motivation to control for confounding by simplifying the user simulation with scalar value feedback, the framework is far from realistic human llm interactions. Several components are significantly out-of-distribution: (a) models are explicitly forced to ask preference-related questions via system prompting, and (b) feedback is reduced to scalar or categorical values, whereas real users express preferences through text. These design choices make the evaluation more of a controlled experiment rather than a realistic personalization eval benchmark, though I acknowledge the inherent difficulty of modeling authentic human feedback at scale.\n2. Personalization through online interaction and exploration is not a novel topic. Several papers have explored this under llm setting, not even to mention previous bandit literatures. The method authors explored in the previous setting seems to be a soley based on the system prompt, rather than sth like uncertainty quantification. I would wonder if more advanced method can boost the performance.\n3. The exact system prompt for exploration is not shown, which I assume would be a very important part of the whole algorithm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lVAxDkLoOV", "reviewer": "Reviewer_9SvA", "summary": "This paper defines personalized reasoning in just-in-time scenarios, where the model actively seeks answers to discover user preferences and adjusts its reasoning accordingly. It then proposes PREFDISCO, transforming a static benchmark into an interactive task using psychology-driven sparse preference profiles. Experimental results demonstrate the limitations of current LLMs in personalized reasoning.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Extending from personalized content to personalized reasoning is interesting and reasonable.\n\n2. A psychology-driven evaluation method is proposed.\n\n3. Experimental results highlight the limitations of current LLMs, emphasizing the need to consider this characteristic.", "weaknesses": "1. Many details in the proposed evaluation framework remain unclear. For example, how are sparse preference profiles generated? What are the 20 scenarios? What are the relevant and irrelevant attributes for each scenario? How are importance weights calculated? How is LLM-based assessment performed? The numerous vague descriptions make the paper incomplete.\n\n2. While passive user simulation can simulate more challenging scenarios, it may overlook more informative interactions in general. Furthermore, real-world scenarios are not always extreme and may not accurately reflect actual situations. Therefore, this rationale requires further discussion.\n\n3. The paper's analysis is not sufficiently thorough. For example, the paper attributes the significant drop in accuracy of the mathematical task to \"over-optimization,\" but can this be further demonstrated experimentally? Why does the accuracy of the mathematical task continue to decline with increasing question count?\n\n4. Case studies would be beneficial. For example, showcasing various failure cases to support the paper's arguments.\n\n5. Minor: The range of the value of $g_j(r,v_j)$ is inconsistent between Sections 2.3 and 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JVTP0AT7WM", "reviewer": "Reviewer_zi3F", "summary": "The paper introduces personalized reasoning, the ability of large language models (LLMs) to adapt their reasoning and explanations to individual user preferences rather than offering one-size-fits-all answers. It presents PREFDISCO, a benchmark that converts static reasoning tasks into interactive personalization challenges where models must elicit user preferences through questioning and tailor responses accordingly. Evaluating 21 frontier models across 10 domains, the authors find that 29% of personalization attempts reduce alignment compared to generic responses and often lower task accuracy, especially in mathematical reasoning. The study concludes that effective personalization requires dedicated mechanisms for preference discovery and adaptive reasoning, as current LLMs fail to perform just-in-time personalization despite strong general capabilities.", "questions": "Please read the weaknesses section.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "They have assessed the 21 frontier models across multiple benchmarks.", "weaknesses": "I believe addressing the following concern might help to improve the quality:\n\n1. Although the task is interesting, the exploration of the user preference has already been conducted in UserBench [1]. The paper lacks a comprehensive comparison with existing works. No table or section is provided for such comparisons. I encourage the authors to conduct a thorough literature review and include a detailed comparative analysis with current studies.\n\n\n2. Another concern relates to the five interaction turns used in the experiments. The rationale behind selecting this specific number is not explained. An ablation study exploring the impact of different conversation counts would be helpful to justify this choice.\n\n\n3. I believe that preference reasoning presents a valuable direction to explore for planning-related tasks, such as travel planning. I suggest that the authors include this domain in the benchmark to enhance the coverage and applicability of the study.\n\n4. The process used to generate user instructions is not clearly described. Only one incomplete example is presented in Figure 1. It is recommended that the authors release a portion of the benchmark and provide a detailed visualization of the data generation process.\n\n\n5. The authors mention that the user simulator is passive. However, in practice, an assistant faced with a passive user would likely disengage and terminate the conversation. The reasoning for considering a passive user in the experiments remains unclear and requires further justification.\n\n\n6. The authors state that MATH and LogiQA exhibit the most performance degradation, whereas SocialIQA benefits the most from interactive personalization. However, no analysis or explanation is provided to clarify these observations. A deeper analysis would help readers better understand the underlying performance dynamics.\n\n\n7. The results in Table 1 show that, for most benchmarks, the performance of 3-Opus surpasses that of Sonnet-4. However, Sonnet-4 is a stronger model. The authors should provide an explanation for this phenomenon.\n\n\n8. The judgment process used in the evaluation is insufficiently explained. Including the evaluation prompts and methodology details would enhance transparency and reproducibility.\n\n\n---\n**Reference**\n\n[1] UserBench: https://arxiv.org/pdf/2507.22034", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)