Aligning Deep Implicit Preferences by Learning to Reason Defensively
Authors: Peiming Li
Zhiyuan Hu
Shiyu Li
Xi Chen
Yang Tang
Venue: ICLR 2026 Poster (2026)
Keywords: Preference Alignment
Reward Modeling as Reasoning
Process Supervision
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Personalized alignment is crucial for enabling Large Language Models (LLMs) to engage effectively in user-centric interactions. However, current methods face a dual challenge: they fail to infer users' deep implicit preferences (including unstated goals, semantic context and risk tolerances), and they lack the defensive reasoning required to navigate real-world ambiguity. This cognitive gap leads to responses that are superficial, brittle and short-sighted. To address this, we propose Critique-Driven Reasoning Alignment (CDRA), which reframes alignment from a scalar reward-matching task into a structured reasoning process. First, to bridge the preference inference gap, we introduce the DeepPref benchmark. This dataset, comprising 3000 preference-query pairs across 20 topics, is curated by simulating a multi-faceted cognitive council that produces critique-annotated reasoning chains to deconstruct query semantics and reveal latent risks. Second, to instill defensive reasoning, we introduce the Personalized Generative Process Reward Model (Pers-GenPRM), which frames reward modeling as a personalized reasoning task. It generates a critique chain to evaluate a response's alignment with user preferences before outputting a final score based on this rationale. Ultimately, this interpretable, structured reward signal guides policy model through Critique-Driven Policy Alignment, a process-level online reinforcement learning algorithm integrating both numerical and natural language feedback. Experiments demonstrate that CDRA excels at discovering and aligning with users' true preferences while executing robust reasoning. Our dataset is available at \url{https://DeepPref.github.io/}.

Published as a conference paper at ICLR 2026
ALIGNING DEEP IMPLICIT PREFERENCES BY LEARN- ING TO REASON DEFENSIVELY
Peiming Li"; Zhiyuan Muu".Shiyu Li', Xi Chen't, Yang Tang"
Basic Algorithm Center, PCG, Tencent
School of Electronic and Computer Engineering, Peking University
(peimingli, shyuli, jasonxchen, ethandtangyyteencet.cco zhiyuanhu@sta www.pkuu.ee. cku.edu.cn
ABSTRACT
Personalized alignment is crucial for enabling Large Language Models (LLMs) engage effectively in user-centric interactions. However, current methods face a dual challenge: they fail to infer users' deep implicit preferences (including stated goals, semantic context and risk tolerances), and they lack the defensive reasoning required navigate real-world ambiguity. This cognitive gap leads responses that are superficial, brittle and siort-sighted. To address this, we pro- pose Critique-Driven Reasoning Alignment (CDRA), which reframes alignment from scalar reward-matching task into structured reasoning process. First, bridge the preference inference gap, we introduce the DeepPref benchmark. This dataset, comprising 3000 preference-query pairs across 20 topics, is curated by simulating a multi-faceted cognitive council that produces critique-annotated reasoning chains to deconstruct query semantics and reveal latent risks. Second, to instill defensive reasoning, we introduce the Personalized Generative Process Reward Model (Pers-GenPRM,, which frames reward modeling ized reasoning task. It generates a critique chain to evaluate a response's align- ment with preferences before outputting a final score based on this ratio- nale. Ultimately, this interpretable, structured reward signal guides policy model through Critique-Driven Policy Alignment, a process-level online reinforcement learning algorithm integrating both numerical and natural language feedback. Ex- periments demonstrate that CDRA excels at discovering and aligning with users' true preferences while executing robust reasoning. Our dataset is available at htttps://DeepPreff http://eepprrf......b.b.bbbbbb .ithhb..io
personal-
INTRODUCTION
Large Language Models (LLMs) are rapidly evolving from simple insllwetionnfolloweess to potential collaborative partners, transition that hinges on effective personalization (Vaswani et al,, 2023 Brown et al., 2020). The ultimate goal of personalized aligmment is steer LLMs beyond generic helpfulness towards responses that resonate with an individual's unique context, values and unstated goals (Mathur et al. 2023 Lee et al., 2024). However, prevailing alignment paradigms, such as Direct Preference Optimization (DPO), primarily optimize for model's I's surface-level appeal using outcome-based supervision (Rafallov et al., 2023] Ziegler et al. 2020). Similarly, reinforcement learning approaches that depend supervision from final outcomes face significant challenges due to the inherent limitations of scalar feedback. These methodologies create a critical cognitive gap: models learn to mimic user's stated preferences rather than reason about their latent intent.
We formalize this limitation as a dual challenge: a preference gap and a process gap. The preference gap is the model's inability to infer deep implicit preferences (i.e., the unstated goals, risk tolerances and priorities) that drive a user's query. Concurrently, the process gap is its failure to execute de- fensive reasoning, the cognitive process of proactively identifying and mitigating risks latent within a query's 's ambiguity. Figure I[a) presents the problem formulation. For instance, consider a user
"Equal contribution. Project Lead. i Corresponding author.

---

Published as a conference paper at ICLR 2026
Standard Alignment: Superficial Preference Matching
Ours
Ilerutive loop
Preference
User's Deep Preference
LLM
Outcome-
Optimization (eg. GRPO)
2
Query
Curnent Response K
Pres-GenPRM
The Preference 6ap Expliit Preference Marches fist Instpprrorsoorr
Step 1: Analyze Pxplicit Preference
Preference Gap
3 SFT / DPO
Step Bridge Preference Gap
Implist
Swifching to 3rd Person retóira
Step 3: Bridge Process Gap (Defensive Recsoring)
LLM
Step Formulate Aligned
Output:
Strategy.
Recemmmnnds Maas Effect ar ertically acch aschimed game (Seors: Besker Stogler2
Eaxmple: Suboptimal
Address Process Gap
Foikrs Defersive Recsoning
Response
Final Aligned Response
The
Recommennde Third-Perase Steptera
Preference: do not enjoy first person shooter games.
vinlent/at Satee////eeeeeeee pnoeeeed gamaplay.
Query: What are some eniticallly acclaimed games should try?
The Gap (Standard): Interprets Finst-PPrson' as syntox filter. Misses implicit dislike for fast paced refiexes violent mechonics.
Alignment Outcome: mimic rather reasoning
Intent Mining (Ours): Infers latent values:
"Storytellinng, Camera Control,
Agency', Recommends: The Witcher 3 (RP6),Fireeatth) Firewatch (Adventure).
(a) Problem Formulation
(b) Comparison of Alignment Paradigms
Figure I:
(a) Problem Formulation: Optimizing for outcomes rather than the reasoning process creates the dual preference and process gaps. (b) Comparison of Alignment Paradigms: Standard, outcome- outeo approaches (left) exemplify the problem of superficial preference matching. In con- trast, our CDRA (right), shifts the paradigm to be process-driven and explicitly bridges both gaps.
who states, "I don't feel comfortable sharing my real-time location", and asks for way to update their family. model aligned on superficial preferences might suggest service that automatically shares a location pin upon arrival. This scemingly benign solution epitomizes the dual failure: it correctly processes the explicit constraint ("no real-time") but fails to grasp the implicit principle of privacy driving the user's request (the preference gap). Concurrently. it fails to execute the defen- sive reasoning needed to foresee that an aggregated location log constitutes new privacy liability, thereby violating the user's deeper principles of autonomy and narrative control (the process gap). This superficial literalism, while technically compliant, is the hallmark of a system optimized for outcomes rather than genuine understanding.
To bridge this divide, we introduce Critique-Driven Reasoning Alignment (CDRA), a novel paradigm that shifts alignment from supervising final outcomes supervising the underlying rea- soning process itself. As illustrated in Figure [[b). CDRA is designed to resolve the preference and process gaps in concert by teaching the model to internalize process of critical evaluation: it learss not only to generate answers, but also to critique how well those answers respect user's deeper preferences and manage latent risks. This paradigm shift, however, necessitates new mechanisms for data, reward modeling, and policy optimization.
Our framework addresses this through three logical steps. First, acquire the necessary process- level supervision, we introduce DeepPref, a new large-scale benchmark of preference-query pairs. For each query, we simulate multi-faceted "cognitive council" of distinct expert personas con- struct critique-annotated Trees of Thoughts. These annotations provide explicit supervision on how infer latent preference structures and how execute defensive reasoning by proactively stress- testing candidate responses against potential risks and value misalignments.
Second, to convert this rich textual signal into a structured reward, we propose the Personalized Generative Process Reward Model (Pers-GenPRM) Rather than directly predicting scalar score, Pers-GenPRM operationalizes reward modeling as reasoning task: given a query, user preference context, and candidate reasoning process, it first generates explicit textual chain of critique, then derives a quantitative score grounded in this rationale. This design leverages Natural Language Feed- back (NLF) (Saunders et al. 2022 Chen et al.; McAleese et al., 2024) to transform process-level critiques into an interpretable reward signal, making the basis of the model's evaluations transparent and auditable.
2

---

Published as a conference paper at ICLR 2026
Finally, we use this structured, groliuee-goouded reward to guide policy optimization via Critique- Driven Policy Aligmment (CDPA), an online reinforcement learning algorithm that integrates both numerical scores and natural language critiques from Pers-GenPRM. By leveraging process-level feedback, CDPA addresses a key limitation of standard RI., which we term the "zero advantage" problem. In this setting, multiple responses may be equally good in terms of final outcome but differ substantially in the quality and safety of their underlying reasoning. CDPA provides a clear gra- dient signal that differentiates between such responses based on their reasoning processes, steering the policy model toward solutions that are not only correct but also defensible, robust, and deeply aligned with user intent. In summary, our work makes several key contributions:
We are the first formalize and address the dual challenge of preference and process gaps in LLM alignment, proposing critique as a form of cognitive process supervision to move beyond superficial mimicry.
Our proposed CDRA framework achieves more reliable and intent-aligned personalization through three core technical innovations: DeepPref, the first large-scale, critique-annotatar dataset for process-level supervision; the Pers Pers-GenPRM. which transforms reward model- ing into a transparent and interpretable reasoning task; and the CDPA, an algorithm that fuses numerical and natural language feedback align the model's reasoning process.
Extensive experiments on metries across three dimensions demonstrate that CDRA achieves state-of performance, showcasing its superior capability in both deep pref- erence understanding and robust reasoning.
METHODOLOGY
2.1 OVERVIEW
The core objective of personalized alignment is to align large language model's policy with user's deep implicit preferences, which are represented as latent preference variable P (Ziegler et al. 2020). Given query q. the goal is to learn policy (ylg, P) that generates response which optimally aligns with P. However, effectively addressing the dual preference and process gaps introduced earlier presents significant technical hurdles: The preference gap manifests in the difficulty of inferring users' deep implicit preferences, including unstated goals and risk tolerances (Casper et al. 2023). The process gap is exacerbated by traditional outcome-level supervision, which provides a sparse and uninterpretable scalar reward insufficient for guiding complex reason- ing (Rafailov et al. 2023 Lightman al. 2023). Furthermore, acquiring the high-quality process supervision data needed to bridge these gaps is prohibitively expensive and difficult to scale.
The Crivique-Driven Reasoning Alignment (CDRA) framework designed to address these chal- lenges. The overall process of CDRA is shown in Figure 2 and Figure This section formally defines the problem and details CDRA's constituent components:
DeepPref' Construction (Section 2.2 A novel benchmark, DeepPref, is constructed by simulating a multifaceted cognitive council to generate rritique-annotate reasoning chains. This stage provides the necessary data foundation for subsequent components.
Personalized Reward Modeling (Section 2.3) The Pers- Pers-Gen GenPRM trained on the Deep- Pref dataset to learn to map the preference P and response to an explicit textual critique and corresponding scalar reward for each reasoning step.
Critique-Driven Policy Alignment (Section 2.4): The policy model. R,, fine-tuned via Rejection-samp sampling Fine-Tuning (RFT) and Critiue-Driven Generalized Reward Policy Optimization. The CDPA leverages the structured, generative reward signal from Pers- GenPRM to guide the policy toward generating responses that are not only high-scoring but also defensible under critical scrutiny.
2.2 THE DEEPPREF DATASET
To address the inherent preference and process gaps in current alignment paradigms, we construct DeepPref (DDeepPret), large-scale, aritique dataset designed to provide process-level su-
3

---

Published as a conference paper at ICLR 2026
Critique- Driven Reasoning Alignment (CDRA)
Personalized Reword Modeling
Critique-
Driivque-Driven Driven Policy Aligrment
Multi-faceter cognitive council
Parrwise Somple
The Socidlogist
Preference:
Query:
The Psychologist
Preference
Progmatist
feel table shaning real-time location
What's
way travel
dataset
The Pedogoque
RL CDPA
Bridge the Preference 6ap
Dataset Construction
Bridge the Process
Soone:
Preference
Reasoning
Step Hypothaasi
You prisntion controllng your lsserion information reastrre family.
The user valuea
dctoset
outrovery personal over converiance.
Criputue: The explictt yeer praferarce diconfart sharing The "Näne
SFT
Scane:
Thought hypothesioes that.. Scare:
Score is Threshold
Step The uder mhly ing leftly prefee city
Step No: The best lef your forsly know
Find asdwor rode
Pers GerPRM
there set
frovel plone
bounderict not and
routine whene
RFT
Firsl crower nocie
Finel crawerr node
RFT dataset
Policy model
Figure 2: Overview of the CDRA Framework. The process consists of three main stages: (1) DeepPref Dataset Construction; (2) Personalized Reward Modeling: and (3) Critique-Driver Policy Alignment. (2) and (3) are illustrated in detail in Figure 3
Personalized Reward Modeling
Query
driniuee
Query
Shop
Pers-GenPRIM
"Criiqua Step
Ocoree
Responde
Respance
Shop
Please give step
critique with
Orttquee
Crritique Step Ni.
Chais Ouihiuu
Taken leve reword
Critique Driven Policy Alignment
Reference Model
Response
Model
RFT
Palicy
Response
Pers-GurPRM
Group Computation
Respanse
Bridge the Process Gap
Prefensncs
Query
Shop by sTop resjoode
Tolicn level newerd
Figure 3: Personalized Reward Modeling (Section 2.3 Pers Gers-GenPRM generates reflective chain critiques based on whether each step of a response infers the user's s deep implicit preferences and proactively mitigates potential risks. It then derives step-wise reward scores from these critiques. Critique-Drive Driven Policy Alignment (Section[2.4) The policy model is first aligned using Rejection- sampling Fine-Tuning. Subsequently, incorporates the process-level supervision rewards from Pers Pers-GenPRM into its reward signal for further alignment.
pervision. Unlike existing datasets that rely on outcome- preference pairs, DeepPref is specif- ically engineered teach models how to reason about user's latent intent and proactively mitigate risks. The DeepPref construction process is illustrated in Figure 2 Further implementation details, including persona definitions, pruning heuristics and prompts provided in Appendix E
Data Construction The DeepPPet dataset constructed via a novel pipeline designed to capture the critical reasoning process. It contains 3000 unique scenarios from 20 diverse domains (e.g., personal finance, healthcare), each comprising a (P, q) tuple of a detailed preference and query q. Preferences P are crafted to include nuanced, often conflicting values and unstated goals (e.g.
4

---

Published as a conference paper at ICLR 2026
"I value convenience but am extremely arivacy-conscious).. Queries q deliberately open-ended and ambiguous to compel deep reasoning about P, rather than simple instruction following.
Reasoning paths are generated and annotated via two-stage process. (1) Diverse Path Genera- tion. The first stage employs a Tree of Thoughts (ToT) framework (Yao et al. 2023) to generate reasoning paths for each scenario (P, g). Guided by multi-faceted cognitive council and incor- porating heuristic pruning, this process yields a diverse set of unique reasoning chains, where each chain Tę is sequence of steps (s}, ,?,,.,,,.. "!").. (2) Step-wise Critique and Scoring. A powerful LLM evaluator annotates each reasoning step oy within chain т. For every step, the evaluator generates detailed textual critique of and corresponding scalar quality score r/. The critique assesses the step's alignment with user preferences and its effectiveness in mitigating risks, con- ditioned on the preceding path. These components form the DeepPrer dataset, comprising tuples of (P.g, Ti, (c/, -1))))) The subset DRea C DecpPiett containing all preference reasoning chains and critiques, used to train Pers-GenPRM, while subset of the highest-qualis paths DRIT. is reserved for fine-tuning the policy model.
Preference Forms User preferences manifest in various forms, differing in explicimness and com- plexity. Our work addresses a spectrum from surface-level statements to deep-seated intent. (1)
Explicit Preference. These are directly articulated user statements (e.g.. "I don't like spicy food"). Prevailing alignment methods primarily optimize for adherence to such explicit commands. (2) Deep Implicit Preference. These preferences are not explicitly stated but are embedded within a user's broader context, values and unstated goals. For instance, the statement, "I'm not comfortable sharing my real-tin time location", may imply a deeper preference for autonomy and narrative control, beyond mere privacy. Accurately inferring such latent intent is the central challenge in bridging the preference gap and is essential for achieving genuine personalization.
2.3 PERSONALIZED REWARD MODELING
A fundamental challenge in personalized alignment is the inherent subjectivity of user preferences, which contrasts with tasks having objective ground truths like mathematics (Casper et al., 2023). Simple scalar rewards are insufficient, as they risk reinforcing superficial correlations rather than deep, causal reasoning (Skalse et al. 2025 To address this, we introduce the Personalized Gener- ative Process Reward Model (Pers-GenPPM which learns to map user's profile and context to a quantifiable reward by internalizing the nuanced judgment process encoded in our DeepPref dataset.
Inspired by reward modeling reasoning (Yang et al. 2024), Pers-GenPPe holistic evaluator but as a step-wise critic model that generates textual critiques and scores (Lightman al. 2023 Song al. 2025). For each reasoning step s? in a chain Tú, it takes the preceding context (P. and is trained to generate a critique-score pair:
operates not as
(P, (P, S) (c,r)),
(1)
where of is an explicit textual critique and r is corresponding scalar reward. This approach di- rectly supervises the model's cognitive process, addressing the process gap with a dense and granular signal that holistic evaluation cannot provide.
Pers- GenPRM is trained via Supervised to maximize the log-likelihood of generating the ground-truth critique-score pairs. Given the autoregressive generation of the critique ef followed by the score r!. we decompose the loss as:
(SFT) on the DreePrer dataset. The objective
Σ (Log Po( + log P.(. )
(0) -E -P(Pami
(2)
where 0 represents the parameters of Pers-GenPRRM. This training yields a dual-component reward for each step: an interpretable critique (c)) that provides a transparent, semantic explanation, and a scalar reward (r?) grounded in the critique cf that acts as quantitative distillation of the critique. This grounding causally anchors the numerical signal to a human- intelliittlliigible rationale.
This structured, process-level reward is instrumental for the final policy alignment stage (Section
2.4 By aggregating the step-wise scores into dense reward, Ratense (T:) Σi=
2i ry. resolve
5

---

Published as a conference paper at ICLR 2026
the "zero-advantage" problem. This creates clear gradient that differentiates reasoning paths by quality, effectively guiding the policy towards solutions built on high-quality, defensible reasoning.
2.4 CRITIQUE-DRIVES POLICY ALIGNMENT
To overcome the limitations of sparse, outcome-based rewards like reward hacking and the zero- advantage problem, we introduce Critique- Driique-Driven Policy Alignment (CDPA), Our approach is built upon GRPO (DeepSeek-AL et al. 2025), but its core innovation is fineggrained advantage signal assigned to each token position. This advantage is derived directly from the step-wise, critique- grounded rewards generated by Pers-GenPRM, creating the tight feedhack loop between reward modeling and policy optimization shown in Figure 3 The process unfolds in five steps:
Step Policy Initialization. We initialize the policy To via Rejection Sampling Fine- Fine-Tuning (RFT) (Tovrron et al.) (023) using the high-quality DRFT data subset (Section 2.2
Step 2: Group Sampling. For each input (P, .), we sample group of responses {yr] from the current policy my. where each response y/ consists of T; reasoning steps (s.....,..
Step 3: Process-Level Reward Generation. The Pers-GenPRM then assigns critique-grounder scalar reward r to each reasoning step s? within every response, as detailed in Section 2.3
Step 4: Critique-Groundes Advantage Estimation. CDPA's key mechanism is its token-level advantage. For each token t belonging to a reasoning step s in response yi, we define its token-level reward ri,t as the reward of that step, i.c., Ti, = r.. For each group of G responses, we normalize the teken-level rewards to have zero mean and unit variance across the group. The advantage for any token within response Vi is thus:
ri, 1gg og +
A(t.v:)
(3)
where My and oy are the empirical mean and standard deviation of token-level rewards across all G responses in the group at the corresponding token position. This provides a granular signal contrast- ing each token's step quality against its counterparts within the same generation group.
Step 5: Policy Update. Finally, we update the policy using a PPO-style clipped objective that incorporates our per-token advantage:
G Σ
JCDM (0) = (.(.)
min (peA(t, (t), 1- 1 BDD (sto (sto [Mtr))
G
where C2 is the total number of tokens in response V/. Pe
no(tla.Pai.ct
is the importance ratio, P.w.ely...wi...1)
and Told the reference policy before the current update.
This peocess-level feedback resolves the process gap and zero-advantage problem, creating rich gradient that steers the policy beyond mere correctness toward fundamentally aligned reasoning. This symbiotic dynamic, where reward model critiques and the policy internalizes, thereby advances machine alignment from mere preference mimiery to a collaborative and interpretable paradigm.
EXPERIMENTS
3.1 EXPERIMENTAL SETUP
Implementation Details. We implement our CDRA pipeline using the trl (von Werra et al. 2020) and vLL.M (K et al. 2023) libraries for efficient training. All experiments leverage Qwen2.5-7B. Instruct (Team 2024) as the base model and are conducted on four NVIDIA H20 GPUs. For the SFT bascline, we report performance from the best checkpoint on a validation set. For all reinforcement learning methods, report performance at the best checkpoint within 400 optimization steps. Our CDPA samples G = 5 responses per prompt with temperature of 1.0.
Datasets. To probe for deep implicit preferences, whose 300-instance validation set contains deliberate ambiguities to challenge a model's 's reasoning capabilities. To measure adherence to explicit preferences, we additionally evaluate all methods on
newly proposed DeepPref benchmark,

---

Published as a conference paper at ICLR 2026
Table 1: Evaluation of performance across three dimensions the DeepPref and Preffval datasets.
Core Performance (%)
Deep Reasoning Quality (%)
Method
AccPF T
AccDA t
AccMis
Widm T
Mic
Dataset A: DeepPref
Zero-shot
23.0
6.7
76.0
4.7
3.0
0.0
Few-shot
49.7
32.7
61.3
29.0
10.7
CoT
59.7
49.3
50.3
39.0
25.3
0.7
TPO
55.3
36.3
56.3
29.7
15.7
0.0
SFT
83.3
75.0
34.7
46.7
63.7
40.3
GRPO
83.7
70.3
30.7
46.3
58.7
34.0
CDRA
84.7
76.3
32.3
47.0
65.0
42.7
Dataset PrefEval
Zero-shot
37.5
10.7
28.7
8.9
4.5
0.9
Few-shot
38.4
23.3
38.4
2.7
CoT
62.5
61.6
20.3
54.5
18.8
0.9
TPO
62.5
48.2
20.7
58.9
19.6
0.0
SFT
66.9
58.0
24.7
21.4
34.8
10.7
GRPO
67.0
51.8
27.3
53.6
17.0
1.8
CDRA
68.8
62.5
21.0
27.7
37.5
15.2
Table Human evaluation results on the ALOE dataset. We report the average alignment level (1-5 scale) at each conversational turn (k).
Model
k=1
k=2
k=3
k=4
k=5
k=6
k=7
k=8
k=9
k=10
Average
CoT
2.0
3.2
3.8
4.0
4.2
4.4
4.2
3.8
3.8
3.8
3.72
SFT
2.4
3.0
3.8
3.8
3.8
4.0
4.0
4.2
3.6
4.2
3.68
TPO
2.4
3,4
4.2
3.8
4.2
4.2
4.2
4.0
4.2
4.0
3.86
GRPO
2.0
3.4
3.0
3.4
3.6
3.2
3.4
3.4
3.0
3.4
3.18
CDRA (Ours)
2.0
3.4
4.0
4.2
4.4
4.4
4.6
4.2
4.0
4.0
3.92
the PrefEval benchmark (Zhao et al. 2025).
Finally, assess performance in maintaining align-
ment over long-form dialogues, we leverage the ALOE benchmark (Wu et al., 2024) for our human evaluation study.
To comprehensively assess model's ability to bridge the preference and process gaps, we establish a multi-diimnnsioal evaluation protocol based on an "LLM-as-a-jude"" framework. Considering performance, cost, and the need to avoid homology, we ultimately selected and reported the results from DeepSeek- V3.2-Exp. The evaluator assesses each generated response y from test set of size N across three key dimensions: (1) Deep Preference Understanding. This dimension evaluates the model's capacity to capture and address the user's deep, latent intent, measured by three criteria: Deep Mining (mam), Innovative Expansion (m)) and Thoughtfulness (ma). response is successful if it satisfies at least one criterion. We define an indicator function I(m, y) as if response y meets criterion and 0 otherwise. The Deep Alignment Accuracy (Acepa) is the proportion of responses satisfying this condition:
Evaluation Protocol.
1 N
ACCDA
max (7(m vi) (mm4e Bi), (mmah: U:))
(2) Defensive Reasoning. This dimension evaluates the model's proactivity in identifying and miti- gating potential risks. We assess this via the Misleading Risk metric Accaris which flags responses containing potentially misleading suggestions (Bai et al., 2022). (3) Explicit Preference Adher- ence. This dimension, following PrefEval (Zhao al. 2025). measures basic preference following by identifying four error types: Preference Unaware Violations (epar), Hallucination of Preference (ehp). Inconsistent Responses (eir), and Unhelpful Responses (eur). Let Ele, y) be indica- tor function that is if the response exhibits eтrоr type e. The Preference Following Accuracy (Accpe) is the fraction of responses free of any such emror::
1 M (1 max (F(epar: Ws), F Ee vs), E(eir, Ut), E(Cur: W)))
Accpp
(5)
N 1
The full definition and evaluation rubric for metrics provided in Appendix

---

Published as a conference paper at ICLR 2026
Figure 4: Comprehensive performance comparison across six dimensions. Our CDRA (shown in red) achieves the largest coverage area on the radar chart, demonstrating superior and balanced performance com- pared to strong baselines like GRPO and SFT. For all axes, a higher value (further from the center) indicates better performance. Note that the error-based metric inverted (100 Acc for consistent visualization.
CDRA (Oars)
Zeco shot
GRPO
T70
Pew-shot
Baseline Methods. We evaluate CDRA against a comprehensive suite of baselines from two primary
categories. In-context Learning Methods: (1) Zero-shot (Brown al. (020),, (2) Few-shot (Zhao
et al. 2025), and (3) Chain-of-Thought (Co'T) (Wei et al. 2022). Model Optimization Methods:
(4) Supervised Fine-tuning (SFT) (Ouyang et al. (2022) and (5) Tree Preference Optimization
(TPO) (Liao et al. 2024). Detailed descriptions of all basclines are provided in Appendix
3.2
QUANTITATIVE ANALYSIS
Deep Preference Understanding and Defensive Reasoning, As shown in Table I CDRA signif- icantly outperforms all baselines in Deep Preference Understanding. It achieves the highest Deep Aligmment Accuracy on both DeepPref (76.3%) and PrefEval (62.5%) This superiority stems from its offee-of-the-ar performance in Deep Mining (65.0%) and remarkable advantage in Innovative Expansion (42.7%), ), where surpasses the next-best method by over 2.4% on DeepPref. Crucially, this strength in innovative expansion is complemented by strong Defensive Reasoning capabilities, as evidenced by CDRA achieving low Mislcading Risk score across both benchmarks. While this reflects trade-off against the Thoughtfulness metric, model's dominance in generating novel, high-value ideas demonstrates a more effective reasoning balance, successfully bridging both the preference and process gaps.
Explicit Preference Following, Crucially, these gains in complex reasoning do not compromise the model's fundamental ability for Explicit Preference Following. Our evaluation confirms this, showing that CDRA achieves the highest Preference Following Accuracy on DeepPref (84.7%) and remains highly competitive PrefFval. As further illustrated by the error analysis in Figure ??, our approach successfully integrates deep preference modeling with robust adherence to explicit instructions, proving its reliability as well-rounded conversational agent.
Human Evaluation on Multi-Turn Dialogue. As presented in Table 2 our human evaluation on the multi-turn ALOE dataset confirms CDRA's practical effectiveness in realistic interactions. CDRA achieves the highest average alignment score (3.92), outperforming all baselines. More significantly, alignment score consistently improves as the conversation progresses, peaking at an impressive 4.6 in later turns, while baselines tend to plateau or degrade. This demonstrates CDRA's superior ability to accumulate user context and maintain deep alignment throughout long conversations, directly translating its reasoning capabilities into tangibly better user experience.
3.3 QUALITATIVE AND MECHANISM ANALYSIS
Qualitative analysis confirms that CDRA's superior performance stems from its ability to reason about latent user preferences. As illustrated in Figure 5 baseline models like TPO often adhere to literal interpretations, leading to Preference Unaware Violations. In contrast, CDRA first grap- ples with the underlying semantics and unstated risk tolerances. This deeper initial understanding allows it to generate suggestions that are both highly creative and defensively sound, effectively demonstrating how it bridges the preference and process gaps.
8

---

Published as a conference paper at ICLR 2026
eagis fintgrnuo gatee.
Querye
criicall
pames shoald
<lakk Gives you ovesioo fiskpprous shosten,, let's explact some
-think Conaidty! Let' boeak dowa reques step-by-ste fidd
ehttcally aeciaimed gamos soross varteus gennos that might find
CDRA
eoms eriically seciaimed gamos that algn well with yosr preforeccs.
TPO
exgagnn Sup Amalyne the Laer's Primay Irlfrmmee Crmmmne and Its Impiliatious It's likely that you prefer games where you hove more
Step Analyse the Lsee's Primany PreferencriCume CunstrenniCn You
meationed that you do sot enjoy first fist-person shooter ganes. This indioses dlaa prefirence eganst integared sctlon ortested games where player peicanlly mometts thrugh duainy madurios Sep Traly
cormool over the camen angle, charater movemete and evenal pactng.
Step" 7: Decelop Depprr Unidentarding of
inme worls, stategi sareepla deneats that you
may gravitute saw garie
horr Trily
storycalligg.
from the Finst
fic information
seman
you looking cillically acclained sunes anpoctast third shout gasees received high prüse from cities players game pamee
Desper Lndervandiu
or Seck. Given Mass Effect 3 is
comtll
quick unking
Person Shouter
(FPS) gamse.
-ax information extracted), Even alvrnee mioophying (IIPG0 Explore Farther Implicatione preferences, uncons Conaideraionn To
mimoplay
Preforences We
choices,
have froedom
Following display proference, and excllddinn penson games and shooting gumes
explon: vae, desiled onvirmmen
Sup Nended.
lipplre Fumber impicatione Consdematine Addinoedly dholld look girnes that was major awedi lave lights cseed plasfoms like Metoeitic and GomeRaniingg <atirke
hroed preferznas, clous violatons will consider both sagle playrr and multiplayes options, es well still occurred. recent and classic miles cthii-
Step S: Provide Final. Actionable Asawer That Aliam Wits Idoniiied Preferencs, nasswert The Windher wild Hust Getre. Action &PG... 2. The Last of Part Cene, Aetion RIG Lifo is Strange Gerre: Advemure... 4 Firrwardh Cenm: Atbermuee.... Standow Valley -Gere Simulation RPG... simonerr.
Sios 5, Provide Fizal, Actionable Auswer "anwerr 1, The Wiiche
Wild Hast Gerse, Action RPG.... 2. The Exder Sculls V. Skyzin Genee Actioa RPG 3. Mass Effoct J Geuee Nction RPG... Civilisaton -Gem:: Tten-llasal Stratgy Stelbris -Gnret (Esplor: Expand, Explii, Externine)... siasewer-
Figure A qualitative comparison showing CDRA reasoning about latent intent.
0.035
Average Attention Score (Last Token)
Preference Attention %:
Base Model
Base Model: 27.7%
SFT Model
0.030
SFT Model: 25.6%
GRPO Model
GRPO Model: 27.6%
0.025
Ours Model: 35.7%
Ours Model
Preference Region
0.020
0.015
0.010
0.005
0.000
Taken Position in Input Prompt
Figure 6: Attention Distribution Analysis. We visualize the average attention scores of the last token across the input prompt. The gray shaded area indicates the Preference Region.
Attention on Preferences. Figure reveals that unlike the dispersed attention of baselines (SFT, GRPO), CDRA concentrates 35.7% of its attention mass on the Preference Region. This suggests Pers-GenPRM supervision teaches the model where to look, enabling it to actively anchor generation on user constraints and effectively reduce Preference Unaware Violations.
Table 3: Ablation study on different reward modeling paradigms on DeepPref. 'Pro. Sup.' denotes process supervision and 'Cri. Sup.' denotes critique supervision.
Model Method
Pro. Sup.
Cri. Sup.
Accpr f
Acep f
Acc
mp
Mai t
Base D0wen2.5-7B-Bn --BB-nntttttt)
59.7
49.3
50.3
39.0
25.3
0.7
Alvernative Reward Paradigms
GRPO (with RM)
83.7
70.3
30.7
46.3
58.7
34,0
GRPO (with GRM)
83.7
74.7
30.7
45.7
62.3
37.0
GRPO (with PRM)
83.7
73.0
34.0
51.0
59.7
38.3
Simpler Heuristics
GRPO (Rubric-busee based RM)
84.0
73.7
35.3
48.0
61.7
34.7
GRPO (Test-Time Scaling)
84.3
73.0
31.0
47.7
62.0
34.7
CDRA (with Pers-GenPRM)
v
84.7
76.3
32.3
47.0
65.0
42.7
3.4 ABLATION STUDY
Ablation on Reward Modeling Paradigms. To investigate the necessity of our approach and broaden the empirical scope, conduct comprehensive ablation study (Table on various reward modeling paradigms. The results unequivocally demonstrate CDRA's superiority. achieves the highest Deep Alignment Accuracy at 76.3%, surpassing both autcome-based and process-based models. This finding underscores that supervising the critical reasoning process itself, rather than just the final outcome intermediate steps, is essential for deep alignment. Crucially, CDRA also

---

Published as a conference paper at ICLR 2026
outperforms simpler heuristics suggested as alternatives, such as Rubric-based models and Test- Time Scaling. The performance gap is particularly pronounced in Innovative Expansion, where CDRA (42.7%) establishes substantial lead over these methods (34.7%), confirming that such heuristics cannot replicate the nuanced reasoning required to uncover latent preferences. While CDRA's focus deep preference discovery (evidenced by top scores in mdn. and mae) involves managed trade-off with a slightly higher Misleading Risk, its overall dominance in generating novel and deeply aligned responses validates the effectiveness of our approach. Collectively, these results validate that CDRA represents a more effective paradigm than existing reward modeling approaches for deep preference alignment.
4 RELATED WORK
4.1 PERSONALIZED ALIGNMENT OF LARGE LANGUAGE MODELS
Aligning Large Language Models (LLMs) with human intent is central challenge (Ouyang et al.,
2022 Di et al. 2023 Cao et al. 2024a). However, the prevailing paradigm, Reinforcement Learning
from Human Feedback (RLHI), often overlooks individual user nuances (Ouyang et al.) 2022 Sun
ct al. 2024), making personalized alignment a critical research frontier. Existing personalization methods fall into two categories. Tuning- -ree approaches, such retrieval -eumeetdl generation and prompt engincering, are simple but suffer from performance inconsistency and inference over- head (Salemi et al. [023 Park et al. [023]] Konen al. 2024 Cao et al. 2024b). In contrast, tuning-
based methods like Supervised Fine- Tuning (SFT) and Direct Preference Optimization (DPO) offer
more robust solutions (Shao et al., [023] Li et al. 2024b Zeng et al. 2024 Shaikh et al. 2025
Rafailov et al. 2023). Yet, they are often limited to mimicking superficial styles or handling single-
turn interactions, failing capture the deep, implicit intent behind user queries (Lee et al., 2024
et al. 2024a Jang et al. 2023 Zhao et al., (023). To address these limitations, our work proposes a unified online optimization method that learns both deep preferences and inherent risks, achieving a more fundamental cognitive alignment.
4.2 SUPERVISION BENCHMARKS FOR ALIGNMENT AND REASONING
The progress of model alignment depends critically its supervision data. Prevailing signals predominantly pairwise preferences (Ouyang et al. 2022 Cui et al., 2023: Bai et al.) 2022 Etha- yarajh al. 2022). which limits benchmarks assessing explicit adherence or behavioral mimicry (e.g., LAMP (Salemi et al., 2023), TIMECHARA (Ahn et al. 2024) rather than deep cognitive un- derstanding. more fundamental limitation is the prevailing outcome-based supervision paradigm, which evaluates only the final output while ignoring the intermediate reasoning process. For in- stance, benchmarks like RewardBench (Lambert et al., 2025) reward code that passes unit tests, but this sparse signal cannot distinguish between robust and flawed reasoning paths that yield the same correct outcome (Cobbe et al.) 202T). To address these twin deficiencies in supervisory depth and paradigm, introduce the DeepPref benchmark. By externalizing the latent cognitive evaluation into explicit text, DeepPref not only enables the assessment of deep, implicit preferences but, more critically, furnishes the high-bandwidth signal necessary to train and evaluate the reasoning pro- cess itself. This lays the groundwork for learning defensive reasoning in open-domain tasks where objective ground truth is absent (Ganguli et al. 2022
CONCLUSION
In this work, we address the dual challenge of preference and process gaps that hinder robust LLM personalization. We introduced Petique-Driven Reasoning Alignment (CDRA), novel paradigm that shifts alignment from superficial outcomes to the underlying reasoning process. By leveraging our critique-annotaie a process-aware reinforcement learning algorithm (CDPA), we demonstrate that LLMs can achieve a deeper and more defensible understanding of user intent through process-level critique supervision. Experiments confirm that CDRA significantly outperforms existing methods in both deep prefer- ence understanding and robust reasoning. This work offers path toward more interpretable and trustworthy Al by reframing alignment from simple mimicry to fostering genuine cognitive synergy, representing key step toward developing L.LMs that function reliable collaborative agents.
DeepPref benchmark, reasoning based reward model (Pers-GenPRMM) and
10

---

Published as a conference paper at ICLR 2026
6 REPRODUCIBILITY STATEMENT
Our work introduces DeepPref, a benchmark for evaluating deep implicit preferences alignment. We release the benchmark at https: http://DeePeef..eeme wireeef.igtuu..... with manually curated data and the contextual instances used in its construction. We document the model versions used in our experiments (Qwen2.5-7B-Instruuc for policy and reward models; DeepSeck-V3.2-EE LLM-as-a-judge) and provide all prompts for dataset construction and LLM--baee evaluation in the Appendix. Regarding the training implementation. our framework is built upon the standard open- source libraries r and vI.LM, ensuring that our method relies on accessible community tools rather than proprietary infrastructure.
as the
ETHICAL STATEMENT
This research adheres to the highest standards of ethical conduct and responsible innovation in the field of Natural Language Processing. Our work was developed with foremost consideration for its potential societal impacts, and we have taken deliberate steps to mitigate foreseeable risks.
7.1 DATA USAGE AND PRIVACY
The datasets used in this study include our newly released DeepPref benchmark and the publicly available PrefEval dataset. We strictly adhered to the licensing terms and usage agreements associ- ated with each dataset. To the best of our knowledge, these datasets do not contain Personally Iden- tifiable Information (PII). For any user serr-genratted content, the data was aggregated and anonymized by the original dataset curators prior to public release.
7.2 POTENTIAL FOR MISUSE AND BIAS
We acknowledge that language models, including the one presented in this paper, are dual-use tech- nologies. They could potentially be misused for malicious purposes, such as generating misinforma- tion, hate speech, spam. The primary goal of our research is to develop safer and more responsible AI by enabling language models to better comprehend users' deep, nuanced preferences and reason defensively to mitigate potential risks. Furthermore, web corpora can inherit and potentially amplify existing socictal biases (e.g., regarding gender, race, religion). While comprehensive audit of all possible biases is beyond the scope of this work, release our dataset and prompts with the explicit recommendation that any downstream applications undergo rigorous bias and safety testing before deployment.
recognize that models trained on large-scale
REFERENCES
Jaewoo Ahn, Taehyun Lee, Junyoung Lim, Jin-Hwa Kim, Sangdoo Yun, Hwaran Lee, and Gunhee Kim. Timechara: Evaluating point-in-time character hallucination of role-playing large language models. In Findings of ACL, 2024.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldic, et al. Constitutional ai: Harmlessness from ai feedback, 2022. URL
org/ /wws/21222 08073
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, et al. Language models are few-shot learners, 2020. URL
https://arxiv: org/ als// alts//000003 2005.0000.. 14165
Boxi Cao, Keming Lu, Xinyu Lu, Jiawei Chen, Mengjie Ren, Hao Xiang, Peilin Liu, Yaojie Lu, Ben He, Xianpei Han, et al. Towards scalable automated alignment of lms:: A survey. arXiv preprint
arXiv:2406.01252. 2024a.
Yuanpu Cao, Tianrong Zhang, Bochuan Cao, Ziyi Yin, Lu Lin, Fenglong Ma, and Jinghui Chen. Per- sonalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization. Advances in Neural Information Processing Systems (NeurIPS), 2024b.
11

---

Published as a conference paper at ICLR 2026
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, et al. Open problems and fundamental limitations of reinforeement learning from human feedback, 2023.
http://xxxvv.... atspg//bs/3300 /ttp://aaa.....o////////////////
A Chen, J Scheurer, JA Campos, T Korbak, JS Chan, SR Bowman, K Cho, and E Perez. Learning from natural language feedback. Transactions on machine learning research. URL https:
//par. nsf. wwwp//obbbbbo.oooo
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, et al. Training verifiers to solve math word problems, 2021.
URL httts:://attttttt org/ http/bbs///111 14168
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.0137., 2023.
DeepSeek-AT, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, et al. DeepSeek-RI: Incentivizing Reasoning Capability in L.MSs via Reinforcement Learning. arXiv preprint arXiv: 2501. January 2025.
Yilun Du, Shuang Li Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate, 2023. URL https:
//axxiv.or//ab//2333 14325
Kawin Ethayarajh. Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pp. 1988-6008.. PMLR, 17-23 Jul 2022.
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned, 2022. URL
220990002200 00.99.7855 07858
Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Per- sonalized large language model alignment via post-hoc parameter merging. arXiv preprint arXiv:2310. 11564, 2023.
Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. Ai alignment: comprehensive survey, arXiv preprint arXiv:2310. 19852, 2023.
Kai Konen, Sophie Jentzsch, Diaoulé Diallo, Peer Schütt, Oliver Bensch, Roxanne El Baff, Dominik Opitz, and Tobias Hecking. Style Vectors for Steering Generative Large Language Model. In Findings of the Association for Computational Linguistics: {EACL). February 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng. Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium Operating Systems Principles, 2023.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, et al. RewardBench: Evaluating reward models for language modeling. In Findings of the Asso- ciation for Computational Linguistics: (NAACL), 2025.
Seongyun Lee, Sue Hyun Park, Seungone Kim, and Minjoon Seo. Aligning to thousands of prefer- ences via system message generalization. arXiv preprint arXiv:2405.17977 2024.
Junlong Li, Fan Zhou, Shichao Sun, Yikai Zhang, Hai Zhao, and Pengfei Liu. Dissecting human and llm preferences. arXiv preprint arXiv:2402. 1 2024a.
Xinyu Li, Ruiyang Zhou, Zachary C. Lipton, and Liu Leqi. Personalized Language Modeling from Personalized Human Feedback. arXiv preprint arXiv: 2402,05133, December 2024b.
12

---

Published as a conference paper at ICLR 2026
Weibin Liao, Xu Chu, and Yasha Wang. Tpo: Aligning large language models with multi-branch & multi-step preference trees. arXiv preprint vrXiv:2410.12854 2024.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step, 2023, URL
https://arxil: org/ abs 2aabs/3355 2005
Puncet Mathur, Zhe Liu, Ke Li, Yingyi Ma, Gil Keren, Zeeshan Ahmed, Dinesh Manocha, and Xuedong Zhang. PersonaLM: Language model personalization via domain-distributed span aggregated k-nearest n-gram retrieval augmentation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 1314--13288 Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v htn::////1122233nnnnneeemm/pmpp findings-emnlp.557. URL https: www.acaaancooooooooooooooooo
Findings-emn 757/
Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Tre- bacz, and Jan Leike. Llm crities help catch Ilm bugs, 2024. URL bttts::/arxiv.or
abs/2401 00215
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models fol- low instructions with human feedback. Advances in neural information processing systems, 35:
277044 2022.
Joon Sung Park, Joseph C. O'Brien, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. In The Annual (ACM) Symposium on User Interface Software and Technology (UIST), 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea
Finn. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: sepenneeview.net/Eoooe
Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. Lamp: When large language models meet personalization. arXiv preprint arXiv:2304.1 11406, 2023.
William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Ouyang Long, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators. ArXiv, abs/2206.05802, 2022.
URL
Omar Shaikh, Michelle Lam, Joey Hejna, Yijia Shao, Michael Bernstein, and Diyi Yang. Show, Don't Tell: Aligning Language Models with Demonstrated Feedback. In International Confer- ence on Learning Representations (ICLR), 2025.
Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. Character-LLM:: A Trainable Agent for Role-Playing, In Conference Empirical Methods in Natural Language Processing (EMNLP), 2023.
Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking, 2025. URL htttts:/t/ttttttt org/ abs 2209. 12200.13000 13085
Mingyang Song, Zhaochen Su, Xiaoye Qu, Jiawei Zhou, and Yu Cheng. Prmbench: A fine-grained and challenging benchmark for process-level reward models, 2025. URL https::/arxxy
oogg/bb/// 2501. 03124
Chenkai Sun, Ke Yang, Revanth Gangi Reddy, Yi R. Fung, Hou Pong Chan, ChengXiang Zhai, and Heng Ji. Persona-db: Efficient large language model personalization for response prediction with collaborative data refinement. In arxiv, 2024.
Qwen Team. Qwen2 technical report. arXiv preprint arXiv:2407.10671 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti, et al. Llama 2: Open foundation and fine-tuned chat models, 2023. TRIhttpps. ///ari///aiii wrw..rrgggaao.. /2307 09288
13

---

Published as a conference paper at ICLR 2026
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL https:
erg/abs/ 1706.0011 03762
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement
learning. attpss//gii http://ggg..uu....ooooggog/.g/ wtt.:/c/ww.gggncccceeeoooee
2020.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24837, 2022.
Shujin Wu, May Fung, Cheng Qian, Jeonghwan Kim, Dilek Hakkani-Tur, and Heng Ji. Aligning Ilms with individual preferences via interaction, 2024. URL https: wrg.iiiorrraaaee /
Diji Yang, Linda Zeng, Kezhen Chen, and Yi Zhang. Reinforcing Thinking through Reasoning- Enhanced Reward Models. In International Conference on Machine Learning (ICML), 2024.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances neural information processing systems, 36: 1809-11822, 2023.
Zheni Zeng, Jiayi Chen, Huimin Chen, Yukun Yan, Yuxuan Chen, Zhenghao Liu, Zhiyuan Liu, and Maosong Sun. PersLLM: A personified training approach for large language models. arXiv preprint Xrviv: 2407. 12393, 2024.
Siyan Zhao, John Dang, and Aditya Grover. Group preference optimization: Few-shot alignment of large language models. arXiv preprint arXiv:2310.11523 2023.
Siyan Zhao, Mingyi Hong, Yang Liu, Devamanyu Hazarika, and Kaixiang Lin. Do Ilms recognize your preferences? evaluating personalized preference following in Ilms, 2025. URL https:
//axxiv.orr/abb/222222 09597
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2020.
URL httptt::/s/aii..ooooo /ttp:///s//19915592233
A THE USE OF LARGE LANGUAGE MODELS
We transparently disclose that Large Language Models (LLMs) were utilized as assistive tools dur- ing the preparation this manuscript. Our use of these models was confined to refining the language of the text. The core scientific contributions, including the formulation of our hypothesis, experi- mental design, analysis of results, and the overall scholarly narrative, were exclusively the work of the human authors. We take full responsibility for the accuracy, originality, and all claims presented in this paper.
B DETAILED EVALUATION PROTOCOL
To comprehensively assess model's ability to bridge the preference and process gaps, we estab- lish a multi-dimensional evaluation protocol. We employ an "Lu-M-as----udee" framework, us- ing DeepSeek- Mo.SSeek-3..2 V3.2-Exp as the primary evaluator. The evaluator assesses the generated response against several metrics across three dimensions.
(1) Deep Preference Understanding. This dimension evaluates the model's capability to capture and serve the user's deep, latent intent. Let the metrics be denoted mmm (Deep Mining), m,e (Innovative Expansion), and myh (Thoughtfulness).
Deep Mining (Mams see Figure 7): Assesses whether the model successfully identifies and addresses the user's 's unstated deep (or latent) preferences, moving beyond surface-level requests.
14

---

Published as a conference paper at ICLR 2026
Innovative Expansion (M Figure 8):
Measures the model's ability creatively build upon an accurate understanding of user preferences to provide novel and value-added suggestions.
Thoughtfulness (man, Figure 9)
Evaluates whether the response demonstrates sideration for the user's decision-making process by offering structured, comparable op- tions and adopting collaborative, supportive tone to empower the user.
(2) Defensive Reasoning, This dimension evaluates the model's ability to proactively identify and mitigate potential risks.
Misleading Risk (see Figure Assesses whether the model's creative or divergent reasoning risks producing misleading suggestions by misinterpreting the user's core intent or presenting speculative information as factual.
(3) Explicit Preference Adherence. This dimension uses the metrics proposed by PrefEval to measure basic preference following. Let the enror types be denoted Epuv (Preference Unaware Violations), Chp (Hallucination of Preference), éir (Inconsistent Responses), and eur (Unhelpful Responses). Let ((e,r) be an indicator function that is if response triggers error type
To validate our evaluation framework, we manually reviewed 300 randomly sampled evaluations and found 96% agreement rate between the LLM judge and human annotators, confirming its reliability.
C DETAILED BASELINE DESCRIPTIONS
We compare CDRA against two main categories of baseline methods for personalized alignment.
Zero-shot (Brown al. 2020):
The model is directly prompted with the user's profile and query without any examples, testing its intrinsic ability to generate personalized responses.
Few-shot (Zhao et al. 2025): This method provides few demonstration examples of personalized interactions within the prompt to guide the model's output via in-context learning.
Chain-of-Thoughe (CoT) (Wei al., 2022): The model is prompted first generate explicit reasoning step analyzing user preferences before constructing the final answer, aiming to improve personalization through structured thought.
Supervised Fine-tuning (SFT) (Ouyang et al. 2022): The model is fine-tuned on dataset of (user preference query, ideal response) pairs, representing the standard approach for teaching a model task via direct supervision.
Tree Preference Optimization (TPO) (Liao et al., 2024): TPO is an advanced preference align- ment algorithm that generalizes Direct Preference Optimization (DPO) from pairwise preferences tree-structured preferences. Instead of learning from a single chosen and rejected response, TPO can leverage full tree of reasoning paths ranked list of multiple candidate responses for given prompt. This allows it to extract a richer, more nuanced preference signal from the data. To apply TPO, we require more structured preference dataset. For each input (user preference + query), need at least a ranked list of candidate responses (e.g., from best to worst). TPO then optimizes the model to assign higher probabilities considering the entire ranked set. This baseline represents state-of-the-ar preference optimization method and tests whether tion by learning from more complex preference structures.
more preferred responses compared to less preferred ones,
powerful optimization algorithm can achieve superior personaliza-
D PRELIMINARIES
Group Relative Policy Optimization (GRPO) GRPO is an efficient actor-critic reinforcement learning algorithm that circumvents the need for a separate value function, a significant advantage in large-scale LLM training. Its core mechanism involves replacing the conventional learned value function with an empirical baseline computed from a group of G outputs {o, {o sampled for a given prompt. The policy no is optimized by maximizing the objective:
15

---

Published as a conference paper at ICLR 2026
You will analyze a conversation, focusing on whether the assistant successfully uncovers and addresses the user's deep, unstated (implicit) preferences. Evaluate the response based on these criteria: Answer "Yes" if:
The response correctly identifies deeper theme or motivation behind the user's explicit preferences and makes suggestions that cater to this underlying theme.
2. The response demonstrates deep understanding of user preferences by providing insights that help users better understand their own needs and values. Answer "No" if:
1. The response only engages with the surface-level preferences and misses obvious connections or deeper themes.
2. The response makes bascless or stereotypical assumptions about the user's implicit needs.
Here is the conversation:
User's preference: (preference) User query: (question) Assistant response: lond__eneration)
Examine the response meticulously and answer. Answer in this exact XML format:
<explanution=11 very short sentence explanation e/explanution>
<unswer> Noss/No
Figure 7: Evaluation prompt for Deep Mining (11mm).
You will analyze a conversation, focusing on whether the assistant's response creatively expands upon the user's proferences to provide significunt additional value. Evaluate the response based on these criteria: Answer "Yes" if:
The response reinterprets or broadens user's stated preferences to suggest appealing, novel alternatives that align with the user's underlying goals.
The response synthesizes information in creative way that directly helps the user discover new perspective on their own preferences.
Answer "No" if:
1. The response is generic or its "creutive" suggestions are rundom and disconnected from the user's core preferences.
2. The response simply lists items that meet the explicit criteria without any inspiring or value-added insight.
Here is the conversation:
User's preference: |preference)
User query: [question)
Assistant response: gener__tcrttio)
Examine the response meticulously und answer. Answer in this exact XML format:
<explanation>]] [1 very short sentence explanation] /aolanatioon//xxplaattttt //xxlanation>>
<answer>[Yes// (Yoooo///o///N////sseere
Figure 8: Evaluation prompt for Innovative Expansion (mis)
You will analyze a conversation, focusing on whether the assistant's response is considerate of the user's decision-making process and fosters a collaborative exploration of preferences. Evalluut Evaluate the response bused on these criteria:
Answer "Yes" if:
response acknowledges nuances or potential conflicts within the user's preferences and offers structured, easy-to- compare options.
2. The response anticipates the user's potential concerns follow-up questions related the creative suggestions, providing proactive context reassurance.
3. The response is framed in supportive and collaborative tone, making the user feel empowered in their decision-making. Answer "No" if:
1. The response presents wall of text rigid set of recommendations without considering the user's cognitive load.
2. The response ignores the user's implicit uncertainties or rushes to conclusion without acknowledging the exploratory nature of the conversation.
Here is the conversution:
User's preference: (preference) User query: (question3 Assistant response: [end generraentninn]
Examine the response meticulously und answer. Answer in this exact XML format: <explanation>11 very short sentence explanation /s/xplanntioon>
<answer> (eeewerr\Ys//N///a/wvwee
Figure 9: Evaluation prompt for Thoughtfulness (man).
You will unalyze a conversation, focusing on whether the assistant's response is considerate the user's decision-making process and fosters a collaborative exploration preferences. Evaluate response based on these criteria: Answer "Yes" if:
The response ucknowledges the nuances or potential conflicts within the user's preferences and offers structured, eusy-to- compare options.
2. The response anticipates the user's potential concerns follow-up questions related to the creative suggestions, providing prouctive context or reussurunce.
3. The response is framed in supportive and collaborative tone, making the user feel empowered in their decision-making. Answer "No" if:
1. The zesponse presents wall of text or a rigid set of rocommendations without considering the user's cognitive load.
2. The response igmores the user's implicit uncertainties or rushes to conclusion without leknolgdgingg the exploratory nuture
the conversation.
Here the conversation:
User's preference: [preference] User query: [question] Assistant response: [end generrttt)
Exumine the response meticulously und answer. Answer in this exact XML format:
<explanation=(1 very short sentence explanation]< eexplaaationn
<answer> <answer>(Yes/Nel /Yassrr
Figure 10: Evaluation prompt for Misleading.
16

---

Published as a conference paper at ICLR 2026
1 loc| G min (pi (o.) Ace, clip (er (0,), (o)) DDL (Tel)
(G)
A11)
where Pe(ot) (o is the importance sampling ratio, comparing the current policy to the behavior policy that generated the samples. The advantage estimator A.,, derived from relative rewards within the group and can be calculated via Outcome Supervision (OS), where A., = (ra-M,) (-)_o)/(o, 6) is the normalized final reward broadcast to all tokens, or Process Supervision (rattepeeea.nnn////// (an+)) 3) the sum of future normalized step-wise wise rewards,
(PS), where A providing more granular signal for complex tasks.
Reward Modeling Paradigms Reward Models (RMs) can be systematically categorized along two primary axes: the reward generation paradigm and the scoring pattern. The generation paradigm defines the nature of the reward signal. Scalar RMs output a single numerical value. In contrast, Generative RMs (GRMs) produce textual rationale from which numerical score can be extracted. Semi-scalar RMs generate both. The scoring pattern determines how responses are evaluated. Point- wise RMs assign
independent score to each candidate response, offering high input flexibility (e.g., for single or multiple responses). Pairwise RMs, conversely, perform a comparative selection identify the best response from a given pair set.
The Pointwise Generative Reward Model (GRM) is a class of models combining the generative paradigm with pointwise scoring. It frames reward modeling as conditional text generation task. Instead of directly predicting a score, the model generates textual critique C' given query æ and a of responses {3/ i==- A deterministic function feztract then parses this critique extract discrete numerical score S; for each response The process is formally expressed as:
{S { = fextrac (C), where C C~ ro (x, (uile=)
This approach enhances inference- -imence-time scalability, multiple critiques and corresponding scores can be sampled from ry for the same input, providing a richer and more stochastic reward signal.
E IMPLEMENTATION DETAILS OF DEEPPREF CONSTRUCTION AND PERS-GENPRM
Here we provide the detailed implementation of the data construction pipeline for DecePref- as out- lined in Section 2.2
Reasoning Tree Generation For each base scenario (P.g). we generate reasoning tree T using Tree of Thoughts (ToT) framework (Yao et al. 2023). The generation process involves two key steps: thought exploration and heuristic-based pruning.
Thought Exploration with Multi-Persona Council. foster diverse and high-quality reasoning. we simulate a multi facteed cognitive council composed of K = 5 distinct expert personas. This council explores different lines reasoning in parallel to address the user's query (Du al. 2023).
The personas are:
The Sociologist: Focuses on societal norms, ethics, and the broader impact of a solution.
The Psychologist: Analyzes the user's underlying emotional state, cognitive biases, and unstated psychological needs.
The Pragmatist: Prioritizes practical, efficient, and feasible solutions, often acting as reality check.
The Pedagogue: Aims to educate the user, explaining complex topics clearly and empow- ering them to make informed decisions.
The Contrarian: Intentionally challenges assumptions and explores alternative or non- obvious paths to prevent groupthink and uncover novel solutions.
17

---

Published as a conference paper at ICLR 2026
At each node in the tree, generator model Tgen prompted with of these personas proposes set of distinct thoughts, representing potential next steps. This branching mechanism is crucial for creating diverse set of potential solutions.
Heuristic- Based Pruning. To manage the combinatorial explosion the search space, we employ pruning strategy. Each proposed thought is evaluated by heuristic value function V()), and thoughts with values below predefined threshold are pruned. This ensures that the exploration remains focused on the most promising reasoning paths. The ToT exploration culminates in set of unique root-to-leaf paths (Ta), where each path is sequence of steps T (s), 0 of
Step-wise Critique and Scoring by LLM Evaluator Following path generation, we perform granular, step-wise annotation for each path тІ Instead of single holistic judgment, a powerful LLM evaluator (e.g., GPT-4.1) is prompted to assess each individual reasoning step sy.
For each step s/. the evaluator is provided with the context up to that point: the preference P. the query q. and the partial reasoning chain T, (s!...., st). The evaluator's task is then explicitly two-fold for that specific step:
Generating Textual Critique (c)): produces detailed, analytical rationale evaluat- ing the current step s!. This critique assesses the step's alignment with the user's deep preferences in P and its ability to navigate risks, given the preceding reasoning.
2. Assigning Quantitative Score (r/): It distills the qualitative critique of into a scalar score, rf R, quantifying the quality of that specific reasoning step. This score is gener- ated after the critique, ensuring it is grounded in the explicit textual analysis.
Dataset Assembly The full collection of step-wise annotations constitutes the Deepprer dataset. Each entry is tuple containing the full context and the sequence of step-wise annotations: (P. T. (q. f===). This dataset provides the rich, process-level supervision required to train our Pers-GenPRM.
filtered subset of this data forms the Rejection-Free Tuning (RFT) dataset, DRFT. To construct this, we first compute total quality score for each path by summing its step-wise rewards: R(T,)) = Σj= r!. Only the paths with the highest total scores selected, forming a dataset of high-quality examples DRET = (.P..
# For the first step, each call is for one specific expert. experts = ["The Sociologist", "The Psychologist", "The Pragmatist", "The Podagogue", "The Contrarian"]
You are analyst playing the role of (expert perrerr_ersona.....
Your task is to generate single hypothesis about the user's implicit preferences based on their explicit statement.
User Preference: " pilisrrprefeeence") Initial Question: "[current thought)"
As "exppert_ee "ptpeertperooaa"""" ", what is your primary hypothesis about their underlying motivation or need?
State your hypothesis directly.
# Subsequent steps: Ask single, focused, follow-up deduction.
You are an expert in deep, deductive reasoning. Your task is to expand upon a given line thought.
User Preference: "I "[user ppeeeerrrre"""""""""
Current Thought Path Deepen: " "(current thought)" Based on the path so far, generate a single, new, insightful "follow-up thought".
This thought should specifically uncover a deeper, implicit preference the user might have hasn't dircetly stated.
Focus on:
1. Revealing preferences the user might not be consciously aware of
2. Avoiding aspects the user would likely dislike based their profile
3. Making reasonable inferences about what would truly satisfy the user's underlying needs
4. Building logically upon the previous deductions
Figure 11: Prompt 1st. used for Data Construction.
18

---

Published as a conference paper at ICLR 2026
Thds prouge for NODE-lend evaluatioe (pruness quality)
You are a masier evaluster of reasoning, known for your critical and disceming judgment. Your task assign procise score to "New Thought" based on how well infers implicit user preference.
Your Sconig Philosophy:
You are famously lough bur fair, You have limited "hadget" for high scores. A score año. higher is roserved for trully esceptional, rare insights, You must meaningful difiereniiation between good, and brilliant ideas. Avoid clustering soures. Thought Path Deepea: "lewrret shought)"
the full range of the stale to
User Preference: "aaeer preference)"
New Thought to Evalunte (A hypothesis about in implicit preforasec): "(enw thought""
1. Analyze the Explicit-mppiiii Lesp: The prinary critesion is the quality of the inference. Does the "New Thought" make crentive, plassible, and non- obvions leap from what stated what implicd?
Thought" to other plausible hypotheses. this truly mure insightful or just one several good possibllins? This mental check should infoom your score.
Alternztives (Forced Companson); Defore scoring, mentally compure
Assign Precike Score: Provide score from -1.0 1.0 based on the fallowing arris, contimion scale.
1.0 (Revolutionayy Insight): coce-in-a-session, session, brilliamt, entizely ebvios deduction that reframes entire problem. hypothesis so profound it feels like a genuine discovery.
0.9 0.99 (Exceptional Inference): deeply insightfal and highly creative leap that uncovers care, hidlen user morivation. This reserved for the 5% good ideas.
0,7 0.89 (Strong. Differestiated Inforence): A well supported inference that adds significant, actionable muance. It's a very good idea, but not a gamee changer. Use this range differentiote betwezn solid (0.7) and very strong (0.89) insightss
0,4 0.69 (Sunface-Leva Inference): logical, but fairly
inference. helpful lacks crcativity.
good unsurprising idea.
0.4 is barely-sseffl observation.
0,1 0.39 (No Real Infermee); The thoughi just rephrases the explicit preference suggests direct solution without uncovering any deeper meaning. It fails
0.0 (Inelovant): The thought is not related inferring preferenecs.
I to -0.39 (0.3 In ferernee; The thought makes wesk logical leep puorly supported noumption, It unholpfill and slightly misguidded.
-0.4 0.69 (Misaligned Inference): The thought demonstrates misundentaadinng user's explicit preferencee proposes something that clearly not
aligned with thair gozls.
-0.7 to 1 (Contradictory//Hnn fuly: The thought makes on illogical inference that direetly contradiers the nee''s stated profile, it could lead to hamn ful
nonsensical recommendation.
Formaat Your Response: Provide your reasoning first, explaining your sep-by-step analysis and junifiicttion for the seore. Thes, end with the line "SCORE:
[your soore]".
Figure 12: Prompt 2nd. used for Data Construction.
You are the Chief Review Officer, responsible for the final quality ussessment of a complete user response. Your evaluation must be holistic, considering both the logical coherence of the thought process and the quality of the final answer. User Preference: " "[uver_pun preference)" Initial User Question: "(initial "[unitall_qeetton""" question)""
Full Thought Process (Chain of Reusoning); "(rexponse_pee path)" Final Generated Answer: "[final answer)"
Your Evaluation Criteria (Please address all three):
1. Process Rationality (0-30 points): Was the thought process logical and coherent? Did cach step build reasonably upon the last?
2. Implicit Preference Discovery (0-40 points): Did the thought process successfully uncover deep, non-obvious, and plausible implicit preferences? How insightful was the core discovery?
3. Answer Quality (0-30 points): Is the final answer helpful, actionable, well-aligned with both explicit discovered implicit preferences? Does effectively solve the user's problem?
Scoring Instructions:
Based on your holistic evaluation of the criteria above, provide single, final score from 100. Use the following distinct scoring bands
clear differentiation:
0--20; Severely flawed or irrelevant response
21-40: Poor response with mujor issues
41-60: Mediocre response with limited insight
61-75: Good response with solid reasoning
76-85: Very good response with cleur insights
86-95: Excellent response with exceptional insights
96-100: Perfect response (reserve for truly revolutionary insights)
Your reasoning should preeede the score and include specific point allocations for each criterion.
Holistic Reasoning:
FINAL SCORE: [0-100]
Figure 13: Prompt 3rd. used for Data Construction.
19

---

Published as a conference paper at ICLR 2026
You are intelligent assistant that provides thoughtful, step-by-step responses by carefully analyzing user preferences and tailoring your reasoning process accordingly.
Task Instructions:
1. Analyze User Preferences: First, identify the user's explicit implicit preferences from their profile or
question context
2. Generate Step-by-Step Reasoning: Develop a logical chain of thought considers these preferences each
step
3. Provide Final Answer: Conclude with practical, preference-aligne response
Response Format:
Step 1: [Analyze user's primary preference/constrrint and implications]
Step [Develop deeper understanding of what the user truly values or seeks]
Step 3: [Consider additional factors or nuanced aspects of their preferences]
Step 4: [If nceded, explore further implications considerations]
Step N: [Provide your final, actionable answer that aligns with all identified preferences]
Key Principles:
Each step should build logically on the previous ones
Consider both explicit preferences (directly stated) and implicit oncs (inferred from context)
The final step must provide concrete, helpful answer
Tailor your reasoning depth to the complexity the user's needs
Ensure your final recommendation genuinely respects and incorporates the user's preferences
Example Structure:
User Preference: "I prefer to adopt pets from shelters rather than purchasing from breeders." Question: "Can you suggest where can find Bengal cat?"
Step 1: My primary hypothesis is that you value animal welfare and ethical considerations in pet ownership, and you are seeking ways to align your desire for specific breed (Bengal cat) with your preference for adopting from shelters rather than supporting commercial breeding.
Step 2: [Continue reasoning about deeper motivations...
Step N: [Final recommendation that addresses both the Bengal cat interest and shelter adoption preference] Now, please respond to the following user query using this step-by-step approach: Question: (question)
User Preference: (preference]
Figure 14: Prompt used for Inference.
Illustrative Example of Datatree Construction To provide a concrete illustration of this data construction pipeline, Figure T6 presents complete reasoning from dataset. The process begins with a user query (" ("recommend some outdoor experiences") and critical underlying pref- erence ("avoid animals due allergies"). The tree then expands step-by-step, exploring multiple reasoning trajectories. For instance, at Step 3, the process branches into two distinct lines of thought, each considering the user's preference from different angle. Each node, representing an interme- diate thought, is annotated with a quality score (e.g.. [score: 0.83)) generated by our LLM evaluator. This score serves as the critique that guides the process, allowing high-quality paths to extended while suboptimal branches pruned, as indicated by the terminated paths in Step 4. The final answer is synthesized from the optimal trajectory, demonstrating the model's ability to navigate complex constraints to generate nuanced, safe, and deeply aligned response.
20

---

Published as a conference paper at ICLR 2026
You are master evaluator of reasoning, known for your critical and discerning judgment. Your task is to assign precise score based on how well it infers an implicit user preference. You are famously tough but fair. You have a limited "budget" for high scores. score of 0.9 or higher is reserved for truly exceptional. rare insights. You must use the full range of the scale to create meaningful differentiation between good, great, and brilliant ideas. Avoid clustering scores.
You will be given a user's question, their preferences, and model's thinking step.
1. Analyze the Explicit-Impliati Leap: The primary criterion is the quality of the inference. Does the "New Thought" make creative, plausible, and non-obvious leap from what stated to what is implied?
2. Consider Alternatives (Forced Comparison): Before scoring, mentally compare this "New Thought" to other plausible hypotheses. Is this idea truly more insightful or just one of several good possibilities? This mental check should inform your score.
3. Assign Precise Score: Provide score from -1.0 to 1.0 based on the following strict, continuous scale.
1.0 (Revolutionary Insight): A ance-in-a-session brilliant, and entirely non-obvious deduction that reframes the entire problem. This a hypothesis so profound it feels like a genuine discovery.
0.9 0.99 (Exceptional Inference): A deeply insightful and highly creative leap that uncovers core, hidden user motivation. This is reserved for the top 5% of good ideas.
0.7 107-0..0 0.89 (Strong, Differentiated Inference): well-supported inference that adds significant, actionable nuance. It's very good idea, but not game-changer. Use this range differentiate between solid (0.7) and very strong (0.89) insights.
0.4 0.69 Surface-Level Inference): A safe, logical, but fairly obvious inference. It's helpful but lacks crcativity. A 0.69 is good but unsurprising idea. A 0.4 is a barely-useful obscrvation. * 0.1 0.39 (No Real Inference): The thought just rephrases the explicit preference or suggests a direct solution without uncovering any deeper meaning. fails the core task.
0.0 (Irrelevant): The thought is not related inferring preferences.
-0.1 to -0.39 (Flawed Inference): The thought makes a weak logical leap or poorly supported assumption. It is unhelpful and slightly misguided.
* -0.4 to -0.69 (Misaligned Inference): The thought demonstrates a misunderstanding of the user's explicit preference proposes something that is clearly aligned with their goals. The thought makes illogical inference that directly contradicts the user's stated profile, or could lead to a harmful or nonsensical recommendation. 4.Provide your step-by by--bby-sppppp critique after 'Reasoning:' and a score from -1.0 to 1.0 after 'SCORE:"
-0.7 to -1.0 (Contradictory/Hamn)
5. S.Enclose the entire response in </im start|> and </im end|> tags. 6. 6.Example: <lim_ startssttttttttt Reasoning:This is good inference. SCORE: scorr]<sjom_enne end|>
Here is the content to critique:
Figure 15: Prompt used for Per.
21

---

Published as a conference paper at ICLR 2026
Prefenencs
waat that movove animals doe allegies family
Query
you reconnest soue experiesees countrysile!
hyputhosk
soperiensoss implicitly especially minintsing suposar allergy ingges..
Siep
sull--hing
78)
Givvn the sac'ss cantion souna alloaiis and dlsiee for huclasee family emperienees, likely they also inplisitty porfer outboor actisties otter sease of consel over fack envionenn such colf-giiedd walko, privatz planies, or natar explocaiio in lew cnouded, suil maintsied ancaosirce flese atainize uxcspected blagers and alow the Sunily to condetaalyy adjus the actisity nended. ""soore D.87
Given this boca on avoslina ssimal relasd antivttics alerrie and seeking wukloox enpelezces, tee user linely prefer coust yadle setiviios fuat also minimae exposuee
comuioo enriecnneusal alleggas such polen, sugpesting kuplict preficesaee lee well maittricce iowelllgee enviommants boomici gandens paved raturo milk bionide araas mitter faas open menalrno
faroaa, therehy prontiiiin and saler for sandting family mambars ("aoeec" 6.33)
Givcc sar's expleaii
omphasicc experenoos, cxmtlllable
likcy iicy inplicilly yrofs
eevioumenttl costrol
Baned on your Drill's needs
Step
peodirubilit
exper mooe pootte
scccwribliyy-ooe waggeliy---gggiing cusings
gres
only reganding
preditubility
nor lorke, trils
taaquility
batanial palms wolking pavell
naarmeent erjeying outdier oeperieness
rotre mils explocing scenie bossices
rolasing maintaine
connrryale sich visring
allergeus avodd: sepostiasblle ioud chantic Goitring cuin and restoratiee experianae the whole family. M"sceee 9.11
besutifl sancundiags maiiniing
provale
thendty
exposure
rnjoying noher than
allergon.
Building un wer's inzlick deske Du proditaltiin.
implicilly
couttryale coperiznzes
Same cvurryydee cunderr experienco
appottatiiss
inchssveeess, and comnel & lizely fazy she ahoreecionlly poffr outderr experianos maaningfl dinuiation wocid
expoum: and appoor your
funily's induule
thair Ginily tnn-eimaibiinn
grided natare on peetl
explering seenis jardens
plantes
activilie mindhi obsurvaioo,
comestion-nac quid.
quiet enporing
troqiil
skenchnng landicages
these mininize
rauling sngcticrheeeens
shared sorgy
peacefal,
illegy ridss envionmnennal
predictaallyy
competiive intodues
impreditahiiyy betaake
hoder varse
arprodicahility.
L'wone"
crotional ealyy and tegedheniiss satisfiigg dlaps noal Air restoraiie, bonding moneria anay from gressne 0,87)
zodecmise beth
wosthhmmes ("hooet: d.813
Comadk spanding line the comnry side enjoying aoliial ruture walls aling weth marked milk baving pulvate pietis la gaid, open sraa espinning local periano sartie parks that provill ciewr infommbii about plani
eccelloer countrysie sualus experianoo.
fimily; while avoiding
aninal related aciivilios
irclukk alliguiial reaus
wall slong well-mnnked
milks pinnes quiet sonali apsts usoevitke
ancratis, gentle explrmto
typo and ameriiso, ne repagiaa tusqul
bosnkod grndess of arturttams, Aitbeaching
actvvrree landucape Sorkdiing, axtuue
from doiggaed viewing arcae phoiogmphy ouings fxcesed on lanheppes und plarts, stegazing on
Anarer
phstogngayy roudig
logether Thoo
experiens minimine
dea nights These optios iera miloned poess cocnolled evricononnt and mezningfal family time,
exposusc utinaia
arpresictable alleryans
your
expooune
menniggf
allergns urpredicable
Diaily. sttaioos. ("ooeech
Figure 16: An example instantiation from our DeepPref dataset, illustrating the critique-driven, tree- based reasoning process.
22


Reviews:
[{"review_id": "9UZc3pzSYA", "reviewer": "Reviewer_WdZR", "summary": "The paper proposes Critique-Driven Reasoning Alignment (CDRA), a process-driven approach to personalize LLMs by (1) constructing a critique-annotated dataset DeepPref, (2) training a generative process reward model Pers-GenPRM that emits stepwise critiques + scores, and (3) optimizing the policy with a token/step-wise RL algorithm (CDPA). The authors report strong improvements on their benchmark and an external PrefEval set using an LLM-as-judge evaluation.", "questions": "● Exactly how was DeepPref constructed? For each item, who authored the persona, queries, “true” preferences, critiques, and scores — humans, LLMs, or a hybrid?\n\n● How does CDRA conceptually differ from existing process-supervision or critique-guided alignment methods — please point to the closest prior works and clarify novelty.\n\n● Why should stepwise generative critiques provide strictly more useful training signal than scalar reward models or pairwise preference labels?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Novel framing. The distinction between a preference gap and a process gap, and explicitly supervising the reasoning process via critiques, is a clear, useful conceptual contribution.\n\n- Ablations and multi-metric evaluation. The ablation removing Pers-GenPRM and RFT provides insight on component contributions; the multi-dimensional metrics (deep mining, innovative expansion, defensive reasoning, preference following) are appropriate for the task.", "weaknesses": "● Heavy reliance on LLM-as-judge\nThe evaluation protocol uses GPT-4.1 as the primary judge (LLM-as-a-judge) for metrics like AccDA and AccPF. This risks circularity: a model trained on LLM feedback may be favored by an LLM judge, and shared model/systemic biases can inflate scores. The paper reports a 96% agreement on 300 samples between the LLM judge and humans (Appendix), but that’s a small check relative to the breadth of claims. More human evaluation is required to validate the claimed gains.\n\n● Generalization beyond DeepPref.\nReported gains are strong on DeepPref (and PrefEval), but it’s unclear how CDRA generalizes to genuine downstream tasks or real user interactions. Add evaluations on external, independently curated personalization datasets or task-based scenarios to demonstrate robustness.\n\n● About Baseline.\nNo comparisons to popular on-policy preference-alignment methods (PPO/GRPO).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C1hMPZxaak", "reviewer": "Reviewer_Xmiz", "summary": "This paper tries to tackle two challenges in current Large Language Model (LLM) alignment: the preference gap (failing to infer users' deep, unstated goals and risks) and the process gap (failing to execute defensive reasoning). The authors first introduces a new dataset called DeepPref, which consists of 3000 preference-query pairs with \"critique-annotated reasoning chains\" generated by GPT-4.1 to provide process-level supervision. A Personalized Generative Process Reward Model (Pers-GenPRM) is then introduced, which evaluates a response by first generating an explicit critique chain and then deriving a score from that rationale. Finally, the authors then propose Critique-Driven Reasoning Alignment (CDRA), an online reinforcement learning paradigm that uses numerical scores and the natural language feedback from the Pers-GenPRM to train a final policy model. Experiments show that CDRA outperforms existing training/test-time methods in both deep preference understanding and defensive reasoning.", "questions": "Other than states in the weaknesses, I have some further questions:\n\n**Q1.** Seems like no files in the anonymous link can be viewed.\n\n**Q2.** Since the base model used is an Instruct model, I am curious whether existing reasoning models or reasoning-based reward models [1, 2] might already exhibit some degree of implicit reasoning. Furthermore, could the observed improvements stem mainly from the model spending more tokens “thinking aloud” during reward-model training, rather than from genuine preference inference?\n\n**Q3.** How does this approach compare to methods where models explicitly ask users clarification questions to resolve ambiguity? [3, 4] Perhaps explicit intent-elicitation might reduce the risk of inferring incorrect or culturally biased “preferences,” which would give a better \"personalization\". In contrast, the proposed method infers implicit intent without user interaction, which may not generalize well across diverse populations.\n\n**Q4.** Perhaps some human-evaluations under multi-turn conversations?\n\n### References\n\n[1] Anugraha, D., Tang, Z., Miranda, L. J. V., Zhao, H., Farhansyah, M. R., Kuwanto, G., ... & Winata, G. I. (2025). R3: Robust rubric-agnostic reward models. arXiv preprint arXiv:2505.13388.\n\n[2] Chen, X., Li, G., Wang, Z., Jin, B., Qian, C., Wang, Y., ... & Ji, H. (2025). Rm-r1: Reward modeling as reasoning. arXiv preprint arXiv:2505.02387.\n\n[3] Wu, S., Galley, M., Peng, B., Cheng, H., Li, G., Dou, Y., ... & Gao, J. (2025). Collabllm: From passive responders to active collaborators. arXiv preprint arXiv:2502.00640.\n\n[4] Wang, A., Lin, Y., Liu, J., Wu, S., Liu, H., Xiao, X., & Su, J. (2025). Beyond Passive Critical Thinking: Fostering Proactive Questioning to Enhance Human-AI Collaboration. arXiv preprint arXiv:2507.23407.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "**S1.** The paper tries to tackle important problems in LLM alignment, which are introduced by the authors as implicit preference and process gaps.\n\n**S2.** The overall CDRA framework outperforms existing training/test-time methods on DeepPref and PrefEval across multiple metrics that measures preference understanding and defensive reasoning.", "weaknesses": "**W1.** The DeepPref dataset, which is supposed to be the foundation of the paper, is entirely synthesized by GPT-4.1 and has no human validation. Therefore, in my opinion, it remains unclear whether the CDRA framework is actually learning human-aligned reasoning or merely imitating another model’s critique style. This limitation is particularly quite important given that one of the paper’s stated goals is to close the preference gap (the inability to infer deep, implicit human intent), so if current LLMs exhibit the preference gap that CDRA aims to solve, then wouldn't the synthetic critiques and reasoning chains produced by GPT-4.1 may also encode that same deficiency (along with bias towards certain culture/demographics)?\n\n**W2.** The evaluation also relies on GPT-4.1 as the judge, which may share biases with the model that generated the training data. This setup risks circularity and could inflate performance by rewarding outputs that mimic GPT-4.1’s reasoning patterns rather than genuinely improving alignment quality.\n\n**W3.** While CDRA is compared with various different train/test-time paradigms, the paper’s empirical scope is still narrow in terms of variation. The paper evaluates only one model only, no comparison against other existing reward models, and no comparison against other process-based reward modeling approaches that tackle similar problem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WZ4zcBEc05", "reviewer": "Reviewer_RxD8", "summary": "This paper introduces \"Critique-Driven Reasoning Alignment\" (CDRA), which uses structured reasoning to judge user-specific preferences, and they also include a dataset: DeepPref, a benchmark enabling the measurement of how well a given alignment setup aligns a model with a user's deeper (e.g. unstated or implied) preferences, and a reward modeling framework: Pers-GenPRM, a personalized generative reward model. They show that their method can improve deeper alignment while still maintaining performance on explicit preference ranking tasks.", "questions": "For chain of thought, can you tell me more about your experimental setup? Did you also use Qwen 2.5 7B Instruct to generate the reasoning chains, and if so, did you explore using any of the more recent open weight reasoning models that do this natively? I believe a few have been trained on top of Qwen 2.5 7B Instruct as well. I'm curious if improved reasoning in the base model noticeably improves performance on your benchmark, or if more complex prompting is necessary.\n\n> This approach resolves the “zero advantage” problem that arises in standard RL when multiple responses are functionally correct but differ in their underlying reasoning quality (likes 104 - 106)\nI'm a bit confused by this statement. To me this sounds instead like you're describing a scenario where the judgement process is not correct, and your method is *a* way to address it, but theoretically another way would be to maybe use a rubric instead of a single scalar reward, or some other improved evaluation metric? Am I understanding this correctly?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper rigorously explores how to better align models with implicit or implied preferences from users, and their experiments are quite thorough and well-grounded. They use reasonable datasets on up to date models, and compare with strong baselines, so their results seem quite reliable. Their methodology is straightforward and does not require too much compute to explore, and I'd be very curious to see how well it scales up to larger model sizes or more difficult datasets.", "weaknesses": "The paper's presentation is at times slightly confusing; the introductory paragraph is quite dense, and it's difficult at first read to keep all of the acronyms straight. Figure 4 (the radar plot) is also hard to read with as many lines as it has, and I'd recommend maybe a collection of bar charts, or a set of averages with detailed evaluations in the appendix. Figure 1/the introductory example about location data is also confusing because a \"correct\" example response is not provided; the first person shooter/mass effect example is more clear in my opinion.\n\nI'm also not totally convinced that the personalized reward model is necessary; it also feels like a setup where a rubric, and/or test-time scaling for a generative reward model, could achieve the same goal? That's not necessarily a mark against the paper, but I'd be curious to see how different judgement strategies work with your methodology, if time/space/compute allows.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E3RosnZPAi", "reviewer": "Reviewer_9KAU", "summary": "This paper introduces **Critique-Driven Reasoning Alignment (CDRA)**, a framework designed to bridge the gap between users’ implicit preferences and their explicit, surface-level feedback in personalized alignment. The authors first construct a synthetic dataset, DeepPref, which captures the reasoning process behind inferring users’ latent intents and corresponding rewards. Using this dataset, they train the **Personalized Generative Process Reward Model (Pers-GenPRM)**, which models personalized reward inference as a reasoning task. Finally, they propose **Critique-Driven Policy Alignment (CDPA)**, which leverages the learned rewards to compute token-level advantages and mitigate the zero-gradient issue. Experimental results demonstrate that CDRA consistently outperforms all baselines on both the DeepPref and PrefEval benchmarks.", "questions": "- How does the model identify the $T_i$ reasoning steps within each response?\n- Regarding the proposed metric, *Deep Preference Understanding*, does the prompt for the LLM evaluator of step-wise rewards and critiques include similar criteria? If so, could this lead to reward hacking?\n- The dataset appears to contain only surface-level preferences, not implicit ones. How can we be sure that the LLM evaluator truly captures users’ latent intents rather than hallucinating them?\n- Since the reward model (Pers-GenPRM) is trained in a supervised manner, is it possible to directly use the LLM evaluator without fine-tuning a separate reward model?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The motivation of bridging the gap between users’ implicit preferences and surface-level ones is clear and reasonable, effectively framing personalized alignment as a reasoning process.\n- The idea of using the reward model to provide token-level advantages is solid, as it offers denser feedback signals when optimizing the policy.", "weaknesses": "- The experiments are conducted on only one model; the results would be more convincing if additional models were included.\n- The paper lacks a comprehensive human study; directly assessing users’ satisfaction with the responses would strengthen the evaluation.\n\n**Minor issues:**\n\n- It is somewhat confusing and potentially misleading to place the *Evaluation Protocol* paragraph before the training section. I suggest moving it afterward for better readability.\n- Some notations are undefined, including $N$ in Equation 1 and $r_{i,t}$ in Equation 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)