P-GenRM: Personalized Generative Reward Model with Test-time User-based Scaling
Authors: Pinyi Zhang
Ting-En Lin
Yuchuan Wu
Jingyang Chen
Zongqi Wang
Hua Yang
Xu Ze
Fei Huang
Yongbin Li
Kai Zhang
Venue: ICLR 2026 Oral (2026)
Keywords: personalizd alignment
generative reward model
test-time user-based scaling
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: The first personalized generative reward model with test-time user-based scaling for preference alignment

Abstract: Personalized alignment of large language models seeks to adapt responses to individual user preferences, typically via reinforcement learning. A key challenge is obtaining accurate, user-specific reward signals in open-ended scenarios. Existing personalized reward models face two persistent limitations: (1) oversimplifying diverse, scenario-specific preferences into a small, fixed set of evaluation principles, and (2) struggling with generalization to new users with limited feedback. To this end, we propose **P-GenRM**, the first **P**ersonalized **Gen**erative **R**eward **M**odel with test-time user-based scaling. P-GenRM transforms preference signals into structured evaluation chains that derive adaptive personas and scoring rubrics across various scenarios. It further clusters users into User Prototypes and introduces a dual-granularity scaling mechanism: at the individual level, it adaptively scales and aggregates each user’s scoring scheme; at the prototype level, it incorporates preferences from similar users. This design mitigates noise in inferred preferences and enhances generalization to unseen users through prototype-based transfer. Empirical results show that  P-GenRM achieves state-of-the-art results on widely-used personalized reward model benchmarks, with an average improvement of ~2.31\%, and demonstrates strong generalization on an out-of-distribution dataset. Notably, Test-time User-based scaling provides an additional ~3\% boost, demonstrating stronger personalized alignment with test-time scalability.

Published as a conference paper at ICLR 2026
P-GENRM: PERSONALIZED GENERATIVE REWARD MODEL WITH TEST-TIME USER-BASED SCALING
Pinyi Zhang!?; Ting-En En Lin?, Yuchuan Wu?t
Jingyang Chen', Zongqi Wang", Hua Yang'. Ze Xu². Fei Huang", Yongbin Li. Kai Zhang
School of Computer Science and Technology, East China Normal University
2 Tongyi Lab, Alibaba Group AAlibaba Group
Institute of Science and Technology for Brain-inspired intelligence, Fudan University
51 (165900220sstuu ecnu. edu.
en, cn, kzhang i 1. coom
{ting lte, shengxiu. www.://uwwyyyallaaab........aa
com.
ABSTRACT
Personalized alignment of large language models seeks to adapt responses to indi- vidual user preferences, typically via reinforcement learning. A key challenge is obtaining accurate, user-specific reward signals in open-ended scenarios. Existing personalized reward models face two persistent limitations: (1) oversimplifying diverse, scenario-specific preferences into small, fixed set of evaluation princi- ples, and (2) struggling with generalization to new users with limited feedback. To this end, we propose P-GenRM, the first Personalized Generative Reward Model with test-time user-based scaling. P-GenRM transforms preference signals into structured evaluation chains that derive adaptive personas and scoring rubrics across various scenarios. It further clusters users into User Prototypes and intro- duces a dual- granulraallrrity scaling mechanism: at the individual level, it adaptively scales and aggregates each user's scoring scheme; at the prototype level, it incor- porates preferences from similar users. This design mitigates noise in inferred preferences and enhances generalization to unseen users through prototype- prototype-based based transfer. Empirical results show that P-GenRM achieves state-of-the-arr results on widely-used personalized reward model benchmarks, with average improve- ment of 2.31% 2,,1%, and demonstrates strong generalization on an out-of-distribution dataset. Notably, Test-time User-based scaling provides an additional 3% boost, demonstrating stronger personalized alignment with test-time scalability[]
INTRODUCTION
Reinforcement learning from human feedback has become a prevailing paradigm for aligning large language models (LLMs) with broadly accepted human values, such as helpfulness and harmlessness (Askell et al., 2021: Bai et al. 2022). Central to this approach is the reward model, which provides reliable scoring signals to steer the LLM's outputs towards desired behaviors (Wang al. 2024).
While conventional alignment targets universal values, personalized alignment aims to tailor model behavior to the diverse preferences of individual users (Jang et al., [203) (aalemi et al., 2023 Zollo
et al. 2024 Ryan et al. (025). This shift poses challenges, especially in open-ended tasks like dia- logue (Salemi & Zamani 2025), where evaluation depends heavily on subjective standards: Explicit preference signals (e.g.. "I prefer concise answers") are often sparse, while implicit signals (e.g.. conversation history) are richer but noisy (Guan et al. (025). Hybrid approaches that combine de- mographics. behavior, and context have been explored Zhang et al.) [222bb Maghakian al.) 2022 Zhao et al. 2023 Poddar et al. (024), but two key limitations remain: (I) Static modeling of prefer- ences. Most methods reduce diverse and dynamic user preferences to fixed set of evaluation rules, failing to capture -enarit-deeendentt variability, even within a user (e.g., preferring brevity while driving but expressiveness in casual settings); (2) Weak generalization to new users. Current models struggle to adapt to new users with sparse feedback, limiting effectiveness in cold-start scenarios.
WWrk done while interning at Tongyi Lab. Yuchuan Wu and Kai Zhang are corresponding authors. Code available at: https://github.com https///ttuubbooo///gggcCoooo// http:////wwwwoeae//////////////a

---

Published as a conference paper at ICLR 2026
User- Peer-Prutotype Initialization and Optimization (Offline)
Usae-Prasatpp Dhirbsstion
Protedype Userr
Query A
Culestive
Pebakking
Kmeans
Histury-aware Atlemüive Protetype Refinement
Uher Group
Ulier Motcing
Profovosee
User
(Troining Data)
Query
Proforgsce
P-GenRM
Percee and cereeae analysh Peruensie beapuay woing
Scating n
Response Responde
User a
the
Final Score
Preference Signals
P-GenRM
Final Sougtes Schrpp
(scare 1, srare
Agpregation
Bridealia Urtkona (heagen
Scao
Sceling
H
Usaay:
Scaling x m
Ellttrr
Stated Criietta
(score s.sore
User ut's Preference Signals
Tepliseza)
empotheli....
Wey pegpastton astwon
Iccore scure
Inddiidduat level
Implicit Preference
Explicit Preference
Evaluation Chain
Scating
Figure I: Workflow of P-GenRM. P-GenRM infers scenario-specific user persona and preference analysis from hybrid preference signals, generates dynamic scoring rubrics, and assesses candi- date responses accordingly. At test-time, P-GenRM can aggregate multiple individual-level scoring schemes and incorporate similar users' preferences to improve scoring accuracy and generalization.
To address these challenges, we introduce P-GenRM, a Personalized Generative Reward Model with test-time user-based scaling. Our approach leverages user personas as priors for interpreting im- plicit signals, enriched by explicit preference cues when available. We design a three-stage training framework: (1) persona--uidee Scoring Induction (PSI) via supervised fine-tuning, which translates hybrid preference signals into explicit evaluation chains; (2) Criteria based Reasoning Enhancement (CRE) with reinforcement learning, which strengthens evaluation chain generation, especially in set- tings with missing preference information; and (3) hard-negative-awart curriculum learning, which progressively improves robustness in handling challenging instances.
After the three-stage training, the P-GenRM can now interpret multi-faceted user preference indica- tors into comprehensive, structured evaluation chains that derive adaptive user personas and scoring rubrics in variable contexts. We further investigate two questions: (1) How mitigate the noise inherent in inferred user preference; (2) How to develop a transfer mechanism that enhances gen- eralization new users by adapting learned preferencex? Inspired by the collaborative filtering algorithms (Goldberg et al. 1992; Sarwar et al.) 2001) in recommendation systems, we propose a prototype-basee baase,, dual-granulan larity scaling mechanism address this issue in unified manner. As shown in Figure [ at test-time, P-GenRM dynamically scales the generation of scoring schemes for each user according to their individual preferences, while simultaneously incorporating ratings from similar users within the same prototype. This mechanism reduces noise in inferred preferences and provides strong generalization unseen users through prototype- based transfer.
Extensive experiments demonstrate that P-GenRM not only sets new state of the art on person alized reward model benchmarks but also achieves an additional 3% gain from test-time scaling, highlighting its scalability and effectiveness in real-world personalization.
In summary, the main contributions of this paper are as follows:
1. We present P-GenRM, to the best of our knowledge, the first personalized generative re- ward model that transforms diverse preference signals into structured evaluation chains, including personas and rubrics, in open-domain settings.
2. We fully leverage the scalability of GenRM and propose a Test-time User-based Scaling mechanism, which substantially improves model performance.
3. Experimental results demonstrate that P-GenRM achieves sfate-of-the art performance on personalized reward benchmarks and exhibits strong generalization to new users.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORKS
Personalized alignment of Large Language Models Personalized alignment of large language models aims to tailor responses diverse user preferences. This can be achieved both by train- ing models with user-specific parameters (Li et al. 2023a; Wang et al. 2023b Li et al. 2024), or steering the LLMs' behavior at inference time (Wang al. 2023a Salemi et al. 2023 Lee al., 2024). We focus here on an important paradigm of the former- Personalized alignment via rein- forcement learning, in which one essential component is the personalized reward model. PPoddrr et al. (024) constructs a reward model conditioned on novel, inferred user-specific latent. [Chen al. (024) model user preferences as convex combination of finite prototypes construct re- ward models. Rame et al. (2023) promote diversity by combining a mixture of experts conditioned on user preferences to learn diverse rewards. Jang et al. (2023) decompose preferences into multi- ple dimensions and train separate reward models to balance different objectives. Building user responses, [Shenfeld et al. (2025) represent user-specific rewards as a linear combination of base reward functions. Ryan et al. (2025) propose SynthesizeMe, which infers synthetic personas from historical preferences to build personalized prompts and reward models, but its static design cannot adapt context-dependent and shifting user preferences.
User Preference Modeling User preference modeling is a prominent research topic across diverse fields. Here, we introduce several User preference modeling methods used in LLM alignment tasks; more details can be found in Appendix A.4 Dong et al. (202)) define explicit multi-diiensional attributes to model human preferences and enhance response customizability. [ee et al.] (2024) encode thousands of user-specified preferences as combinations of values within system prompts. Zhao et al. (2023) train a transformer module predict group preferences and guide LLM generation in a sew-shot setting.
Generative Reward Models Recently, generative reward models (GenRM) have attracted increas- ing attention for their strong generalization, support for test- time scaling, and effective utilization of the powerful generative capabilities of LLMs. Zhang et al.] (2024a) train generative verifiers via the ubiguitous next-token prediction objective, thereby seamlessly integrating reward modeling. [Li et al.) (2023b) propose generative judge to enhance generality, flexibility, and interpretability align- ment evaluation tasks. Liu al. (2025) introduce Self-Principled Critique Tuning, which scales the generation of high-quality principles and precise critiques to enhance scalability, Zhao et al. (2025) incorporate generative chain-of of-thought reasoning and code verification into process-level reward- ing. Xiong et al.] (2025) develop a generative judge that evaluates a policy model's intermediate reasoning steps for stepwise reward modeling.
3 PROBLEM FORMULATION
In a dialogue system, at turn I, the current user u issues a query qt- For each previous turn T < I, the user's preferred response and dispreferred response denoted yテ and y, The user's historical interaction up to turn t is therefore
H(") = { (q) Vi Mi )...,, (UT: V7 17 ,..., (1--1, vT-)
In practice, to avoid excessive reliance historical data, the history size is limited to h via random selection, i.e., H h. In addition to these implicit interaction signals, the user may also provide explicit preference criteria E(w) (e.g., desired style, tone, etc.). Both explicit criteria and implicit signals extracted from historical interactions are treated as signals of the user's preferences. Based on this, the personalized generative reward model designed to first infer a context-aware textual description of the user's personalized preference modeling P(() and subsequently to generate set of scoring rubrics with associated weights, thereby defining a scoring process s(()) that evaluates responses based on their adherence to these rubrics. This process can be formalized as:
[P!")) s(""] Ro (q.. H!!)) .EE"),,) vv)), {s)}] Extract(s!!")
where Rg denotes the personalized generative reward model parameterized by 8, si represents the scalar reward assigned to the i-th candidate response v/. and b denotes the number of candidate responses (typically& = 2). Extract denotes the operation of extracting each response's 's score from text-based scoring process.
3

---

Published as a conference paper at ICLR 2026
Persona-uuide Scoring Induction (SFT)
P-GenRM
Evaluation Chuin
Regenni.2
Noves on
Flltring
SFT
T
Prefenanc Madicler
Trdieaau Hazzn
porss pallo
Remsoning Enhuncement (RL)
Assec dae fostoe soding
Qairy
caluiii dial pn1
M.M-Mas--Judee
Process Reward
Reward
P-GenRM
xB
Rule
Prefemer Coleete cenoina
Penesa and pechracc
Ealarie Criteia
Eatl
score
Reward
GRPO
Hard negarive-awaae Curriculum Learning (RI)
GRPO
ATI Bara
P-GeiRM
Requarch.2
Onty
Evakedias Crherts
score
Rule
Reward
Dedgge Hilerr
Tridiig
score
(b)
Figure 2: (a) The stree-stage training framework of P-GenRM (b) An illustration of the person- alized evaluation chain, showing how preference modeling and derived scoring schemes lead to interpretable, criterion-weighted judgments for responses.
4 METHODOLOGY
As shown in Figure given the user's current query Qi and preference signals (H) EE")), P- GenRM transforms these signals into structured evaluation chains that derive adaptive personas and scenario-specific scoring rubrics, built upon our proposed three-stage training strategy:
4.1 MULTI-STAGE TRAINING FRAMEWORK
Our training pipeline consists of three stages, as illustrated in Figure 2 SFT equips the model with basic personalized scoring abilities; RL further improves the quality of the evaluation chains; and curriculum learning enhances the model's robustness difficult negative cases for such highly subjective task setting.
Persona-guided Scoring Induction (SFT). The basic premise of P-GenRM is to integrate self- generated analyses of user preferences into the generative reward modeling process, so as to enhance scoring accuracy in subjective settings. We conduct a preliminary experiment investigate how various preference-related factors affect the accuracy of personalized scoring. Detailed information provided in Table and the conclusions are as follows:
(i): User persona inferred from interaction histories can serve an effective prior to infer the user's preference, while (ii) user's explicitly stated criteria further sharpen the evaluation precision.
Building on these findings, we first construct a Structured Evaluation Chain (SEC) dataset for re- sponse scoring. As shown Figure an instruct LLM prompted with both the user's implicit and explicit preference signals [ ( to first induce the user's preference modeling p(()) con- sisting of ceenario-speciffe persona and then derive the corresponding preference criteria. Then each candidate's 's response of is evaluated against these criteria to yield personalized score s We utilize the data filtered through rejection sampling as the SFT stage, enabling P-GenRM to acquire the initial capability of transforming hybrid preference factors into an adaptive personalized scoring scheme.
Criteria- -baee Reasoning Enhancement (RL). To generate high-quality evaluation chains in sce- narios lacking explicit user feedback, we propose Criteria-based Reasoning Enhancement via rein- forcement learning.
Our method builds on the standard GRPO Shao et al. 2024) algorithm and incorporates both process-level and rule- outcome rewards. Specifically, instruct P-GenRM with limited number of history interactions to first infer plausible explicit preference from historical interactions. Based on these, P-GenRM likewise generates the evaluation chain in the Persona-guided Scoring
4

---

Published as a conference paper at ICLR 2026
Induction manner. We then adopt LLM-as-a-judges (Gu al. 2024 to the process, where the judge tasked to assess how well the evaluation process covers user-stated or synthetic (details in Appendix C explicit preferences and output score (ranging from 0 to 1), denoted as PR,
Moreover, we define rulc-based outcome rewards based on the correctness of the final output OR, [( > s))), where and the indices of the responses labeled as chosen and rejected, respectively. Additionally, in cases of formatting error, a penalty -0.1 is imposed.
The overall reward is computed weighted sum of the process reward and the outcome reward.
R₂ = a PR₁ +B- OR,,
(3)
where a and B are hyperparameters to balance the relative contributions of the two reward signals. R2 is further utilized to compute the advantage function in the GRPO algorithm.
To To(C 9 H Uer 2,5,) A!
(O)ro(0)
=B ia. ui wWDDD
min
K
Thau (G. (ee, H!!) .V c,,5)
(K)
(k)
gel 2 1 1 mu
To To(c
Ves ( )
(K)
1- 1- A((1) A(t)} DKL. (To (ref)
clip
Teau Ct.s Iet, 11, (1)
vl. c,e))
(k)
where (k) (8)
[P( (u) s(u)] denotes the k-th sampled structured evaluation chain, 1c (k) denotes the length of 6 and A (R) is the relative advantage computed from the corresponding reward R!(a)
This reward function encourages the P-GenRM not only generate correct outputs but also ensure that the evaluation chain reflects the user's s preferences. Moreover, by adjusting the weighting factor a, the model's sensitivity explicit user preferences can be modulated. This helps prevent the model from overfitting to specific preference dimensions and preserves its ability to freely explore the broader preference space. We provide the impact of variations in and Appendix [A.3
Hard negative-aware Curriculum Learning (RL). Following the SFT and RL stages, we in- troduce a hard negative-aware curriculum learning (Bengio et al. 2009) phase enhance the P- GenRM's ability to learn from challenging cases. Specifically, we gradually increase the proportion of hard negatives during the training phase. In addition, enable a larger exploration space for hard negative samples, we disable process-leve reward here. Accordingly, the objective function in this stage retains the same form as the previous stage, except that the reward function R, omits the process reward PR.
4.2 TEST-TIME USER-BASED SCALING
In this subsection, we propose a Test-time User-based Scaling mechanism to jointly address the two key challenges of personalized reward modeling: (1) the inherent noise in inferring user preferences; (2) the limited generalization on new users with sparse feedback. It comprises two aspects: offline prototype initialization and optimization, and test-time dual gual-granularity scaling, as detailed below.
4.2.1 OFFLINE PROTOTYPE INITIALIZATION AND OPTIMIZATION
0.6B (Zhang et al. 2025 to com- pute embedding for each p(u) Rd We then concatenate these embeddings to form P, an overall cross-scenario user-preference embedding matrix. Subsequently, we apply K-means cluster- ing P to obtaín & centroids, denoted A Rexd which serve the initial user prototypes.
User Prototype Initiation. We first employ
History-aware Attentive Prototype Refinement
The goal of prototype refinement is to transform them from mere semantic centers into effective priors that can represent subordinate users' preference choices, which consists of the following steps
(1) Historical selection and prior construction. Given the j-th prototype aj -Ag.] and the historical records of its associated user u turn t: H(() (gr. vf v7 e Random(t 1, 1.h)) (M) Random(t h) denotes that we randomly sample h items from the history up to step f.
5

---

Published as a conference paper at ICLR 2026
For brevity, assume that these letters represent their embeddings, where 9r R is the 7-th input embedding and y V- E Fd are the embeddings of positive and negative feedback, each history triple is first encoded o, o ( concat(g.. y )-))
A prototype-augmenter attention mechanism assigns importance weights to historical records:
O +p
o
UH
a, a, = softmax,
(4)
Vd
This procedure ensures that the prototype selectively exploits those historical records most informa- tive for the current query.
(2) Discriminative prior update with regularization. We integrate the salient historical information with the prototype representation, forming prior that is expected to guide the current query qe in effectively discriminating between y," and V-
Zt = 0j Wg X W,N,, 2 R
The prior 2, then employed to quantify the preference for y." over ye This preference captured by discriminative score difference, A., which is then used to formulate pairwise loss Lpair. The model aims to maximize this difference by minimizing the loss:
A vi
2, De
Lpair log (Ai)
(6)
To avoid excessive drift, the overall objective function for given prototype aj is augmented with two regularization terms. The final loss is defined as:
C = - Acenulla M3 m3l12 + Aerlla, -p,11.-
Here, Hj is the center of the j-th cluster (i.e., the mean of its associated sample embeddings), and pj the state of the prototype a, from the previous update step. The first regularizer encourages the prototype to stay close to its cluster center, while the second ensures that the prototype's evolution remains smooth across updates. The hyperparameters Acent and Åir control the strength of these regularization effects. This L will backpropagate to update the prototype aj. We perform this proce- dure on all samples within each prototype, subsequently reassigning them to the nearest prototype. An algorithmic procedure is in Appendix 1
4.2.2 TEST-TIME DUAL- GAAL-LANNLLARTY SCALING
We propose two scaling mechanisms that fully leverage the inherent test-time scalability of GenRM: individual-Level scaling and prototype-level scaling. The former performs parallel sampling on the user's current query generate multiple scoring schemes, while the latter incorporates preference signals from similar users when evaluating the candidate responses. Details are as follows:
We employed the resulting updated user-prototype distribution in 4.2.1 to perform test time scaling. Specifically, given the user's query gt. preference signals H,"" candidate response vÊ. P-GenRM performs parallel sampling constructs m individual-level scoring schemes that reflect the user's own preference. Moreover, given the user's preference embedding, P-GenRM assign it to the nearest prototype and then select the most similar users {uw)" according to the embedding. Based on these references, P-GenRM simultaneously incorporates additional scores inferred from the preferences of similar users. This process can formalized as follows:
tt ~ Ro(ge H(")) Vir P(W))
(S:) (Mw) ~ Ro Rolge. H(")), wis p(!w))
(8)
m Extract (St,2) 1
Extract ((S:) (mw))
8
m
n2
where P!()) the preference analysis obtained from the z-th sampling, Pf pu represents the pref- erence signals of a similar user U.., Extract denotes the operation for extracting the score from the generated textual analysis, and of is the final score assigned yf after aggregating all scaled results.

---

Published as a conference paper at ICLR 2026
The effectiveness of this approach can be attributed as follows: At the individual level, it explores multiple hypotheses about a user's preferences to obtain richer and more robust scoring rubrics, while at the prototype level it refines these inferences using preferences from similar users. More- over, for new users under sparse historical data, assigning them to suitable prototypes allows the model approximate their preferences through those of similar users.
EXPERIMENTS
5.1 DATASETS AND EXPERIMENTAL SETTINGS
We evaluate P-GenRM and various baselines on three popular personalized alignment datasets: Chatbot Arena-personalized and Prism-personalized. Ryan al. (2025) devise data-filtering pipeline to extract challenging and highly personalizable user data from Chatbot Arena (Zheng et al., 2023 and PRISM (Kirk al. 2024
LaMP-QA: LaMP-QA(Salem)) & Zamani 2025) evaluates personalized long-form question answer- ing by testing how well LLMs generate informative, coherent, and contextually relevant answers given a user profile. More information on these three datasets is in Appendix B.]
Experimental Settings:
For experiments on PersonalRewardBBenchh we implement both P-GenRM and multiple baseline models across two model scales: LLaMA-3.1-8 and -LAMA-3.1-70BB Ex-
periments on LaMP-QA are conducted only with LLaMA-3.1-888 Experiments involving 8B/70B models utilize 8/32 GPUs, respectively. LLaMA-3.1 1.0-70BB training is performed using LoRA (Hu et al.. 2022). The weight parameters ar and 33 of process reward and outcome reward in Equation are 0.5 and 1,0 in our RL stage training. The impact of variations in their values is shown in Table
Appendix A..3 The model used for embedding the prototype is Qwen3- Freenn--mmeddinn-O.. The
instruction model used in Persona-guided Scoring Induction is OpenAI o3 (https://ppnaai.com)
5.2 PERSONALIZED ALIGNMENT PERFORMANCE ON REROONAIRRWARDBENCCC
Baselines. We compare P-GenRM with various baselines on PersonalRewarrBencch
In-Context LLM as a Judge: We provide the LLMs with varied personalization information in the prompt, instructing them to infer the user's preference and determine which response is superior.
Finetuned Reward Models: (a) Following Ryan al. 2025 we train the Bradley-Teryy (Bradley & Terry 1952) Reward Model on the unfiltered data from PersonalRewardBendde to capture and
fit the overall preference distribution of this specific population. (b) We implement other Existing Personalized Reward Models with distinct motivations and methodologies, including GPO(Zhao
al. 2023). (PLdPod et al.) 2024), PAL(Chen et al. (2024), and SynthesizeMe. (Ryan et al.)
2025
More details of these basclines are in Appendix B.2
Overall Results. Performances of different baselines and P-GenRM are shown in Table We find that P-GenRM consistently outperforms the previous ofate-of-the-ar (SOTA) across model scales. Specifically, on the 8B model, P-GenRM achieves an average improvement of 2.77% over the prior SOTA, while on the 70B model (trained with LORA), it delivers an average gain of 1.99%. Further- more, P-GenRM-SB surpasses the previously best-performing 70B model by an average of 1.04%. In addition, as shown in Table 4 P-GenRM also significantly outperforms the leading proprietary model, OpenAI-o3, instructed with strong prompting. Morcover, to ensure that the preferences of minority groups given equal consideration, we provide additional comparative experiments in Appendix A.5 using macro accuracy as the metric, where accuracy is computed separately for each persona group and then averaged across all groups. P-GenRM still achieves the highest macro accuracy (65.21%) among all evaluated baselines and does not overfit to any majority persona. The effectiveness of Test-time based Scaling. Leveraging the test-time scalability of P- GenRM, P-GenRM generates multiple scoring schemes tailored to the current user and aggregates the corresponding results to improve scoring accuracy. More importantly, by incorporating scoring schemes from similar users, P-GenRM achieves higher accuracy while requiring only mod- est increase in scaling operations. For instance, the best result comes from the setting of Ind- 16 and Pro-8, which surpasses the performance of Ind-32 with fewer scaling steps (16+8), yielding an
Ryan al. 2025 name the combined benchmark PersonalRewardBende which we also adopt hereafter.

---

Published as a conference paper at ICLR 2026
Table 1: Results on PersonalRewardBench. P-GenRM outperforms all baselines on both datasets and model scales, while Test-time User-based Scaling brings further gains. Best and second-best results are marked in bold and underline. Ind and Pro denote the Individual and Prototype level scaling, respectively. Results are reported as "mean + standard error" over independent runs.
Chatbot Arena-Personalized Llama 3.1 70B
PRISM-Personalized
Base Model
Llama 3.1 8B
Llama 3.1 8B
Llama 3.1 70B
In-Context LLM as a Judge
Default
56.37 2.16%
57.02
52.04
54.02
CoT
57.05
57.61
52.58
54.09
Demographics
52.96
54.21
Preference History
58.53
58.65
56.24 0.48%
57.11
SynthesizeMe
61.07
63.14
54,70
58.19
Persona-guided Scoring Induction (Ours)
62.20 1.41%
65.55
58.33 0.515
61.61
Finetuned Reward Models
Bradley-Terry
Finetuned Reward Model
67.21 2.07%
71.12 1.71%
63.27 0.62%
63.44 0.77%
Existing Personalized Reward Model
GPO
$7.87 1. 2.2%%
58,50
1 2.37%
$7.29
1.06%
59.16
1.25%
VPL
58.12 2.64%
59.02
2.08%
58.25
0.68%
59.70
1.10S
PAL
$7.31
4: 2.49%
59.40
2.69%
56,74
1.18%
57.75
0.83%
RM Se
69.78
1.98%
72.05
2.24%
62.84
0.85%
63.74
0.60%
Personalized Generative Reward Model
G-GenRM
72.68
73.42
65.32
66.21 0.76%
Test-time User-based Scaling
Ind-8, Pro-4
74.30
67.54
Ind-16, Pro-8
75.92
68.06
average improvement of 2.99 over P-GenRM itself and a substantial average over both SOTA open-source and proprictary models. This demonstrates the effectiveness of our proposed Test-time User-based Scaling strategy. Notably, simply increasing the number of ratings from similar users does not necessarily yield performance gains (as shown in the last two rows of Table 4 further underscoring the highly user-specific nature of the personalized scoring task.
Notably, measured the end-to-end inference time of P-GenRM-8B M-B at different scaling levels test set and compared against several baselines. The proposed
on Chatbot test-time user-based scaling incurs only limited increase in inference time while yielding supe- rior performance and retaining lower latency than prior SOTA methods. Details are presented in Appendix A.9
Ablation Study, We conduct ablation studies to assess the contribution of each component of P- GenRM (Table 2). Removing any component leads to significant performance drop. Notably, the RL-stage ablation indicates that both process and outcome rewards are necessary for this task.
Table 2: Ablation studies of P-GenRM compo- nents: CL (Curriculum Learning), PR (Process Reward), OR (Outcome Reward). Results are reported "mean standard error" over in- dependent runs.
Table 3: Comparison of adaptive (PSI, Persona- guided Scoring Induction) and static (SMe, Syn- thesizeMe) persona methods across base models. Results are reported as "mean standard error" over 5 independent runs.
Chatbot
PRISM
Chathot Arema
PRISM
P-GenRM
72.68 1855%
65.32 0.56%
Qwen3 8B
61.82 1.47%
$5.01 0.77%
wfo CL.
71.07
1.44%
63,82
0.64%
Qwen3-8B + SMc
62.57 +: 1884
56.33 4: 0.91%
w/o CL. PR
70.22 + 1.74%
62.70
0.73%
Qwen3-8B + PSI (ours)
64.22 1.58%
58.01 0.83%
69.05
1.59%
60.94
0.77%
64.47 1.62%
56.34 0.64%
wio CT_ RL.
66.76 + 1.42%
57.08 0.89%
03 SMe
67.73 + 1.94%
58. 18.49 1.22%.
CL. RL, SFT
56.37 2.16%
52.04 0.54%
+ PSI (ours)
69.14 1.46%
63.87 0.85%
Adaptive vs. Static Personas, A key design of our method to infer dynamic user persona and corresponding evaluation criteria, unlike methods that treat user persona static priors. To validate
8

---

Published as a conference paper at ICLR 2026
Retained Veriance Ratio 198)
The Determination of Prototype Numbers
Scaling performance varying prototype numbers
60-
Chalbol Priam
Performance
72
74
73:69
73.45
60-
70-
Chatbot Arena
68-
67.54
67:28
67.19
66.29
66
65.32
20
40
60
80
100
120
20
60
80
100
120
Number of Singular Vectors
Protolype Numbers
Figure 3: Determination of prototype numbers and their effect on scaling performance, Left: re- tained variance ratio as function of the number of singular vectors on Chatbot Arena and PRISM. Right: performance P-GenRM with different prototype numbers.
its effectiveness, we conduct all experiments under the LLM-as-a-judde setting in Table where our method consistently outperforms SynthesizeMe (SMe) across base models.
We also investigate the number of samples required for generating reasonable user preference, and details are in Appendix [A.I
Broader Preference Space. As previously discussed, major challenge in personalized reward modeling within open-domain settings is comprehensively capturing user preferences, rather than confining them to a limited set of static dimensions. The Prism dataset offers predefined pref- erence criteria, such as {Style, Values, Fluency, Factuality, Safety, Diversity, Helpfulness, etc. P-GenRM. however, exploits broader range of highly personalized scoring dimensions, encom- passing (Philosophical Engagement, Openness. Structure, Depth, Nuance, Sensitivity, Breadth of Ideas) and beyond. We present in Figures 6 and Figure 7 the diverse preferences of non-cherry- picked user across different scenarios.
5.3 ANALYSIS OF PROTOTYPE
We analyze how the number of prototypes de- termined and examine its impact on the results.
Table 4: Comparison of different scaling strate- gies on Chathot Arena and PRISM benchmarks, where Ind, Pro denotes the Individual and Pro- totype level scaling, respectively. Results are ported as "mean 1 standard error" over indepen- dent runs.
Determining Prototype Numbers. User pro- totypes serve representative characteriza- tions of the overall user preferences. To select an appropriate number, we perform PCA dimensionality reduction on the cross- scenario user-preference embedding matrix P and record the proportion singular values re- tained when preserving only the leading singu- lar vectors, as illustrated in Figure 3 We set the number of prototypes 50; beyond this point, additional prototypes provide only marginal in- formation gains while incurring substantially higher inference costs.
Model Proprietary Model o3
Chatbot Arena
PRISM
0.64%
PSI
0.85%
P-GenRM (8B)
72.68 :: 1.85%
65.32 0.56%
+ Ind-8
73.61 + 1.54%
65.79 = 0.68%
+ Ind-4, Pro-4
73.66 :: 1.39%
66.20 = 0.75%
+ Ind-16
73.87 :: 1.69%
66.66 0.82%
+ Ind-8. Pro-4
74.30 :: 1.60%
67.54 0.58%
Impact of Prototype Numbers. We investi-
+ Ind-8, Pro-8
74.89 + 1.75%
67.44 0 84%
gate the effect of prototype number on test- time user-based scaling under the (Ind-8, Pro-4) setting, evaluating 0, 25, 50, 100, and 125 pro-
Ind-32
75.59 + 1.64%
67.65 0.66%
Ind-16 Pro-8
75.92 4: L.70%
68.06 0.69%
+ Ind-O Pro-8
66.90 4: 1.54%
57.65
0.86%
Ind-16, Pro-16
72.59 :: 1.61%
64.61
0.72%
totypes. As shown Figure 3 across both
datasets, performance improves substantially validating the effectiveness of introducing prototypes in test-time user-based scaling. However, be- yond 50, performance plateaus and slightly degrades at 100, suggesting that excessive prototypes may introduce inference noise owing to overly fine- fine-graine grained partitioning. We provide the mean and variance of P P-GenRMM8B3 performance under different numbers of prototypes in Appendix A.
the number of prototypes increases from 0 to 50,
Visualization and Case Study. To better understand the role of prototypes in the user-based scaling process, visualize urettpro distributions and their representative preference patterns (Fig-

---

Published as a conference paper at ICLR 2026
P36
P46
User-Pretotype Distribution
factually correct... and well-develpp...
favors
Fluent and read...
Respectfull and balanced values language; dislike forced cultuall
Nuanoe & Creativity...
immodiately deliver substantive.
information...
P15
Fluesey Clarity - writing quality is expected go beyond mere disclaimer and actually outline key points
P8
appreciate ncutrality ....
P49
Two users in P36 values
Conciseness valued: chosen ancoeess tend to be tighter
points) and flow Courtcous, moderately formal, empathetic...
structure
A smalll amount of engageneent (asking follow questions) is plus but
P40
They value brevity and clamity.... comsisienlly pick the mure concise answer friendly... light, crisp sentences preferred
is acceptable even
Trulissiumal/Frinndl Tome respeciful, mildly upbeat
Figure 4: Visualization of User-prototype distributions and representative preference patterns. Blue highlights show shared intra-group preferences, red highlights show individual diversity. Distinct clusters capture inter-group heterogeneity, validating prototype-bast based modeling.
ure with further details in Appendix 8 Users form distinct clusters around prototypes, validating optimization approach described in Section 4.2 Within each cluster, users share core preferences (e.g., fluency, factuality) yet still exhibit individual variation (e.g., creativity), whereas users from different clusters display clearly divergent preferences. This balance between intra- group similarity and inter-group heterogeneity underpins the effectiveness of prototype- based test-time scaling.
5.4 PERFORMANCES ON LAMP-QA WITH SPARSE FEEDBACK
We evaluate the generalization of P-GenRM to unseen users with limited interaction histories us- ing the Lamp-QA dataset, an OOD benchmark. Since Lamp-QA lacks candidate responses with preference ground truth, we construct an evaluation framework (Appendix (A.7) in which six LLMs generate responses that are scored by three advanced models based on personalized rubric aspects to form ground-truth ranking.
Reward models, trained solely PersonalRewarddBnnce and tasked with ranking the same responses. We measure agreement with the ground-truth ranking using Spearman correlation (Spearman 1961). As shown in Table P-GenRM (8B) with the (Ind-8, Pro-4) setting outperforms all baselines, even surpassing the much larger Qwen3-235B-A222 This demonstrates its robustness under sparse feedback and strong generalization new users.
then fed with sparse user histories
5.5 P-GENRM FOR POLICY MODEL'S TRAINING
We train policy models with P-GenRM under DPO and GRPO settings to further validate its effectiveness. P-GenRM boosts an 8B policy model surpass the performance of 70B-sized models, demonstrating its strong efficacy. More details are Appendix A.13
Table 5: Performances with cold-start settings
measured by Spearman's rank correlation on Lamp-QA (f = better). Arts Arts & Entertain- ment, Pers. = Personal Life & Development, Soc.
Society & Culture, Avg. = Average.
Reward Model
Arts
Pers.
Soc.
Avg.
Qwen3-8B 8B Qwen3-32B
0.486
0.543
0.600
0.543
CONCLUSION
0.543
Qwen3-235B-A22
0.600
0.657
0.600
0.619
I.MMA 1-8B
0.486
0.543
0.543
0.524
We introduced P-GenRM, personalized gen- erative reward model that transforms diverse preference signals into structured, scenario- aware evaluation chains and uses test-time user-
SynthMe 8B
0.486
0.657
0.600
0.581
I. 1-70B
0.543
0.657
0.600
0.600
P.GenRM.BB SB+ + Ind-8, Pro
0.543
0.714
0.657
0.638
based scaling to combine individual and prototype-leve preferences. Across personalized reward benchmarks, P-GenRM sets new state of the art. with test-time scaling offering additional gains at modest compute cost. P-GenRM improves subjective evaluation fidelity, generalizes well to users with sparse feedback, and offers interpretability through explicit personas and rubrics.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by Alibaba Research Intern Program and East China Normal University Graduate Student Special Fund for International Conferences.
REFERENCES
Gati V Aher, Rosa Arrigga, and Adam Tauman Kalai. Using large language models to simulate multiple humans and replicate human subject studies. In International conference on machine learning, pp. 337-371. PMLR, 2023.
Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate, Out of onc, many: Using language models to simulate human samples. Political Anal- ysis, 31(3):337-351, 2023.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, al. A general language assistant laboratory for alignment. arXiv preprint arXiv:21 2021.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204. 05862, 2022.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference machine learning, pp. 41-48, 2009.
Ralph Allan Bradley and Milton E Terry. Rank analysis incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3 345, 1952.
Daiwei Chen, Chen, Aniket Rege, and Ramya Korlakai Vinayak. Pal: Pluralistic alignment framework for learning from heterogeneous preferences. arXiv preprint arXiv:2406.00 2024.
Yi Dong, Zhilin Wang, Makesh Narsimhan Sreedhar, Xianchao Wu, and Oleksii Kuchaiev. Steerlm: Attribute conditioned sft (user-steerable) alternative to rlhf. arXiv preprint arXhv:2310.05344 2023.
David Goldberg, David Nichols, Brian M Oki, and Douglas Terry. Using collaborative filtering to weave an information tapestry. Communications of the ACM, 35(12):61-70, 1992.
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Ying- han Shen, Shengjie Ma, Honghao Liu, et al. A survey llm-as-a-judge. arXiv preprint arXiv:2411.15594 2024.
Jian Guan, Junfei Wu, Jia-Nan Li, Chuanqi Cheng, and Wei Wu. A survey personalized alignment-the missing piece for large language models real-world applications. arXiv preprint arXiv:2503. /7003, 2025.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Per- sonalized large language model alignment via post-hoe parameter merging. arXiv preprint arXiv:2310.1 15544, 2023.
Hannah Rose Kirk, Alexander Whitefield, Paul Rottger, Andrew M Bean, Katerina Margatina, Rafael Mosquera-Gomes Juan Ciro, Max Bartolo, Adina Williams, He He, et al. The prism alignment dataset: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models. Advances in Neural Information Processing Systems, 37: 37:105236-105344 2024.

---

Published as a conference paper at ICLR 2026
Seongyun Lee, Sue Hyun Park, Seungone Kim, and Minjoon Seo. Aligning to thousands of prefer- ences via system message generalization. Advances in Neural Information Processing Systems, 57:73783-73829, 2024.
Cheng Li, Mingyang Zhang, Qiaozhu Mei, Yaqing Wang, Spurthi Amba Hombaiah, Yi Liang, and Michael Bendersky. Teach Ilms to personalize-an approach inspired by writing education. arXiv preprint
2023a.
Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. Generative judge for evaluating alignment. arXiv preprint arXiv:3310.05470, 2023b.
Xinyu Li, Ruiyang Zhou, Zachary C Lipton, and Liu Leqi. Personalized language modeling from personalized human feedback. arXiv preprint arXiv:2402.05733, 2024.
Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-time scaling for generalist reward modeling. arXiv preprint arXiv:2504.02495, 2025.
Jessica Maghakian, Paul Mineiro, Kishan Panaganti, Mark Rucker, Akanksha Saran, and Cheng Tan. Personalized reward learning with interactiongrrundde learning (igl). arXiv preprint arXiv:2211.15823 2022.
Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, and Natasha Jaques. Personalizing reinforcement learning from human feedback with variational preference learning. Advances in Neurul Information Processing Systems, 37:52516-52544, 2024.
Alexandre Rame, Guillaume Couairon, Corentin Dancette, Saan-Bapisste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by in- terpolating weights fine-tuned on diverse rewards. Advances in Neural Information Processing Systems, 36:7 (66771095-7114, 2023.
Michael Ryan, Omar Shaikh, Aditri Bhagirath, Daniel Frees, William Held, and Diyi Yang. Syn- thesizeme! inducing persona- guided prompts for personalized reward models in llms. arXiv preprint arXiv:2506.00 055509 2025.
Alireza Salemi and Hamed Zamani. Lamp-qa: A benchmark for personalized long-form question answering. arXiv preprint arXiv:2506.0013: 2025.
Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. Lamp: When large language models meet personalization. arXiv preprint arXiv:2304. /406, 2023.
Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. Item-based collaborative filtering recommendation algorithms. In Proceedings of the 10th international conference on World Wide Web, pp. 285-295, 2001.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint urXiv:2402.03300 2024.
Idan Shenfeld, Felix Faltings, Pulkit Agrawal, and Aldo Pacchiano. Language model personalization via reward factorization. arXiv preprint arXiv:2503.06355 2025.
Anikait Singh, Sheryl Hsu, Kyle Hsu, Eric Mitchell, Stefano Ermon, Tatsunori Hashimoto, Archit Sharma, and Chelsea Finn. Fspo: Few-shot preference optimization of synthetic preference data in Iims elicits effective personalization to real users. arXiv preprint arXiv:2502. 19312, 12, 2025.
Charles Spearman. The proof and measurement of association between two things. 1961.
Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, et al. Secrets of rlhf in large language models part ii: Reward modeling. arXiv preprint arXiv:2401.066 06080, 2024.
Danqing Wang, Kevin Yang, Hanlin Zhu. Xiaomeng Yang, Andrew Cohen, Lei Li, and Yuandong Tian. Learning personalized alignment for evaluating open-ended text generation. arXiv preprint N..0:23300003000 2023a.
12

---

Published as a conference paper at ICLR 2026
Zekun Moore Wang. Zhongyuan Peng. Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Jian Yang, et al. Rolellm: Benchmarking, eliciting, and enhancing role- playing abilities of large language models. arXiv preprint arXiv:2310.07746, 2023b.
Wei Xiong, Wenting Zhao, Weizhe Yuan, Olga Golovneva, Tong Zhang, Jason Weston, and Sain- bayar Sukhbaatar. Stepwiser: Stepwise generative judges for wiser reasoning. arXiv preprint (rXii:2508.1927) 2025.
Jiarui Zhang. Guided profile generation improves personalization with Ilms. arXiv preprint UrXiv:2409.13000 2024.
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling next-token prediction. arXiv preprint arXiv:2408.152400 2024a.
Mengxiao Zhang, Yuheng Zhang, Haipeng Luo, and Paul Minciro. Provably efficient interactive- grounded learning with personalized reward. Advances in Neural Information Processing Sys- tems, 37:121033-121098 2024b.
Yanzhao Zhang, Mingxin Li, Dingkun Long. Xin Zhang, Huan Lin, Baosong Yang. Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05/76 2025.
Jian Zhao, Runze Liu, Kaiyan Zhang, Zhimu Zhou, Junqi Gao, Dong Li, Jiafei Lyu, Zhouyi Qian, Biqing Qi, Xiu Li, et al. Genprm: Scaling test-time compute of process reward models via generative reasoning, arXiv preprint wxXi:2254.008%) 2025.
Siyan Zhao, John Dang, and Aditya Grover. Group preference optimization: Few-shot alignment of large language models. arXiv preprint arXiv:2310. 1523, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt- bench and chatbot arena. Advances in neural information processing systems, 36:46595-46623, 2023.
Thomas P Zollo, Andrew Wei Tung Siah, Naimeng Ye, Ang Li, and Hongseok Namkoong. Person- allim: Tailoring Ilms to individual preferences. arXiv preprint arXiv:2409.20296 2024.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 THE USE LARGE LANGUAGE MODELS
We utilized ChatGPT 5 (bttps://chatgpt.com// solely for language refinement and for checking gram- mar and typographical errors. The study design, experimental procedures, data analysis, interpre- tation of results, and formulation of scientific ideas were conducted entirely by the authors. All substantive intellectual contributions remain exclusively those of the authors, who thoroughly re- viewed and approved the final manuscript.
A.2 PRELIMINARY EXPERIMENTS ON EFFECTIVENESS USER PERSONA
In this section, we conduct preliminary investigation into the effectiveness of incorporating user preference indicators through prompt engincering, i.c., the LLM-as-a-Judge paradigm. For the dataset, we employ PRISM as it contains diverse and well-structured user preference descriptions that are suitable for controlled experiments. We sampled 15% of the data from this dataset for testing. The judge model used in experiments is OpenAl 03-2025-04-16.
To evaluate the impact different user preference indicators, we directly append the corresponding preference description to the original prompt, thereby enabling the model to explicitly condition its reasoning and judgment on the stated user preferences. The only exception is when using persona: first let the model infer the current user's persona, and then score the response accordingly. The evaluation is calculated by mean accuracy (ACC) averaged over five independent runs to mitigate randomness in model outputs.
As summarized in Table 6. the experimental results indicate that explicitly providing preference- related descriptions lcads to consistent improvement in scoring accuracy compared to the bascline without persona information. Among them, the most pronounced performance gain is obtained by Persona, OSR, and SDim, suggesting that current LLMs may not be able to effectively infer the users' explicit preferences, and persona can serve as a useful preference indicator.
Table 6: Accuracy(%) of LLM-as-a-Judee with different types of user preference indicators.
Condition N-CoT
Description
ACC (%)
Baseline model instructed by simple Chain-of- ofTThough prompt without any user-specific information.
62.42
+ Persona
First, develop
persona that characterizes the user and out- lines their potential preferences, subsequently assign score based on this analysis.
64.02
+SD
User provides self-description. Example: "I value honesty and integrity above all. Trust essential in building and maintaining both personal and business relationships ...
63.22
+OSR
System String, i.e. explicit output style requirements specified. Example: "The Al should be kind and respectful, and produce only truthful factual content...
64.24
+BDI
Basic demographic information, such as age, gender, employ- ment status, education level, English proficiency, marital religion, ethnicity, and location.
63.34
status,
+SDim
Choice autributes, i.c., user-defined scoring dimensions and their weights. Example: "values": 61, "fluency": 98, "factuality": 98, "safety": 29, "diversity": 20, "creativity": 9, "helpfulness": 100)
63.63
+Persona, OSR, SDim
66.17
A.3 THE IMPACT VARIATIONS IN AND 3
We experimented with different combinations of and B. documenting their influence on the out- comes of reinforcement learning in Table 7 The experimental results indicate that: (1): Both process-related and outtome-related rewards essential; removing either leads to significant degradation in performance. (2): Excessive emphasis on process-related rewards may cause the model overfit to certain specific criteria, thereby impairing overall performance.
14

---

0.5, Published as a conference paper at ICLR 2026
Table Performance changes of the model after reinforcement learning under different a- settings
and setting
Chatbot Arena
PRISM.
71.07
63.82
a 0.5,
G 0.5
70.65
63.33
1
69.05
60.94
0,
3 =1 1
70.22
62.70
A.4 USER PREFERENCE MODELING
User preference modeling is a prominent research topic across diverse fields, including psychol- ogy, marketing, recommender systems, and natural language processing. For LLM alignment tasks, Argyle et al. (2023) and Aher et al. (2023) both infer preferences from demographic information. Dong al. (2023) define explicit dimen-idimeni attributes model human preferences and en- hance response customizability. [ee et al. (2024) encode thousands of user-specified preferences combinations of values within system prompts. Zhao et al. 2023) train transformer module to predict group preferences and guide LLM generation in a few-shot setting. [Singh et al. (2025) apply meta-learning rapidly adapt an LLM to individual preferences using small number of la- beled examples. Zhang (2024) generate personal profiles to extract key features from user histories, tailoring responses individual needs.
A.5
PERFORMANCE ACROSS DIFFERENT PROTOTYPES
Prototype Sample Count (Bar) + Accuracy (Line)
800
0.8
800
0.6
Accuracy
400
0.4
200
0.2
0.0
30
Prototype by sample
Figure 5: The number of samples assigned to each prototype and the corresponding performance of P-GenRM across them.
To ensure that the preferences of minority groups are given equal consideration, in the context of personalized preference learning, evaluating performance across groups of different sizes provides a more faithful measure of personalization capability.
We conduct additional prototype-wise analysis using macro accuracy, where accuracy com- puted separately for each persona group and then averaged across all groups, ensuring that minority personas are equally weighted. As shown in Figure 5 and Table based on our clustering method, the user groups in the PRISM dataset exhibit clear long-tail distribution.
Despite this pronounced long-tail tail distribution, our method maintains stable performance across pro- totypes: The prototype-level macro accuracy is 0.6521, and the sample-level average accuracy is 0.6532, differing by only 0.0011, indicating that the model does not overfit to majority personas. Moreover, persona-level accuracies have moderate dispersion (median 0.6500, std 0.0401, IQR 0.0544), showing that the model performs consistently across both large and small persona groups, provided in the Figure 5 and Table
We further compare the macro accuracy performance P-GenRM against other baseline methods, shown in Table I0, P-GenRM achieves the highest macro accuracy (65.21 %) among all eval- uated baselines, outperforming both open open-source and proprictary models with strong prompting strategies.
15

---

Published as a conference paper at ICLR 2026
Table 8: Distribution of user groups in the PRISM dataset
Min
Max
Mean
Median
25th pet
75th pet
Num of samples
18
930
154.8
105
63
176
Table 9: Stable performance P-GenRM across prototypes
Macro
Min
Max
Median
Std
IQR
Prototype-level Acc
0.6521
0.5806
0.7508
0.6500
0.0401
0.0544
A single user's preference analysis under music recommendation setting
Query:
Can you recommend music?
PART User Preference Model Analysis (Chain-of-Tie -foonhhhTh
1. Persona and Core Preference Synthesis:
The user is pragmatic but inquisitive music-lover who also enjoys thoughtful, open-ended discussions. They explicitly ask the AI "share correct information and avoid misinformation" and appreciate answers that fcel useful, specific, and conversational. the music domain, their past "chosen" replics either asked clarifying questions to personalise suggestions or supplied concrete lists (artists plus actual song titles). Those choices consistently scored extremely high on "helpfulness." indicating that actionable detail is their priority.
In historical exchanges, they preferred nuanced, balanced statements that recognise complexity rather than absolute claims. They assigned relatively high scores "factuality," "fluency." and "values," showing that they like articulate, well-reasoned, value-sensitive answers that acknowledge multiple perspectives.
Overall preferences inferred: Helpfulness / actionability is king. Accuracy (factuality) is very important. Clear, fluent writing is expected. They enjoy nuance and will punish over-certainty or dogmatism. Creativity and diversity matter, but only after the need (useful, correct content) respectful tone are necessary hygiene factors.
Safety and
2. Scoring Model Derivation:
Weights are assigned according to how strongly each attribute seemed influence their historical choices: Helpfulness/ Specificity 30 Factuality / Correctness 20 Fluency / Clarity 15 Creativity / Depth 10 Diversity Breadth of ideas 10 Values Alignment and Openness 10 Safety Tone
Positive marks: concrete recommendations, nuance, accuracy, fluent wording, some creative variety. Neg- ative marks: generic or repetitive phrasing, factual slips, lack of actionable detail, dogmatism, unsafe or disrespectful wording.
Figure 6: A single user's preference analysis under music recommendation setting
A.6 EXAMPLE OF A SINGLE USER'S 's DIFFERING PREFERENCES ACROSS VARIOUS SCENARIOS.
In figure 6 and figure we provide examples of single user's differing preferences across various scenarios.
16

---

Published as a conference paper at ICLR 2026
Table 10: P-GenRM outperforms baselines methods using macro-accuracy as the metric
Llama3. 1-8B
Llama3.1
Fine-tuned B1-70B
SynthesizeMe 708
P-GeaRM-8B
macro 80c
56.24 0.78%
58.27 +: 0.76%
63.3 + 0.68%
0.84%
0.73%
65.21 0.64%
A single user's preference analysis under serious discussion setting
Query: Are people really responsible for their immoral actions?
"PART I: User Preference Model Analysis (Chain-of- -foo-oughth
1. Persona and Core Preference Synthesis Conversational style: The user prefers concise, straight-to- the-point answers that feel like a real dialogue (e.g.. "Yes, can recommend music! What genre... ?" got picked over longer, fluffier alternative). Helpfulness first: In every pair the chosen answer either offered concrete next steps (song titles, meditation effects) clearer structure. The "helpfulness" attribute peatedly carries the highest numeric weight (90-plus in music examples, 80-plus in meditation). Balanced factuality: The user likes nuance and avoids over confident claims. In the philosophy examples the winning replies admitted uncertainty but still outlined contrasting views; they avoided dogmatic statements ("there is scientific evidence to definitively answer" was preferred to "it is unlikely will ever... "). Fluency matters: Chosen are tidy, error-free, and casy read. Philosophical engagement: The request history shows repeat interest free-will / moral responsibility. The stated preference asks that engage in hypothetical and philosophical conversations." So openness to multiple perspectives is valuable. Creativity and diversity: Moderate importance chosen music answers include specific, varied titles. But creativity never outweighs clarity helpfulness. Values and tone: For ethical topics, the user's "values" suggesting desire respectful, neutral, non-judgmental tone. Safety: Always acceptable in the examples; user hasn't 't highlighted safety concerns, so it gets the lowest weight. 2. Scoring Model Derivation Based on the pattern above, I'll rate future responses six principles, cach with a weight mirroring the implicit priorities:
"be willing to
weight jumps
1. Helpfulness and Depth (30) actionable, structured, insight-producing content. 2. Factuality and Nuance (25)- avoids misinformation, acknowledges complexity. 3. Fluency and Clarity (20) - readability. coherence, concision. 4. Philosophical Engagement Openness (15) explores multiple viewpoints, en- courages reflection. 5. Values Alignment and Tone (5) respectful, balanced, non-dogmatic. 6. Safety (5) complies with policy, harmful content.
Lower scores given for: shallow generic answers, dogmatic certainty, lack of nuance, verbosity without value, grammatical issues, or unsafe content.
Figure 7: A single user's preference analysis under serious discussion setting
17

---

Published as a conference paper at ICLR 2026
A.7 EVALUATION FRAMEWORK USED IN LAMP- QA DATASET
LaMP-QA does not provide fixed of candidate responses or corresponding ground truth. Conse- quently, the procedure for evaluating the quality of reward models becomes somewhat more intricate. We adopt the following testing pipeline to enhance the stability of the evaluation results. We invoke GPT-5, GPT-4o, and Gemini-2.5-pmm to
six LLMs (Qwen3-8B, Qwen3-32hh generate, for each user query, a set of candidate responses. To assign scores to these responses, we adopt the following procedure: provide three highly-advanced LLMs (Gemini-2.5-pro, Claude- 3.7-Sonnet, and GPT-4o) with rubric aspects corresponding to current user query, and instruct them to each response according how well each aspect is addressed. We sum up the scores obtained by these models across all responses to represent their overall performance.
Swen-235B-A221
We then evaluate set of reward Sesiels-Qwee3--35 3.1-8B, 1--MA--.1170B, LLaMA-3. 1-BB-BBB- + SynthesizeMe, and P-GenRM-scale under feedback scenario, where each model received either three of user's historical interactions input. For fairness, the scoring of all reward models was normalized by repeating the scaling procedure eight times and taking the average.
Qwen3-32B, Qwen-235B-A22 LLaMA- under a sparse-
The LaMP-QA dataset comprises three subsets. We examine whether the rankings of the six genera- tion models. as induced by these reward models on each subset, are consistent with the ground-truth rankings. To this end, we compute the Spearman rank correlation coefficient between the reward- model rankings and the ground-truth rankings.
A.8 VISUALIZZATION OF USER-PROTONE PROTOTYPE DISTRIBUTION
Here provide broader visualization of user-prototype distributions and representative prefer- ence patterns.
P36
P46
P37
P15
P34
P49
P47
P2
P40
P17
P7
PI
P27
P13
fictually well-Guvelop...
P7
Accuracy relevance
P36
Fluent and cary to read...
Cunciseness valued: chosen
tighter
Respeciful and balanced values lingage;; dislike foroed cultral slang
Clear structure (ofiea bullet points)
liked,
lttuially (contai disclaimers when violenee illegaiiyy discused)
Courteous, moderately formal, and empathetic...
Nuunce Creativity...
small amount engagement (esking follow-up questions) is plus
immediately deliver substamive, factual infomation...
Helpfulness
Ascuncy
Fluency & Chrity wriling quality expected...
They value brevity and clarit... consistently pick the more coucise znswer
beyond disclaimer actually outline key points
Bullet- structure crisp sentences preferred
approsinte ncutrality ...
friendly... light, informal souch accepaalee und even welcume Professional Frienill Tone regpectfll mildly ophest
Safttyyhrnlesnes must be peserved...
Figure 8: Visualization of prottppo distributions and representative preference patterns. Each cluster corresponds to a learned prototype, around which users with similar preferences are grouped. (1) Within the same cluster, users share common preferences (highlighted in blue), yet also exhibit subtle variations and diversity (highlighted in red). (2) Across different clusters, users demonstrate clearly distinct preference tendencies, underscoring the effectiveness of prototype-based modeling for capturing both intra-group group commonality and inter-group heterogeneity.
18

---

Published as a conference paper at ICLR 2026
Table 11: Inference time comparison between P-GenRM with test-time user-based scaling and base- line methods.
Model
Inference Time (Wall-clock)
Performance
1--.A33.--8---nsuuuee + PSI
00:14:06
62.20
LLaMA3.1170B-Imttne PSI
00:39:17
65.55
SynthesizeMe + FT RM-8B
00:24:10
69.78
SynthesizeMe + FT RM-70B
01:29:59
72.05
03+PSI
01:25:45
69.14
P-GenRM-8B P-GenRM-8B + Ind-8. Pro-4
00:14:16
72.68
00:18:22
74.30
P-GenRM-8B + Ind-16, Pro-8
00:23:05
75.92
A.9 COMPARISONS INFERENCE TIME
We measured the end-to-end inference time of P-GenRM-SB different scaling levels the full Chatbot Pecaa-Person test set and compared it against several baselines. Both the open-source models and our method were deployed using vLLM on 8 NVIDIA A100 GPUs, while proprietary models were accessed via the Alibaba Cloud API with a maximum concurrency of 40. The results are presented Table 1T
The observed increase in inference time for our method remains limited (00:14:16 - 00:23:05), and outperforms larger models while requiring less inference time. believe this mainly due to two factors:
First, the task requires including a certain number (specifically, 3) prior user preference selections in the prompt to allow the model to infer meaningful user preferences. This leads to long input quences, making the construction the KV cache for the prompt the major component of inference latency. This prompt encoding is performed exactly once per query and shared across all samples.
Second, our test-time user-based scaling introduces only limited additional cost. It consists of two parts: For individual-level scaling, perform parallel sampling via the OpenAAl-coppatbble parameter, allowing multiple outputs to be generated in a single model call with latency comparable single-output generation. For prototype-leve scaling, similarity computation lightweight and the preference-based scoring over similar users can also be parallelized efficiently handled through vLLM's batching capabilities.
As result, the overall increase in inference cost remains modest, especially considering the perfor- mance improvements brought by the Test-time User-based Scaling, which yields 3.24% gain over P-GenRM--8 -BB without scaling and 3.74% advantage with less inference time over the previous tha
A.10 HISTORY- ATTENTIVE REANEMENT ALGORITHM
Here, we provide an algorithm procedure of History-aware Attentive Refinement.
A.11 NUMBER OF SAMPLES REQUIRED FOR GENERATING REASONABLE USER PREFERENCE
Intuitively, a single preference instance only reflects a one-off choice, and two instances are still insufficient to form consistent pattern. In contrast, three preference samples provide the minimal structure needed to assess preference consistency, reduce randomness, and support reliable person- alization by the model. We also conducted experiments Chatbot Arena- Penn--Persnnllzed evaluating P-GenRM-8B trained with 1-4 preference samples per user, and the performance differences are shown in Table [2
We observe that the model performance improves significantly when the number of preference sam- ples reaches 3. Adding more samples beyond this point primarily contributes to improved evaluation
19

---

Published as a conference paper at ICLR 2026
Table 12: P-GenRM's performance with different numbers of preference pairs
Preference Pairs
2
3
Accuracy (%)
59.78 + 2.66
64.62 + 2.07
72.68
1.85
72.50
1.64
Table 13: P-GenRM-8B performance under different numbers of prototypes with (Ind-8, Pro-4) setting
# Prototypes
0
25
50
100
125
Chatbot Arena
1.85%
1.69%
1.60%
73.69
1.59%
PRISM
65.32 0.56%
66.29
0.75%
67.54 + 0.58%
67.28 + 0.68%
67.19 + 0.84%
stability, but does not necessarily lead substantial performance gains. Therefore, set the num- ber of preference samples to for both training and evaluation.
A.12 P-GENRM's PERFORMANCE UNDER DIFFERENT NUMBERS OF PROTOTYPES
The performance of P-GenRM-8B with (Ind-8, Pro-4) setting under different numbers of prototypes are listed in Table
A.13 P-GENRM FOR POLICY MODEL'S TRAINING
Assessing the downstream policy model is essential for further validating the effectiveness of the reward model. We conduct extensive experiments using both GRPO and DPO train policy models with P-GenRM and evaluate their personalization performance. The details are as follows:
We train the policy model Llama 3.1 8B-Instruct using the Chatbot vrena-Personalized dataset under two different setups.: (I) P-GenRM is integrated into GRPO to score each response and compute the corresponding relative advantage. Specifically, given the user's historical preference pairs, P-GenRM assesses how well each candidate response aligns with the user's 's preferences (using same prompting format as in P-GenRM's training). These scores are then incorporated into advantage estimation and loss computation to update the policy model.
(2) P-GenRM serves as the implicit reward model in DPO. Samples labeled as chosen by P-GenRM treated as positive instances, whereas those labeled as rejected are treated as negative instances. These preference pairs are then utilized to train the policy model via DPO.
The policy model is evaluated as follows: given user's historical preferences, we prompt the model to generate responses that align with the user's 's tastes. We then employ three advanced LLMs (GPT- 4o, Claude-Sonnet-4 Gemini 2.5-Pro) as judges to rate the personalized quality of each response on a 1-5 scale. For each query, we compute the average score across the three judges. We record the overall score and repeat 5 independent runs of the evaluation and report the mean performance along with the standard error (SF) and 95% confidence intervals (CI) in Table 14
Across five independent runs, the 8B policy models trained with P-GenRM achieve 95% confidence intervals whose lower bounds are 3.303 and 3.334, both of which exceed the upper bounds of the 70B models (3.174 and 3.228), showing no interval overlap. This confirms that the performance advantage is robust and statistically significant. We also provide full per-run results in Table 15]
A.14 LIMITATIONS
Despite strong empirical performance, there exist two current limitations of the method: (I). It requires generating an evaluation chain to obtain reliable personalized scores, which may be less efficient than reward models that directly produce scalar values when considering inference speed alone; (2). It relies on three historical preference selections to construct reasonable preference anal- ysis, which implies moderate amount of data collection in practical scenarios.
20

---

Published as a conference paper at ICLR 2026
Table 14: Comparisons policy models' performance over 5 independent runs
Policy Model
Mean
SE
95% CI
1 10mma3.1-8B-..
Instruutt
2.954
0.0074
[2.939.
2.969]
Dwen2.5--BBIInttrn
2.970
0.0089
[2.952 2.988]
Llama3. 1-0...-1-7----tuuu
3.156
0.0093
[3.138 3.174]
3.174]
Owen2.5-72B-Inse.
3.214
0.0089
23.19233.22)
Llama3. ISB--nstrrut--PO
3.316
0.0068
[3.303 3.329]
Llama3. 100taa..1880 1-8B--88-mmn 1--------ttttt-GRRRIII Instruct-GRPO
3.354
0.0102
[3.334, 3.374]
Table 15: Full per-rum results of policy models' performances over 5 independent runs
GPT-4o
Claude-sonnet-
Gemini-2.S-pp
Policy Model
2
2
4
L.ama3.1 1-88-Instruct
3.15
3.
3
10
1.19
3.18
2.90
2.94
2.58
2.95
2.80
2.78
2.83
2.76
22.22
Qwen2.5.
2.94
2.89
LJama3,1- 1-mmm3,1-78--111 Insttet
3.33
3.337
3.33
3:31
3.37
3.05
3.08
3.05
3.00
3.13
3.08
3.06
3.04
3.09
3.04
Qwen2. Instuut
3.42
3.47
3.44
3.44
3.41
3.14
3.14
3.14
3.16
3.12
3.17
2.97
3.04
33.05
Llama3,1 -B--Isttrutt--t----ee Instruct-DPO
3.49
3.44
3.46
3.44
3.40
3.15
3.13
3.20
3.18
3.21
3.28
3.33
3.31
3,40
3331
LJama3.1 8B Instruct. GRPO
3.47
3.51
3.51
3.50
33.57
3.22
3.31
3323
3.28
3.24
3.39
3.36
3..27
3.21
13.27
B DATASETS AND BASELINES
B.1 DATASET
Chatbot Arena. We use the Chatbot Arena subset in PersonalRewardBench Ryan al. (2025) as the evaluation set for this part our study, which includes data from 13T users. Chatbot Arena Zheng et al. (2023) is an interactive platform where users engage open-ended conversa- tions with two anonymous LLMs and provide pairwise preference judgments based on the quality of their responses. It collects in-the-wild prompts and user feedback, making it valuable source for evaluating models under realistic and diverse conversational settings.
PRISM. We use the PRISM evaluation set from PersonalRewardBenct Ryan al. (2025), which contains data from 720 users. PRISM Kirk et al. (2024) maps detailed survey responses of diverse participants onto their live, multi-turn conversations with various I.LMs, with a particular emphasis on values and controversial topics. In each turn, users rate multiple candidate completions cardinal 1-100 scale and provide fine-grained feedback on attributes such as factuality, creativity, and value aligmment. These N-way preferences converted into pairwise comparisons for reward model training, and pairs with less than 10% quality difference are removed. By combining stated preferences from surveys with observed contextual preferences in conversation, PRISM enables research on personalized and cross-cultural alignment beyond simple binary A/B judgments. In PersonalRewardBennch all (N) comparisons from each PRISM turn are extracted to form a pairwise dataset.
LaMP-QA. We also adopt the LaMP-QA benchmark Salemi & Zamani 20255) for evaluating per- sonal reward ability. LaMP-QA is built from dataset collected from the StackExchange CQA plat- form. LaMP-QA focuses on tailoring responses to the specific information needs expressed by the user, leveraging both their current question and historical questions as a user profile. Each question is accompanied by detailed narrative outlining personalized requirements. from which key eval- uation aspects are extracted using L.LSs and quality- by human annotators. These aspects remain hidden during generation and are used exclusively for evaluation, enabling fine- -raiirddd,,, aspect-based assessment rather than binary or ordinal judgment. The dataset covers three major cat- egories-Arts & Entertainment, Lifestyle & Personal Development, and Society & Culture-with over 45 subcategories
to its two advantages First, it provides personalized rubric aspects for each user where responses be scored on how well they align with these concrete preferences. Second, it offers lorg-form historical queries, allowing us to extract only a limited subset to evaluate the generalization of different models.
21

---

Published as a conference paper at ICLR 2026
Algorithm History-aware Attentive Refinement
1:
Input: Initial prototypes A {ai....,.x. ax); cluster means {uy}; user u, turn t: current query
Gt. pair (VE, and history H(a) {ar, v=F | e Random(t 1, h) (w); parameters (Ag, As; Atr A)z weights (W, W W.); learning rate 77-
Output: Updated prototypes A.
3: for each prototype aj E A do
4: Pj <-
keep previous prototype state
5: for
to do
0, of concat(qr, V² v)
D encode each history feedback triple or
7: end for
0r q: vd vd
oaj
8: a, softmax,
+p
& assign prototype prototype-augmente augmented attention weights historical
records
9:
CH + Σか= ar Or
> aggregate attentive historical context > form the prototype-nnfomed informed prior discriminative score difference (preference for yi over y: D- pairwise loss to maximize the preference keep prototype near its cluster center and
10:
21 + a; W ge A.W.
D, f- 21 1/2 21 y1c
12:
Lpair t- logoo(()
13:
++ Epate
Aoout |a H5ll + Au la, p,1l
ensure smooth evolution
0j - Uソ - VayL
loss backpropagates to update the prototype aj
15: end
16: Reassign sample embeddings to the nearest prototype
17: return A
B.2 EXISTING PERSONALIZED REWARD MODELS
GPO Zhao et al. (2023) Group Preference Optimization aims to adapt LLM outputs to different group preferences despite having very limited data per group. It adds a separate Transformer-baa preference module the base model, encodes the meta-learning framework for few-shot, in-context preference prediction. This enables group-specific alignment inference time without fine-tuning, using the module as a reward ranking function.
pair, and trains the module in
VPL Poddar et al. (2024). Variational Preference Learning addresses the limitation of standard RLHF that assumes single utility function for all users. It treats preference modeling latent- variable problem, with a hidden variable representing user context, estimated via variational infer- ence from few pairwise annotations. A reward model r(s, 2) and autent--onditioned policy capture multi-modal preferences, with stability improved via reward scaling and active query selection to quickly refine 2 test time, achieving personalization without identity or demographic data.
Chen et al. (002)). The Pluralistic Alignment Framework models diverse and heterogeneous user values by representing each user as mixture of "prototypical preference points" in trans- formed representation space. It jointly learns the mapping function and prototypes from comparison data, and infers mixture weights for each user. This allows fast personalization with minimal data, while matching the performance of much larger reward models.
SynthesizeMe Ryan et al. (2025). SynthesizeMe tackles data scarcity and the difficulty of inferring latent preferences from pairwise comparisons. Without identity data or fixed preference axes, it uses LLMs to infer possible explanations for user choices, synthesize a persona capturing these prefer- ences, and select the most informative past examples to form interpretable personalized prompts. These prompts improve reward models or LLM-as-a-judee systems in matching user-specific pref- erences without fine-tuning.
22

---

Published as a conference paper at ICLR 2026
C PROMPTS
Explicit Preference Synthesis
ROLE
You are a preference analysis expert. Based on the user's previous inputs and historical choices, you need to infer what explicit preference criteria they may have.
# Examples from user's preference history (The Start of User's preference history]
few shots}
(The End of User's s preference history]
# Output Instructions
Your output must set of concise descriptions, listing point by point the possible preference criteria that the user may have.
Figure 9: Prompt of Explicit Preference Synthesis.
23

---

Published as a conference paper at ICLR 2026
Persona-guided Scoring Induction
ROLE
You are meticulous user preference analysis expert. Below will provide you with this user's desired response style requirementts, and the user preference history examples, each detailing the user's choice attributes and score of chesen or rejocted response
PRIMARY GOAL
Your task is deeply understand user's inherent preferences from their stated requiremeents and historical choices. Based this mnderstanding, you will first construct a personalized scoring model for this user in current scenario. Finally, you will apply this model to score new set responses determine which better.
INPUTS
User's Desired Stated Response Style:
[The Start of Desired stated response requirements] [dcsired style)
[Tbe Ead Desired stated respense style requirements)
User Preference History Examples:
Each example details the user's choice altribules and (he chosen rejected responses. higher altribute sigmifies greater importance.
[The Start of User's preference history] whotss,wthhhhhik [The End htw..cooocccaatbbtee..... Lattributes.scoo scores
preference historyl
OUTPUT INSTRUCTIONS
Your output must consist two parts, in this exact order:
PART User Preference Model Analysis (Chain-of- Thouhht)
Refore the ISON cutput, provide detailed analysis outlining the personalized scoring model have derived for this user.
Follow these stops:
Persona & Core Preference Synthesis: First analyze the user's 'Desired Stated Response Style' and 'Preference History Examples'. Deduce the user's likely persona, communication style, and core preferences. All theses derivations shoold apply to the current scenario. State only highly confident conclusions.
2. Scoring Model Derivation: Baved on the examples (bistory preference paris, choice attributes and scures), explain the logic of your characteristics get positive marks? gets negative marks? How do you weigh different This section explains the "rules" you will use for scoring.
personalized scoring
PART Final Scoring and ISON Output
your analysis, output "TARRT, followed immodiately any text after "JSON,END".
object, "ISON_END".
# ISON OUTPUT SPECIFICATION
The ISON object must contain exactly keys: "rationale" and "better rresponse".
(string):
A step by step application of your model to the current responses. The content of this string must follow this structure: Evaluation Criteria: the evaluation principles you derived in Part each principle, state its percentage weight determined assessment of current responses. The sum of all weights must 100* B. Scoring Breakdown: For each response (Response Response etc.):
Evaluate against each principle, assigning
from (Poor) (Excellent).
Show the calculation for the tinal weighted score. Score (Score Principle] Weightl) (Score Principle2 Weight2) +
Example: 'Response
besterrresposse (object): A key-value object containing the final calculated scores for each response. Keys must be the response ideatifiers (e.g.. 'response I', 'response 2").
Values must be the linal mumerical scures.
Example ISON Output Format:
"(( ("rationale": "your rationale adhering affreementinedd instructions", response": ("responseex x": sponse y"; "score.y")]] ")
"score
# Iaput Data
[The Start of User Ingut]) (n (user (user_input) \n Thee End User Input] \
Response aa fresponse_l A/n End of Respense 1/ [The Start of Response 2) ( fresponse 2) An The End of Response
Figure 10: Prompt of Persona-guided Scoring Induction.
24

---

Published as a conference paper at ICLR 2026
Criteria-based Scoring Enhancement
ROLE
You are meticulous user preference analysis expert.
"CNTTEXT
You will be given user's 's perfenecc history, which consists of pairs of chusen and rejected respunses from past interactions.
# Examples from user's preference history
[The Start of User's preference history]
[lew shots)
[The End User's preference history]
GOAL
Your primary goal huild and apply personalized scoring model for this Finally you will apply model responses and determine which is better.
To uchieve this, you will infer the user's 's plausible pre ference criteria: Infer the user's 's desired response style from their historical choices.
2. Derive of weighted scaring criteria bused this protile.
Then, you will
of responses. 4. Provide a detailed, step-by-step rationale for your ovaluation, culminating in final JSON output.
Apply this model to ovaluate
OUTPUT STRUCTURE
Your output must strictly consist parts, in this exact order:
PART Chain-of-Thougee Analysis
section your "scratchpad" where you huild model. must precede JSON output.
User Preference Synthesis:
Rased the 'User's Preference History'. analyze the chosen rejected examples. Synthesize coherent personn of the user's preferences in this sccnario. Besides, deseribe their preferred Style (c. formal, casual, empathetic). Content Structure (e.g.. prefers lists, dotailed explanations, answers). any other discemible Core Values (e.g.- values accuracy. creativity, safety).
only highly confident conclusions drown directly from evidence.
Personalized Scoring Model Derivation:
Rased 'User Preference Synthesis', define the evaluation criteria for These the "rules" you will for scoring. For cach criterion, briefly explain why it's important to this user.
PART 2: Final Scoring and ISON Output
Immediately after analysis. oupput SOORRTTART' followed by a single valid ISON objoct, and then 'IOON END', Do not add any text before 'SON START" or after 'SON END".
# JSON SPECIFICATION
The ISON object must contain exactly two keys: "rationale" and "scores".
'rationale' (string): step-by-step application of your scoting model the new responses. The content of this string must
Kollow this exact structure:
Evaluation Criteria Weights: List the evaluation criteria derived in I. Assign pereentage weight each, reflecting its importance for *thi specitic evaluation*. The sum of all weights must be Weight) B. Scoring Breakdown: For each response
(e.g.. Response X, Response
Evaluate it against each criterion, assigning score from from (Poor) 10 (Fxcellent).
Provide brief justification for each score.
Show the calculation for the final weighted score.
Example: 'Response Final Score = (Score Principlel Weightl) + (Score Principle2 Weight2) +
2. beetter.respume (object): A kayuv object containing the linal calculated scores for each response. Keys the response inlentifiers (e.g.. 'response I', 'response 2*).
Values must be the final numerical scores.
Example ISON Output Format:
"((" ("rationale": "your rationule adhering to the affrrementioeed instructions", reetter.respmsee: {( ("response "scorexx", "re. sponse.y": scorr_y")]]
Inpat
[The Start of User Toput)) (n (nsurtinput)>/ \n [The End of User Inpur] \ [The Start of Response 1a fresponse_ _1 An The End of Response 1/
[The Sturt of Response ja fresponse 2) \n The End of Response
Figure 11: Prompt of Criteria-based Scoring Enhancement.
25

---

Published as a conference paper at ICLR 2026
LL.M-as-a-Judge + System String
# ROLE
You are meticulous user preference analysis expert. Below I will provide you with this user's desired stated response style requirements, and this user's preferred responses from their input history.
Primary
Your primary goal is to understand this user's inherent preferences from the desired stated response style inntummmentssitraattiooo history and use that understanding to judge responses to the user's current input.
User's Desired stated response style requirements
(The Start of Desired stated response style requirements
desired style} (The End of Desired stated response style requirements)
Examples from user's preference history
[The Start of User's preference history]
(totsshots) }
[The End of User's preference history]
INSTRUCTIONS
Follow these steps precisely:
Analysis First, perform your detailed analysis.
Analyze User Persona Preferences: Based on the history, deduce user's likely persona, communi- cation style, demographic information, and core preferences. Retain only the conjectures of which you are highly confident.
b. Evaluate Responses against User Persona & Preferences: For each user's input and its corresponding responses, evaluate how well it aligns with the user's persona & preferenees you inferred from the history above.
c. Final Decision: State which response you believe is better and briefly summarize the single most important reason.
Final Output (ISON) After completing your analysis and decision, generate single, valid JSON object as your final answer. The JSON should be the only thing you output.
# #SOON OUTPUT SPECIFICATION
The ISON object must have exactly keys: "rationale" and "hetter response".
"rationale": concise string that explain the user's preference and the advantages of the better response.
"better An integer, which must be 'I" Response is better, or '2" if Response 2 is better.
Example Output Format:
{("rationalc": "your explanation", "eetoorr.ssponee:: x/
# Input Data
(The Start of User Input] n (user_inputl n [The End of User Input]
The Start of Response 1] in (response 13 1 The End of Response i
(The Start of Response
(response 2) The End Response 2]
Figure 12: Prompt of LLM-- + Output Style Requirements.
26

---

Published as a conference paper at ICLR 2026
LL.M-as-a-Judge + Self-Deseription
# ROLE
You are a meticulous user preference analysis expert. Below will provide you with this user's self description and this user's preferred responses from their input history.
Primary Goal
Your primary goal is to understand this user's inherent preferences from the self description, interaction history and use that understanding to judge responses to the user's current input.
User's self description
(The Start of User's self description)
self description
(The End of User's self description]
Examples from user's preference history
[The Start of User's preference history]
(towsshots) }
[The End of User's preference history]
INSTRUCTIONS
Follow these steps precisely:
Analysis First, perform your detailed analysis.
Analyze User Persona Preferences: Based on the history, deduce user's likely persona, communi- cation style, demographic information, and core preferences. Retain only the conjectures of which you are highly confident.
b. Evaluate Responses against User Persona & Preferences: For each user's input and its corresponding responses, evaluate how well it aligns with the user's persona & preferenees you inferred from the history above.
c. Final Decision: State which response you believe is better and briefly summarize the single most important reason.
Final Output (ISON) After completing your analysis and decision, generate single, valid JSON object as your final answer. The JSON should be the only thing you output.
# #SOON OUTPUT SPECIFICATION
The ISON object must have exactly keys: "rationale" and "hetter response".
"rationale": concise string that explain the user's preference and the advantages of the better response.
"better An integer, which must be 'I" Response is better, or '2" if Response 2 is better.
Example Output Format:
{("rationalc": "your explanation", reetoorr.ssponee:: x/
# Input Data
(The Start of User Input] n [user (user_input]
n [The End of User Input]
The Start of Response 1] \n (response 1 The End of Response i
(The Start of Response
(response 2) n The End Response 2]
Figure 13: Prompt of LLM--ss--Judge + Seff-Desrription.
27

---

Published as a conference paper at ICLR 2026
LL.M-as-a-Judge + Basic Demographic Information
# ROLE
You are meticulous user preference analysis expert. Below will provide you with this user's basic demographics and this user's preferred responses from their input history.
Primary Goal
Your primary goal is to understand this user's s inherent preferences from the basic demographics, interaction history and use that understanding to judge responses to the user's current input.
User's basic demographies
(The Start of User's basic demographies)
basic demographics)
(The End of User's 's basic demographies)
Examples from user's preference history
[The Start of User's preference history]
[few_shots) shwtss }
[The End of User's preference history]
INSTRUCTIONS
Follow these steps precisely:
Analysis First, perform your detailed analysis.
Analyze User Persona Preferences: Based on the history, deduce user's likely persona, communi- cation style, demographic information, and core preferences. Retain only the conjectures of which you are highly confident.
b. Evaluate Responses against User Persona & Preferences: For each user's input and its corresponding responses, evaluate how well it aligns with the user's persona & preferenees you inferred from the history above.
c. Final Decision: State which response you believe is better and briefly summarize the single most important reason.
Final Output (ISON) After completing your analysis and decision, generate single, valid JSON object as your final answer. The JSON should be the only thing you output.
# #SOON OUTPUT SPECIFICATION
The ISON object must have exactly keys: "rationale" and "hetter response".
"rationale": concise string that explain the user's preference and the advantages of the better response.
"better An integer, which must be 'I" Response is better, or '2" if Response 2 is better.
Example Output Format:
{("rationalc": "your explanation", "eetoorr.ssponse:: x
# Input Data
(The Start of User Input] n (user_input) n [The End of User Input]
The Start of Response 1] in (response 1 The End of Response i
(The Start of Response
(response 2) n The End Response 2]
Figure 14: Prompt of LLM-assa-Judge + Basic Demographic Information.
28

---

(Te Start of Response 1] [response_1| Published as a conference paper at ICLR 2026
LL.M-as-a-judge +Choice Attributes
# ROLE
You are meticulous user preference analysis expert. Below I will provide you with examples of this user's preferred responses from their input history, each followed by the user's choice attributes in this conversation, which indicate the rationale for the user's preference for response others. A higher score signifies that corresponding attribute greater importance.
# Primary Goal
Your primary goal understand this user's inherent preferences from each conversation history together with its corresponding choice attribucts and use that understanding to judge responses the user's current input.
# Examples from user's preference history and corresponding choice attribucts [The Start of User's preference history and corresponding choice attribuets] [[ew_shots, wss.sssosssss. whihtswwthhh.o wwt.tttwhhhheeeee.... www.eecoooeeeteeeee )
[The End of User's 's preference history and corresponding choice attribuets]
# INSTRUCTIONS
Follow these steps precisely:
1. Analysis: :
First, perform your detailed analysis.
Analyze User Persona & Preferences: Based on the history, deduce the user's likely persona, communi- cation style, demographic information, and core preferences. Retain only the conjectures of which you highly confident.
Evaluate new Responses against User Persona & Preferences: For each user's input and its corresponding responses, evaluate how well it aligns with the user's persona & preferences you inferred from the history above.
Final Decision: State which response you believe better and briefly summarize the single important reason.
2. Final Output (ISON) After completing your analysis and decision, generate single, valid JSON object as your final answer. The JSON should be the only thing you output.
#
JSON OUTPUT SPECIFICATION
The ISON object must have exactly two keys: "rationale" and "hetter
"rationale": A concise string that explain the user's preference and the advantages of the hetter response.
"hetter response": An integer, which must *1" if Response is better, '2" if Response 2 better.
Example Output Format:
(("rationale": "your explanation". "betterrresponse: x}}
Input Data
[The Start of User Input]
[user input] (The End of User Input]
[The End of Response 1]
(The Start of Response
(response 2} [The End of Response 2]
Figure 15: Prompt of LLM-as-a-juddee +Choice Attributes.
29


Reviews:
[{"review_id": "IqUiyYiu8p", "reviewer": "Reviewer_Ehiq", "summary": "The paper tackles personalized preference modeling for LLM outputs with a generative reward model, P-GenRM, tailored to each user. The method trains a model that outputs a short persona and a weighted scoring rubric from a user’s history and any stated preferences, enabling clearer, user-aware judgments. The paper proposes a complex three-step training recipe to train their reward model that supports test-time scaling that averages multiple runs for the same user and also brings in signals from similar users to cut noise and handle cold-start. The experiments validate the approach on personalized benchmarks with consistent gains over prior methods.", "questions": "1. The paper proposes a three-stage training process. I think the training of generative reward models has been studied in previous works, as mentioned in the related works. What is the relation between your training procedure and previous works exactly? Why do you do so intuitively?\n2. For the experimental parts, I have two problems. a) To use your test-time scaling, what is the computation cost to have your results compared to the baselines? b) How many samples are required to generate a reasonable preference for each user?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper is generally a solid paper. The strengths include:\n\n1. The motivation is clear and important: personalized preference modeling is a real bottleneck for aligning LLM outputs to individual users. \n\n2. Conceptually, the method advances online preference handling by turning user evidence into a contextual persona and rubric, and by scaling judgments at test time with both multiple runs for the same user and signals from similar users; this is a novel and well-argued design. \n\n3. The approach integrates several effective components—supervised imitation, RL with process and outcome signals, hard-negative curriculum, and prototype-based test-time scaling—and the ablations substantiate that each piece contributes.", "weaknesses": "There are still some weaknesses listed as below.\n\n1. The method in the main text is too abstract. Key I/O and losses are not clearly written there or clearly pointed to the appendix. The algorithm, including both training and test-time, is generally complex, and I have some questions to be answered in the question section.\n\n2. Computation cost is underreported. I assume the additional test-time user-based scaling may take much more than the baselines. A computation cost ablation study may be necessary.\n\n3. Minor writing/format issues:\n   1. Capitalization after a comma: “Based on this, The personalized generative reward model…”\n   2. In table 3, \"w/o Cl, RL, SFT\" should be w/o CL, RL, SFT", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wD35MZIYFC", "reviewer": "Reviewer_qy8U", "summary": "This paper introduces P-GenRM, a personalized generative reward model that aligns large language models to individual user preferences. This is by generating structured evaluation chains, clustering users into prototypes, and applying a dual-granularity scaling mechanism that adapts scoring both at the individual and group levels. This approach reduces noise, improves generalization to new users, and achieves good performance with additional gains from test-time user-based scaling.", "questions": "What are the practical limitations of this method? I suggest including a limitation section in the paper.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper address an important problem of user personalization by providing the full pipeline of collecting data, clustering users, refining the personalized reward model and adapts the output. \n2. The experimental evaluation is comprehensive. \n3. The pipeline uses both implicit and explicit preference signals, which fully utilizes the preference dataset.", "weaknesses": "1. The main concern is the limited novelty of the paper. It seems that the main contribution of this paper is proposing the overall pipeline of obtaining personalized outputs, by using existing methods such as generative reward models and clustering users. It is not very clear what the technical contributions are. \n\n2. Lack of analysis of inference costs. It would be nice if some analysis on the costs of personalization can be done, including analysis of the baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E5YgJ67Rbw", "reviewer": "Reviewer_ZALW", "summary": "This paper proposes P-GenRM, a personalised generative reward model to address personalised alignment. In more detail, the authors aim to address the challenge of oversimplifying scenario-specific preferences and generalising new users with limited feedback. P-GenRM is a combination of various methods: (i) transforms preference signals into evaluation chains, (ii) clusters users into prototypes, (iii) various stages of training form SFT, to RL, to hard-negative aware curriculum learning.", "questions": "- How would you implement P-GenRM for generation in a policy model?\n- Could this potentially even be extended to Direct Preference Optimisation (DPO) as an implicit reward model?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper has various strengths, particularly I wish to point out:\n- The achieved results on first look are indeed promising, especially Table 1, which makes a strong case for their proposed method\n- The experiments, especially the ablation experiments, are extensive and cover most of the questions I had while reading this paper.\n- Their overview figures (1 and 2) help to understand what is happening in their method.", "weaknesses": "I wish to point out some weaknesses that this paper has, upon which the authors could improve to make a stronger case:\n\n- **missing errorbars**: While Table 1 reports the error bars (I assume this is the standard error?), all other experiments do not report any error bars, which makes it difficult to gauge the statistical significance of the experiments. Especially the results in Table 3 and Figure 3b could have overlapping errors.\n\n- **Composition of many methods**: While I appreciate the work of the authors in explaining and displaying the various parts of their methods, to me (personally), the final method seems a bit over-engineered. This makes it difficult to gauge which part has a more substantial effect on good performance, and I also found it hard for the reader to follow. Table 3 does a good job in slightly mitigating this problem\n\n- **What about Generation?** To me, the reason to have a reward model is to train a policy model which demonstrates personalised alignment characteristics subsequently. While I understand the authors' focus on the accuracy of the reward model, it would also be beneficial to test whether this RM can be used to train a policy model and whether personalisation still works.\n\n- **Is average accuracy the right metric?** If I understand it correctly, the authors use average accuracy as a metric in most tables. This would make sense if we are interested in general preference learning. However, as we are looking into personalised preference learning, it is essential to investigate accuracy across the different types of personas in the dataset. Because if there is a majority class/persona in the dataset, and the RM learns its preferences, the average accuracy will look good, while not actually achieving personalised alignment.\n\n**Misc**\n- I did not find any code available to inspect what is happening\n- The authors refer to the LLM-as-a-judge output as process reward, despite only giving a single value in return. I would abstain from using this terminology, as process reward models (at least from my understanding) are (semi-) dense reward models that assign intermediate rewards to tokens, rather than just one value at the end.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)