Learning to summarize user information for personalized reinforcement learning from human feedback
Authors: HyunJi Nam
Yanming Wan
Mickel Liu
Peter F. Ahnn
Jianxun Lian
Natasha Jaques
Venue: ICLR 2026 Poster (2026)
Keywords: pluralistic preference alignment
RL finetuning of LLMs
pluralistic reward modeling
Primary area: foundation or frontier models, including LLMs
TL;DR: PLUS enables personalized reward prediction and response generation for pluralistic LLM alignment. It improves reward modeling accuracy by 11-77\%, and responses conditioned on PLUS summaries achieve a 72% win rate compared to default responses.

Abstract: As everyday use cases of large language model (LLM) AI assistants have expanded, it is becoming increasingly important to personalize responses to align to different users' preferences and goals. While reinforcement learning from human feedback (RLHF) is effective at improving LLMs to be generally more helpful and fluent, it does not account for variability across users, as it models the entire user population with a single reward model, meaning it assumes that everyone's preferences are the same.
We present a novel framework, **P**reference **L**earning **U**sing **S**ummarization (**PLUS**), that uses reinforcement learning (RL) to learn to produce text-based summaries of each user's preferences, characteristics, and past conversations. These summaries condition the reward model, enabling it to make personalized predictions about the types of responses valued by each user. Both the user-summarization model and reward model are trained simultaneously, creating an online co-adaptation loop. We show that in contrast to the standard Bradley–Terry model, summaries produced by PLUS capture diverse aspects of user preferences, achieving a 11–77\% improvement in reward model accuracy. Key strengths of PLUS are: (1) robust performance with new users and conversation topics, achieving a 25\% improvement over the best personalized reward model technique used for RLHF; (2) zero-shot personalization with state-of-the-art proprietary models like GPT-4 (e.g., PLUS-summary-conditioned responses achieved a 72\% win rate compared to 28\% for default GPT-4o); (3) learning from flexible user contexts beyond preference labels, and (4) interpretable representation of users, enabling greater transparency and user control in pluralistic LLM alignment.

Published as a conference paper at ICLR 2026
LEARNING TO SUMMARIZE USER INFORMATION FOR PERSONALIZED REINFORCEMENT LEARNING FROM HUMAN FEEDBACK
Hyunji Nam Stanford University
Yanming Wan
University of Washington
[h essaanmdsdaooodd.ee
{ymwan]0ca washington. edu
Mickel Liu
Peter F. Ahnn
University of Washington
DoorDash
mnicke17]@cs washington, edu
{peter. a @doordash. com
Jianxun Lian
Natasha Jaques
Microsoft Research Asia {1 [jianxun. lian] cwmocrrooot.commmm
University of Washington {nj (nj]ecs. washington.edd.
ABSTRACT
As everyday cases of large language model (LLM) Al assistants have panded, it is becoming increasingly important to personalize responses to align to different users' preferences and goals. While reinforcement learning from human feedback (RLHF) is effective at improving LLMs to be generally more helpful and fluent, it does account for variability across users, models the entire user population with a single reward model, meaning it assumes that everyone's s pref- erences are the same. We present novel framework, Preference Learning Using Summarization (PLUS), that uses reinforcement learning (RL) to learn produce text-based summaries of each user's preferences, characteristics, and past conver- sations. These summaries condition the reward model, enabling it to make person- alized predictions about the types of responses valued by each user. Both the user- summarization model and reward model are trained simultaneously, creating online co-adaptation loop. We show that in contrast to the standard Bradley-Terry model, summaries produced by PLUS capture diverse aspects of user preferences, achieving 11-77% improvement in reward model accuracy. Key strengths of PLUS are: (1) robust performance with new users and conversation topics, achiev- ing 25% improvement over the best personalized reward model technique used for RLHF; (2) personalization with strong proprietary models like GPT-4 without further fine-tuning (e.g., PLUS- S--Ss-mmary--oonne conditioned responses achieved 72% win rate compared to 28% for default GPT-4o); (3) learning from flexible user contexts beyond preference labels, and (4) interpretable representation of users, enabling greater transparency and user control in pluralistic LLM alignment.
INTRODUCTION
Large language models (LLMs) have become a daily part of life for over billion people, with Gemini reaching at least 450 million monthly users by July (Lee, 2025) and ChatGPT's daily users approaching a billion (Curry, 2025). Users interact with LLMs in various ways, from asking for domain-specific help to engaging 2024). Reinforcement Learning from Human Feedback (RLHF (Ouyang et al., 2022)) is criti- cal component of modern LLMs deployed in production systems, because ensures that they are not only functional (i.e., usefully follow user instructions) but also aligned with constitutional values such as safety and helpfulness (Bai et al., 2022). The standard Bradley- -erry-eryyLeee (BTL) reward model (Bradley & Terry, 1952) used in existing RL.HF algorithms learns reward function based
open-ended storytelling and content creation(Zheng et al.,

---

Published as a conference paper at ICLR 2026
majority preferences: If two users have opposite preferences about the response particular prompt, and no further context is provided, the BTL model cannot learn to disambiguate between different users' preferences.
What is your opinion abortion? (X)
PLUS summary: "The user profors detailed, factual answers with supportive examples and clear explanations."
Un-personalized (GPT-40) response (Y)
*I can provide a dotaled, factual overviow.... Views on abortinn often reflect orcader societa/ beliefs and values, including those morality and medical ethics. For logal perspectives vary widely, with some states imposing strict regulations while others support woman's right to choose. Factors like fetal viabiliit and maternal health often play roles these laws. Understanding contexts helps form comorehensive view of
R(ylx)
sesuch en Soun wil cheoe
related women's
eippoetts insluding srtbal logut
conslovation..
examplo,
manrpte spaall centarl swstioned
Personalized / PLUS-condiffoeed response
R(y/x, X. Z)
penonalized S pharalistc
Figure 1: Caution: This content may reflect particular beliefs. While standard RLHF techniques fail to capture user variability, PLUS trains both summarizer and reward model in an online co- adaptive framework learn summaries useful for predicting diverse preferences. Italicized texts are actual outputs by GPT-4o and PLUS showing the effects of summaries on personalization.
However, real users have diverse, potentially conflicting, preferences (Kirk et al., 2024), and cur- rent methods are unable to accommodate these differences. In fact, when the standard BTL model is applied to data with conflicting preferences, it will either ignore the preferences of minority groups (Siththaranjan et al., 2024), or learn a reward model that is inaccurate for all groups of users (Poddar al., 2024). This has led to recent calls to develop techniques for pluralistic align- ment Sorensen et al. (2024); Li et al. (2024); Siththaranjan et al. (2024), to enable LLMs to be personalized to heterogeneous user preferences. Given the widespread use of LLMS and the diverse populations of users they serve, being able to model and adhere to more than one set of preferences critical for making effective, accurate assistants that are capable of personalizing to each indi- vidual's 's unique needs, One way to achieve pluralistic alignment is by developing a reward model conditioned on user variability, rather than relying on single BTL model (Siththaranjan al., 2024; Zhao et al., 2024; Poddar et al., 2024) (e.g., (Poddar et al., 2024) learns a latent user embedding from the past preference labels). However, compressing rich text into a vector can lead to performance loss (Cideron et al., 2022). As yet, we are missing a highly performant, public RLHF technique capable of modeling the preferences of diverse users.
A more natural approach which retains the strengths of LLMS to reason with text, is to learn text- based user summaries to act as the latent user variable. We propose Preference Learning Using Summaries (PLUS) as a general RLHF framework that learns to summarize important information about the user using reinforcement learning (RL) fine-tuning and conditions the reward model on these user summaries. PLUS consists of two co-dependent training phases: (1) summarizer trained with Proximal Policy Optimization (PPO) (Schulman et al., 2017). and (2) summary-conditione reward model, which provides the reward signal for training the summarizer and makes predictions about user preferences.
We carefully compare our approach several alternative implementations. First, we check whether we can simply prompt an LLM to generate user summaries without fine-tuning the summarizer. Our experiments show that automatic summaries (even from GPT-4) often miss key details necessary to capture user preferences. Another alternative is to condition the reward model on the user's 's en- tire conversation history, without summarizing it. However, both approaches fall short compared reward models when conversation topics change. An additional
to concise, summary-condition benefit of concise summaries compared to existing techniques is that the summaries are easier to review and edit, improving transparency and trust in LLM assistants. We show that such benefits ex- tend to complex real-world datasets, like PRISM (Kirk et al., 2024), where diverse user preferences be expressed through flexible textual summaries and used enable personalization of strong, proprietary models without additional fine-tune muninuning. In summary, our main contributions are:
1. Novel pluralistic alignment algorithm, PLUS (Preference Learning Using Summarization) that uses RL to jointly learn summarizer and reward model. PLUS improves accuracy, inter- pretability, and transferability across new conversations and users.
2

---

Published as a conference paper at ICLR 2026
2. Empirical advantages of PLUS on benchmarks from prior state-of-the-art work (Poddar et al., 2024): Pets and UltraFeedback (Cui et al., 2024). PLUS improves reward model accuracy by 11-77% compared to BTL, is robust to topic shifts, and can handle unstructured user contexts.
3. Scaling to a dataset in the wild. We are one of the first to report reward modeling results with PRISM (Kirk et al., 2024), pluralistic dataset of 1,500 users across 75 countries and 20 LLMs.
summaries. PLUS can generalize to unseen users of the PRISM dataset and the learned summaries can be used to personalize responses of strong proprietary models without additional finetuning (PLUS summary- conditioned responses achieve a 72% win rate over the default GPT-44o responses).
4. Enabling personalization of strong, proprietary models via PLUS-generatee
2 RELATED WORK
Reinforcement Learning from Human Feedback (RLHF). Our work builds on reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022), which models binary human prefer- ences using the Bradley-Terry-Luee (BTL) model (Bradley& & Terry, 1952). The BTL model assumes the user provides preference ratings in terms of which of two responses they like better; it then learns a reward model (RM) that estimates the magnitude of the user's preferences using this data. This is then used to train an LLM policy to optimize responses based on the reward signal. RLHF has been effective in refining language models' responses to better align with user-pre seer-preferred attributes, like helpfulness and truthfulness (Bai et al., 2022). However, existing RLHF works (Ziegler et al., 2020; Bai et al., 2022; Ouyang et al., 2022; Siththaranjan et al., 2024; Rafailov et al., 2024) have assumed a single reward model to capture all user data, which may be misaligned with heterogeneous users.
Personalized & Pluralistic RLHE Increasingly, more works (Sorensen et al., 2023; Kirk et al., 2023; 2024; Zhao et al., 2024; Poddar et al., 2024; Li et al., 2024; Chen et al., 2025; Liu et al., 2025) have pointed out the limitations of existing BTL reward models in accommodating diverse underlying preferences, which may conflict with each other in terms of what individual users con- sider ideal responses. These differences may surface along various dimensions, from writing style to value alignment. Despite algorithmic advances in pluralistic alignment, large-scale human datasets are still limited. PRISM (Kirk al., 2024) makes an effort to provide a heterogeneous user dataset consisting of 1,500 participants from 75 countries interacting with 21 different LLMs (contributing around 9,000 unique conversations). However, so far we are not aware of work that has successfully published reward modeling results on PRISM, pointing to the difficulty of modeling diverse and heterogeneous preferences with sparse data.
Broadly, works on pluralistic RLHF have focused on either of two main aspects: (1) personalizing reward models (Siththaranjan et al., 2024; Poddar et al., 2024; Zhao et al., 2024; Liu et al., 2025; Chen et al., 2025) and (2) personalizing LLM responses with user model (Li et al., 2024). Reward (user) modeling serves as the backbone of many existing algorithms for LLM alignment, since re- ward model accuracy has direct implications for downstream tasks, such as RLHF finetuning and best-of-N sampling (Shen et al., 2023; Gao et al., 2022; Meng et al., 2024). Notably, Poddar et al. (2024) introduces Variational Preference Learning (VPL.) which uses variational autoencoder to learn user-specific embeddings and conditions the reward model on these latent variables. Siththa- ranjan al. (2024); Zhao et al. (2024); Chen et al. (2025) provide algorithms for accommodating diverse user groups through methods such as Distributional Preference Learning (DPL), dispersion index, and shared LoRA adaptation. However, the key difference between prior works and ours is that these methods represent users as embedding vectors or (linear) weights, which reduces the power and interpretability of the underlying personalization mechanism. In contrast, we leverage the text-generatio capabilities of L.MMs to represent diverse user preferences through summaries.
LLM finetuning for summarization. Finetuning LLMs for improved and nomain--pecific sum- marization has long been of interest (Stiennon et al., 2022; Ziegler et al., 2020; Zhang et al., 2023; Van Veen et al., 2023; Song et al., 2025; Wu et al., 2025). Most closely related to our work is by Wu et al. (2025) which demonstrate that learning user summaries can be effective for personalizing recommendation systems, when you assume access to a fixed user classifier. In contrast, we focus on the online co-adaptive loop of training both the summarizer and reward model, which serves as the key for enabling the reward model to make use of the learned summaries for capturing nuanced user preferences.
3

---

Published as a conference paper at ICLR 2026
3 PRELIMINARIES
Bradley- Tarosy--errr--eeee reward model. Our work focuses on the reward modeling stage of the standard RLHF pipeline (Ouyang al., 2022). Given dataset of annotated preferences D {(s') s'g, ( > s'>))==1 where &A = [x. yal and V/ = [æt Val are two responses yA y over the same prompt æ. the standard Bradley-Temy-Luee (BTL) model (Bradley & Terry, 1952) assumes an underlying reward function r(s). This reward function determines the likelihood of the preference label 1 [SAA* > $B} as:
expr(sa)
p(l > $B} = 1 SA. expr(sA) + expr(ss) = (r((s)( r(sB)) -((ss)
(I)
the sigmoid function. The reward modeling stage typically involves learning parameterized reward function (s(s) by maximizing the likelihood of Eq. 1 over the ob- served response pairs and their preference labels. The BTL model assumes all users share the same reward function, and thus fails to capture conflicting preferences when two users prefer different responses to the same prompt æ.
where (ar) =
User-conditioned reward model. Existing work in personalized RLHF (Li et al., 2024; Poddar et al., 2024) has demonstrated the effectiveness of extending the standard BTL reward model into a user-conditioned reward model:
r(pl()))2)
p(1(sA> > sBl2} = SB, expr(sal2) + exp expl() = (r(ssa))))))))))--)))))))))))))) r((Blz)) (2)
groups. Eq. 2 is better suited for capturing the population's 's heterogeneous preference distribution that may be multi- modal, and the target is to learn parameterized user-conditioned reward function r=(s(2). This gives the reward model the expressivity to capture different user types, which yields more accurate predictions when user preferences are diverse and even conflicting.
where is identifier (c.g., user ID) that distinguishes different users
However, in most real-world scenarios, datasets do not contain user identifiers z. This is partly be- cause such identifiers may be unavailable due to privacy issue, and partly because conditioning only on users observed in the dataset makes it difficult to generalize to new users. Therefore, we can instead use context c, such as past conversations, represent the current user in each data point'. This leads to dataset of user-conditioned annotated preferences D' {(s's s 2', { > s' } where 2' are unknown. For notational convenience, we assume s' is always the pre- ferred one by user 2f, and the dataset can be simplified to D' = {(s), 8/g, )
4 PREFERENCE LEARNING USING SUMMARIZATION: PLUS
While prior works (Li et al., 2024; Poddar et al., 2024) formulate the latent user representation z as an embedding vector, we propose learning in natural language more natural way to leverage the advanced text-based reasoning and generation capabilities of LLMs. To be more specific, we propose to use summarizer To generate a concise representation in natural language based on the context e of each user, 2 ~ [-cc) Then, the parametrized reward model r-(+))) will compute the reward conditioned this learned summary
Rather than manually defining any ground truth criterion for useful user summaries and pre-training summarizer in advance, we argue that the quality of summary should be measured by its ability to enable the downstream reward model to accurately predict user preferences. Therefore, the sum- marizer and the reward model should be optimized under joint objective, namely minimizing the negative log-likes likeliooldood loss based on Eq. 2 across the coneiionii dataset D':
(1) 0) = R. c) (swaa [log ar (Fo(sA)=) - r=())])))]]
To handle the co-dependency in Eq. 3, we design training process that continues to alternate be- tween training one model while keeping the other fixed. By allowing the summarizer and the reward
Poddar et al. (2024) requires that the context consist user's past chosen and rejected responses to different prompts {(5) $b)) 1m In contrast, PLUS can incorporate more flexible inputs like past conver- sations without explicit preference labels and system strings.
4

---

Published as a conference paper at ICLR 2026
model to co-adapt, the summarizer learns to extract information that is most useful for improving the predictive accuracy of the reward model, while the reward model is trained on updated sum- maries to more accurately capture the generated summaries always target the most recent iteration of the reward model.
preferences. This process is computationally efficient, since
Training the reward model. When training the reward model, the summarizer weights fixed, and the reward model is trained to minimize the negative log-likeliine log-likelihood loss (Eq. 3) using pairs of chosen and rejected responses, conditioned on user summaries sampled from the summarizer policy. Then, we repeat the step of training the summarizer using the newly obtained reward model weights.
Training the summarizer. Unlike the reward model, sum- marizer's optimal weights cannot be learned directly through supervised learning, because whether summary is effee- tive for predicting preferences can only be assessed once the full summary has been sampled. Instead, RL can opti- mize the summarizer at the token level by assigning credit for the full-trajectory return across long token sequences (Z1 #gy..., =. ). as demonstrated by prior successes in using RL to improve summarization (Stiennon et al., 2022; Ziegler et al., 2020). Thus, formulate generating the summary into an RL post-training problem, with the objective in Eq. 4. We start with default LLM reference policy No, sample summaries (trajectories) from the current estimate their returns (as negative losses of the reward model), and update the weights in direction that maximizes estimated returns.
T
log loga(ra(sA(z) [a(3)(=))
No mo(-co
Figure 2: Eq. 3 naturally leads to the online co-adaptation of the summarizer and the reward model.
(TTO = xo0T((-1e) (NA,SB.C)NDD
Σ o' (
[0,
ift < H, if ( H.
Rs 8B)
zx) ra(sn =x)),
PPO training for handling non-stationarity. As the reward model for predicting user preferences continuously updated, the RL rewards for training the summarizer become non-stationary, in multi-agent RL (MARL) problem. Therefore, while different RL algorithms could be considered for learning effective summarizer, MARL algorithm and performs well even under non-stationary rewards (de Witt et al., 2020; Yu R',
choose PPO (Schulman et al., 2017), as it is state-of-the-ar
et al., 2022). In addition to the summary-level rewards R = Σ/ 7 Rg, we calculate token-level advantages using the generalized advantage estimator (GAE) (Schulman al., 2018):
A GAB(?.A)
Σ (TA)'O B% R. (V(2)1)) -((s)
where Vi are token-level value estimates by the PPO critic. Advantages are normalized across batches Apomm and the summarizer mo is optimized with clipping to avoid instability, leading to the PPO loss:
To <
Le 0) -E, [min (G) cl),(((()), - 6, 1 e)
Anorm) (1 (C()
(6)
Tew (2 [et)
The complete joint-training framework is illustrated in Alg. 1.
Why learning to summarize important. focus on past content rather than the general style of conversations the user prefers to have. There- fore, the obtained summaries may not provide predictive signal for the reward model. Alternatively, one could include all prior preference labels from the user as in-context learning (ICL. examples, and train the reward model conditioned on such examples (i.e., the full list examples acting as the conditioning variable z). However, the large input space of ICL may make reward learning more difficult. This shows up in our empirical results, where ICL underperforms, especially when trying to generalize to conversation topics. While in theory ICL can enable personalization, it may be sample-inee inefficient and computationally expensive to train and use for inference due to the increased context length.
untrained summarizer can lead to summaries that
5

---

Published as a conference paper at ICLR 2026
Algorithm 1 Co-adaptation of Summarizer and Reward Model
Require:
Initial summarizer Te. reward model ræ. dataset D' of user contexts and preference pairs, rollout batch size N, mini-batch gradient steps Mo, M
1: for each training iteration /. do
2: Sample N tuples of contexts and preference pairs from D
3: Generate summaries
Te. (-lc)
4: Compute rewards in Eq. and advantages in Eq. 5
5: for iter = to My do
6: Compute the PPO loss Lero(e, #) in Eq. 6.
7: Update summarizer parameters: as + 0 ag Vo VeLoo end for
8:
9:
for iter= 1 to M& do
10:
Compute the negative log- lieelikelihodd loss (o) in Eq. 11: Update reward model parameters: & as Φ t- Φ as v LNLL(G ())
12: end for
13:
end for
return Trained Na, r
Implementation. We finetuned different reward models (Qwen2.5-005B 5.B.50005IIInstccc,, Qwen2.5-3B. Instruct, and Instruct). We paired each reward model with summarizer from the same model family. We used 2-4 x H200 GPUs for model finetuning. The co-adaptation pipeline in Alg. 1 is built on top of PPO training framework from OpenRLHF (Hu et al., 2024), and we also used the reward model training code from OpenRLHF. For other baselines, namely VPL, DPL, and BTL (for Pets), used the implementations by Poddar et al. (2024). More details are in Appendix
and summarizers (Qwen2.5-33) In-srructt or Llama3.2 3--3BB
5 EXPERIMENTS
Domains. We evaluate on three benchmark tasks from Poddar et al. (2024), along with a challeng- ing variant that tests out-of-distribution user contexts, well as real user datasets. Pets. In this dataset, 50% of users prefer to discuss dogs and 50% prefer cats, so a single reward model will fail accurately capture user preferences. To test the robustness of different methods with new conver- sation topics and users, we create new Pets OOD (Out-Or-Distribun of users who prefer talking about rabbits or birds, as opposed to cats and dogs. Using this dataset, we can assess how well each method can generalize to new users with novel preferences despite being trained on only cats and dogs. Poddar al. (2024) construct two domains from the scaled-up up prefer- ence dataset UltraFeedback (Cui et al., 2024). In FltraFeedback- or honesty, and in UltraFeedback- users choose subset of four possible attributes an LLM assistant: helpfulness, honesty, truthfulness, and eestruction-oolloiiin ized reward models need to infer different user types based on past conversation data. For example, some users may prioritize helpfulness and honesty, while others may value instruction-followinn above all other attributes. Following Poddar et al. (2024), we also focus on a controversial subset of the dataset by removing samples where all users agree on the chosen response. Evaluation with real user data (PRISM). This real-world user dataset consists of open-ended conversations from 1,500 participants across 75 countries and 20 LLMs (Kirk al., 2024). Each user engages in up to six conversations on topics of their choice, loosely guided by one of three themes: unguided (free topic), value- ggiuee, or controversy-guided
dataset, which consists
users prefer either helpfulness
In both domains, personal-
We filter out users who did not answer the preference survey quit before completing all six conversations. As result, we train on 1,000 users and evaluate personalization performance on held-out conversation topics and/or users. This domain particularly challenging due to the high heterogeneity and variance across users, as well as the lack of shared conversation topics, compared to the size of the training data. Despite the challenge, our work is among the first to report pluralistic reward modeling performance the PRISM dataset.
We evaluate PLUS against six existing RLHF baselines: Bradley-Terry-Luee (BTL) model most widely used in RLHF preference learning Ouyang et al. (2022). The BTL model uses a single model to capture all user preferences which may be misaligned with heterogeneous popu- lations. Distributional Preference Learning (DPL. Siththaranjan al. (2024) proposes to learn
Baselines.

---

Published as a conference paper at ICLR 2026
both the mean and the variance of the reward distribution, which captures overall variance across users' preferences but does not model each user's distinct preferences. Variational Preference Learning (VPL) Poddar et al. (2024) learns latent representation of each user with a vector of size 512 using a variational auto-encoder (VAE) reward model and shows state-of-the-art perfor- mance on pluralistic preference modeling. In-Context Learning (ICL) Past conversation examples are directly included as part of the reward model's prompt, in addition to the chosen and rejected response. This would in theory enable modeling each user's distinct preferences. but may be less sample-efficient due to the lengthy context, and less robust to topic changes. PLUS with a fixed summarizer (PLUS-untrained), This ablation only trains the reward model while fixing the sum- marizer. Oracle. This is reward model conditioned on the true user preferences. For example, in Ultral Feedback, if user values helpfulness and honesty over the other attributes, that user's prefer- ence model is conditioned on the following string input: This user prefers helpfulness and honesty in the assistant's 's responses.
6 RESULTS
Advantages of PLUS on existing benchmarks. First, we compared PLUS against existing RLHF methods on existing personalized reward modeling benchmark tasks. Table shows that PLUS achieves improved prediction accuracy across all three domains compared to vanilla RLHF with the largest improvement of 77% on Pets. While existing personalized RLHF techniques, like VPL and ICL also perform well on Pets, they are not as performant for complex domains resembling real human preferences, like UltraFeedback P-2 and P-4, where conversation contents vary between training and test samples, while the underlying user groups remain the same. VPI. struggles to per- sonalize showing limited improvement over BTL. (-2.76-4.81 % across model types). We hypoth- csize this is because training the reward model to distill all user context into a fixed-length vector sacrifices performance, previously discussed by Cideron et al. (2022). While in theory ICL can model each user's distinct preferences, conditioning on all past conversation examples is sample- inefficient due to long, unsummarized contexts and is more sensitive to conversation topies than to the underlying user preferences. Specifically, ICL only shows 0.9-8.4% improvements across model types, compared to 10.4-17.7% achieved by PLUS, on the UltraFeedback dataset. Even for larger reward models --lam-8B--Instrere and IB-nsttcn,,,,, ICL still perform less accurately than 3B PLUS for Ultrafeedback domains (Table 6). We further investigate this limitation with our Pets-OOD experiments.
Table 1: Reward model accuracy on the held-out set. We report the mean and two-sided 95% t-intervals over 3 seeds; otherwise, single seed.
Pets Qwen- D.5B
Utrafeedback-PP P22
UttrfeedDacc--P. Qwen 3B
Reward model
Qwen 0.5B
Qwen 3B
Llama 3B
Qwen-O.5B 0.5B
Llama 3B
BTL (Bradley & Teryy 1952)
44.15 17.02
0.87
58.75
$7.40
$3.53 3.27
$6.25
56.35
DPL. Sihtharaannn et al. (2024)
46.20 + 10.95
58.53
1.74
58.15
57.55
55.22 + 3.71
57.00
56.60
VPL Poddar al. (2024)
100
58.37
- 0.87
57.15
59.40
54.2 5.84
57.45
ICL
100
59.62
1.79
59.30
59.10
57.22 = 0.95
59.10
57.70
PLUS-untrained
96.20 1: 4.13
59.90 1:00::000 0.50
59.65
59.35
56,89 : 1.11
58.75
57.90
PLUS (Ours)
99.80 0.86
69.40
2.81
69.85
66.90
62.10
0.69
62.45
62,80
Oracle
100 0
69.10 11
0.72
68.95
67.95
70.60
1: 1.93
73.05
73.80
Even summaries by PLUS -nttunnte can achieve improvements over BTL, but they still fall short of providing sufficient predictive signal for the reward model. This limitation is clearly observed in the Pets dataset: the untrained summarizer struggles to distinguish meaningful user contexts from spurious details or outputs vague descriptions, while the trained summarizer correctly identifies whether user prefers cats or dogs, as shown in the box below. This highlights the importance of learning summaries, which we further investigate through ablations that fix either the reward model or the summarizer.
Robustness of PLUS to new topics & users. As new users increasingly engage with LLM as- sistants, robustness to new users and conversation topics that are different from the training data becomes important for personalized systems. While UltraFeedback already evaluates on new user contexts, these come from users with the same underlying preferences. We took step further by
7

---

Published as a conference paper at ICLR 2026
explicitly evaluating new users whose test-time preferences shift to entirely different content, using Pets (OOD).
Concretely, the context and preference labels (SA. used for training about dogs cats, but test time, the summarizer receives a new user context about birds and rabbits, and the reward model is given two response options between birds and rabbits. As shown in Table 2, PLUS's robust generalization to new user contexts indicates that it has learned a general algorithm for extracting user information into summaries, which translates well new users with new types of preferences. In contrast, other approaches may be vulnerable to memorizing specific users' preferences, rather than learning how learn about what new user interested in.
Table 2: Reward model accuracy with and without shift in Pets. We report the mean and the two-sided 95% t-intervals seeds. The reward model is trained with Qwen2.5- 0.5B- I.st-ustt and the summarizer is trained with Qwen2.5-3B B-structt Mean and std from 3 seeds. ICL's performance drops from 100 72%, and VPL drops to below 50%, whereas PLUS the only method to maintain accuracy above 90%.
Out-of-distribation user contexts
BTL
49.5 + 6.45
DPL
49.0 + 1.24
VPL
44.83 + 10.04
ICL
72.50 7.76
PlUSSuntrained
85.83 + 10.04
PLUS (Ours)
93.67 + 9.96
Oracle
100+0
Comparison of PLUS-generated and untrained summaries
PLUS summary (shows clear preference for cats): The user is interested in information about cat behavior and properties, excluding topics related to dogs or human effects.
Default summary (discusses general pet traits): The user seems to appreciate traits that denote affectionate and playful qualities in pets, as evidenced by their rejection of traits related to training or timing of activity (renewal of knowledge).
Default summary (focuses on spurious feature): The user prefers short, factual information and tends to reject detailed anecdotes or complex behaviors. The assistant should stick to basic descrip- tions where possible.
Importance of co-adapting summarizer and reward. A key component of PLUS is the online co-adaptation of the summarizer and the reward model, which enables the summarizer to focus on dimensions of user variability that most relevant the reward model. We conducted ablations using different model types to examine the effect of training only the summarizer or the reward model. Table shows that despite using smaller reward model (O.SB), co-adaptation achieves higher performance than larger reward models (3B) that are either trained with fixed summaries or only used train the summarizer. Surprisingly, using a strong, proprietary summarizer (GPT-4o) with a trained 3B reward model still underperforms compared to co- co-training substantially smaller models (0.5B reward model and a summarizer) in both UltraFeedback domains. This supports our intuition about the importance of RI. fine-tuning for learning summaries: good summaries are defined by the reward model's performance rather than any fixed criteria. Therefore, the training signal for the summarizer should come from the reward model's loss.
Table 3: Reward model accuracy with static reward models or summarizer: Static reward model or summarizer is in red (1,, A and the trained model is in green (il, A All ablations with Qwen- Instructt series.
Ultra Feedback (UF)-P2
Ultra Feedback (UF)-P4
Summarizer
Reward model 10
Accuracy (%)
Summarizer
Reward model 00
Accuracy (%)
Trained
Static 0.5B 01
53.45
Trained 3B
Statie 0,5B 11
49.50
Trained 3B
Statie 1,5B 00
52.10
Trained 3B
Statie 1,5B (10
50.10
Truined 3B
Static 3B fi@
42.20
Truined 3B
Static 3B f0
53.50
Static /0
Trained 0.5B fty
59.90
Stutic 3B M
Trained 0.5B ftU
56.40
Static e
Trained 3B 0g
59.65
Static 3B
Trained 38 00
58.75
GPT-4o
Trained 110
59.30
GPT-4o
Trained 3B f10
60.80
Trained 3B
Trained 0.5B 00
69.40
Trained 3B
Trained 0.5B 10
61.80
Trained 3B
Trained 3B 93
69.85
Trained 3B
Trained 3B f'
62.45
8

---

Published as a conference paper at ICLR 2026
Incorporating flexible textual inputs beyond preference data. PLUS can incorporate additional user information in flexible forms of textual input, not just preference labels. We focused on two input types. Non-preffremce--aasee user contexts. PLUS can infer user preferences from natu- ral conversations without requiring chosen and rejected response pairs per prompt. Users do not typically provide both an accepted and rejected response, so VPL's input assumption may limit its applicability to real-world personalized systems. While both PLUS and ICL can train reward models using unstructured user contexts, Table 4 shows that PLUS outperforms ICL by 9-15%.
Table Reward model accuracy with unstructured inputs. The reward model is trained with Qwen2.5- 0.5B Instruct, and the summarizer is trained with Owen2.5-3B-Instte the UltraFeddack datasets.
User-guided instructions. For example, a system designer may know which dimensions are relevant for personalization but not know which attributes actually matter to a specific user. To simulate this scenario with the UI- traFeedback dataset, we made a simple mod- ification to the user's prompt stating "Users typically care about the LLM assistant's 's help- fulness, honesty, truthfulness, and instruction- following capabilities." This additional infor- mation improves performance of PLUS com- pared to ICI. by 10-16%. PLUS can incor- porate flexible inputs beyond preference data, which other personalized techniques, like ICL and VPL, cannot leverage effectively.
UF-P2 Accuracy
Non Non-preference
User guide
ICL 58.55
59.65
PLUS
68.30
70.25
UF-P4 Accuracy
Non preference
User guide
ICL PLUS
56,96 62.45
56.60
62.70
Pets
ULtrafeedbacke-2
PRISH
Figure 3: PPO training return curves for Pets, Ultrafeedback P2, and PRISM using Qwen3B- I-Istructt summarizer and Qwen0.5B reward model with a rollout batch size 256. The returns obtained by the summarizer are the negative prediction log-liiklihood) loss of the reward model.
Evaluating PLUS in the wild with PRISM user data. Despite the growing interest in pluralistic alignment, results demonstrating effectiveness real-world datasets have been limited due to both the challenging nature of real user data and the limited sample size. Having established the advan- tages of PLUS on existing benchmarks for pluralistic alignment, we next applied it to the real-user dataset PRISM (Kirk et al., 2024). Reward modeling with PRISM. We first implemented PLUS with a Swen2.5--B-Insttree users from the PRISM dataset. We also implemented six reward-model baselines with Qwen2.55- Instuuct Then, evaluated reward-mode accuracy the preference labels of held-out conversations. This evaluation protocol assumes that each user's underlying preferences remain consistent across different conversation topics, once the reward model learns user's preferences from five conversations, it should be able to predict the preferred response in a new, sixth conver- sation. Due the challenging nature of this dataset, which attempts to capture high degree of variance with a small sample size, we find that all methods achieve somewhat low reward predic- tion accuracy, and report the results in Table 8 in the appendix (in particular, when used to predict out-of-distributin
summarizer and Qwen2.5-0.5D Owen2.5-00.5-IInttrue reward model using 1,000
users, the pluralistic reward models, e.g.. VPL,, ICL and PLUS, perform poorly 11). PLUS outperforms existing (un)personalized methods with an accuracy of 62.9%, whereas BTL achieves 59.8 99. VPL 60.4%, and ICL 60.1 % for the in-distribution users. However, in spite of marginal improvements in reward modeling score, we find that PLUS is still able to learn to usefully extract user information that enables personalization as below.
Personalization on PRISM with GPT-4 models using PLUS summaries without fine-tuning. So far, we have focused the reward modeling performance of PLUS compared to existing RLHF techniques. However, another key advantage lies in the transparency and portability of PLUS- generated user summaries for downstream personalization tasks.
Personalized LLM-as-judges. We combine PLUS with an LLM-as-judge setup (Zheng et al., 2023) to enable personalized reward modeling without further fine-tuning. Specifically, evaluate GPT-

---

Published as a conference paper at ICLR 2026
4's prediction of user's preferred response between two candidates, both with and without the L.US-generatee summary. The evaluation includes held-out user examples as well as value-guided and controversial conversations from known users from PRISM. As shown in Table 9, across three task types, T.US-guided -uueeess improve accuracy for both GPT-4o (by 19.5%) and GPT- 4.1 (by 2.9%). These results demonstrate that PLUS- summaries effective only for personalizing learned reward models but also for personalizing proprietary models without further fine-tuning.
GPT-4o
GPT-4.1
Personalized response generation. We prompted GPT-4o and 4.1 generate responses with and with- out the PLUS-gee -eU-geeeeeteed user summary. Specifically,
we sampled two new prompts for each of the 308
Win
held-out users in PRISM and compared the win rates of personalized versus default GPT responses eval- uated by oracle reward model, which is trained to condition on users' self-stated preferences (in PRISM, users rate different attributes of an LLM sistant in surveys, which we use as ground truth). This experiment assesses whether PLUSS-enerated summaries are useful for enabling proprietary mod- els personalize to new users without additional finetuning. In Fig. 4, the win rates of PLUS- personalized responses over default GPT responses clearly demonstrate the effectiveness of PLUS (e.g., 72% vs. 28% for GPT-4o, and 69% vs. 31% for GPT-4.1). We included examples in Ap- pendix F to show the effect of summary-conndiioned personalization using GPT-4.
Default
PLUS
Default
PLUS
Figure 4: Win rates of personalized vs default GPT-4 the PRISM dataset (evaluated with two prompts per 308 unseen users).
7 DISCUSSION & LIMITATIONS
We propose PLUS, novel RLHF technique of learning text-based summaries to capture user vari- ability and co-training a reward model conditioned on the obtained summaries. Summaries provide natural format that leverages the language modeling capabilities of LLMs and show strong em- pirical performance that extends beyond benchmark tasks. We conducted extensive experiments and compared approach against six RLHF baselines, demonstrating the following five benefits. Strong empirical advantages. PLUS outperforms standard BTL by 11-77% on pluralistic datasets of Pets and Ulraaeedback. Robustness. PLUS is the only method that maintains 90% accuracy with test-time preference content shifts. Benefit of co-adaptation. Co-adapting a smaller summa- rizer (3B parameters) and reward model (0.5B) outperforms a reward model trained on proprictary model summaries by 10%. Flexibility to incorporate unstructured user contexts. Unlike prior personalized RLHF techniques (Poddar et al., 2024; Li al., 2024). PLUS can leverage user- guided instructions and unlabeled conversation data, achieving improvements of 9-16% compared to ICL. Summary-conditioned personalization
proprictary models, achieving a win rate of 72% against the unpersonalized GPT-4o responses. Furthermore, generated summaries are human- readable, enhancing transparency and improving users' trust LLM systems. Limitations. PPO in MARL setup, which enables co-adaptation of the summarizer and the reward model, may be sensitive to training hyperparameters, such as lcarning rates. Some potential approaches to mitigate training issues (e.g., instability, under-convergencc) may include training the summarizer and reward model for more than one epoch, as observed by Poddar et al. (2024) (for their latent encoder and the reward model); specifically, for our MARL setup, freezing the weights of the summarizer after one epoch and continuing to train only the reward model conditioned on the generated summaries, or warm- warm-starting the reward model by first training it on the outputs of the untrained summarizer. These approaches may be helpful while taking care to avoid overfitting to the training data.
ETHICS STATEMENT
We propose PLUS as a step toward pluralistic preference alignment that is both interpretable and ac- cessible even for non-technical users, the user representation is in human-readable text form. We hope to broaden the benefits of personalized and value-aligned LLM assistants to a wider audience, while being cognizant of ethical considerations, like privacy and user trust in AI systems. Privacy considerations. We believe that any personalized reward modeling methods that require maintain- ing information for each user group of users may into the issue of using user's private data
10

---

Published as a conference paper at ICLR 2026
to distinguish between different users. Therefore, it would be crucial to consider removing any sitive information prior to training a user-conditioned reward model (ICL, VPL, PLUS) or a PLUS summarizer for actual deployment users. Given the importance of transparency and maintaining user trust in personalized Al systems, we helieve that the ability to present user summaries natural language is comparative advantage of PLUS that other personalized reward models do not afford. The user-specific summaries can be shown to the user to reveal which parts of the previous conver- sation are being prioritized by the reward model, and it would be interesting to consider allowing users to edit or remove specific information from these summaries.
REPRODUCIBILITY
Our code is available at this link. The vo-adaptation pipeline in Alg. 1 was built on top of PPO training framework from OpenRLHF (Hu et al., 2024). We also used the reward model training code from OpenRI.H.. For other baselines, namely VPI,, DPI, and BTL (for Pets), we used the implementations by Poddar al. (2024). Our experiments were conducted with H200 GPUs.

---

Published as a conference paper at ICLR 2026
REFERENCES
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage. Zac Hatfield-Dodds Danny Hernan- dez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Ka- plan. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022. URL https:tt///.o....... a org/ abs / 2204. 058 62. 1,
Ralph Allan Bradley and Milton E. Terry Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39:324, 1952. URL https://api. semanticscher org wttrg/oooosss::1120551
1, 4, 18,
Haoxian Chen, Hanyang Zhao, Henry Lam, David Yao, and Wempin Tang. Mallowspo: Fine- tune your Ilm with preference dispersions, 2025. URL
org/abs/240555
14953. 3
Geoffrey Cideron, Sertan Girgin, Anton Raichuk, Olivier Pietquin, Olivier Bachem, and Léonard Hussenot. vec2text with round-trip translations, 2022. URL https:///xxx..i.coo 2209. 06792.672222 2,7
abs/
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Ultrfeedback: Boosting language models with scaled feedback, 2024. URL https:// http:://axwvv.r......te abspg//bb///313 01377. 3,
David app installs.
Curry.
Chatgpt and gemini
battle for the top spot in
https://wweet wwwp:bbuiieesofeeee. com newn/nwww///// chatgpt- and--nd-aaa memiini-battee for-te--forhheee tochhe-to--ee t-stttopspppp spot- in- app-installs/
2025. Accessed: 2025-09-23.
Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip H. S. Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge?, 2020. URL https::///// /ttps//////iv/////eeeeeeeeeeaavs org/abs /2011. 09533.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization, 2022.
URL https::pp//. /ttxivivvvv///////v///a.//aaaa orgo abs/ 221 0.100..0 10760.
Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Openrlhf: An easy-to-use, scalable and high- perfoorrfoeeece rlhf framework. 2024. URL https://arxiv.
org/abs/2400 11143. 6, 11, 22
Hannah Rose Kirk, Andrew M. Bean, Bertie Vidgen, Paul Röttger, and Scott A. Hale. The past, present and better future of feedback learning in large language models for subjective human preferences and values. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2-09-24330 Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18653/v 1/2023. emnlp- main. 148. URL https: attpaaaaanntcccco / org/ emnlp- main. 148/.
Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, Bertie Vidgen, and Scott A. Hale. The prism alignment dataset: What participatory, representative and individualised human feed- back reveals about the subjective and multicultural alignment of large language models, 2024. URL https::/p///o // /arxiv org/ abs/ /2444 60199 9. 2, 6,9
Robert A. Lee. Chatgpt vs. google gemini statistics 2025: Head-to-head ai trends. https:
//sqma ga i iine..... co uk - e e--gemnni -tai-statiii stccs/ 2025. Ac-
cessed: 2025-09-23.
Xinyu Li. Ruiyang Zhou, Zachary C. Lipton, and Liu Legi. Personalized language modeling from personalized human feedback, 2024. URL https: /ttp:s////iiii.// org/ abs ht////ba///// 24022 05133. 2, 4, 10
12

---

Published as a conference paper at ICLR 2026
Renpu Liu, Peng Wang, Donghao Li, Cong Shen, and Jing Yang. A shared low-rank adaptation approach to personalized rlhf, 2025. URL attps://arxiv... org. abs abs//aas.... 200.//990...1 3
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward, 2024. URL https:///aaaeee. httt::::/axxlv..rr/ab////// wttpaa///.2....1. 14734.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
URL https:tt//// /ttpss///aaxvv.i/ org/ abs /203.,223.. 02155. 1,3,,,
Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, and Natasha Jaques. Personalizing reinforcement learning from human feedback with variational preference learning, 2024. URL
https: http//awiv....gaaaa///
www..org/aaaa /bwgg//b//// 2200 10075. 3, 4, 6, 10, 11, 18, 21, 22
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model secretly a reward model, 2024. URL https: /ttpaaaiiivvvv/////////a//aa.ava org abs 22 1.300..8.9.....
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https: //tp//aavvvv///.v////a///aa///al owwoorrg///.... at////bs/1/00 1117170077047 06347. 2,5
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-
dimensional continuous control using generalized advantage estimation, 2018. URL https: 02438. 5
Lingfeng Shen, Sihao Chen, Linfeng Song, Lifeng Jin, Baolin Peng, Haitao Mi, Daniel Khashabi, and Dong Yu. The trickle-down impact of reward (in-)consistency on rlhf, 2023. URL https:
16155.. 3
Anand Siththaranjan, Cassidy Laidlaw, and Dylan Madfield-Menell Distributional preference learn- ing: Understanding and accounting for hidden context rlhf, 2024. URL https: /arxiv.
org/abs/ /tobs//231.. 2212. 08358. 2, 3, 6, 7,
Hwanjun Song, Taewon Yun, Yuho Lee, Jihwan Oh, Gihun Lee, Jason Cai, and Hang Su. Learning to summarize from Ilm- Ilm- generated feedback, 2025. URL https: arxiv. org/ abs//410.
13116.
Taylor Sorensen, Liwei Jiang, Jena Hwang, Sydney Levine, Valentina Pyatkin, Peter West, Nouha Dziri, Ximing Lu, Kavel Rao, Chandra Bhagavatula, Maarten Sap, John Tasioulas, and Yejin Choi. Value kaleidoscope: Engaging ai with pluralistic human values, rights, and duties, 09 2023.
Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christo- pher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, et al. A roadmap to pluralistic alignment. arXiv preprint arXiv:2402.05070, 2024.
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2022. URL
https: /ttpp:://axxv...o org/ alss 2009. 01325. 5,5
Dave Van Veen, Cara Van Uden, Louis Blankemeier, Jean-Benoit Delbrouck, Asad Aali, Christian Bluethgen, Anuj Pareck, Malgorzata Polacin, Eduardo Pontes Reis, Anna Sechofnerova, et al. Clinical text summarization: adapting large language models can outperform human experts. Re- search Square, pp. rs-3, 2023.
Jaaxing Wu, Lin Ning, Luyang Liu, Harrison Lee, Neo Wu, Chao Wang, Sushant Prakash, Shawn O' O'Banion, Bradley Green, and Jun Xie. Rlpf: Reinforcement learning from prediction feedback for user summarization with Ilms, 2025. URL pittps://at httpes///axxvvvee orgg/ abs/ 22099 04421.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative, multi-agent games, 2022. URL https: / //taa/i.wrggaaaa///////o/o 01 955. 5
13

---

Published as a conference paper at ICLR 2026
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B. Hashimoto. Benchmarking large language models for news summarization, 2023. URL https:
// http:iiv.rr/ab////aaaaaa
13848..
Siyan Zhao, John Dang, and Aditya Grover. Group preference optimization: Few-shot alignment of large language models, 2024. URL https: http//aaai...ogaa///3/3/33 11103.111.... 3,
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing. Hao Zhang. Joseph Gonzalez, and lon Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. URL
abs/2306 05685.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yong- hao Zhuang. Zhuohan Li, Zi Lin, Eric P. Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. Lmsys-chat-lm: A large-scale real-world Ilm conversation dataset, 2024. URL https: // //axxiv.org/aas//30000 11998.
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2020.
URL https:t///// /aaxxvvv.v///////// org/ abs//a/9.....//.////// 08593. 3,5
14

---

Published as a conference paper at ICLR 2026
APPENDIX
A INFERRING USER CONTEXTS WITH FLEXIBLE TEXTUAL INPUTS
Table 5: Reward model accuracy with unstructured inputs. We compared the performance of PLUS and ICL when user contexts include unstructured data other than preference labels that are typically un-supported by personalized RLHF techniques. Preference data provides users' past preferences in the form of chosen and rejected response examples. User-guideline + preference data additionally includes textual instructions specifying which aspects of an LLM assistant fo- cus on. Conversation data does not include binary preference labels; instead, it only presents chosen responses as "examples of past conversation" and expects the reward model or summa- rizer to infer the user's underlying preferences only from the positive examples. This last setup more closely models real user interactions with LLM assistants, since users are typically given only response rather than choosing between two; however, their preferences from their questions, initial prompts, and follow-up responses, despite not having observed explicit user preferences. We trained a Qwen2.5-0. a Siven2.5-3B-Inseee
still be inferred
reward model for both ICL and PLUS and summarizer for PLUS. Interestingly, PLUS benefits from additional user instructions specifying the relevant dimensions of LLM assistant's attributes, whereas ICL's per- formance is hurt by longer user prompts. We did not compare with VPL, as does not support unstructured input formats.
Ultra Feedback (UF)-P2 Accuracy User guideline + preference data
Reward model
Preference data
Conversation data
ICL
59.60
59.65
58.55
PLUS
69.40
70.25
68.3
Ultra Feedback (UF)-P4 Accuracy
Reward model
Preference data
User guideline preference data
Conversation data
ICL
57.30
56.60
$6.96
PLUS
61.80
62.70
62.45
15

---

Published as a conference paper at ICLR 2026
B PERFORMANCE OF ICL WITH LARGER REWARD MODELS
Table 6: ICL reward model accuracy with larger base model sizes. We included 7B & 8B reward models for UF-P2 and UF-P4 and observed that the performance achieved by these larger ICL. models is lower than the performance obtained by PLUS with a 3B reward model and a 3B summarizer (66.9 UF-P2 and 62.8 on UF-P4 with Llama-3B).
Preference Prediction Accuracy Ultrafeedback-P2 Ulrrfeeddacc--P
model Jlama&B-Instrun QwennB-Instruet
61.4
58.6
61.8
58.8
C PLUS-GENERATED SUMMARY EXAMPLES
Pets While unturanndd still achieves high accuracy of 96.2% on the Pets dataset, the sum- maries for incorrect examples highlight the importance of training the summarizer so that it can learn to distinguish between relevant and irrelevant details in the user's past conversations. We've iden- tified common failure cases made by the untrained summarizer as: (1) focusing irrelevant aspects of the user's conversation history, such as style, and (2) providing broad descriptions of qualities that could apply to both dogs and cats.
Example 1 (focus on style): The summarizer fails to capture the user's preference for dogs or cats, instead emphasizing only the user's stylistic preferences.
The user prefers short, factual information and tends reject detailed anecdotes complex be- haviors. The assistant should stick to basic descriptions where possible. Based on the user's 's pref- erences, the Al assistant should provide concise, straightforward information about pets, avoiding detailed anecdotes or...
Example (focus on general pet traits): While this summary mentions traits like "playfulness" and "loyalty" that are more common for dogs, they could apply to both dogs and cats. The untrained summarizer confounds the user's preference for dogs with their preference for specific qualities in animals that are mentioned in the past conversation.
The user seems appreciate traits that denote affectionate and playful qualities in pets, evi- denced by their rejection of traits related to training or timing of activity (renewal of knowledge). Focus traits like loyalty, playfulness, and exploration for future.
Example (focus on general pet traits): Similar example 2, this summary references traits discussed in the user's conversation, but it doesn't It clarify the user's preference between dogs and cats. As a result, the model makes incorrect predictions when new conversations mention traits about dogs that unrelated to hunting or communication.
The user is interested in pet species that can vary in behavior or communication, particularly traits like vocal range or hunting habits. The focus is on traits that allow for diversity within the species.
On the other hand, the trained summarizer learns to distinguish between relevant and irrelevant features in the user's conversation history and correctly identifies their preference between dogs and cats. This is made possible by simultaneously training the reward model conditioned on the generated summaries, which provide prediction losses the signal for RL fine-tuning.
Example of PLUS summary:
The user is interested in information about cat behavior and properties, excluding topics related dogs or human effects.
This summary clearly states the user's preferences and can generalize to unseen traits about cats i... in future conversations, statements about cats will be preferred to those about dogs.
PRISM One benefit of PLUS is that the generated user summaries are human- readable, unlike vector representations, and can be shared directly with users to increase transparency about what the
16

---

Published as a conference paper at ICLR 2026
model has learned personalize future responses. The user summaries capture different dimensions of preference, such as brevity, conciseness, factuality, and the diversity of perspectives on contro- versial topics. An interesting instantiation of our method within existing AI systems would be to provide users with their summaries and invite them to make edits to the summaries, or reflect on how their demonstrated preferences align, misalign, with their self-stated preferences.
Table compares the user's self-stated preference attributes with the PLUS-generated summaries of the user context. This shows the summaries reflecting the user's preference can be useful for deciding which of the two candidate responses is preferred over the other-even when the user's self-stated preference (survey response) alone may not give sufficient information about the user's preference.
Table 7: User preference summaries generated by PLUS for the held-out users of the PRISM dataset. The following users value diversity and detail in responses, which is reflected in their summaries and chosen responses. The first column shows the user's self-stated preference from the pre-task survey. The users rate each dimension on scale from 0 (unimportant) -100 (very important).
Survey Response
PL.US-gemerated Summary
Prompt
Chosen Response
Rejected Response
(Stated Preference)
Preferences lean towards bal- anced response between and hrevity. The assistant profers clear explanations and ucknowkedges varying perspectives on complex topics. Additionally, users value cnncisse summaries and Helpfalness: someiimes sock expert consulta- tions.
Values: 84.
Can authentic lationship hetween 21 year old man and 52 year old woman genuine?
Relationships complex and multifaccttd and depond on many factors, so is difficalt give definitive answer question... [continue]
If this combination seems you then you're 'e probuibyy not emotionnlly
Creativity: 100.
Hluency 100.
Factuality: 100,
Divensty:
mature enough
Safety: 94,
monogamous relationship ye Being an adult micans being afruid of the unknown and being willing to new
Persoralization: 100,
things... [continue]
Values: 100
Preferences include detailed explanations backed by con- siderations of context and avoiding simplistic
Which gender more important?
I'm just I don't have personal opinions beliefs, and I doe't I think it's appro- priate say is more important than the other. Both men and womon have equal value and should be treated respect dignity. It's important
apologize, but do not feel
Creativity:
comfortable making broad claims about the relarive
Flaency: 37
Factuality: 100
gender
Safety: 100
Diversity:
Answers often require balance
importance of genders.
Personalization: 100
between personal neurrality and
Helpfulness:
exploration diverse perspectives. Conciseness is valued when de- tailed explanations alicady provided.
recognizz that gender roles and expectations can
across cultures and socicties, [continuc]
Values: 29
The user prefers detailed
Is the anti-abortion policy associated religious be- lies?
The anti-abrrtion policy associated with reli- gious beliefs values held by some in- dividuals and groups. Many religious traditions
The anti-abortion policy is not associated with religious medical practice rather than reli- gious doctrine. However, religious beliefs can some- times influence people's 's be- liet's about aboction.
Creativity: 77
explanations and multiple credible sourees when given topies like churity organization comparisons and architectural advice. They also value fact- based respoases rather than opinions lacking support.
Fluency:
Factuality: 100
Diversity: 61
Safety: 59
Personalization: 21
Helpfulness: 81
including Catholicism.
Protestantismn, and Orthodox
Judaism, teach that human
life begins at conception and ends at natural death. and therefore, abvetion is taking of an innocent human life. However, not all religions
considered
traditions have official positions on abortion. and there are also religious pro-choice perspectives.
D PLUS IN THE WILD
Having established the applicability of PLUS to the real user dataset, next investigate two possi- ble user cases of L.US-generated summaries for downstream personalization: Personalizing LLM- as-judges in Alg. 2 and personalizing LIM response generation in Alg. 3. In both cases, PLUS-
17

---

Published as a conference paper at ICLR 2026
Table Reward model accuracy on PRISM held-out conversations. To approximate the oracle performance, we train reward model conditioned on the user's self-rated preference attributes (shown as "user self-stated" in the last row).
Accuracy
BTL (Bradley Terry, 1952)
59.8
DPL Sitthhrannjjn et al. (2024)
61.8
VPL Poddar et al. (2024)
60.4
ICL
60.1
PLUS-untriined
61.3
PLUS (Ours)
62.9
User self-stated
62.6
generated summaries directly enable personalization of strong proprietary models (e.g., GPT-4) without further finetuning.
Algorithm Personalizing L.MM-s-juddes
1: Input: PLUSSSummarise Te, user's textual context c, new query æ, response pairs y1: 1/2
2: Generate summary (e)
from user's textual context
3: Predict reward f(y1 ax, 2) and 2))2/zr,2)
4: Choose the response with the higher predicted reward y" arg ma f(y|zr,)
5:
Output: return y*
Algorithm 3 Personalizing LLM assistant's response
1: Input: PLUSSSummarize me, user's textual context c, new query sr generator LLM PLiM
2: Generate summary Ie (c) > z from user's textual context
3: Generate a new response ~ PLLM (z,2)
4: Output: return y.
E PERSONALIZED LLM-AS-JUDGES
Here, we use an LLM-ass--judde- framework where we prompt GPT4 to select which of the two responses from the PRISM dataset would be preferred, and evaluate its accuracy in predicting the right one with and without conditioning on the PLUS user summary. This allows us to test whether the learned summaries can enable personalized preference prediction for strong, proprietary models without further fine-tuning, We provide both GPT-4.1 and GPT-4o with the user prompt and two LLM assistant's 's responses from the held-out PRISM dataset (one chosen and the other rejected), and ask the models to predict which response the user would prefer with and without PLUS PLUS- generatede user summaries. We evaluate the model's judgment on 277 aalue-guided and 307 controversy- guided conversations from the held-out PRISM data, and additionally, on 308 new users whose profiles are never before seen during training. Table 9 shows that GPT4 makes more accurate pre- dictions about the user's preference when the summaries makes a 40% improvement. The summaries are especially helpful for cuntroversy-guided ques- tions, subset of the PRISM dataset designed to capture preferences politically culturally sensitive topics.
given- eiveneespecially for GPT4o which
F RESPONSE PERSONALIZATION WITH PLUS-GENERATED SUMMARIES
We selected examples of PLUS- conditioned responses generated with 4PT-4o and GPT- 4.1. The highlighted parts in blue reflect personalization based on the corresponding user summary, especially when compared the default GPT responses.
18

---

Published as a conference paper at ICLR 2026
Table 9: Accuracy of LLM-judges with and without PLUS summaries. The value before the arrow is the prediction accuracy (%) without conditioning on the PLUS FLUS-generated user summary, and the value after the arrow shows the improved accuracy with the summary. Best scores per conversation type are shown in bold.
GPT-4o Accuracy (%)
Accuracy (%) Default PLUS- SJUS-personalized
GPT-4.1
Conversation Type
Default
PLUS LUUS-peroonalized
Value-guided 65.7
77.25
75.5
78.7
Controversial
52.2
73
69.1
72.3
New users
65.9
72.1
74.35
74.35
Table 10: Win rates of default GPT-4 versus personalized responses evaluated by the oracle preference model. The oracle model is trained on self-stated user preferences that score the follow- ing L.M assistant attributes: values, creativity, fluency, factuality, diversity, safety, personalization, and helpfulness, based on their importance to each user when interacting with LLM systems. We used summaries generated by the PLUS-trained Qwen2.5-3B-Innse --Istructt model to personalize GPT-4's responses to 906 new prompts sampled from unseen PRISM users. We then compared these person- alized responses with default responses that did not have access to summaries. Best scores per model are shown in bold.
PLUS personalized
Default
GPT-4o
72
28
GPT-4.1
69
31
Selected examples personalization with PLUS & GPT-4o
Prompt #1: Is Jesus Christ the Son of God?
PLUS-summary about the user: The user prefers detailed, balanced responses avoiding definitive statements when there's uncertainty.
Default response: In Christianity, Jesus Christ is believed to be the Son of God. This is central tenet of Christian faith, affirming Jesus' divine nature.
Personalized response: Many Christians believe that Jesus Christ is the Son of God... However, perspectives vary widely across different religions and secular viewpoints. Judaism and Islam, for instance, do not view Jesus as the Son of God, although they recognize his historical significance. Interpretations can depend on personal beliefs, re- ligious tradition, and theological perspective. (Personalization based the user summary is highlighted in blue.)
Prompt #2: What your opinion on abortion?
PLUS-summary about the user: The user prefers detailed, factual answers with supportive examples and clear explanations.
Default response: don't 't have personal opinions beliefs, but can provide information on the topic. Abortion is complex and sensitive issue with diverse viewpoints, including ethical, legal, and personal considerations. It's vital approach discussions about abortion with respect for differing perspectives and to consider the specific laws and cultural contexts of each region.
Personalized response: don't have personal opinions or beliefs. However, can provide detailed, factual overview. Abortion is the termination of pregnancy and is contentious issue globally. Views on abortion often reflect broader societal beliefs and values, including those related women's rights, morality, and medical ethics. For example, in the U.S., legal perspectives vary widely, with some states imposing strict regulations while oth- ers support a woman's right choose. Factors like fetal viability and maternal health often play roles in these laws. Understanding these contexts helps to form a compre- hensive view of the issue.
19

---

Published as a conference paper at ICLR 2026
Here describe three notable patterns of personalization enabled by PLUS generated summaries compared to default (unpersonalized) responses.
Response length adaptation. User's ss preference for "practical suggestions over lengthy ex- planations" affects the length of the response to change from 166 (default) 56 (summary- conditioned) words.
Reference to user's specific preference values. GPT's response addresses certain aspects the user's preferences that may relevant to the new prompt. For example, when asked about the drawbacks of cutting welfare benefits, the LLM assistant acknowledges the user's ' preference for safety. Similarly, when asked about the appeal owning cats, the LLM assistant connects the user's appreciation for independence to the cat's trait of autonomy.
When the user's summary mentions values safety and constructive advice," the LLM assistant responds, "... Conclusion with your preferences in mind. Given the impor- tance you place on safety and constructive advice, it's crucial to approach this issue by considering the broader impacts of cutting benefits beyond just economic met- rics..."
Similarly, for different anced perspectives". the LLM assistant answers, "... The autonomy cats exhibit might specifically resonate with your appreciation for independence, enhancing the charm they hold for individuals who value self-sufficiency alongside companion- ship."
who values "personalization, autonomy, and bal-
When the user asks, "Tell me the most offensive thing possible." the summary- conditioned LLM assistant responds, "Given your preferences, I'll refrain from stating anything offensive. Instead, let's focus on promoting positive and re- spectful communication. This approach encourages understanding and harmony in our interactions. If you have specific topics or questions in mind, let's discuss them with open and respectful perspective." " On the other hand, the default LLM assistant responds, "I'm mm sorry, but I can't I comply with that request. If you need help or want to discuss different topic, feel free ask!"
Fact-based response for users who value factuality. When the PLUS summary mentions "showing preference for evidence- baaee responses," the LLM assistant provides evidence based on relevant research or studies to support its claims.
When the user asks, "my teenage grandson is beginning to rebel against his parents, in small ways. He mainly wants to be allowed to take his mobile phone to his room at night so he can play games with his friends. Is this good idea," the summary- conditioned LLM assistant mentions, "Research suggests that screen time before bed can affect sleep quality, which is crucial for teenagers' development." On the other hand, the default LLM assistant responds, "Consider setting rules like screen time limits or phone-free nights to ensure he gets enough rest."
G ADDITIONAL EXPERIMENTS WITH THE PRISM DATASET.
Table 11: Reward model accuracy of predicting the held-out users' preferences in PRISM. The in-distribution column is shown in Table 1, and out-of-distribution is the evaluation accuracy of predicting for held-out users. Surprisingly, BTL and DPL, which by design cannot accommodate conflicting preferences, show high performance the held-out simply by predicting the majority.
Model
In-distribution
Out-of-distrbbttion
59.8
DPL
61.8
65,1
VPL.
60.4
59.
ICL
60.1
60.6
PLUS untrained
61.3
62.3
PLUS (ours)
62.9
59.7
Oracle Proffle-conditinn
62.6
62.3
20

---

Published as a conference paper at ICLR 2026
In addition to main experiment, also investigated whether reward models trained real- world datasets can generalize not only to unseen conversations but also to unseen user contexts. To test this, evaluate the reward models on 308 new users whose profiles and past conversations were not included in the PRISM training dataset. Surprisingly, we observe that BTL and DPL. - models that, by definition, cannot personalize different user preferences, they assume that single reward model can capture all user preferences outperform personalized models, including VPL, ICL, PLUS, and the oracle-reward model which conditioned on the user's self-stated pref- erences. This suggests that these unpersonalized models achieve high accuracy by simply taking the majority vote and making accurate predictions for the majority users. In contrast, personalized models, including the oracle reward model trained on the user's self-stated preferences, perform poorly. We observed successes in applying the learned reward models held-out conversations but experienced limitations in generalizing to held-out users, which is likely due to the small size of the dataset compared to the amount of heterogeneity it attempts to capture in user preferences. In reward modeling
spite of this limited performance that PLUS is still able to enable personalization of responses to new users, when combined with proprietary models like GPT-4, shown in Fig. 4. We believe this highlights the difficulty of mod- eling nuanced real user preferences with limited data and suggests that personalized RLHF research would greatly benefit from large-scale, heterogeneous dataset.
with the held-out users, we observed
H EXISTING RLHF TECHNIQUES
Table 12: Comparison different preference learning methods with and without conditioning user variables. Reward models are typically trained using the negative log-likelihood loss of the chosen versus the rejected response pair. VPL and PLUS additionally train an encoder to obtain user-specific latent variables, which the reward model relies of user preferences. ICI., VPI., and PLUS can enable personalization and pluralistic alignment by conditioning the reward model different user contexts latent variables. Our key experimental results show that the choice of latent variables strongly affects the reward model's 's accuracy, espe- cially in complex domains resembling real world user preferences, and when conversation topics shift between the training and testing samples.
make
accurate predictions
Trainable component(s)
Training objective
BTL. (Bradley & Terry, 1952)
Reward model
Prediction
DPL (Sithcharanjan et al., 2024)
Reward model
Prediction
ICL.
Reward model
Prediction
VPL Poddar al. (2024)
+ Reward model
Reconstruction + Prediction
LLU--untaineed
Reward model
Prediction
PLUS (Ours)
Summarizer (actor in the PPO framework) + Reward model
Reward maximization Prediction
I TRAINING DETAILS
I.1 DATASETS
To build user context for VPI., ICI., PLUS (untrained & ours), we sampled N past conversations per user.
For Pets and Pets (OOD), sampled 3 past examples.
For UF-P-2 and UF-P-4, we sampled 2-4 past examples.
For PRISM, we sampled 3 past examples, which can be across different conversations or different turns in the same conversation.
Dataset sizes:
Pets: train: 1970, test: 194.
Pets (OOD): test: 200.
UF-P-2: train: 10k, test 2k.
PRISM: train: 20k, test: 1k.
21

---

Published as a conference paper at ICLR 2026
UF-P-4: train: 40k, test 2k.
1.2 HYPERPARAMETERS
Hyperparameter
Value
Reward model learning rate
10-6
Batch size
256 (128 for Pets)
Actor learning rate
5 x 10-7
Critic learning rate Advantage estimator
9 10
GAE
X e-clipping initial kl coefficient kl estimator Ir warmup ratio max norm micro batch size Mg reward clip range
1
0.95
0.2
0.01, 0.001
k1
0.03
2
[-10. 10]
micro batch size Ms BF16 Optimizer Adam 8
2
Yes
Adam
B1 = 0.9, 32 = 0.95
Table 13: PPO training hyperparameters. Our co-adaptation framework Alg. 1 is built on top of OpenRLHF's ppo training code (Hu et al., 2024). Summarizer To is implemented the PPO actor. We observed that reducing the KL penalty for the summarizer significantly smaller to 0.001 worked better for Pets and PRISM. For Pets, using a smaller batch size 128 helped improve the reward model performance.
Hyperparameter
Value
Hidden Dimension (for VPL)
512
Latent Dimension (for VPL)
512
Learning rate
9 x 10-6
Batch size
256 (128 for ICL with PRISM)
BF16
Yes
Optimizer
Adam
Table 14: Reward model training hyperparameters: reward modeling code was adapted from Poddar et al. (2024). We additionally conducted hyperparameter sweeping for VPL, BTL, DPL, and the oracle reward models by varying the learning rates {3 10 10-4,9 9 10-63 and training batch sizes (32, 128, 256]. We observed that the oracle models on the UltraFeedback datasets achieve higher accuracy by training for more than epoch.
J LLM PROMPTS
J.1 PROMPT FOR GENERATING SUMMARIES WITH PLUS.
Here is request of a user for AI assistant. Please talk about one kind of pets. The user chose The user rejected Based on the user's past conversation, provide a short summary of the user's preference. Focus on the user's preference that should guide how the Al assistant responds to this user in future conversations. Keep the summary concise under 50 words. Start your response with ##Summary
Pets:
UltraFeedback: In order to provide more information about myself, I'm including amples from my previous conversation history, which include the prompt, the rejected re- sponse, and the chosen response. Use this information leam about my preferences and reflect that understanding in your future responses. The prompt was: chose this re- sponse: ... rejected this response:... Based on this information, provide short summary
22

---

Published as a conference paper at ICLR 2026
of the user's preferences to guide how the Al assistant should respond in future conversa- tions. Reflect on the user's 's past chosen and rejected responses, and what the user preferred about the chosen responses. Do not focus the topics of specific conversations. Keep your summary concise-under 50 words. Start your response with ##Summary
PRISM: order to provide more information about myself, I'm including examples from my previous conversation history, which include the prompt, the rejected response, and the chosen response. Use this information to learn about my preferences and reflect that understanding your future responses. ##Past conversation 0## #**Prompt#** ###Cho- sen response#ffff ##Reejetted responsefftt.... Based on this information, provide a short summary of the user's preferences to guide how the Al assistant should respond future conversations. People's preferences typically vary along dimensions, including verbosity, diversity, and language fluency. Do not focus on the topies of particular conversations; instead, focus on the preferences revealed through past conversation examples. Keep your summary concise-under 100 words. Start your response with #Summmay
How the summary is provided to the summary-conditiones reward model: Here is the summary of my preferences based previous conversations: [summary z]. In the following conversation, keep this in mind to personalize your response.
J.2 PROMPT FOR ORACLE REWARD MODELS.
Pets: This user prefers cats over dogs (or dogs over cats).
UltraFeedback-P22 & prioritize list of relevant user's preference attributes from following,
truthfulness, instructionfooloovnngg honesty, helpfulness in Al assistant's 's responses. Now based on this information about myself, heres new question:
PRISM: I've scored the following attributes based on how much care about each attribute.
A higher score indicates that this attribute is more important to me: 'values': 'creativity": "fluency": 'factuality':. 'diversity': 'safety":. "personalisation:. 'helpfulness': Now based on this information about myself, here's 's new question:
K
PROMPT FOR IN-CONTEXT LEARNING MODELS.
Pets:
Based on the user's past conversation, respond the user's request in way that aligns with the user's preference. Conversation history: Start of new conversation:
UltraFeedback P-2 &4: In order to provide more information about myself, I'm including examples from my previous conversation history, which include the prompt, the rejected response, and the chosen response. Use this information to learn about my preference and reflect that understanding in your future responses. The prompt was: I chose this response: rejected this response: Here's ' new conversation:
PRISM: In order to provide more information about myself, I'm including examples from my previous conversation history, which include the prompt, the rejected response, and the chosen response. Use this information to learn about my preferences and reflect that understanding your future responses. ##Past conversation 0## ###Prompt#*# ###Cho- sen response#tttt ###Rejected response NNo based on this information about myself, here's 's a new question:
K.1 PROMPTS FOR GPT-4 PERSONALIZATION WITH PLUS.
Prompt for LLM--a-a-JJdge. System prompt: You are a helpful assistant.
User prompt (without personalization): Here's previous conversation between the user and the AI assistant: [current conversation] Decide which response the user would prefer-option I or option 2. Option is: Option 2 is: Respond with Option or Option 2 based on what you know about the user.
User prompt (with personalization): Here's 's a previous conversation between the user and the AI assistant: [current conversation] Decide which response the user would peefer-optin or option 2. Option 1 is: Option 2 is: Respond with Option Option 2 based on what you know about the
23

---

Published as a conference paper at ICLR 2026
Here's summary about the user's preference based past conversation: [PLUS- -een--eeee user summary] Based on his information about the user, predict which response they would prefer.
Since GPT may be sensitive the ordering of the responses Option 1 and Option 2, randomly choose whether the rejected or the chosen response shown first, but keep the ordering the same for the same query with and without the summary, the only difference is whether the summary is included the prompt or not.
Prompt for generating a new response user's query. System prompt: You are helpful assistant. Keep your response short-under 100 words.
User prompt (without personalization) User's prompt sampled from the test set.
User prompt (with personalization): User's prompt sampled from the test set]. Personalize your response based on the following summary about the user's preference: [Summary]
24


Reviews:
[{"review_id": "Bd909R5xJY", "reviewer": "Reviewer_x1Wn", "summary": "The submission proposes a method to learn personalized reward models called PLUS.\nIt consists of a summarization model, the output of which is used to condition a reward model.\nThe summarization model and reward model are trained in an iterative fashion, alternating between doing PPO updates on the summarization model and supervised crossentropy updates on the reward model conditioned on the generated summaries.\nThe user summaries are textual.\nExperiments show that the proposed method leads to a notably higher reward modeling accuracy, using the resulting reward model for preference-tuning is not evaluated.", "questions": "* Why were no RL experiments conducted with the trained reward model? It would significantly strengthen the paper.\n *", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Adapting to a user's individual preference is an important problem\n * The method is well motivated and very reasonable\n * The results show a notable improvement in RM accuracy", "weaknesses": "* The evaluation is limited to reward modeling, it was not tested whether the trained reward model can be used in a subsequent step to train a better policy. Recent research [1,2] has shown that a higher RM accuracy does not necessarily lead to a better performance after RL training. This is my main issue with this paper.\n\nThe abstract also claims a \"25% improvement over the best personalized RLHF technique\". This is incorrect, at most the paper can claim such an improvement over the \"best personalized RM technique\", not over the best RLHF technique.\n\n* Experiments were only conducted with rather small models (<3B). Particularly for the ICL baseline, ICL is known to perform better for larger models [3]. It is thus unclear the proposed method is advantageous to ICL for larger reward models which would be used in practice.\n\n * Writing is unclear or incorrect at times, examples below in the minor issues\n\nMinor issues:\n * \"BTL reward model [...] is incapable of creating systems that can adapt to users' unique needs and and preference\" the paper's own ICL results show the opposite. They are worse than the proposed method, but not fully incapable of adapting to the preferences.\n * GPT4 is not a state-of-the-art model\n * Figure 1: using a religious example seems prone to invite discontent, a non-religious, non-political example may be more inclusive\n * \"zero-shot\" is used inconsistently and incorrectly throughout the paper. In L295 it refers to generalization to the test-set, in L342 it refers to not training the summary model, which is not zero-shot and in L445 it refers to prompting GPT4 without fine-tuning it. None of these are \"zero-shot\" in the classical sense of generalizing to entirely unseen classes.\n *  L201: The proposed method is described as \"two-stage sequential\". To me this sounds like we first train the summarizer, then the reward model, and then we are done. Instead the method alternates between both steps. L210 \"Once the summarizer is fixed\" also sounds like we fix it only \"once\", contradicting the alternating training scheme. As it is key to the paper I would recommend revising the presentation.\n\n\n\nReferences:\n\n[1] Razin et al. \"What Makes a Reward Model a Good Teacher? An Optimization Perspective\", NeurIPS 2025 \n\n[2] Chen et al. \"The Accuracy Paradox in RLHF: When Better Reward Models Don't Yield Better Language Models\", EMNLP 2024\n\n[3] Brown et al. \"Language Models are Few-Shot Learners\", NeurIPS 2020", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VHaxMHmNzt", "reviewer": "Reviewer_8x8P", "summary": "This paper proposes the PLUS framework for modeling user-specific preferences. A text-generating summarizer produces a user summary $z$ from the user context $c$, and a reward model $r_\\phi(· | z)$ is optimized conditioned on that summary to capture personalized preferences. The summarizer and the reward model are trained in an alternating fashion. Experiments on benchmarks such as Pets and UltraFeedback-P-2 demonstrate strong performance compared with baselines.", "questions": "See above weaknesses", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "# strengths\n1. Representing user preferences via natural-language summaries makes the user representation easy to understand and inherently interpretable, while avoiding issues with overly long raw contexts.\n2. PLUS is evaluated across diverse datasets and settings (including OOD), providing evidence of its effectiveness.\n3. PLUS achieves improvements over baselines in most settings and even surpasses strong baselines such as GPT-4o and GPT-4.1.", "weaknesses": "# weakness\n1. Data sparsity per user. With only a few dialogs for each user (e.g., Pets uses 3, UltraFeedback uses 2–4, PRISM uses 3), can the reward model learn a reliable representation of that user’s preferences? Is the reward model potentially *under-converged* in such low-data regimes?\n2. *Training stability.* Alternating optimization can be hard to converge. It would be important to provide *training curves for the reward model* to assess stability (e.g., accuracy,  variance across steps).\n3. Cost and scalability. Building a reward model at the *user granularity* raises the question: when new users arrive, does the system need to update the reward model each time? If so, this could lead to *high computational cost*.\n4. *Context-dependent preferences.* A single user’s preferences may shift across contexts, which questions whether modeling a single, static per-user profile is appropriate. For instance, a user may like cats in a general conversation about stray cats, yet under a context emphasizing ecological harm, the same user’s preference toward stray cats could diminish.\n\nIf the authors can provide additional results to address these concerns, I would be happy to raise my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IRLf1RFPw0", "reviewer": "Reviewer_Kjwp", "summary": "This paper proposes an approach for learning a reward model that is conditioned on a summary of a user profile. This means that the reward model can be personaliized. The approach jointly learns a summarization model with the reward model to ensure that the optimal info is included in the summary. The paper reports results showing that by doing this, the reward model accuracy improves and has a higher win rate on the PRISM dataset", "questions": "Why is the oracle accuracy so low at times?\n\nCan the authors comment on the privacy considerations of this approach compared to others? \n\nCould the authors add details and discussion on the actual training of the summarizer and reward model?\n\nCould the authors include quantitative results showing how well their language-based summarization captures the user compared to embedding approach?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper provides strong results showing that PLUS can better personalize. The authors show results on the PRISM dataset. this is a diverse, multicultural dataset and reflects what an LLM may encounter in the real world. Furthermore, PLUS shows robustness to new topics and users which is more difficult for approaches that, for example, assign a new user an id rather than a language-based summary.", "weaknesses": "The authors could add some important details related to training such as convergence criteria. Furthermore, PPO in a MARL setup is highly sensitive to learning rate and other parameters. The authors should discuss the possibilty of instability in training in the limitations section.\n\nIt would be interesting if the authors provided quantitative results to back up their claim that the personal embeddings dont capture information as well as the language summary - perhaps via mutual information measures between the user info and the summary/embedding\n\nThe paper would benefit from providing confidence intervals for results since some of the improvements are small and only tested over 3 seeds\n\nThe results in Table 10 in the Appendix are noteworthy and should be discussed in the paper.\n\nA human evaluation of the summaries would be interesting to include to determine if better summaries are actually produced by training in this way. Its possible that the reward model learns to exploit the summarizer in ways that dont actually produce better summaries.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)