Language Model Personalization via Reward Factorization
Authors: Idan Shenfeld
Felix Faltings
Pulkit Agrawal
Aldo Pacchiano
Venue: COLM 2025 (2025)
Keywords: RLHF
Personalization
TL;DR: We present an extension of the RLHF framework for LLM personalization

Abstract: Modern large language models (LLMs) are optimized for human-aligned responses using Reinforcement Learning from Human Feedback (RLHF). However, existing RLHF approaches assume a universal preference model and fail to account for individual user preferences, limiting their effectiveness in personalized applications. We introduce a framework that extends RLHF to enable user personalization by leveraging the assumption that user preferences lie in a low-dimensional space. Instead of training a separate model per user, we represent user-specific rewards as a linear combination of base reward functions. Using only 10 user responses, our method can infer user-specific rewards and align LLM outputs accordingly. We validate our approach through experiments with both synthetic and real users, demonstrating significant personalization achieved by our method. In human evaluations, our method achieves a 67% win rate over default GPT-4o responses.

Published as a conference paper at COLM 2025
Language Model Personalization via Reward Factorization
Idan Shenfeld*, Felix Faltings' & Pulkit Agrawal Massachusetts Institute of Technology (idanshen, fal @mi e.
Aldo Pacchiano Boston University pacchian@bu. edu
Abstract
Modern large language models (LLMs) are optimized for human-aligned responses using Reinforcement Learning from Human Feedback (RLHF). However, existing RLHF approaches assume a universal preference model and fail to account for individual user preferences, limiting their effective- ness in personalized applications. We introduce a framework that extends RLHF enable user personalization by leveraging the assumption that user preferences lie in a -iw-dimeniional space. Instead of training a separate model per user, we represent user-specific rewards as linear combina- tion of base reward functions. Using only 10 user responses, our method can infer user-specific rewards and align LLM outputs accordingly. We validate our approach through experiments with both synthetic and real users, demonstrating significant personalization achieved by our method. In human evaluations, our method achieves 67% win rate over default GPT-4o responses.
1 Introduction
A major driver of the success of modern large language models (LLMs) is their ability to generate responses aligned with human preferences. This alignment is typically achieved through Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022), a technique that optimizes a reward function based on user preference data. Current approaches to RLHF assume a universal preference model across all users and cannot cater to individual user preferences, a key limitation to personalization (Casper et al., 2023; Sorensen al., 2024).
User preferences vary widely across individuals and tasks. For example, one user might use an LLM as a professional assistant for workerelated tasks, while another might use as a virtual friend. Naively extending RLHF to cater to different user preferences, such as training separate model for each user, is often infeasible. This is mainly due to the large amount of user-specific data required (typically thousands of data points (Gao et al., 2023)) and the significant computational cost of training and maintaining user-specific LLMs.
We present framework called Personalization via Reward Factorization (PReF) that extends RLHF to support user personalization. PReF assumes that user preferences share structure (i.e., lie on low-dimensional manifold (Rentfrow et al., 2011). Under this assumption, we represent the reward function of the ith user, V;, which maps a prompt and response linear combination of "base" reward functions:
E N/p.. The coefficients A/ determine the contribution of each base function (x,y) to the 5t user's reward function. We further assume that individual user preferences follow the standard Bradley-Terry formulation that computes rewards based on the user's pairwise rankings of responses (Bradley & Terry, 1952). Given a prompt and a pair of responses, 1/2, the preference of ith user whether y1 > y/ is modeled as a Bernoulli random variable with parameter a (r)((x,))) y2)). With these assumptions, PReF reduces the problem of personalization to estimating user-specific coefficients X/, which simpler and more data-
y to scalar,
"equal contribution

---

Published as a conference paper at COLM 2025
User Preference AUC ROC
30x data efficiency
User Specific Reward
Weights
Functions
9.8
Reward of user over response
4
Σ
2 $
0.6
Separate per
PRef (Ours)
10a
10a
10
10a
Number of User Answers
Figure We factorize each user's personal reward as a linear combination of base functions. The linear structure enables us to perform personalization in an efficient manner, needing up to x30 fewer answers from the user achieve the same performance the standard RLHF approach.
efficient than learning separate reward models per user. PReF, therefore, greatly reduces the need for user-specific data and computation.
Previous work on LLM alignment developed methods combine set of pre-defined reward functions linearly but did not focus on personalization (Han et al., 2024; Guo et al., 2024; Yang et al., 2024b). In particular, these approaches do not address the core problems necessary for data and compute efficient user personalization: (1) determining the base reward functions and (2) identifying the user-specific combination of base reward functions that personalize responses. Our work addresses these questions.
In PReF, we first collect preference data in which each pair of responses the same prompt is annotated with the user's preference (i.e., which response is preferred) and the user's s identity. We use this dataset to learn the base reward functions. Once the base reward functions are determined, the next step is to infer the coefficients for each new user. To achieve this, we generate a sequence of questions and a pair of responses and ask the user to indicate which response they prefer. Based on the responses, we estimate the user coefficients and, thus, its specific reward function.
The challenge in estimating the user's 's coefficients is to do with minimum number of questions presented to the user. To do so, we adopt an active learning approach where the sequence of answers is adaptive to the user, meaning that the questions are selected based on the user's 's prior responses to efficiently refine their preference model. Specifically, we select a question and responses that minimize the uncertainty of the user's coefficients. We adapt and extend results from the logistic bandit literature to efficiently compute uncertainty scores of response pairs. Using our method, we can determine the user coefficients using only 10-20 questions.
Once the user-specific reward function has been identified, the next step is to align the LLM to it. We leverage recent advances in inference-time alignment methods (Han et al., 2024; Yang et al., 2024b; Rame al., 2024) that can generate reward-aligned responses from an LLM at deployment without modifying the weights of the LLM. This allows for efficient, scalable adaptation to individual users without requiring costly model updates.
We validate our framework with extensive experiments. On synthetic data where we use LLMs to mimic real users, our framework outperforms standard RLHF by a large margin. Our personalized reward model needs only five samples from new users to perform better than a standard reward model. Finally, we demonstrate our method's 's ability to cater to real human preferences. In our experiment, we aligned GPT4o with real users, achieving 67% win rate over the model default answers.
2 Related Work
Personalization of LLMs has become important research direction, enabling models to better serve individual users' needs (Sorensen et al., 2024; Kirk et al., 2024b; Zhang et al., 2024). Broadly, personalization can take several forms: incorporating user-specific knowledge, fine-tuning models to develop domain expertise, or adjusting response styles to align with user preferences (Ning et al., 2024; Wu et al., 2024; Richardson et al., 2023; Kirk
2

---

Published as a conference paper at COLM 2025
et al., 2024a; King & Cook, 2020). Our work focuses the last category personalization through uper-ipecific preference aligmment.
A leading approach for aligning LLMs with human preferences is Reinforcement Learning from Human Feedback (RLHF), first introduced by (Christiano et al., 2017) and further refined in later works (Ouyang al., 2022; Ziegler et al., 2019; Stiennon et al., 2020; Bai et al., 2022b). RLHF trains a reward model using datasets of response pairs annotated with human preferences (Wang et al., 2024a), often requiring thousands to hundreds of thousands of labeled examples (Gao et al., 2023).
To improve the alignment process, researchers have proposed decomposing human prefer- ences into distinct aspects, such as helpfulness, harmlessness, and factuality (Bai et al., 2022a; Wang al., 2024b; Dorka, 2024). In these approaches, separate reward function is trained for each of these properties and reinforcement learning is performed on their weighted sum. This decomposition facilitates learning each how to maximize each property independently and allows for control over their balance in downstream applications. Extending this idea, multi-reward formulations have been proposed for personalization, where each user has different combination of these reward functions (Guo et al., 2024; Zhou et al., 2023; Yang et al., 2024b; Wang et al., 2024c). Although this supports personalization, a key limitation is that it typically requires training separate models for each reward combination.
Several approaches have tackled this challenge by reweighting reward functions at inference time, allowing for dynamic model adaptation without retraining (Han et al., 2024; Chen et al., 2024b; Khanov et al., 2024; Mudgal et al., 2023). Others have trained separate models for different reward functions and later combined them in weight space (Jang et al., 2023; Rame et al., 2024). However, these methods rely on the assumption that reward functions are pre-defined and that user preferences are explicitly specified. In contrast, our work develops personalization algorithms that relax these constraints, enabling more flexible and adaptive model behavior.
3 Preliminaries
Our objective to generate responses y to a given prompt x that align with the preferences of an individual user. To capture these preferences, we assume that each user i has a reward function ri(x,y), y), which quantifies how well a response y satisfies the user's expectations for given prompt x. In order to produce user-aligned responses to a given prompt, it is imperative to learn the user-specific reward function and then use the LLM to generate responses that maximize it.
Instead of requiring users to assign explicit scores to responses, we follow common practice (Ouyang et al., 2022) and rely on pairwise comparisons, where users indicate their preference over pair of responses. We adopt the Bradley- Terry (BT) choice model (Bradley & Terry, 1952; Christiano et al., 2017) for pairwise comparisons. The BT model defines the probability that user prefers response yt over y² as:
ply' > 12xx, -
(1)
where p(y1 > y2)x, is the probability that user i prefers y1 over y2, and ((w) Ite s the sigmoid function.
1
is
In standard RLHF, a single, global reward function r(x, y) is learned by maximizing the likelihood of all pairwise comparisons across the dataset. This is formalized optimizing the objective:
C(0) = Σ logr(y' > yxxx;)
(2)
xx)
where 0 represents the reward model. This objective assumes homogeneous preferences across users, treating all pairwise comparisons as arising from the same reward function
r(x,y).
3

---

Published as a conference paper at COLM 2025
While effective for general alignment tasks, this approach fails to account for user-specific variations in preferences. By aggregating data from all users into a single global reward model, standard RLHF overlooks individual differences, potentially leading to suboptimal personalization.
4 The PReF framework
In this work, we model the reward function of an individual user as a linear combination
R, where each g (x, E quantifies the score of the response y the prompt x based on the j-th base function.
of J base reward functions y) ( (x, (x, .x..y)..... ...,) (x,
Each user i is characterized by preference vector A; = [1] A/.,,,...... A/T E R', where A represents the weight that user assigns to the j-th base reward function. The overall reward for user is then defined as:
(x = Σ N (x,1)) = No (xx,)
(3)
This formulation provides a compact representation of user-specific preferences, with the weights A; capturing the unique importance each user assigns to the / base reward functions. Plugging it into Equation 1 gives us the PReF pairwise preference model 1:
ply² > 1² = c(A p(x,y') A, (xx,y²)
(4)
In practice, train neural network to estimate Φb This network gets a concatenation of the prompt x and response y as input and outputs a J-dimensional vector. To train this neural network, we assume access by multiple users, each with individual preferences. Formally, the dataset is represented as { {xn, Vu you in, An IN n=1 where in is the index of the user providing the annotation, and An E {0,1} denotes the user's binary preference, with A = 1 indicating that the user prefers y/ over y Given U different users and M pairs of responses, we can represent the dataset in a matrix form:
pairwise preference dataset where each prompt is annotated
~ Bernoulli(P), P= o (A)
where E R U contains the observable binary preferences in matrix form, P E R contains the preference probabilities as per Equation 4, R|xu is the matrix of user preference vectors, and & IR I is the matrix of base reward function embeddings for all response pairs.
Such a representation of reward function enables us to leverage existing algorithms that can adapt the response of the large language models to linear combination of multiple reward terms at deployment time (Han et al., 2024; Chen et al., 2024b; Khanov et al., 2024; Mudgal
et al., 2023).
4.1 Learning the Base Functions
We train the base reward function model and user embeddings À using the Maximum Likelihood Estimator (MLE) objective of Equation 4:
((),4) = Σ An log o (0. (x
==
An) 1og(1- (( v 22)),,
Unlike standard MLE in RLHF (Eq. 2), our formulation introduces significant challenges. First, the number of users parameters À scales with the number of users in the training
I For simplicity of notation, when dealing with pairwise comparisons of responses y' and y² for the same prompt x, we will denote them o (xx,y²) $(x,y²) = ((x,y' ,y2).
4

---

Published as a conference paper at COLM 2025
set, increasing complexity, More critically, the reward model exhibits bilinear dependency between A and 4(x, yy,)), which makes the optimization landscape non-convex with many local minima. This coupling makes the optimization sensitive to initialization and prone to degenerate solutions (e.g,, trivial or uninformative user vectors). Results in Section 5 show that such instability leads to high variance in the performance of the trained model.
To mitigate these instabilities, we leverage the linear structure of our framework. Specifically, we recognize that Since I(P) A T <b, when the preference probability matrix is known, we can recover A T ① by applying the inverse sigmoid function and reducing the problem of learning (A) and (+) to a matrix factorization problem. However, since P is unknown and and only sparse binary observations in A available, the learning task becomes an instance of Logistic Matrix Factorization (Johnson et al., 2014) problem.
Using these insights, we propose a two-step approach to overcome the instability challenges when training & (see formal description in Algorithm 1):
1. Initialization via SVD: We initialize training using Singular Value Decomposition (SVD) of the observed annotation matrix A, treating it noisy proxy for the underlying preference probability matrix P. The low rank outputs of the SVD are used as initialization for A and Φ, offering a structured initialization that reduces sensitivity to random starting conditions. While the binary nature of A introduces noise, SVD still captures the dominant components of P, providing a meaningful starting point.
2. Refinement via MLE: Although SVD provides strong initialization, it does not directly optimize the likelihood of observed preferences. Therefore, we refine the factorization using the MLE objective. In our experiments we have found that the magnitude of either & or Í tends be big, which hurts downstream performance. We tracked the core of the problem to the fact that the reward factorization A Φ is not unique. For any invertible matrix R, we have A T + = A R- Rœ. Therefore, to stabilize the training we add L2 regularization of the user vectors À to the MLE objective. This prevents extreme parameter values, reduces instability, and addresses scale ambiguity in matrix factorization. As result, training converges more consistently.
This combination of SVD initialization and regularized optimization addresses the instability issues associated with bilinear optimization and ensures a consistent and stable learning process.
4.2 Adaptation to New User
After learning the base reward functions, the next step is to estimate the weight vector À for a new user based on their preferences. The challenge is do this efficiently, requiring as little user feedback as possible to reduce the effort required from the user. This process involves iteratively collecting pairwise feedback and refining the estimate of A.
In each round E {1,.., T}, we sample a prompt and use an uncertainty-based selection strategy to determine pair y/ v of responses to provide the user. We aggregate the prompt, responses, and the user preference A₁ into a dataset and use to estimate the user preference using the regularized MLE objective:
L
((A) = Σ As logo (xx,
==1
(6)
+ (1 As) log(1 o(1 (xxx, y v2)) + ||a1l2
Where B is a hyperparameter that controls the weight of the L2 regularization. Given that during adaptation the features & are known, the problem of inferring A is a plain logistic regression problem which is concave (Kleinbaum et al., 2002) that does not suffer the instabilities that we had while learning the features.
Our strategy to improve data efficiency is to choose the next response pair that maximizes uncertainty, standard approach in active learning (Ren et al., 2021). In this work the
5

---

Published as a conference paper at COLM 2025
uncertainty for candidate prompt-response pair (x,y1,y2) is defined as the largest potential prediction error:
yixx,y,,y = max |A pxx,y' y22 (xx
AEC
where A, is the MLE estimate of À round f, and C is confidence for A (the true user preferences). Intuitively, this metric quantifies how much the predicted preference for the response pair could vary given uncertainty in A.
For logistic regression, the tightest known confidence set (Faury et al., 2020) can be expressed using the Hessian matrix of the log- logllikelihood function, H,(A):
1-1 = (( (xx + B1
(8)
5-1
where of is the derivative of the sigmoid function. Using this Hessian, we define the confidence set:
Lemma 4.1. (Faury et al. (2020), Lemmna 11) Let E((5 = {A E R2 ||a Aell S ((())
where (0) O ld (f)), and assume || < The following holds with probability at least 1- 8 for all t E N.
A E E,
While (6) is theoretically tight, it is computationally infeasible to directly solve Equation under this constraint since we do not have way to avoid iterating over every À E E,(8). To address this, we introduce relaxed confidence set & ε. (6) that provide simple solution to Equation 7. The new confidence set is constructed by replacing the Hessian (A) with the Hessian evaluated at dy:
A. S [)())) where (i(0) = dlgg) !)) The following holds with probability at least 1 8 for all t E IN.
Lemma 4.2. Let E,T (8) = E R" ||- |
A E E (8).
Using the expanded confidence set2 the uncertainty metric simplifies to: Lemma 4.3. The following holds with probability least 1 5 for all E N:
U,(x, y 122) = p0yy y2, x) (2)) (8).
Therefore, to ensure that we choose y1. that most uncertain about, solve the following:
max
(pxx,y',0 y22) (2) y
(9)
The solution for o is the eigenvector of H, 1 (At corresponding to its largest eigenvalue (Hamming, 2012), which we will denote v. To obtain a response pair y' such that (xx,yy,yy)) = v we will use an inference time alignment algorithm to generate a response y1 such that yxx,y²) 1 and Φxx,y²) = 1v. See full description of the procedure in Algorithm 2.
?While the expanded confidence set introduces an exponential dependence on the dimension, our response selection strategy (Equation 9) is not explicitly affected by this. Empirically, we observe that the approach performs well in practice, suggesting that more refined analytical techniques could potentially yield tighter bound.

---

Published as a conference paper at COLM 2025
Attributes
PRISM
User Proforence AUC R0C
0.85
0.80
0.75
Preforence
0.70
0.65
0.00
10
15
20
25
30
35
40
10
15
25
30
40
Number of user answers
Number user ansswrs
80%
73%
Witrate
Wnrate
70%
60%
6076
007
55%
50%
50%
Number user
Number
Ours
Standard RLHF
Model
Figure ROC AUC and winrates for varying number of user answers on the Attributes (left) and PRISM (right) datasets. Our method quickly achieves high ROC AUC and winrates, outperforming baselines by a large margin.
Experiments
Datasets.
We test our method using the following datasets (more details in Appendix
B):
Attributes. To test personalization, we introduce dataset that simulates diverse user preferences using LLMs as roleplay judge (Dong et al., 2024; Zheng et al., 2023). We defined seven preference attributes, each with a positive and negative trait. For example, the attribute length corresponds to users who either prefer verbose concise responses. Each user is assigned two randomly sampled traits, resulting in 84 unique users. Pref- erence data for each user is collected over responses generated using prompts from the AlpacaEval dataset (Li et al., 2023), resulting in 100 user preferences per prompt.
PRISM. We use the PRISM dataset (Kirk et al., 2024b), which contains user preferences for LLM-generated content from 1.5K global respondents across 3K prompts, often showing significant disagreement. Since the original dataset lacks overlap between users and prompts-required for our method-we augment it with synthetic annotations following the PERSONA protocol (Castricato et al.). PERSONA uses LLMs as judges, shown to correlate well with human preferences. This augmentation resulted in 50 user preferences per prompt.
Training and Evaluation Protocol. All experiments use Qwen 2.5, an open-source state-of- the-art model family (Yang et al., 2024a). Unless stated otherwise, use the 0.5B variant as the reward model backbone with a single-layer linear head. Each experiment is repeated 10 times with different random seeds, and report aggregated results. To demonstrate compatibility with various alignment methods, we use ChatGPT-4 with multi-objectivee Best-of-N for the Attributes dataset and Qwen2.5 7B with VAS (Han et al., 2024) for PRISM. Hyperparameters and further training details appear in Appendix C.
Each dataset is split into four parts: train set, a validation set (same users, new prompts), calibration set (new users, same prompts), and test set (new users and prompts). We first train the base reward functions on the train set. To evaluate PReF 's personalization ability, we infer test users' preference coefficients using the calibration set. Final evaluation is on the test set. We use two metrics: (1) Winrate, measuring how often responses aligned with the learned reward function outperform non-personalized responses (evaluated via LLM-as-a-Judgee and (2) User Preference AUC-ROC, measuring how well the reward function predicts choices from response pairs, independent of downstream alignment.
7

---

Published as a conference paper at COLM 2025
(A)
(B)
(C)
Usar Prefurence AUC noC
User Preference AUC ROc.
0.83
User Prefurence AUC ROC
0.0
0.80
0.75
0.2
Uncertaintyy ection Rerstom
Full
No Reg.
No SVD
Number of base reward functions
Number of user answers
Figure 3: (A) Effect of L2 regularization and SVD initialization model performance. We see that both choices are crucial to reduce instabilities in training. (B) Increasing the feature dimension J leads to better performance. (C) PReF's uncertainty-base selection of response pairs to obtain user preferences outperforms the naive strategy of random selection.
5.1 The Benefits of personalization
To assess PReF's 's ability capture personalized preferences, we compare it to two baselines: Standard RLHF, which trains a single reward model assuming homogeneous preferences, and Model per User, which trains a separate reward model for each user.
The results in Figure 2 shows that across both datasets, PReF (blue) clearly outperforms Classic RLHF (green). With fewer than 10 user responses, PReF achieves a 10-15% AUC- ROC gain, showing strong personalization from limited data. A similar boost is seen in win rate: about +10% on PRISM and +25% on Attributes. addition, the Model per User baseline (orange) performs poorly due to the inability to leverage shared structure across users, confirming that training separate models per user is impractical in real-world settings. Figure 8 in the Appendix shows the performance of the Model per User baseline for much larger number of user answers. It shows that our approach requires 25x less data to achieve the same performance.
5.2 Can PReF capture the preferences of real humans?
We validated our framework on real users by conducting a human evaluation study focused adapting to new with pre-trained set of features.
We use the base functions learned from the synthetically generated Attributes dataset. We recruited 28 volunteers, each shown 30 prompts from the test set with two generated answers, and asked to choose their preferred response. The first 15 comparisons were used to learn the user's preferences and the remaining 15 for evaluation. The user was not aware of this distinction. In evaluation, one response was always generated as a baseline using GPT-4o, while the other was personalized version using the learned user preference. We computed the winrate of the personalized answers over the baseline answers. Additional details are given in Appendix D.
We found that our method achieved a 67% winrate, with 95% confidence interval of [57.4%, 76.6%] winrate. shows that by tailoring the responses each user's preferences PReF improves over GPT-4o. This improvement is notable given that GPT-4o has already been aligned to general human preferences, and given that we used very simple features derived from synthetic data. Moreover, the user preferences were learned from just 15 interactions with the user.
5.3 How PReF performs against other personalization frameworks?
We benchmark PReF against other LLM personalization frameworks Variational Preference Learning (VPL) (Poddar et al., 2024) and Pluralistic Alignment (PAL) (Chen et al., 2024a). VPL conditions the reward model encoding of the user responses and training it to achieve in-context learning ability. PAL represents each user as vector latent space and defines the reward of response as its distance from this point.

---

Published as a conference paper at COLM 2025
To assess performance, evaluate these methods on the Attributes dataset, measuring AUC-ROC for unseen responses after collecting 5, 10, or 20 answers from a new user. For fair comparison,
ensure that each method undergoes the same number of hyperpa- rameter tuning experiments, with results averaged over five random seeds. The results, presented in Table 1, show that while VPL performs well, PReF outperforms it at 10 and 20 user responses. This shows that in-context learning has a hard time utilizing a large number of examples.
Number of User's Responses
5
10
20
PReF (Ours)
0.77
0.83
0.85
VPL (Poddar et al., 2024)
0.78
0.80
0.80
PAL (Chen al., 2024a)
0.56
0.59
0.61
Table 1: AUC-ROC over responses from the initial model. Our method outperforms other proposed frameworks for efficient personalization of LLMs.
Beyond reward-learning approaches, we also compare against two widely used technique for personalizing LLM outputs through prompts. The first is the user-provided system prompt, where we use the prompts in the PRiSM dataset (Kirk et al., 2024b) that were written by each participant. The second is in-context learning baseline, concatenating the user-specific dataset (prompt, responses, and preferences) into one prompt. We measure the win rate of responses generated from PReF against these generated from the baselines. PReF achieves a win rate of 71.9% against the system prompts baseline, demonstrating clear advantage in capturing individual user preferences. For the in-context learning baseline, PReF achieves 56.1% win rate at 5 responses, 62.7% at 10 responses, and 68.4% at 20 responses.
5.4 Ablations
Our optimization framework introduces bilinear dependencies between learning the base reward functions and the user coefficients, that can lead to instability and sensitivity to ini- tialization. To address this, we incorporate SVD-based initialization to provide structured starting point and L2 regularization stabilize the MLE optimization (Section 4.1).
Figure (A) validates the importance of these components by comparing our full method (Full) to two ablations: (1) No Reg., which removes L2 regularization, and (2) No SVD, which replaces SVD-based initialization with random embeddings. The figure reports the mean and standard deviation of the mean over 10 models trained on the same data with different seeds. Removing SVD leads to significantly higher variance, particularly in the new user setting, highlighting its role in reducing sensitivity random initialization. Similarly, without L2 regularization of the user's coefficients, the standard deviation of the mean also increases, suggesting that regularization prevents overfitting and stabilizes optimization.
Additionally, we evaluate the benefits of our active learning approach in determining user weights. In Figure 3 (C), we compare our method to a baseline where questions presented to the user chosen at random. The results clearly demonstrate the advantage of our approach: our method achieves x2.7 increase in efficiency getting the same performance with just 15 samples that random selection requires over 40 samples to reach.
6 Discussion
In this work we propose method to quickly adapt LLM generations to personal user preferences. We model each user's preferences as a linear combination of base features. Given fixed features, feedbacks using a logistic bandit algorithm. These weights can then be used to adapt LLM generations without the need for retraining. Moreover, can effectively learn the base
can learn new user's weights from small number of preference

---

Published as a conference paper at COLM 2025
features from data, circumventing the need to predefine them. A major limitation of our work is that most available datasets do not contain preference feedback from multiple users for the same pair of generations, making the learning problem ill-posed. We thus had to limit many of our experiments to synthetically generated data using LLMs to roleplay human users. Collecting large dataset with such overlapping annotations is important goal for future work.
Acknowledgements
We thank members of the Improbable AT Lab for helpful discussions and feedback. We are grateful to MIT Supercloud and the Lincoln Laboratory Supercomputing Center for providing HPC resources. This research was supported in part by Hyundai Motor Company, Qualcomm Innovation Fellowship, an AWS MLRA research grant, ARO MURI under Grant Number N911NF-23-1-0277, DARPA Machine Common Sense Program, ARO MURI under Grant Number W911NF-21-1-0322, and ONR MURI under Grant Number N00014-22-1-277 The views and conclusions contained in this document are those the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the United States Air Force or the U.S. Government. The U.S. Government authorized reproduce and distribute reprints for Government purposes, notwithstanding any copyright notation herein.
References
Francis Bach. Self-concordant analysis for logistic regression. 2010.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint ar arXiv:2204.05862, 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitu- tional ai: Harmlessness from feedback. arXiv preprint ar 2rXi::2212.080733, 2022b.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 32)34344) 1952.
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. Xiv preprint ar arXiv:2307.15217 2023.
Louis Castricato, Nathan Lile, Rafael Rafailov, Jan-Philipp Fränken, and Chelsea Finn. Persona: A reproducible testbed for pluralistic alignment, 2024b. URL https://arxii. org/abs/2407.17387
Daiwei Chen, Yi Chen, Aniket Rege, and Ramya Korlakai Vinayak. Pal: Pluralistic alignment framework for learning from heterogeneous preferences. arxiv preprint arXiv:2406.08469 2024a.
Ruizhe Chen, Xiaotian Zhang, Meng Luo, Wenhao Chai, and Zuozhu Liu. Pad: Personalized alignment of lims at decoding -errodiingtime... arXiv preprint ar XrXi:22410.04070, 2024b.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Yijiang River Dong, Tiancheng Hu, and Nigel Collier. Can 1lm be a personalized judge? ar Xiv preprint ar Xit:2406.11667 2024.
10

---

Published as a conference paper at COLM 2025
Nicolai Dorka. Quantile regression for distributional reward models in rlhf. ar preprint urXiv:2409.1016 2024.
Louis Faury, Marc Abeille, Clément Calauzénes, and Olivier Fercoq. Improved optimistic algorithms for logistic bandits. In International Conference on Machine Learning, pp. 3052- 3060. PMLR, 2020.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. (0835-10866.. PMLR, 2023.
Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. Scaling synthetic data creation with 1,000,000,000 personas. arXiv preprint ar Xriv:2406.20094 2024.
Yiju Guo, Ganqu Cui, Lifan Yuan, Ning Ding, Zexu Sun, Bowen Sun, Huimin Chen, Ruobing Xie, Jie Zhou, Yankai Lin, et al. Controllable preference optimization: Toward controllable multi-objective alignment. arviv preprint ar arXiv:2402.19083 2024.
Richard Hamming. Numerical methods for scientists and engineers. Courier Corporation, 2012.
Seungwook Han, Idan Shenfeld, Akash Srivastava, Yoon Kim, and Pulkit Agrawal. Value augmented sampling for language model alignment and personalization. ar preprint arXiv:2405.06639, 2024.
Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Personalized large language model alignment via post-hoc parameter merging, arviv preprint XrX::22300.11666 2023.
Christopher C Johnson et al. Logistic matrix factorization for implicit feedback data, Ad- vances Neural Information Processing Systems, 27(78):1-9, 2014.
Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. Args: Alignment as reward-guided search. ar Xiv preprint ar arXio:2402.01694 2024.
Milton King and Paul Cook. Evaluating approaches to personalizing language models. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 2461-2469, 2020.
Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott A Hale. The benefits, risks and bounds of personalizing the alignment of large language models to individuals. Nature Machine Infelligence, pp. 1-10, 2024a.
Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, et al. The prism alignment project: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models. arv preprint ar Xiv: 16019, 2024b.
David G Kleinbaum, K Dietz, M Gail, Mitchel Klein, and Mitchell Klein. Logistic regression. Springer, 2002.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of 2023.
instruction-followiing models. https:
Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, et al. Controlled decoding from language models. ar preprint ar xxX::2310.17022. 2023.
Lin Ning, Luyang Liu, Jiaxing Wu, Neo Wu, Devora Berlowitz, Sushant Prakash, Bradley Green, Shawn O'Banion, and Jun Xie, User-llm: Efficient Ilm contextualization with user embeddings. Xiv preprint 2325:240,.13598, 2024.
11

---

Published as a conference paper at COLM 2025
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Ch Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744 2022.
Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, and Natasha Jaques. Personalizing reinforcement learning from human feedback with variational preference learning. arXiv preprint ar Xiio:2408.10075, 2024.
Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. Advances in Neural Information Processing Systems, 36, 2024.
Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Brij B Gupta, Xiaojiang Chen, and Xin Wang- A survey of deep active learning. ACM computing surveys (CSUR), 54(9):1-40, 2021.
Peter J Rentfrow, Lewis R Goldberg, and Daniel J Levitin. The structure of musical prefer- ences: a five-factor model. Journal of personality and social psychology, 100(6):1139, 2011.
Chris Richardson, Yao Zhang, Kellen Gillespie, Sudipta Kar, Arshdeep Singh, Zeynab Raeesy, Omar Zia Khan, and Abhinav Sethy. Integrating summarization and retrieval for enhanced personalization via large language models. ar preprint arXim:2310.20081 2023.
Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, et al. A roadmap to pluralistic alignment. arxiv preprint ar Xiv: 0500070 2024.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008 2020.
Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, et al. Secrets of rlhf in large language models part ii: Reward modeling. arXiv preprint Xiv:2404 0.60000000 2024a.
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-ofeexperts, arXiv preprint D:Xiv:2406.12855, 2024b.
Kaiwen Wang, Rahul Kidambi, Ryan Sullivan, Alekh Agarwal, Christoph Dann, Andrea Michi, Marco Gelmi, Yunxuan Li, Raghav Gupta, Avinava Dubey, et al. Conditional language policy: A general framework for steerable multi-objective finetuning. arXiv preprint ar Xrx::22407.11: 157622 2024c.
Bin Wu, Zhengyan Shi, Hossein A Rahmani, Varsha Ramineni, and Emine Yilmaz. Under- standing the role of user profile in the personalization of large language models. arXiv preprint Xio::400.117888 2024.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. Xiv preprint ar 2rXie:2412.15115 2024a.
Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. Rewards-in-contet: Multi-objective alignment of foundation models with dynamic pref- erence adjustment. arxiv preprint ar arXio:2402.10 10207, 2024b.
12

---

Published as a conference paper at COLM 2025
Zhehao Zhang, Ryan A Rossi, Branislav Kveton, Yijia Shao, Diyi Yang, Hamed Zamani, Franck Dernoncourt, Joe Barrow, Tong Yu, Sungchul Kim, et al. Personalization of large 2024.
language models: A survey. arXiv preprint ar
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Éric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595- 46623, 2023.
Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, and Yu Qiao. Beyond one-preference-frr for-all: Multi-objective direct preference optimization. ar Xiv preprint arXit:2310.03708, 2023.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human prefer- ences. ar Xiv preprint ar Nii:1999.00893,, 2019.
A Proofs and Derivations
|| A11l S {((()) where (1(6) O log( f The following holds with probability at least ő for all IN.
Lemma 4.2: Let & (0) {A Rd
A' E E
Proof:
Using Proposition 1 from (Bach, 2010), we have that there exists c 2 1 (the self- concordant constant the function) such that:
e 2c e. 1..8-.810 (e.) = H, N 20110. - (e.)
From Lemma 11 in (Faury et al., 2020) we have that, with probability at least 8:
0. 0 (0, S (2 (8))))))
Because H²(O.) is positive semidefinite with minimum eigenvalue B, we get
(2 + 45) (6)
||e. 0ell2 S
VB |l. - H.
/B
With R(5) (5(2+49)m v
This directly gives us:
( H, H
Combining this all together and taking union bound, we have that, with probability at least 28, the following holds:
||e 0.ll (0)) I R 0 H.
Invoking Lemma again:
100 0 011l,,(8) (245)(+44)))
Lemma 4.3 (general version): Let = (0: 110 ر|| < {}} be an ellipsoidal confidence
set in Id around ê, where || VzT is the norm induced by positive semi-definite
matrix A. For any vector E Rd the solution to the optimization problem max
BEC
is given by:
max(0,x) = (0,x) + B||x||g-11 BEC
13

---

Published as a conference paper at COLM 2025
Proof:
The optimization problem can be written
max(0,3) (0, REC
max
(0, x)
0:10 5B
Substituting = d we decompose:
max x) = =0,,x) BEC
max (0,x)
8: :p
Let v' B.
Then |/lal S implies ||o||s - 1, and
max (v,x) = B max 2: Sp
(v/,x) ":
Using the definition of the E-norm, || S 1 implies o < 1. Letting = I this
constraint transforms to ||=||2 & 1, and v' Σ 11122,. Substituting into the inner product:
(d',x) xz-x-1/2-
The problem becomes:
max (0',x) max ZTZ 1/2x o/ <1
=:||=|||<1
By the Cauchy-Schwarz inequalit, this achieves maximum at 12-1/xx]1a with the value:
Σ 1-1122
max ZT 1/2 = ||11/2211 21||=|1211
Substituting back,
max (v,x) -111-11/2x11 ::||p||es
Thus, the original problem becomes:
max = x) + -1|x||=-1
dEC.
Corollary 1: Under the same setting and following the same proof steps, the following also holds:
min (0, x) (0,x) -1|
BEC
Corollary Define U(x) maxØed (0, (0,x)) Using Lemma 4.3 and Corollary we can write:
U(x) = =||x
B Datasets
B.1 Attributes
B.1.1 Data Generation
We simulate users with roleplay (Ge et al., 2024), where each user is defined by two traits that determine their preferences. For example, user A might prefer long and formal responses, while user prefers engaging and confident responses. We define 7 categories, each with positive and negative trait. For example, one category is length, and user could either prefer verbose or concise responses. This results in 84 users, corresponding to combinations of traits.
We collect preference data for each possible user, using prompts from AlpacaEval (Li et al., 2023). For each prompt, generate two responses, reusing the user traits to elicit
14

---

Published as a conference paper at COLM 2025
Table Attributes used for data generation.
attribute
direction
direction 2
length
verbose formal
concise
formality
informal
humour
humorous
serious
elicitation
engaging
unengaging
politeness
polite
rude
enthusiasm
enthusiastic
demure
confidence
confident
uncertain
contrasting responses. For example, one response could be long and formal, and the other engaging and confident. For each user, we collect preferences for the same 100 randomly sampled prompts, resulting in preference matrix A E U where M = 100 and U = 84 in our experiments. This dataset then split into training and test sets (80-20) by splitting users and pairs separately to avoid contamination
When collecting preferences using roleplay, we present the two responses A and the prompt in both possible orders to account for any possible order bias. This gives two preference matrices, A and A2, where A 1 if the simulated user prefers response A and
A 0 if they prefer response B. The final preference is the average, A (A A2)
B.1.2 Prompts
Below we give all the prompts used for data generation. In all cases used OpenAl's GPT-4o model via API.
Preferences prompt.
To collect preferences based on user attributes, we used the following system
You are helpful Al judge. You prefer attrl and attr2 responses.
Preferences then collected using the following prompt from AlpacaEval (Li et al., 2023).
15

---

Published as a conference paper at COLM 2025
Select the output (a) or (b) that best matches the given instruction. Choose your preferred output, which can be subjective. Your answer should ONLY contain: Output (a) or Output (b). Here's an example:
# Example:
## Instruction:
Give description of the following job: "ophthalmologist"
## Output (a):
An ophthalmologist is a medical doctor who specializes the diagnosis and treatment of eye diseases and conditions.
## Output (b):
An ophthalmologist is medical doctor who pokes and prods at your eyes while asking you read letters from chart.
## Which is best, Output (a) or Output (b)?
Output (a)
# Task:
Now the real task, do not explain your answer, just say Output (a) or Output (b).
## Instruction:
(instruction)
## Output (a): (output 1}
## Output (b): {output
## Which is best, Output (a) or Output (b)?
Responses prompt.
Responses generated based attributes by using the following system
You are helpful Al assistant. You generate attrl and attr2 responses.
B.2 PRISM
B.2.1
Data Generation
We construct a dataset of roleplayed user preferences using real human-provided attributes from the PRISM dataset. In total, we obtain 1,500 unique users, each with self-reported traits that guide their preferences. These traits encompass wide range of characteristics, includ- ing familiarity with LLMs, frequency of usage, personal values, preferred communication style, and demographic factors. To simulate user responses, we follow the roleplay protocol outlined the PERSONA paper, utilizing the GPT-4o model to generate responses aligned with user traits. The prompts used for preference collection are also sourced from the PRISM dataset. We apply a filtering process to select prompts that are inherently controversial, resulting in final set of 2,262 prompts.
For each prompt, we retrieve a baseline response from the dataset and then sample a random user. Using Qwen 2.5 7B, revise the response to better align with the sampled user's 's preferences, thereby generating response pairs that exhibit contrasting characteristics. For instance, user who prefers highly factual and fluent responses may receive revision that
16

---

Published as a conference paper at COLM 2025
improves clarity and correctness, whereas user who values creativity and engagement might get a more expressive and imaginative revision.
To construct the preference dataset, sample 50 users for each response pair and simulate their preferences, leading to a dataset of approximately 110,000 preference data points. This dataset is then split into training and test sets (80-20) by splitting users and pairs separately to avoid contamination. Notably, this constitutes only about 3% of the full preference preferencemaateen matrix, which would include all users over all possible response pairs.
As with the preference collection process described in the Altributes section, we ensure robustness against order bias by presenting response pairs in both possible orders when eliciting preferences.
B.2.2 Prompts
Below we give all the prompts used for data generation.
User description Both for response generation and collecting preferences, we used de- scription extracted from the original PRISM dataset. This is an example of such description:
Familiarity with LLMs: Very familiar
Indirect use of LLMs: Yes
Direct use of LLMs: Yes
Frequency of using LLMs: Every day
Briefly describe your values, core beliefs, guiding principles in life, etc.: Be a kind, honest, helpful, and fair person who is generally polite to everyone. Do not do things that I may regret in the future. Follow all norms in the country I'm visiting and living. Be loyal friend. When see someone needs help and I'm capable of helping, step up to help.
Your system prompt for LLMs: You are an attentive listener and loyal Canadian friend who is very honest when I'm asking you for feedback. If something seems wrong, you'll point it out to me to let me know. straightforwarr, don't reframe something negative into something very positive. Also, please be concise in your answer. you have no idea on what feedback to give, just say "I don't t know". Age: 18-24 years old
Gender: Female
Employment Status: Unemployed, seeking work
Education: University Bachelors Degree Marital Status: Never been married
English Proficiency: Fluent
Religion: No Affiliation
Ethnicity: Asian
Birth Country: Hong Kong
Current Country: Canada
LLM use cases: suw.uucessuggstiosns.. 'professional.work/ 'casual converstion,,, 'financialguuddance', 'relation-
ship advice', llanguggelearringg 'other'l
Preferences of LLM behaviour (scale of 1-100): l'values: O', 'creativity: 72', 'flu- ency: 100', 'factuality: 100', 'diversity: 100', 'safety: 100', 'personalisation: 100',
"helpfulness: 100']
Preferences To collect preferences based on user attributes, we used the following prompt taken from (Dong et al., 2024).
17

---

Published as a conference paper at COLM 2025
Given the user profile provided below, select the response from AT assistant A or that the user would most likely prefer. Don't focus on which response is better in general, just which one is better for this user. Declare your choice by using the format: "[[A]]" if you believe assistant A's response is more suitable, or "[[]]] if assistant B's response is better suited.
[User Profile]
user description
[User Question]
[prompt]
[The Start of Assistant A's Answer]
responsee_1
[The End of Assistant A's Answer]
[The Start of Assistant B's Answer]
(response 2}
[The End of Assistant B's Answer] [Answer]
Responses To generate responses based on user attributes, we used the following two prompts, taken from (Castricato et al.):
Examine the COMPLETION:
foriginal response)
in relation to the DEMOGRAPHIC: juser_decrriptin]
and the INSTRUCTION: {prompt).
Put yourself in the shoes of DEMOGRAPHIC, Identify the ways the completion both does and does not resonate with the demographic. Provide a concise explanation, quoting directly from the demographic and completion to illustrate your evaluation. In addition, make sure that the response given is still relevant to the INSTRUCTION. Format: EVALUATION: ... SUGGESTIONS:...
The output is then used input to the second prompt:
Revise the COMPLETION:
roriginallresponse)
with respect to INSTRUCTION: {prompt)
based the CRITIQUE: (critique)
Provide revision of the completion, do not make ANY references to the exact preferences or attributes of the demographic. Just provide the new response, use the format:
REVISED RESPONSE:...
C
Training Details
Table 3 includes the hyperparameters for all models trained in this work. Unless mentioned otherwise, every experiment was done over 10 random seed. To ensure fair comparison, we only performed hyperparameter tuning experiment per algorithm before settling the final ones.
For the Classic RLHF baseline used the hyperparameters of base functions, which equal to this case). For the Model per liser baseline, we fixed the learning rate of the linear head to 1e-3 but experimented with different learning rates for
method (besides number
18

---

Published as a conference paper at COLM 2025
Algorithm 1 Training the base reward functions
1: Input:
Pairwise preference dataset ( J8. A, base reward function(s) R with output dimension ,, randomly initialized user matrix A
2: Construct the observed preference matrix
RRXMM, where is the number of users and M is the number of item pairs in the dataset.
3: Compute rank-J SVD (or an approximation for sparse matrices), obtaining A UEVT.
4: Extract the initial user matrix: A UE3, and the per-pair reward matrix: O E.VT..
5: Fit the reward function Rg to < using Ez-loss.
6: Refine Rg by jointly optimizing A and Rg using Equation 5.
7: Output: Rg, A
Algorithm 2 Uncertainty-Guider User Weight Estimation
1: Input: Reward function & with output dimension J
2: for 1,2,... do
3: if | = 0 then
4: Select a random prompt and response pair (y', y
5: else
6:
Choose prompt x and response pair (y²,y²) that maximize Equation (6).
7:
end if
8: Obtain the user preference for the selected response pair.
9: Estimate new user weights A, based on all collected data using Equation (5).
10: end for
11: Output: User weights À
the backbone. In that, followed common practices in few-shot adaptation that showed that training the entire model with a small amount of data points can lead to extreme overfit. We have found that freezing that backbone entirely works the best in the range of 5-40 answers, and training with a learning rate of 1e-6 works the best in the regime of 100+ user answers.
Table 3: Hyperparameter table
Algorithm
Ours
Ours (PRISM)
VPL.
PAL
Dataset
Attributes
PRISM
Attributes
Attributes
Reward model Learning rate Regularization weight
Qwen 2.5 0.5B
Owen 2.5 0.5B
Qwen 2.5 0.5B
Qwen 2.5 0.5B
1e-3
1e-3
1e-3
le-5
0.02
0.02
N/A
N/A A
# of Gradient steps
500
1000
500
500
Batch size
32
64
32
32
# of base functions
8
6
N/A
D Human Evaluations
In this section we give additional details about our human evaluations.
Volunteer Evaluators The volunteer human evaluators recruited for our study were
Harvard and MIT graduate students post-doctoral researchers with STEM focus.
19

---

Published as a conference paper at COLM 2025
Study Protocol Human evaluators took part in our study via web app. Upon starting the task, users were first shown a set of instructions. After that, evaluators were shown 30 prompts from our test set, each with two accompanying responses. The first 15 responses and prompts were chosen using our online learning algorithm, while the next 15 were ever repeated. For each example, evaluators could
chosen at random. No prompt choose the response they preferred, or they could choose neither. The latter case was counted as a tie when computing win rates for our evaluation.
Figure shows screen captures of the pages in our webapp: the instructions and a single prompt and responses example.
Indrrcciioss
Welcome to Study
jinu uill fill short devalling wrefeeences
Riber thesurveu will, the you caked
from LLM souusheuud bused personal preference cumpple,
you
resdemse, or you rotlyy can't deciole you can obin thoose preference" you will beanoun examples total.
"Finiehh" subrmit resporse.
To keep track resporass, ue need you prazi de narme Pui omy nome thit ue could idencilly your nepprnee Sia
your here:
Survey
Usevio RPS Balt with Gradio Setlings +
Figure Instructions Page
Breakdown of Winrates Figure shows the winrates for the 25 participants in the study, We see that there is fraction of participants that prefer the personalized response almost all the time, while another group is close to indifferent. One reason for this may be that the features we used in our experiment were focused on small set of attributes. Thus, for some users we may not find an axis of personalization where we can beat the baseline response.
Personalized Response Generation In our human evaluations we compare against GPT- 4o, which are unable to finetune. This prevents from aligning the responses based on learned user weights. Instead, we generate large pool of responses using random attributes and select the response that best aligns with the user's preferences. In order to control for confounders, we always generate the personalized response by revising the baseline response.
Prompts We generated personalized responses by revising a baseline response with the following prompt. (Castricato et al.):
20

---

Published as a conference paper at COLM 2025
Instrutton ons Saney
Maln
Question out of 30
Please select your preferred response
Prompt:
you
Respanse A:
Response
anether
Predidents
othernity
Provrdenns
Indernl hoildnng
Woshington rendy, mondo miy celebrotedon on the mondin of Febe unry, bero uhl diorurr the regular day order II nriginally was up in 1ee6 to honor George Woschin lide
thing
Recnvon, originolly pltoneeeed
tres wonor Ceorge ul oerhingoon the first presiden,, uhosse oith day Rebruary
22, Over time, houooy has come oelcrroo not only woshinggtnn olso oll U.S. with porsoullr emohoss Abrohom Uncolit, blishday time
first
years nou, supposed presicents, Just lumping evergcne sogeeher Use they desiene equol spodight or sommal Some colrecaeee Akruhamm Linocl busanuse Lind
Fobouory some
Pabruoyy trreogg the country's hisooy.
busionsd presidents
Resormac
Resporas tal
Flo eeaeeenn
Submit
Use API BLlk with Grodlo
Settings
Figure 5: Response Comparison Page
Here is a user instruction:
(instruction)
And here is possible response: [base_response]
Revise it according to your own tastes. Remember, (sys.prompt)
Only include the revised response your answer and nothing else. Your response must look like response to the original user instruction. If you include any other text in your response other than the revised response, you are bad assistant. Make sure to keep your answer to single paragraph and do not make it too long.
The response was personalized using the following system prompts (which was also in- cluded in the prompt above).
You are helpful Al assistant. You generate (attrl} } and (attr2) responses.
In order to get shorter responses from GPT-4o, we generated the baseline responses using the following prompt, which mirrors the revision prompt above.
21

---

Published as a conference paper at COLM 2025
Eigenvalues of the PRISM dataset
100
80
Magnitude
60
40
20
0 10
20
30
40
50
Eigenvalue
Figure 6: The magnitude of the 50 first eigenvalues of the preference matrix. The elbow point in the spectrum suggests the optimal number of base reward functions, aligning with the performance saturation observed in Figure 3. This indicates that eigenvalue analysis may serve efficient heuristic for selecting the dimensionality of user preference representations.
Human evaluation winrates Histogram
Count
3
2
0%
10% 20% 30%
40%
50% 60% 70% 80% 90% 100%
Winrate
Figure Histogram the results from our human evaluation experiment.
Here is user instruction: [instruction)
Give a response to the user instruction. Your response must look like response to the original user instruction. If you include any other text in your answer other than your response, you are bad assistant.
Make sure to keep your answer to single paragraph and do not make it too long.
E Additional Experiments
E.1 Feature Interpretation
To better understand the base reward functions learned by our framework, we perform automatic interpretation analysis. This helps validate that the learned reward structure
22

---

Published as a conference paper at COLM 2025
0.80
0.75
0.70
0.60
0.55
0.50
10a
Number of
Figure 8: Reward model performance when trained using single user's answers only. To achieve full performance, requires over 500 pairwise preference comparisons from the user, making this method not feasible in scale.
captures meaningful dimensions of user preferences. We first score all responses in our Aftributes dataset using the learned base reward function. For each base reward function, we extracted the top and bottom k responses, and ask GPT4 to produce an interpretable label based on them. For more details, see Appendix E.1.
Figure 9 shows the generated labels for each dimension along with the explained variance. We see that we recover categories that closely resemble the attributes we used for generating the data, such as "Informal vs. Formal" or "Conciseness vs. Elaborateness".
Professional Casual Conciseness vs. Elaborateness
Engagement vs. Analysis
Informal Formal
Practical vs. Creative
Engagement vs. Factual
Practical Theoretical
Formaiity vs. Informality
0.00
0.05
0.10
0.15
0.20
0.25 0.30 0.35
Explained Variance (%)
Figure 9: Sorted principal components of the Attributes dataset along with LLM generated descriptions. We were able to recover some of the axes that were used the dataset generation.
E.2 Scaling data and compute leads better base reward functions
Here, we investigate how the quality of the base reward functions improves as scale the amount of data used in their training and the size of the neural network we use to model them. Our hypothesis is that using more users and response pairs in the training will lead to better, more nuanced reward factorization. Figure 10 shows that, indeed, performance (measured by ROC AUC) improves consistently with both larger models and more training data. While larger models generally perform better, we observe that as the training dataset becomes larger, the performance of all model sizes begins to converge. These results indicate that PReF follows expected scaling trends, reinforcing its potential to benefit from larger models and larger preference datasets.
Another critical factor affecting the performance of our method is the number of base reward functions J. A higher number of base reward functions allows for a more nuanced representation of user preferences, but increases the amount of data required to determine user-specific weights accurately. Figure presents the ROC AUC scores for the PRISM dataset function of the number of base reward functions, under fixed budget of
23

---

Published as a conference paper at COLM 2025
40 user-specific samples. We observe that increasing J beyond six base functions yields diminishing returns, suggesting a sweet spot in the trade-off between expressivity and data efficiency. Interestingly, this trend aligns with the elbow point observed in the magnitude spectrum of the eigenvalues from a SVD of the training dataset (Figure in Appendix). This suggests that analyzing the eigenvalues of the reward preference matrix may serve effective heuristic for selecting the optimal number of base reward functions, potentially reducing the need for hyperparameter tuning.
User Preference AUC ROC
0.86
0.84
0.82
0.80
0.5B
1B
0.78
3B
104
10a
Dataset size
Figure 10: Effect of scaling dataset size (x-axis) and the neural network of the base reward function size (different colors) on the reward model performance in the PRISM dataset.
24


Reviews:
[{"review_id": "2SoeeydOeb", "reviewer": "Reviewer_iLD4", "summary": "This paper proposes PReF, a framework for LLM personalization via reward factorization. It assumes that user preferences lie in a low-dimensional space and can be expressed as a linear combination of shared base reward functions. The method requires only a small number of user responses for effective adaptation and integrates with existing inference-time alignment techniques without modifying model weights. Experiments on both synthetic and real data show promising results, with a 67% win rate over GPT-4o in human evaluations.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Tackles the limitations of existing alignment methods by modeling user-specific rewards through shared base components.\n2. Needs only around 10 examples per user and avoids model finetuning, making it practical for large-scale use.\n3. Demonstrates strong results in both synthetic and real-user settings, with effective use of active learning.", "reasons_to_reject": "1. May not capture complex or evolving user preferences.\n2. It remains unclear how interpretable or complete these learned components are.\n3. The framework lacks mechanisms to track and update changing user preferences.", "questions_to_authors": "1.\tHow robust is the heuristic for selecting the number of base reward functions, and how does it impact performance?\n2.\tCan users inspect or adjust their personalization parameters to improve transparency or control?\n3.\tHow does this method align with broader safety and value alignment goals?\n4.\tHow well does the approach generalize beyond synthetic datasets to more diverse, natural user interactions?", "extra_scores": {}}, {"review_id": "PzMzaqDptp", "reviewer": "Reviewer_fVG8", "summary": "This paper introduces a framework called PReF to utilize the low-rank structure of user preferences. PReF addresses the research problems: 1. learning the low-rank reward models with seen users; 2. active learning for efficiently learning new users' preferences. This paper shows PReF can adapt to a new user and predict their reward better than previous methods. It also validates the personalized responses according to the learned reward function with both static datasets and human evaluation.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This paper comprehensively studies almost all problems of language model personalization and proposes reasonable solutions. \n\n* The active learning part is novel and has theoretical support.\n\n* Table 1 shows the proposed method has better reward prediction accuracy than VPL and PAL.\n\n* This paper really demonstrates the quality gains of personalized responses. It also uses two alignment methods: Best-of-N and VAS, showing the generalizability of the proposed reward modeling.\n\n* The human evaluation also supports the quality of personalized responses.", "reasons_to_reject": "* For the adoption or active learning part, I see maximizing Eq. (7) or (9) by enumeration can work but don't see the proposed method being more practical. In particular, I don't understand how we pick the prompt $x$ before using Eq. (9). Given $x$, I don't understand how we pick a response pair using an inference time alignment algorithm. In particular, how do we guarantee that we can exactly solve $\\phi(x, y^1) = \\frac{1}{2}\\nu$. Why don't we use $\\phi(x, y^1) = 2\\nu$ and $\\phi(x, y^2) = \\nu$ for example?  Also, by comparing Figure 3(C) and Table 1, it looks like random selection already outperforms both baselines. Typically confidence set based method is conservative like other bandit algorithms but also like to see if there is any approximation in maximizing Eq. (9).\n\n* I feel the SVD initialization is a little bit ad-hoc also wonder if we have a better way to learn the base functions. See my questions below.", "questions_to_authors": "* In Section 4.1, why don't we have an L2 regularization term in Eq. (5) as in Eq. (6). Logistic Matrix Factorization does have L2 regularization. I wonder if we can overcome the instability challenges with regularization. We can still initialize via the corresponding WALS [1] solution.\n\n* Again for learning the base functions, have the authors considered constraining $\\lambda \\geq 0$ or modeling the reward function as a convex combination of the base functions like LoRe [2]. LoRe (without active learning) also outperforms both VPL and PAL.\n\n* As said, I don't understand how we pick the prompt $x$ with Eq. (9) during active learning and how we find $y^1$ and $y^2$ using the eigenvector. \n\n* In Figure 3(A), did we require to tune the value of $\\beta$?\n\n* In Algorithm 2 L6: ``maximize Equation (6)`` should be ``maximize Equation (9)``.\n\n* In Algorithm 2 L9:  ``using Equation (5)`` should be ``using Equation (8)``.\n\n[1] Y. Hu, Y. Koren, and C. Volinsky. Collaborative ltering for implicit feedback datasets. In ICDM 2008, pages 263--272.\n\n[2] Avinandan Bose et. al.. LoRe: Personalizing LLMs via Low-Rank Reward Modeling. https://arxiv.org/abs/2504.14439", "extra_scores": {}}, {"review_id": "0kvSne1gPS", "reviewer": "Reviewer_wGH9", "summary": "The authors introduce a new method called \"Personalization via Reward Factorization (PReF)\" that factorizes each individual user's reward as a linear combination of base functions. While prior works required training user-specific reward models, the authors demonstrate that PReF is more data efficient and achieves higher win rate across two datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Section 3 and Section 4 provide a good explanation of what problem the authors are addressing and and theoretical guarantees of how they designed their PReF method to achieve that. \n\n2. The ablation experiments provide sufficient insights on how to design a good personalization system (L2 regularization, SVD initialization, and how the framework could be extended as more number of user's responses are accumulated).", "reasons_to_reject": "1. In Table 2 of Appendix B, the authors define 7 distinct attributes. I think the core of LLM personalization papers is a good definition of different user attributes that could well capture what people care about from LLM responses in real-world interactions. Yet, the attributes that the authors test are confined to a narrow set of stylistic-related attributes. There should be more consideration of what users might take into account beyond stylistic features and an inspection of PReF could also capture those. (I am aware that the authors also tested on the PRISM dataset, but am mentioning this to check if they have any thoughts on which attributes the PReF method could be extended to).\n\n2. Related to 1, in a real-world setting, users might not explicitly verbalize what they prefer or the preferences might be entangled with each other and even contradict (e.g., In setting A, user X might prioritize preference M than preference N and in setting B, user X might prioritize preference N than preference M). The settings that the authors employed (Attributres, PRISM) do not seem to capture this and remains to be quite synthetic. I don't think that this is what the paper should solve, but yet, since the method assumes that we could extract user weights to define a user specific reward, it is concerning of whether this method could be extended in these more practical settings.", "questions_to_authors": "In Section 5.3, I am curious about the reasons why PReF performs better than VPL and PAL. Beyond the quantitative results, could the authors provide thoughts on why user-provided system message and ICL isn't effective?", "extra_scores": {}}, {"review_id": "hOvtXuWcme", "reviewer": "Reviewer_Fo71", "summary": "The paper proposes an elegant way to personalize Reward Models per user preferences. To \"onboard\" new user, few (<=30) prompt-response pairs are necessary per user and the method uses uncertainty-based selection strategy to determine a pair should be presented to the user.\n\nThe authors claim 67% win rate improvement over GPT-4o. It's not very clear from the paper but my understanding is that 67% win rate is GPT-4o vs GPT-4o + inference-time alignment method with Qwen-based personalized RM. I'd appreciate if the authors make it more obvious.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Interesting approach to tackle personalization.", "reasons_to_reject": "Main downside is that the proposed approach is for Reward models. Which makes onboarding new users possible only if LLM is then aligned using inference-time alignment technique.\n\nIt would be interesting to see how this approach compares to a much simpler baseline: just give GPT-4o these 10-30 collected preference pairs *in-context* and see if it will be able to tailor it responses per user better.", "questions_to_authors": "Lines 261 - 262. How exactly to you split the data into (train, validation, calibration, test) set. For example, in the test set, are there unique users not present in other splits? Or each user can have some prompt-response pairs in more than one split?", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors propose a new way of personalizing the output of LLMs: by factoring user preferences in a linear, low dimensional space of reward functions, and combining them on a per-user basis. Experiments demonstrate strong sample efficiency compared to baselines on 2 datasets.

Reviewers raised some concerns that were partially addressed in response.

iLD4's biggest remaining concern is that appendix E1 about interpretablility is not in the main text body: perhaps the authors could surface a pointer in revision?

Fo71 suggested an additional experiment, which was only partly covered in the original submission. The authors added the additional experiment which further shows the strengths of their approach.

Overall, I am pleased with the discussion between reviewers and authors, and believe many of the biggest concerns have been addressed. This work represents an interesting contribution above and beyond existing works, and (while some presentation factors could be improved) many of the shortcomings have been addressed in discussion.
Decision: Accept