LoRe: Personalizing LLMs via Low-Rank Reward Modeling
Authors: Avinandan Bose
Zhihan Xiong
Yuejie Chi
Simon Shaolei Du
Lin Xiao
Maryam Fazel
Venue: COLM 2025 (2025)
Keywords: preference learning
personalization
reward modeling
plurality
TL;DR: We introduce low-rank personalized preference modeling for LLMs, enabling scalable and efficient user-specific reward learning with superior generalization and few-shot adaptation.

Abstract: Personalizing large language models (LLMs) to accommodate diverse user preferences is essential for enhancing alignment and user satisfaction. Traditional reinforcement learning from human feedback (RLHF) approaches often rely on monolithic value representations, limiting their ability to adapt to individual preferences. We introduce a novel framework that leverages low-rank preference modeling to efficiently learn and generalize user-specific reward functions. By representing reward functions in a low-dimensional subspace and modeling individual preferences as weighted combinations of shared basis functions, our approach avoids rigid user categorization while enabling scalability and few-shot adaptation. We validate our method on multiple preference datasets, demonstrating superior generalization to unseen users and improved accuracy in preference prediction tasks.

Published as a conference paper at COLM 2025
LoRe: Personalizing LLMs via Low-Rank Reward Modeling
Avinandan Bose
Zhihan Xiong
Yuejie Chi FAIR at Meta
University of Washington FAIR at Meta
University of Washington
Simon Shaolei Du University of Washington
Lin Xiao FAIR at Meta
Maryam Fazel University of Washington
Correspondence to: avibose@es. wwt.se/dosw.asssngooo...ddu
Abstract
Personalizing large language models (LLMs) to accommodate diverse user preferences is essential for enhancing alignment and user satisfac- tion. Traditional reinforcement learning from human feedback (RLHF) approaches often rely on monolithic value representations, limiting their ability to adapt individual preferences. We introduce a novel framework that leverages low-rank preference modeling to efficiently learn and gener- alize user-specific reward functions. By representing reward functions in low-dimensional subspace and modeling individual preferences weighted combinations of shared basis functions, our approach avoids rigid user categorization while enabling scalability and few-shot adapta- tion. We validate our method on multiple preference datasets, demon- strating superior generalization to unseen users and improved accuracy in preference prediction tasks. The code for our experiments is available at: https://gi thub. cttmm//accbbooeeeeerc///l/oo//.
Introduction
Aligning Large Language Models (LLMs) with human values is paramount for enhancing their relatability and effectiveness. Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., 2017) is the standard approach to achieve this alignment. However, conventional approaches often rely on monolithic value representations, which inadequately address the diverse needs of various populations (Bakker et al., 2022; Durmus et al., 2023).
In recent years, there has been a growing advocacy for pluralistic alignment in AI systems. Researchers (Sorensen et al., 2024; Kirk et al., 2024a; Jang et al., 2023) emphasize the im- portance of designing AT systems that cater to the unique requirements of individuals and groups. This paradigm shift has spurred the development of novel methods, benchmarks, and training datasets. Nevertheless, many existing approaches depend on pre-selected diversity-defining dumensions-such personality traits (Castricato et al., 2024; Jiang et al., 2023; Serapio-García et al., 2023; Zhu et al., 2024), and writing styles (Han et al., 2024; Jang et al., 2023; Bai et al., 2022) egorize individuals into predefined groups, potentially overlooking intra-group variability. The scarcity of large-scale preference datasets has previously hindered personalized L.M development. However, pioneering efforts by (Kirk et al., 2024b; Zollo et al., 2024) have facilitated the exploration of personalization methods beyond predefined user types.
demographics (Moon et al., 2024; Kwok et al., 2024),
cat-
Early attempts to personalize LLMs involved integrating additional inputs-typicallyy learn- able models that generate latent representations of user preferences based on past interac- cions-into the design of LLMs (Li et al., 2024; Chen et al., 2024b; Wožniak et al., 2024) or reward models (Poddar et al., 2024; Chen et al., 2024a). These strategies, however, often need substantial individual user data or rely on categorizing users based on factors such as

---

Published as a conference paper at COLM 2025
demographics, personalities, etc. To address these limitations, we introduce LoRe, novel Low-Rank Reward Modeling framework for few-shot personalization.
LoRe leverages structured low-rank decomposition of reward functions. This approach allows us to model individual preferences as weighted combinations of the basis reward functions, enabling scalable and statistically efficient adaptation with minimal user-specific data. In contrast to prior approaches, LoRe demonstrates superior generalization capabilities to diverse unseen users while maintaining computational efficiency suitable for real-world deployment. By integrating seamlessly with multi-objective alignment frameworks, LoRe supports personalized response generation without the need for extensive retraining.
Preliminaries
crucial step in aligning LLMs through Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., 2017; Ouyang et al., 2022) is learning a reward function that captures human preferences. Unlike traditional supervised fine-tuning, which relies on explicitly labeled data, RLHF enables models to learn from human comparative judgments. This is particularly valuable in settings where direct supervision is impractical, such as optimizing AI systems for subjective qualities like helpfulness coherence. In practice, human annotators provide feedback by ranking responses to the same prompt, and this data is used to train reward model that assigns numerical score to each response.
common framework for modeling such preferences is the Bradley and Terry (1952) (BT) model. The BT model represents preferences by assigning scalar scores/ rewards to stems-where an "item" " could be any decision, option, or response. Given two items i and j with scores y; and rj, the probability that item is preferred over item j follows:
1
(1)
1 exp(-(r; -r)))
In the context of LLMs, the reward function maps prompt-response pairs to scalar score, indicating response quality. This function, typically represented as rg X x y + R, is trained using human-labeled preference data. Here, & is the reward parameterization in the function class Φ Specifically, for a given prompt x E X, if human annotators prefer response Ye E y over y, y, the BT model expresses the probability of this preference as:
Plye > yr(x) 1 + exp (- (x, yy) r (x)))))
(2)
Given a dataset D of pairwise preference feedback consisting of independent samples, where each sample is triplet (x, and Ye E Y is the response preferred over y, y for the prompt x E X drawn uniformly at random. The joint likelihood of the dataset is:
II
Pp(e > yr(x).
(3)
(EDDN.)ED
Assuming preferences follow the Bradley Terry Model Eq. (1), the parameters of the reward model can be learned by minimizing the negative log- ligeliiol defined as:
min #E
Σ
log (1 + exp (x,(x,)r) (x,))))))
min ΦE
Σ (x.
((rg(x,ye)) - rx(xx)))), (4)
(x
where ((z) = log(1 + exp((=2) is the logistic loss function.
3 Preference Personalization using LoRe
While the BT model assumes a single underlying reward function shared across users, real-world preferences often exhibit significant variation due to individual experiences, biases, and cultural contexts. Next we present an overview of classical work on collaborative ranking, which provides way to model diverse user preferences.
2

---

Published as a conference paper at COLM 2025
Offiinn Datoort User Prefeemie
Monolithic Reward Model for Everyone
quea etaer
soodi cer 3
Skine is Alp..
offtine
Bowand Fuuttion
thereby parsonalization
Use X
crek cuntag! Mei with ftet Iist.
Benund Functions mags staloe monsrl
snaki tave N
Offinee Dalovr User Prefenences
LoRe: Low-Rank Reward Modelling
slate
mool
Alp.
Wetds tasiin leees
weights parsonalization
X
Stare wit food BISt.
seate be nucdad!
mape seward dimetodoos
wir
rewell
Passsakeel
Every noet leunaale dlmmnninndd weights
Poot-Shot Eeaepppes
Few shot ecamppis from new user used to learn user weights Wezsr hespingtho the reward basis trazen
New
Revanl Buis Fnero Wrights
Nim Ware
Peroosaton
Statt Soes the intrddation.
wnightt sare Snwnehoot extimpoe
Figure 1: Typically preference data from diverse users is pooled together train a single reward model for everyone. LoRe introduces a more flexible approach by collaboratively learning a shared reward basis from user data. Instead of producing a single reward, this basis generates B latent rewards, which can be combined using B-dimensional weight vector unique to each user to produce personalized rewards. This allows for seamless personalization with minimal effort. For new users, only the user weights need to be learned from few-shot examples, while keeping the reward basis fixed, enabling an efficient and lightweight personalized reward model.
3.1 Collaborative Ranking from Pairwise Comparisons
Collaborative ranking (Koren al., 2009) leverages preference data from multiple users to infer individual preferences across a large item set. Each user provides feedback on only few item pairs, and the goal to reconstruct their full preference profile by utilizing shared information across users. This approach accounts for diverse preferences, by avoiding the need to aggregate conflicting opinions into a single reward function. Instead, it models personalized rewards by learning structured representations of user preferences.
Consider N users and M items, where preferences captured in matrix P NxM The 1th row of P, denoted p,,, represents user i's rewards across all items. The probability of user i preferring item c over item r is given by the BT model Eq. (1) based on the reward difference (pic ). Since users provide comparisons for only small subset of items, recovering the full matrix P is challenging. A common solution (Lu and Negahban, 2015; Park et al., 2015)-taking into account the similarity among users and items teems-assumes is low-rank (has rank B & min(M,N) and can be factorized as:
P WR,
(5)
where the rows of R E R BxM represent a reward basis, and W E NN contains user-specific weights. Each user's preference vector is then given by:
p; w R.
(6)
3

---

Published as a conference paper at COLM 2025
Here, w determines how user i's preferences combine the basis vectors in R. The objective to learn this low-rank matrix from a small fraction of observed pairwise comparisons, enabling personalized and scalable reward learning.
3.2 Low-Rank Reward Modeling for LLM Alignment
Collaborative ranking, by exploiting the low-rank structure in user preferences, enables few-shot learning on new user even when only a handful comparisons are available. The idea is to fix the reward basis R and learn only low dimensional weight vector for this user. The main challenge in adapting collaborative ranking to RLHF is the high dimensionality of the item space, which consists of user prompts X' and LLM-generated responses y É ) forming items (x, y) E x y. User preferences are captured through a limited number of pairwise comparisons (x, yr, indicating preference for Ye over Vr given However, modern LLMs typically use vocabulary sizes between 32000-128000 (V) tokens and support context windows ranging from 2048 million (K) tokens (Touvron et al., 2023; Achiam al., 2023), the item space scales as M ve-rendeiing direct reward basis learning R IR B infeasible.
To address this challenge, we propose framework that models diverse user preferences through set of B basis reward functions represented by the Reward Basis Rş X' x'x x ) - R The individual preference function for user P: : X' y - R, is defined as:
pi = w; Rp.
where w E AB-1 is a normalized weight vector inherent to the user. By leveraging straightforward low-rank matrix factorization, we efficiently capture diverse user pref- erences. This approach, overlooked in favor of more complex methods in prior work, highlights the strength of collaborative ranking across diverse applications. The simplicity of our method is major advantage, enabling significant performance gains (see Sec. 5) while being easy to integrate with various downstream tasks (see Sec. 3.4). The space of the learnable parameters depends the case and the sample size available for training. We provide a few examples below.
Example 1. Fine-tuning only the final layer of a pre-trained reward model (Ziegler et al., 2019): Standard transformer-based models output single scalar reward. To modify the reward model to output B-dimensional representation, we learn a simple linear transformation on top of the embeddings generated by the pre-final layer, denoted as e: X y + RD, while keeping other layers frozen. The reward basis then defined as:
(x,y)) = Ae(x,y),
(8)
where A E BBD is a learnable matrix that projects y) into B-dimensional space.
Example 2. Similar to Example we modify the pre-trained reward model to output B-dimensional representation by applying a learnable transformation to the embeddings from the pre-final layer. Instead of using a linear transformation, we train a shallow multi-layer perceptron (MLP) fo on top of these frozen embeddings (Wang et al., 2024):
R y) = ff fe(xxx,))),
where the parameters of the MLP. This allows for more expressive transformations while keeping the earlier layers of the reward model frozen.
Example 3. Fine-tuning earlier layers with LoRA (Low-Rank Adaptation) (Hu et al., 2021): Instead keeping the earlier layers frozen, we can allow them to be fine-tuned in parameter-efficient way using LoRA. As in the previous examples, we first modify the pre-trained reward model to output B-dimensional representation by applying learnable transformation to the embeddings from the pre-final layer. However, in addition to learning a transformation on top of these embeddings, we also fine-tune the transformer's earlier layers by introducing low-rank adaptation matrices.
4

---

Published as a conference paper at COLM 2025
3.3 The LoRe Workflow
In this section, we outline the LoRe workflow in full generality. The specific choices made for our experiments are discussed in Section 5.
We assume access to preference feedback data from seen users, denoted by the set Useeen- Each user provides a set of labeled pairs of responses, denoted as D, = (xx Ve, Vr))r),, where is the input, and y/ and Yrr are the chosen and rejected options, respectively. The training dataset Dirain is the collection of D, of all users in Useen-
Collecting User Preference Data:
Jointly Learning Basis Reward and User Preference Weights: Assuming that the preference for each user follows the BT Model with their personalized reward function pi, the learning problem boils down to the following maximum likelihood estimation problem:
min (W.E histlren iEMacen
Σ
(Rp(( - Rp(x,y,))),
(10)
Di
(xNo.)))DD
where E(-) is the logistic loss function as described in Eq. (4).
Once the learner has learned these parameters, they are interested in generalizing to new prompt queries by the users in Useen, well being able to adapt to preferences of new users denoted by the set Munseen-
Few Shot Learning for New Users (Unseen User Generalization): This type of generaliza- tion involves predicting well for users whose preference data was not part of the training data at all, i.e., completely new users with unseen preferences. These users are termed as unseen users denoted by the set Munseen- These users have few interaction data points, that termed as D Plewshot/ and this is used few-shot learn these users' preferences (wi) by optimizing Eq. (11).
For any user E Lunseen with few feedback samples denoted by D, DD fewshhotr we estimate their preference Wnew E AB-1 keeping the reward basis Ry fixed from Eq. (10) as follows:
Wnew argmin
Σ
<(ww) (Rp((x)))))) - Rx(x,y)))).
(11)
wEAB-1 8-1 (x.,,)))D
3.4 Personalized Response Generation via Steerable Multi-Objective Alignment
Recently, there has been a growing interest in Multi-Objective Alignment (MOA) in LLMs. Formally, let R.: X' x y + R represent B reward functions for different (often conflicting) objectives. For instance, one objective may favor detailed explanations, while another prioritizes conciseness. The goal is to generate responses with varying emphasis, dictated by weight vector w IR_ yielding the reward function w Rg.. A naive approach would be to train language model for every possible w, but this becomes infeasible as the space of is infinite.
Prior work (Wu et al., 2023; Zhou et al., 2023; Rame et al., 2024; Jang et al., 2023; Shi et al., 2024) demonstrates that it is possible learn only B language models corresponding to these B reward functions and still generate responses for any arbitrary weight w at inference time. However, lhese works assume known reward models (using tff-the-shef reward models for observable objectives like harmlessness, conciseness, etc.) and provided preference weights.
In personalization, objectives are often latent and subjective, requiring learning rather than explicit specification. Furthermore, preferences are inherently subjective and can be difficult to articulate, making it challenging for users to specify precisely what weights they want. Modeling personalized rewards as linear combinations of basis functions enables adaptation to users' implicit preferences. Our research is orthogonal yet complementary to this line of work, enabling seamless integration into advances in multi-objective alignment.
5

---

Published as a conference paper at COLM 2025
4 Related Work and Contributions
Personalized Reward Learning from Human Feedback: This line of research typically captures diversity in user preferences in one of the following ways: 1.Explicitly categorize users based on observable traits (e.g,, demographics, personality traits) and train separate reward models for each category (Jiang et al., 2023; Zhu et al., 2024; Bose et al., 2024). This approach is inherently limited in granularity and struggles with cases where user preferences do not align neatly with predefined categories. 2. Learn per-user reward models by conditioning on latent representations of user preferences (Poddar et al., 2024; Chen et al., 2024a; Lee et al., 2024). These methods require significant data for each user and do not generalize well to new users with limited feedback. We elaborate on two of these methods next.
Personalized reward modeling for pluralistic alignment (PAL): (Chen al., 2024a) assume that the latent preference of each user modeled by an unknown ideal point z; E RD. They propose two methods to represent these ideal points:
PAL-A: In this approach, the ideal points factorize as z; = Qw, where Q E RDx represents prototype ideal points and w; E B represents weights over these prototypes for user
i. Given pre-trained embedding function X' y - IR,, the reward for user for prompt response pair (x.y) E X x y is given by the Fuclidean distance square between e(x, y) and Qw; in learnable representation space defined by function fe : IRD + R (a
shallow MLP with parameters 0):
px(x,y) ()fo(e(x,))) fa (Qw;)
(12)
The learnable parameters are θ, Q W, Wiekiuenn This method aims to generalize to unseen users by covering the user space with the prototype matrix.
PAL-B: Here, the user ideal point is function of the prompt, expressed as 2x Gp(e(x))w, The personalized reward function defined as:
=
px =- 2xxx fo(e(y)) = w; G (e(x)))o(e(y)
(13)
where Gp: RDD + Rd x R and fe RD + R$. Mathematically, PAL-B is special case of Eq. (7) where yx == G (e(x))f((e()) This decomposition of Rp is not novel to PAL-B and has already appeared in a prior work (Wang et al., 2024).
Variational Preference Learning (VPL) (Poddar et al., 2024): The latent preference of each user is denoted by 2, IRD, which is the output of an encoder function Qa : D; RR,, that maps the user preference data Dj to latent code. Given a pre-trained embedding function e: X' x y + RD, the reward for user i for prompt response pair (x,y) E X' x ) is given by learnable function RØ IR2D + R, as:
pi (x, := ():
(14)
The noise in the latent space encourages the reward model to learn over the entire latent space, which encourages the production of meaningful rewards for unseen users, with the learnable parameters being those of R and Qa.
Personalized Response Generation: Following the idea of Direct Preference Optimization (DPO) (Rafailov et al., 2024), which learns a response generation policy (a language model) without explicitly learning reward function, recent approaches directly model personalized responses. Li et al. (2024) use an encoder to generate latent user embeddings, conditioning a language model on them via DPO. Chen et al. (2024b) personalizes responses through user-specified prompts, while Wožniak et al. (2024) incorporates user information input features. Other works explore limited personalization settings, such as multiple-choice questions (Zhao et al., 2023), explicit human corrections (Shaikh et al., 2024), and few-shot adaptation with synthetic users (Singh et al., 2025). In Appendix A, we show that the core idea of LoRe naturally extends to response generation without explicit reward learning too.

---

Published as a conference paper at COLM 2025
Evaluating personalized response generation is particularly challenging. Unlike reward learning, where user data can be held out for validation, there no access to users in offline datasets to label their preferences newly-generated responses. Instead, evaluations rely on heuristic LLM-based evaluators, which are typically trained using monolithic Bradley- Terry reward models assuming single global ranking of responses, failing to capture diverse user preferences. Hence, as discussed in Panickssery et al. (2024); Dong et al. (2024), these models reinforce bias; favoring dominant preference patterns while undervaluing minority preferences and struggle with ambiguous or tied preferences, leading to systematic misalignment with real user satisfaction.
Contributions We state our contributions and how we overcome limitations in prior work:
1. Latent Basis Reward Functions for Personalized Alignment LoRe introduces a struc- tured approach to personalization by learning set of primitive reward functions (basis functions) that span the space of individual reward models. Each user's preference rep- resented as weighted combination of these basis functions, enabling smooth adaptation without requiring predefined user categories or extensive per-user data.
Decoupled Learning for Efficient and Generalizable Adaptation LoRe separates the learning of basis reward functions from user-specific weights, enabling rapid few-shot personalization. Once the basis functions are learned, new user's 's preferences can be captured with only a small number of interactions, making it practical for real-world deployment. By capturing the underlying structure of user preferences, LoRe generalizes effectively to unseen users with minimal data. Unlike latent-code-based methods like VPL or PAL that require separate modules for inferring user representations, LoRe directly learns a compact basis, reducing the number of learnable parameters and improving both efficiency and generalization.
3. Scalability to Large and Diverse User Populations Unlike approaches that either as- sume homogencous reward models (BT) or struggle with scalability (PAL, VPI.), LoRe maintains strong performance on large-scale personalization tasks. The low-rank decom- position reduces computational overhead while preserving expressiveness (cf. Sec. 5). Integration with Multi-Objective Alignment for Response Generation LoRe naturally extends to personalized response generation by leveraging its basis reward functions. Unlike PAL and VPI. which require additional policy networks to generate responses, our method integrates seamlessly with steerable multi-objective alignment frameworks. Bridging the Gap Between Explicit Categorization and Per-User Models Many per- sonalization methods either cluster users into predefined categories (demographics, personality types) or train separate models for each user. LoRe avoids these extremes by learning a flexible, data-efficient representation of user preferences that adapts without requiring extensive individual data.
By combining structured reward decomposition, scalable adaptation, and efficient integra- tion with response generation, LoRe offers a principled and practical approach personal- ized RLHE, addressing key limitations of prior methods while enabling new capabilities.
Experiments
Evaluation Metrics: We evaluate the reward model's accuracy on unseen response pairs for both seen and unseen users:
Σ
I[ (Rp(x,yc) - (x,xx,,)) > 0),
(15)
D
(x
where I[:) is the indicator function, equal to 1 if the condition holds and 0 otherwise.
We define four dataset splits: Dtrain contains labeled preference data from seen users Useen used to train the reward basis, Desen evaluates generalization to new response pairs for seen users, D lewshot provides small set of labeled data for unseen users Munseen, and Dunscen assesses generalization to new users. We evaluate:
1. Seen Accuracy: Generalization to new response pairs for seen users using Dseen.
7

---

Published as a conference paper at COLM 2025
Pe
Setting Method
Very Diverse (a = 0.001) Seen
Moderately Diverse (x = 0.01)
Near Uniform (a 0.1) Overall
Unseen
Overall
Seen
Unseen
Overall
Seen
Unseen
ref
78.4 = 12 76.1 1.5 77.3 = 1.3 78.1 = 14 77.8 = 16
77.9 1.5
82.7 + 83.7
111 83.2 L.0
BT
86.3 10 113 86.4 11
87.8 12
87.2
14
13
93.2 09 93.1
10 93.2 09
VPL
86.4 1.3 86.5 1.2 86.5 1.3
93.9
1.1 84.1
1 15
89.0 1.3
92.0 1: 1.0 92.8
08 92.4 0999
PAL
85.0
36.5.11
85.7
86.1
87.1
15
86.6 1:3
91.7
91.8
0s 91.7
LoRe
94.3 09 93.3 1 1.0 93.8 0.9
94.6
0.8
93.6
# 111
94.1
0.9
96.0 1: 0.7 96.1
0. 96.0 0.7
Table 1: Using PersonalLLM we generate 1000 seen users and 1000 unseen users. In particular, we use 45 examples per seen user and few-shot examples per unseen user.
Unseen Accuracy:
Generalization new response pairs for unseen users on Dunseen,
where user preferences are learned from few-shot examples in Diewshot-
Few-shot Ability: Accuracy on Diest Dunseen upon varying the number of examples in lewshot
Baselines: We use a pre-trained reward model (Liu et al., 2024) to generate fixed embeddings v(x, RD, where D 4096. We compare against 1) Reference Model (Liu et al., 2024), 2) BT (monolithic reward model, applying a learnable linear mapping from fixed embeddings (xx,y) to a scalar reward), 3) VPL (Poddar al., 2024), 4) PAL (Chen al., 2024a) (both using their respective architectures over the same fixed embeddings e(x, y)), and 5) LoRe (applying learnable linear transformation on e(x,y) map embeddings to R (corresponding to Example in Sec. 3.2). A detailed description of all baselines is presented in Appendix B.1.
Semi-synthetic Preference Dataset: The PersonalLLM dataset (Zollo et al., 2024) contains 10,402 prompts, each with responses from eight top LLMs (e.g,, GPT-4o, Claude 3 Opus, Mixtral8x22B). Each response is scored by 10 reward models from Reward Bench, built on popular base models such as Llama3, Mistral, and Gemma, with diverse preferences. Given prompt x and response y, the reward vector Rxx,y) R The dataset is split into 9,402 training and 1,000 test prompts.
Synthetic users are generated by sampling a preference vector Dirichlet(a) and com- puting response scores as w [xx,y). We vary in the range {0.1, 0.01, 0.001), where larger a results in more uniform preferences and a smaller leads to more discrete user types. We then categorize users as follows: Very Diverse (a = 0.001): Aligning closely with one of the 10 reward models. 2. Moderately diverse (a = 0.01): A balance between specific and broad preferences. Near Uniform (a 0.1): The most uniform preferences.
For each user, store the highest/ cachess/lowwer lowest scored responses and simulate 1000 seen and 1000 unseen users. Each seen user gets 45 prompts from the training set Dirain, while each unseen user gets 9 prompts to form Deewshot- All 1000 test prompts form Dseen Dunseen to test the performance of the learnt models.
Summarization Task Real Users: We use the TLDR dataset, where each comparison consists of a Reddit post, two sum maries, and the worker ID who annotated it (Stiennon et al., 2020). After filter- ing out workers with fewer than 50 annotations, we retain 40 workers. They are split into two equal groups of 20, correspond- ing to Useen and Munseen-
Reddit TLDR
Setting Method
100 examples per seen user Unseen
150 examples per seen user
Seen
Overall
Seen
Unseen
Overall
ref
56.3 + 1.3 57.3
56.8 1.2 1.2
57.3
1.5 56.8 + 1.3
BT
60.0 14 60.0 + 12 60.0 = 1.3 63.2 +11 64.3+
1.3 63.7 + 1.2
VPL
63.6 1.3 62.1 + 14 62.9 = 112 16.44442 +
62.7
+ 112 63.1 1.3
PAL
64.1 111 64.9
1.5 64.5
122 64.4
1.3 63.8
1.2 64.1 1.3
LoRe
65.0 111 66.2
1.2 65.6
66.2
66.7
66.5 1.0
Table 2: We split Reddit TLDR into 20 seen and 20 unseen users, with 50 few-shot examples per unseen user. We vary the number of examples per seen user to learn the reward basis.
8

---

Published as a conference paper at COLM 2025
Each worker has average of 4,000 labeled pairs, which evenly divided into training set and a test set. To construct Drrain we randomly sample {100, 150 pairs from each seen user's 's training data. For unseen users, randomly select 50 examples from their training data to form D Evaluation is conducted on the full set of labeled test samples, corresponding to lest and Dunseen.
Real-World Preference Dataset with a Large and Diverse User Base The PRISM dataset (Kirk et al., 2024b) is a comprehensive resource for LLM feedback analysis, featuring 1, 1,500 participants from 75 countries. It provides fine-grained feedback on both contextual and stated preferences, collected from 8,011 live conversations across 21 different LLMs.
After filtering out users with fewer than six dialogues, we randomly split the remaining participants into two equal groups, resulting in [Ugeen| [LAunseen 643 users, with each user averaging seven dialogues. For each user, half of their interactions are used for training (Dtrain D Piewshot), which informs reward model learning and the preferences of unseen users. The remaining interactions are reserved for evaluation (Desrt test
We repeat each experiment times with different random splits, reporting the mean and standard deviation in Tables 1, 2, and 3. To further analyze the scalability of the few- shot adaptation phase, we vary the number of few-shot samples and plot the average performance across 20 runs in Figure 2.
Analysis In the Le dataset, as diversity decreases, all methods improve, but LoRe consis- tently achieves the best performance across all set- tings. VPL and PAL, which are personalization base- lines, struggle to remain competitive when the num- ber of users large, as seen in both PersonalL.LM and PRISM. Their performance is often close to BT, which does not personalize at all, indicating a lack of scala- bility. However, when the number of users is smaller, as in the Reddit TLDR dataset, both VPL and PAL perform well, further reinforcing their limitations in scaling to larger personalization tasks. Addition- ally, VPL exhibits signs of overfitting, performing well on seen users but significantly worse on unseen users, highlighting its inability to generalize effec- tively. In contrast, LoRe consistently outperforms all other methods, demonstrating strong scalability, generalization, and adaptability across varying levels of personalization diversity.
PRISM
Method
Seen
Unseen
Overall
ref
58.8 + 11 57.3 122 58.0 = 100
BT
64.0 1.
61.0
62.5
VPL
64.6+10 12
6.
PAL
70.8
59.
64.92 11
LoRe
71.0 +: 099 71.0 1.0 71.0 0.8
Table 3: We split PRISM into 643 seen and 643 unseen users. On aver- age there are only 3.84 examples per seen and 3.87 examples per unseen user, making generalization to un- seen users challenging for baselines.
We also analyze the dependence of few-shot examples unseen accuracy in Figure 2. The BT and ref models are not capable of personalization to new users and hence demonstrate the same performance throughout. We observe that VPL and PAL do not change their performance much either as number of few shot examples increase. We found that the high-dimensional latent codes do not change significantly for both VPL and PAL as the number of few-shot examples increased. For LoRe, the unseen accuracy steadily increases across all datasets, as the number of examples increases.
Conclusion and Future Work
We introduced LoRe, a novel framework for personalizing LLMs via Low-Rank Reward Modeling. Our approach improves RLHF personalization by leveraging structured de- composition of reward functions, enabling efficient adaptation to diverse user preferences with minimal data. Extensive evaluations demonstrated LoRe's superior generalization to seen and unseen users while maintaining scalability and efficiency. Compared to baseline methods, LoRe consistently achieved better unseen user adaptation and preference predic- tion accuracy. It remains effective even as the number of users increases, breaking key limitation in prior work. Future directions include extending LoRe to online RLHF with explorative data collection.

---

Published as a conference paper at COLM 2025
Reddit (100 examples seen user)
Reddit TLOR (150 examplec seen user)
PRISM
Performanc (Ulnsewn
Number Few-Shot Examples LoRe(Ours)
28
Faw-Shot Examples
PAL
VPL
BT
ref
Figure 2: We vary the number of few-shot samples and repeat each experiment 20 times, randomly subsampling different examples in each run. The plot reports the average per- formance (unseen accuracy) along with standard deviations. Notably, VPL, which infers the latent code from few-shot examples without the ability to relearn it, shows limited improvement as the number of examples increases. While PAL exhibits some gains, our algorithm's 's performance improves significantly faster in comparison.
10

---

Published as a conference paper at COLM 2025
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. ar preprint arXiv:2303.08774 2023.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitu- tional ai: Harmlessness from ai feedback. arXiv preprint arXio:2212.08073. 2022.
Michiel Bakker, Martin Chadwick Hannah Sheahan, Michael Tessler, Lucy Campbell- Gillingham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslamides, Matt Botvinick, et al. Fine-tuning language models to find agreement among humans with diverse preferences. Advances in Neural Information Processing Systems, 35:38176-38189 2022.
Avinandan Bose, Mihaela Curmei, Daniel Jiang, Jamie H Morgenstern, Sarah Dean, Lillian Ratliff, and Maryam Fazel. Initializing services in interactive ml systems for diverse users. Advances in Neural Information Processing Systems, 37:57701-57732, 2024.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/ 4))3444)32--45 1952.
Louis Castricato, Nathan Lile, Rafael Rafailov, Jan-Philipp Fränken, and Chelsea Finn. Persona: A reproducible testbed for pluralistic alignment. arXiv preprint ar XrX:::2407.17377, 2024.
Daiwei Chen, Yi Chen, Aniket Rege, and Ramya Korlakai Vinayak. Pal: Pluralistic alignment framework for learning from heterogeneous preferences. arXiv preprint ar arXiv:2406.08469,
2024a.
Ruizhe Chen, Xiaotian Zhang, Meng Luo, Wenhao Chai, and Zuozhu Liu. Pad: Personalized alignment of Ilms at siroding-time, arXiv preprint ar xrxi:22410000000, 2024b.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Yijiang River Dong, Tiancheng Hu, and Nigel Collier. Can 1lm be personalized judge?
Xrv preprint ar arXto:2406.11657 2024.
Esin Durmus, Karina Nguyen, Thomas I Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Satfield-Dodde Danny Hernandez, Nicholas Joseph, et al. Towards measuring the representation of subjective global opinions in language models. arxiv preprint ar Xxiv:2306..1688, 2023.
Seungwook Han, Idan Shenfeld, Akash Srivastava, Yoon Kim, and Pulkit Agrawal. Value augmented sampling for language model alignment and personalization. arXiv preprint ar arXiv:2405.06633 2024.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. ar Xiv preprint ar Kie:2106.09689 2021.
Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Personalized large language model alignment via post-hoc parameter merging, arxiv preprint ar Xiv:2310.111664 2023.
Guangyuan Jiang, Manjie Xu, Song- Chun Zhu, Wenjuan Han, Chi Zhang, and Yixin Zhu. Evaluating and inducing personality in pre-trained language models. Advances in Neural Information Processing Systems, 36:10622-10644 2023.
Diederik P Kingma. Adam: A method for stochastic optimization. arxio preprint
ar Xrx:::1411 (tt::14126688)) 2014.
11

---

Published as a conference paper at COLM 2025
Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott A Hale. The benefits, risks and bounds of personalizing the alignment of large language models to individuals. Nature Machine Infelligence, 6(4):383-392, 2024a.
Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, et al. The prism alignment project: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models. Xiv preprint ar Xii:2440.11001 2024b.
Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recom- mender systems. Computer, 42(8):30-37) 2009.
Louis Kwok, Michal Bravansky, and Lewis D Griffin. Evaluating cultural adaptability of large language model via simulation of synthetic personas. arviv preprint 01X2:2,408.0699,, 2024.
Yoonho Lee, Jonathan Williams, Henrik Marklund, Archit Sharma, Eric Mitchell, Anikait Singh, and Chelsea Finn. Test-time alignment via hypothesis reweighting. arXio preprin! arXiv:2412.08812 2024.
Xinyu Li, Zachary C Lipton, and Liu Leqi. Personalized language modeling from personal- ized human feedback. arv preprint ar drXi::2402.05113, 2024.
Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-reward: Bag of tricks for reward modeling in llms. aiv preprint ar xxx::2410.18444 2024.
Yu Lu and Sahand N Negahban. Individualized rank aggregation using nuclear norm regularization. In 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages 1473-1479. IEEE, 2015.
Suhong Moon, Marwa Abdulhai, Minwoo Kang, Joseph Suh, Widyadewi Soedarmadji, Eran Kohen Behar, and David M Chan. Virtual personas for language models via an anthology of backstories. ar preprint ar arXi::2407.0653 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 15-27730-27744- 2022.
Arjun Panickssery, Samuel Bowman, and Shi Feng. Llm evaluators recognize and favor their own generations. Advances in Neural Information Processing Systems, 37:68722-68802,, 2024.
Dohyung Park, Joe Neeman, Jin Zhang, Sujay Sanghavi, and Inderjit Dhillon. Preference completion: Large-scale collaborative ranking from pairwise comparisons. In International Conference on Machine Learning, pages 1907-1916. PMLR, 2015.
Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, and Natasha Jaques. Personalizing reinforcement learning from human feedback with variational preference learning. arXiv preprint :xfo:2448.10077 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systemss, 36, 2024.
Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. Advances in Neural Information Processing Systems, 36, 2024.
12

---

Published as a conference paper at COLM 2025
Gregory Serapio-Garcia, Mustafa Safdari, Clément Crepy, Luning Sun, Stephen Fitz, Marwa Abdulhai, Aleksandra Faust, and Maja Matarić. Personality traits in large language models. 2023.
Omar Shaikh, Michelle Lam, Joey Hejna, Yijia Shao, Michael Bernstein, and Diyi Yang. Show, don't tell: Aligning language models with demonstrated feedback. ar preprint (a4:::2406.00888, 2024.
Ruizhe Shi, Yifang Chen, Yushi Hu, ALisa Liu, Noah Smith, Hannaneh Hajishirzi, and Simon Du. Decoding-time language model alignment with multiple objectives. aiv preprint ar Xiv:2406. 18853,, 2024.
Anikait Singh, Sheryl Hsu, Kyle Hsu, Eric Mitchell, Stefano Ermon, Tatsunori Hashimoto, Archit Sharma, and Chelsea Finn. Fspo: Few-shot preference optimization of synthetic preference data in ilms elicits effective personalization to real users. arXiv preprint dXXiv:2502.19312, 2025.
Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, et al. A roadmap to pluralistic alignment. arxiv preprint ar XrXvi:2402.0550 050000 2024.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-300 -0200,, 2020.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint ar 1r311: 2023.
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-expers. arXiv preprint arXiv:2406.12845, 2024.
Stanistaw Wožniak, Bartlomiej Koptyra, Arkadiusz Janz, Przemyslaw Kazienko, and Jan Kocoń. Personalized large language models. arXiv preprint ar xrio:2442..09269 2024.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. Advances Neural Information Processing Systems, 06:59008-59033. 2023.
Siyan Zhao, John Dang, and Aditya Grover. Group preference optimization: Few-shot alignment of large language models. arxiv preprint arXim:2310.11523 2023.
Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, and Yu Qiao. Beyond one-preference-frr for-alll:: Multi-objective direct preference optimization. aiv preprint ar arXivv2310.033008, 2023.
Minjun Zhu, Linyi Yang, and Yue Zhang. Personality alignment of large language models. ar Xiv preprint ar Xiv:2408. 11779, 2024.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human prefer- ences. ar Xiv preprint ar Xiv:1909.08599. 2019.
Thomas P Zollo, Andrew Wei Tung Siah, Naimeng Ye, Ang Li, and Hongseok Namkoong. Personalllm: Tailoring lims to individual preferences. arXiv preprint ar arXiv:2409.2022 2024.
13

---

Published as a conference paper at COLM 2025
A LoRe directly for language generation
A language model is policy network that assigns the likelihood of a response given prompt x X' denoted by 7t X' + A Given a reward function rg X x + R and reference policy ref (typically obtained by supervised finetuning a pre-trained model), the objective is to learn a policy n that maximizes the KL-regularized reward maximization problem:
0- arg max x.y. (rp(x, fID (
(16)
BEO
Rafailov et al. (2024) shows that the optimal policy for the KL-regularized reward maximiza- tion problem satisfies
TT. (y|x) (yex(y (x(x)
rx(xx))) B
TT. (y)2x) + (Zzp(x), (17)
exp
rx(xx -Blog
ret (y|x)
where (x) is normalization factor. Since there is direct correspondence between the reward function r. and the optimal policy that maximizes the KL-regularized reward, Rafailov et al. (2024) proposes Direct Preference Optimization (DPO), which directly learns the policy πg. without first learning reward model r on offline preference data. Then, by using this closed-form expression for rg (x, y) and applying minimizing negative log- likelihood to the Tradley-Terry model, we can obtain
min NEAp (XDOON))DD
Σ
(xx((x,))) -
(Blog To(yclx) (ye()y)
Blog no(yr[xx) (er((y/(x))
-min BEO
Σ (D
(by Eq. (17).)
Learning a Policy Basis: Instead of first learning a reward basis function R : X' x y RD, is equivalent to learning (fine-tuning) basis policies { X A 1 which produces B likelihoods, each corresponding the optimal policy for each of the latent reward dimensions.
Conditioned on a prompt x, and pair of responses (y, f), the probability of user with preference weights w AB of choosing y over y under the Bradley- Terry model can be written as:
1
Ply > 7/
1 + exp (-wT (R - ()))
(by Eq. (17).)
Tg. (v(x) [ert(Mi
Me. x
+ exp
(BB w(i)
log
log
where (i is the fth entry of the vector w. Analogous to Eq. (10), given a dataset of offline preferences, the parameters and { (wi]ietueen can be learnt by plugging in the negative
log- log-likelihood loss as
TT. (Melx) I 10!" Blog [xerlyc(x) - log [E(B
Teer(y, ))
FTg (U-11x)
min
Σ
Σ
(18)
(0:0)) (W.CAP Dettoom ieldvrn
Dj
JED,
For a new user, 0 is frozen, and only the user weight Wnew is estimated using few-shot examples:
(
To, (yclx)
[T)(()) )
To, To, (Mrlx))
Wnew argmin Σ &
wi) Blog
(Te(()) B Blog
(19)
wEAB-1 (ED,
|
14

---

Published as a conference paper at COLM 2025
B Additional Details on Experiments
Evaluation Metrics For any user i E Useen Uunseeen (seen or unseen) we evaluate the performance of the learned reward basis Rs and preferences Wi, on their unseen pairs of responses Dj (ie. data not present in Derain or D teewhhot) as the fraction of responses the learnt reward model classified correctly, defined formally
Σ
II [[w; (x,(x, - (x(x, > 0].
(20)
DD
(x
To test how well the learnt parameters generalize, we consider the following:
1. Generalizing to Unseen Pairs of Responses for Seen Users (Seen Accuracy): This type of generalization involves predicting well for new pairs of responses for users whose preferences have already been learned from the training data. We do so by evaluating the learnt model's (f, [wi) from Eq. (10) classification accuracy (via Eq. (20) on Desee which contains the seen users' labels on prompts and response pairs that are not present in the training dataset Dtrain.
Generalizing to New Users (Unseen Accuracy): This type of generalization involves predicting well for users whose data was not part of the training data D iee, com- pletely new users with unseen preferences. These users are termed as unseen users Munseen- These users come with few labelled data points, that is termed D lewshot and this is used to few-shot learn these users' preferences {wi]iellmmenn by optimizing Eq. (11), keeping & fixed. We evaluate the learnt preference weights' accuracy (via Eq. (20)) D test which contains the unseen users' labels on prompts and response pairs that are not present in the training dataset D Piewshot-
Performance as the number of dialogs increases for few-shot estimation: It is natural that a new user gradually builds up feedback data on LLM server. So far we were working in the setup where the LLM has already collected some feedback for the user based on some conversations, and see how the estimated preferences generalizes to future conversations. We consider how the performance varies as the LLM provider builds up multiple conversations with users. We do so by increasing the dataset size for D; E Diewsho while learning w; via Eq. (11) for all E Uunseen-
B.1 Architecture of Baselines and Training Hyperparameters
Functional Form of Reward Basis Function (LoRe): For all our experiments, we use a pre-trained reward model (Liu et al., 2024) to output embeddings of dimension D 4096, which we denote e X' x ) + Rº. We keep this embedding function frozen and learn linear transformation on top of these fixed embeddings. The reward basis function is thus defined as:
(x,y)) Ae(x,y),
(21)
where A E BBBD is a learnable matrix, representing the linear transformation applied to the pre-trained embeddings. By keeping the embedding function fixed and only learning the linear transformation, we can effectively adapt the pre-trained reward model to user preferences from specific datasets while leveraging its rich feature representations.
Hyperparameters: A is learned by using Adam (Kingma, 2014) on Eq. (10) with learning rate 0.5. For few-shot adaptation, we use Adam with learning rate 0.1 on Eq. (11).
The number of basis B is selected from ( through cross validation a held out validation set.
Experimental Setup for baselines: We follow the exact training code, model architecture, and hyperparameters as described in the original implementations of PAL (Chen et al., 2024a) and VPL (Poddar et al., 2024), and run them on benchmark datasets without modification. Detailed hyperparameter settings for PAL and VPL are listed in Table and Table 5, respectively.
15

---

Published as a conference paper at COLM 2025
Table The training hyperparameter setting of PAL (Chen al., 2024a).
Hyperparameters
Values
B (Number of prototypes)
selected through cross validation from 10, ,0)
Batch size
Projectors
mip-llyye-geul-dhopouuu
Learning rate of projectors
le-4
Learning rate of user weights
5e-3
Weight decay of projectors
0.01
Weight decay of user weights
0.0
Dimension of preference embedding
512
Table Hyperparameters for VPL (Poddar al., 2024).
Hyperparameter
Value
Pair Encoder Architecture
2 llaer MLP with LeakyRel.U
Hidden Dimension
512
Latent Dimension
512
Learning rate
1.0000 10
Learning rate scheduler
Cosine with 3% warmup steps
Batch size
32
Optimizer
AdamW (with weight decay 0.001)
Parameter Efficiency of LoRe As shown in Table 6 and Figure LoRe is significantly more lightweight in terms of total trainable parameters compared to PAL and VPL. While PAL and VPL rely on large MLP architectures and per-user latent representations, LoRe uses only simple linear projection on frozen embeddings, combined with small set of basis-user interactions. This design leads much more compact model, especially as the number of users increases, enabling scalable personalization without compromising effectiveness.
Table 6: Scaling of Total Trainable Parameters for VPL, PAL, and LoRe. Here, N is the number of seen users and B refers to the number of prototypes in PAL, and number of basis (rank) in LoRe.
Method
Architecture Details
Parameter Count
VPL
layer MLP on frozen embeddings
4096 x 512 512 512
PAL
2 layer ML.P on frozen embeddings
4096 x 512 + 512 512 4096 x B
(B prototypes)
LoRe
Linear Transformation on frozen embeddings
B 4096 B
(B basis)
16

---

Published as a conference paper at COLM 2025
Parameter Scaling with Number of Users
Total Trainable Parameters (log scale)
VFL
PALL
200
400
600
800
1000
Number Seen Users (N)
Figure 3: Trainable parameter count vs. number of seen users (log scale). LoRe scales significantly more efficiently than PAL and VPL as the number of users increases. Unlike VPL and PÁL, which rely on large MLPs and high-dimensional prototype representations, LoRe uses a lightweight linear projection with shared basis vectors, resulting in dramatically fewer parameters while retaining personalization capabilities.
17


Reviews:
[{"review_id": "gFz45dr426", "reviewer": "Reviewer_YesT", "summary": "This paper introduces LoRe, a novel framework for personalizing LLMs by leveraging low-rank reward modeling. The core idea is to represent individual user reward functions as weighted combinations of a shared set of basis reward functions, learned collaboratively from multiple users' preference data. This approach aims to enable efficient few-shot adaptation for new users and improve generalization compared to monolithic reward models or methods requiring extensive per-user data. The authors validate LoRe on multiple preference datasets, demonstrating promising empirical performance, particularly in predicting user preferences for unseen users with few examples. While the work tackles an important problem and shows strong results, further clarification on methodological choices, experimental design, and baseline comparisons is needed to fully assess its contributions. I am willing to update my score if the authors address the questions and points raised.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Reads well: The paper is generally well-written and easy to follow.\n* Shows promising empirical results: LoRe demonstrates significant improvements over baselines, especially in few-shot generalization to unseen users across multiple datasets. On the chosen metric, LoRe outperforms the baselines by substantial margins.\n* Simple to implement on top of existing LLMs: The proposed method appears conceptually simple and potentially straightforward to implement, particularly the variant using a linear transformation over frozen embeddings.\n* Addresses a significant problem: Personalizing LLMs to individual user preferences is a crucial step towards more effective and satisfactory human-AI interaction.", "reasons_to_reject": "* Clarity of methodological advantage:\n    * he paper could better articulate the precise advantage of LoRe's specific low-rank decomposition (Eq. 7 and its instantiations) over, for instance, a monolithic reward model with a similar increase in parameter count allocated differently. The intuition behind Eq. (10) – why this particular structure of shared basis functions $R_\\phi$ and per-user weights $w_i$ is optimal or particularly well-suited for capturing diverse preferences beyond simple parameter sharing – needs more explicit justification.\n    * There is a risk of overfitting, especially when learning reward models using the BT model loss. The paper does not discuss what, if any, regularization measures were applied during the training of the BT baseline and LoRe to prevent overfitting.\n\n* Concerns regarding experimental design and baseline comparisons:\n    * The design of the synthetic data experiment (PersonalLLM) inherently assumes a linear relationship between the 10 base reward models and the target personalized rewards, which naturally favors LoRe's linear low-rank assumption. While useful for validating the concept, this should be explicitly acknowledged. More insightful would be an analysis of how performance changes as the rank B is varied, especially around the true underlying rank (10 in this case).\n    * It's unclear how the baseline methods (VPL, PAL) utilize few-shot examples in the experiments depicted in Figure 2. VPL, for instance, shows no improvement with an increasing number of few-shot examples, and PAL's performance fluctuates. This raises questions about the fairness or optimality of the few-shot adaptation process for these baselines in the comparison.\n    * There appears to be numerical discrepancies in LoRe's performance on the Reddit TLDR dataset between Table 2 (accuracies >90%), Table 3 (accuracies ~71%), and Figure 2 (accuracies ~64%). This needs clarification or correction.\n\n* Potential for deeper analysis: The \"Analysis\" subsection (lines 328-352) in Section 5 is very compact. Expanding on these observations could strengthen the paper, possibly by condensing other sections like detailed related work discussions, which could be partially moved to an appendix.", "questions_to_authors": "* Regarding Methodology (Section 3):\n    * Eq. (11) seems to solve a simple logistic regression problem using user-specific data to determine $w_{\\mathrm{new}}$. How does this approach perform effectively with very few examples, such as a single few-shot example as indicated for the PRISM dataset in Figure 2? Do you have any insights into the minimal sample size necessary for robustly learning an effective weight vector $w_{\\mathrm{new}}$?\n    * The placement of Section 3.4 (\"Personalized Response Generation via Steerable Multi-Objective Alignment\") feels somewhat tangential to the main reward modeling contribution. Would this fit more naturally in the Related Work section?\n\n* Regarding Related Work (Section 4): The location of the \"Related Work\" section (Section 4) is somewhat unusual as it falls in between the methodology and related work. Is there a specific reason for this placement rather than a more conventional position (e.g., after the Introduction or before the conclusion)?\n\n* Regarding Experiments (Section 5):\n    * For the evaluation metric in Eq. (15), is the reported accuracy an average of per-user accuracies, or is it calculated on a test set where all user samples are pooled? (The text implies per-user then averaged, but good to confirm).\n    * In the PersonalLLM synthetic data experiments, how does LoRe's performance change as the chosen rank B is varied (e.g., from small values up to and beyond 10)? Was any specific regularization used when learning A or $w_i$?\n    * For the Reddit TLDR dataset, what is the performance of the monolithic BT model (and LoRe) on the aggregated test dataset (i.e., all samples pooled together, not averaged per-user accuracy)? This would help understand if the BT baseline was trained effectively on the general task.\n    * Could you elaborate on how the VPL and PAL baselines are adapted using the few-shot examples in your experiments for Figure 2?\n    * Please clarify the y-axis values in Figure 2 in relation to the values in Table 2 and Table 3.\n    * Would you consider expanding the \"Analysis\" section to provide a more in-depth discussion of your findings?\n\nI believe addressing these points and questions would significantly improve the clarity and robustness of the paper.", "extra_scores": {}}, {"review_id": "T5txhD43z6", "reviewer": "Reviewer_yyA8", "summary": "This paper proposes LoRe, a low-rank reward modeling framework, to model individual preferences. LoRe trains an efficient preference representation from seen users and can adapt quickly to unseen users. The proposed approach, while simple, is novel when comparing with previous methods. This paper presents strong empirical results of LoRe in terms of preference accuracy. It also demonstrates LoRe's scalability to large and diverse user populations. This paper intentionally focuses on reward modeling but deemphasizes personalized response generation.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This paper is easy to follow. Section 4 includes a comprehensive survey of related work.\n* The proposed low-rank reward modeling approach seems effective though simple.\n* The experimental results on the reward model accuracy are promising, covering multiple datasets and showing its scalability.\n* Table 1 with varying $\\alpha$ is especially interesting with a semi-synthetic dataset.", "reasons_to_reject": "* This paper does not provide results on personalized response generation with the LoRe although discusses about it in Section 4 and Appendix A. It would be great to have a human evaluation on this.\n* Would be better to present the result on selecting the number of basis B. Please also list the values of B chosen in Section 5.", "questions_to_authors": "L144: How does LoRe optimize the normalized weight vector $w_i$ subject to probability simplex?\n\nL200: $G_\\psi$ -> $G_\\phi$.", "extra_scores": {}}, {"review_id": "IO4suT5WJZ", "reviewer": "Reviewer_Vjsj", "summary": "This paper addresses the challenge of tailoring LLMs to individual user preferences. The authors introduce a novel framework employing low-rank preference modeling, learning user-specific reward functions by representing them in a lower-dimensional space as weighted combinations of shared underlying components. The proposed approach is designed to avoid rigid user grouping, enhance scalability, and enable rapid adaptation to new users with minimal data. The authors validated their method using multiple preference datasets, finding it demonstrates superior ability to generalize to unseen users and achieves higher accuracy in preference prediction tasks compared to existing approaches.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tUser customization of reward models is an important problem in this field.\n2.\tThis paper possesses originality and a high level of completeness, while also being clearly written and easily comprehensible.", "reasons_to_reject": "1.\tThere is more room for discussion regarding the training of the basis reward model. See the Questions section for details.", "questions_to_authors": "1.\t[important] I have some questions regarding the training of the basis reward model:\na)\tAre the trained basis reward models inherently capable of spanning the entire reward model space?\nb)\tIf so, how is this guaranteed within the LoRe framework?\nc)\tIf not, for example, when the scoring distributions of the trained reward models exhibit strong convergence (i.e., become very similar), how can it be ensured that a new user's reward model can still be effectively represented through a linear combination?\n2.\tCan the authors provide extra evaluation result of the LLM policy trained on the LoRe reward model (comparing to the baselines)?", "extra_scores": {}}, {"review_id": "IaBdlqXB2Q", "reviewer": "Reviewer_q8WV", "summary": "This paper introduces a novel framework called LoRe aimed at personalizing Large Language Models (LLMs) by learning low-rank reward basis functions. The approach draws inspiration from Collaborative Ranking, modeling users' personalized preferences as weighted combinations of shared reward basis functions. The authors argue that this method avoids the rigidity of pre-categorizing users, effectively handles the diversity of user preferences, and enables efficient few-shot adaptation. The paper validates LoRe's superiority in unseen user generalization and preference prediction accuracy through experiments on multiple preference datasets, comparing it against baselines like BT, VPL, and PAL.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. LLM personalization is an important and challenging research direction. This method offers a new perspective to address the limitations of existing approaches.\n2. The paper is well-structured, the method description is easy to understand.", "reasons_to_reject": "1. The lack of discussion on the effects of these more complex models leaves readers uninformed about the trade-offs between performance and complexity.\n2. The paper claims that LoRe can be seamlessly integrated into Multi-Objective Alignment frameworks for generating personalized responses, but this is only a conceptual discussion and not validated through experiments.\n3. There is a lack of deeper analysis on how the choice of Rank B affects model performance.", "questions_to_authors": "Please refer to the reasons to reject.", "extra_scores": {}}]

Author rebuttal:

Meta-review: While terse, all the reviews (except YesT) recommend acceptance. With the reviewer who recommended rejection, while the review was substantial - there appears to be a good faith attempt by the authors to engage and address the reviewer's concerns. While there is limited overall novelty, the work has potential to be practically useful by way of providing researchers with more efficient ways of using low rank parameter efficient tuning methods for personalizing LLMs. Many of the small details are generally abstracted over and hotly debated about, so such a work has value.
Decision: Accept