Capturing Individual Human Preferences with Reward Features
Authors: Andre Barreto
Vincent Dumoulin
Yiran Mao
Mark Rowland
Nicolas Perez-Nieves
Bobak Shahriari
Yann Dauphin
Doina Precup
Hugo Larochelle
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning from Human Feedback
Reward Models
Preference Learning
Preference Optimisation
Primary area: reinforcement_learning
TL;DR: We formally study the problem of learning an adaptive reward model and propose a concrete architecture for it.

Abstract: Reinforcement learning from human feedback usually models preferences using a reward function that does not distinguish between people. We argue that this is unlikely to be a good design choice in contexts with high potential for disagreement, like in the training of large language models. We formalise and analyse the problem of learning a reward model that can be specialised to a user. Using the principle of empirical risk minimisation, we derive a probably approximately correct (PAC) bound showing the dependency of the approximation error on the number of training examples, as usual, and also on the number of human raters who provided feedback on them. Based on our theoretical findings, we discuss how to best collect pairwise preference data and argue that adaptive reward models should be beneficial when there is considerable disagreement among users. We also propose a concrete architecture for an adaptive reward model. Our approach leverages the observation that individual preferences can be captured as a linear combination of a set of general reward features. We show how to learn such features and subsequently use them to quickly adapt the reward model to a specific individual, even if their preferences are not reflected in the training data. We present experiments with large language models illustrating our theoretical results and comparing the proposed architecture with a non-adaptive baseline. Consistent with our analysis, the benefits provided by our model increase with the number of raters and the heterogeneity of their preferences. We also show that our model compares favourably to adaptive counterparts, including those performing in-context personalisation.

Capturing Individual Human Preferences with Reward Features
André Barreto Google DeepMind
Vincent Dumoulin Google DeepMind
Yiran Mao Google DeepMind
Mark Rowland Google DeepMind
Nicolas Perez-Nieves Google DeepMind
Bobak Shahriari
Google DeepMind
Yann Dauphin Google DeepMind
Doina Precup Google DeepMind
Hugo Larochelle
Google DeepMind
Abstract
Reinforcement learning from human feedback usually models preferences using a reward function that does not distinguish between people. We argue that this is unlikely be good design choice in contexts with high potential for disagreement, like in the training of large language models. We formalise and analyse the problem of learning a reward model that can be specialised to a user. Using the principle of empirical risk minimisation, we derive probably approximately correct (PAC) bound showing the dependency of the approximation error on the number training examples, usual, and also on the number of human raters who provided feedback on them. Based on our theoretical findings, we discuss how to best collect pairwise preference data and argue that adaptive reward models should be beneficial when there is considerable disagreement among users. We also propose a concrete architecture for adaptive reward model. Our approach leverages the observation that individual preferences can be captured as a lincar combination general reward features. We show how to Iearn such features and subsequently use them to quickly adapt the reward model to a specific individual, even if their preferences not reflected in the training data. We present experiments with large language models illustrating our theoretical results and comparing the proposed architecture with non-adaptive baseline. Consistent with our analysis, the benefits provided by our model increase with the number of raters and the heterogeneity of their preferences. We also show that model compares favourably to adaptive counterparts, including those performing in-context personalisation.
set of
Introduction
Reinforcement learning from human feedback (RLHF) can be useful when we do not have explicit reward function but can distinguish between good and bad behaviour (15). This is often the case in the training of large language models (LLMs), in which RLHF has been applied with great 58).
success 46
The most common way to collect human feedback is to ask humans to rank examples of an agent's s behaviour. Usually, all the feedback is integrated derive a single reward function that reflects as well as possible the preferences of the population of interest. Although this is a sensible strategy when preferences are homogeneous across the population, may be less effective when there considerable disagreement. This is likely the case in the training of LLMss.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

As an illustration, suppose that, given a pair of alternative responses to subjective question, 51% of the target audience prefer the first option while the remaining 49% prefer the second. If we do not distinguish between users, we are left with two options: either we pick the preferred answer and leave 49% of the users unhappy 100% of the time, or we sample the answers proportionally to how often they are preferred and leave 100% of the users unhappy approximately half of the time. Both solutions are clearly unsatisfactory.
We need reward models that can be specialised to users. Crucially, we want the model to be able to adapt to people outside of the group who provided feedback for training. In this paper we formalise and analyse the problem of learning a reward model with this ability. Using the principle of empirical risk minimisation, we derive a probably approximately correct (PAC) bound that shows how the approximation error depends not only on the number of training examples, but also on the number of human raters who provided feedback on them. This is an interesting problem from a theoretical standpoint because the data involved is not identically and independently distributed (i.i.d.). The resulting analysis provides formal framework to discuss strategies for pairwise preference data collection and to assess the trade-off's associated with the use of an adaptive reward model.
We then turn our attention to the question of how to adapt a reward model to a new user in practice. Generalising to unseen users is challenging because it requires one to capture the subjective criteria that underlie human preferences- a non-trivial endeavour, as humans often cannot fully articulate the reasons why they prefer one behaviour over the other. A possible approach is to leverage precisely the type of pairwise preference data used in RLHF, since it allows the preference criteria be indirectly elicited rather than explicitly spelled out. Building on our theoretical findings, we propose a simple, principled method to unveil the preference criteria underlying the RLHF data.
Our model leverages the fact that individual preferences can be captured as a linear combination of a set of general reward features. These features are unknown (in some cases to the person themself), so we need a systematic way to extract them from the data. We show how this can be accomplished using a very simple architecture and training. During the regular RLHF process, we use data coming from different individuals to learn common features that capture the preferences of the group. When the reward model is being specialised to an unknown user, the features are frozen, and only the coefficients of the linear combination must be learned. This results in a simple classification problem that can be reliably solved with a few training examples provided by the user.
2 Background
Our goal is to use human preference data to learn a reward function that can be used in RL to learn a policy or LLM 15 (56). We will focus on and adopt the terminology from LLMs, although our analysis and proposed approach are more general. Let T be finite set of symbols called tokens, let X = U T' be the context space. where lg is the maximum context length. and let 3 ;= u T be the response space. An LLM is a mapping from X' to a distribution over V. Let H. be the set of human users we are interested in. Given a context x E X' and two responses v') ) we can ask a user h E H for their preferred option. We use y > y [a., h to indicate that h prefers over y' given x 47 2. We encode this outcome with the value = = and encode the opposite preference as z 2=0 O. We want to model humans' preferences as accurately as possible.
Define 2 = {0, } and let D be a distribution over H x X' x )22 x Z. We will use subscripts to refer to the marginal distributions of 'D and superscripts to indicate conditioning. For example. D is the marginal distribution over X' conditioned on H h. We will use numeric superscripts to indicate the joint distribution induced by independent draws from the same distribution; for instance, (D) )a is the distribution over X"" resulting from n independent draws from D'x-
We will formalise the problem through the lens of empirical risk minimisation [11) Let Co {X x ) - [0. 1}} be a classifter set (or a hypothesis class) and let é [0, x Z - R be an appropriately defined loss function. Our goal is minimise the generalisation loss defined as Lp(e) := EH...Y.Y',..a (((((, K,Y, Y, Y,Y"), To do so, we will use data generated as follows. First, a user is sampled, h ~ DH- Second, n contexts are sampled, (x;)²= ~ (D²)". Third, for each context Fú, two responses are sampled, yr, v/ (D)...)2 )² We then show each of the n resulting tuples (I, Vi. 5/) to h, who ranks them by sampling z D Dh........ (we will refer to the users h who ranked the examples as "raters"), Note that Dz
A. is a Bernoulli distribution whose mean is
2

---

F(y; > y |2 h), the probability that h will prefer ys over v given 2,. This entire process is repeated mn times, resulting the dataset S. == {[z Ys: V., 2,) mn We then use So to define the training k=)
loss: T.s, (c) = 1 mm mn E(c(aT, Vr. v(), =.), which we will minimise as proxy for IDD [51.
The Bradley-Terry model. The ultimate objective of the process above is to derive a re- ward function r : X' x y - R to be used in RL. This is possible if we assume that hu- mans' preferences are generated based on r. A common assumption is the Bradley-Tery model: F(y > zz) [r((z, y) r(z,'(()))))) y'),, where a is the sigmoid function [10]. Let rg : X x y R of classifiers Co whose ele- We can write the likelihood of 0 with respect to z as (()) = P(2 x. v, = (2; y) re (x; y (r)))(((r(()))))))) v²) re( -))+-= Since the samples in the dataset So are ii.d. the likelihood function of 0 given the data can be written as C(0 (6) II, ((z Ií Vr, y/: 0). Thus, if we define
be reward function parameterised by 0 E R This induces
ments are co (z. y. v/) == ((ra(1, y) -ro(z,y'))
(I) log (()). The minimisation of T.s, (co) yields a vector of parameters 0" E RR. The corresponding ræ- can then play the role of the reward function in RL, resulting in an LLM that reflects the preferences of the population H. [15] [15].)..
V. = -2 log co y. v/) + +2 log co (z, v)) naturally have (ce) = -log _(0) Sa)) and Lp(ce)
3 Reward-model personalisation
One of the key assumptions underlying ① is that
(2) That is, we are modelling the probability of response being preferred over response y' given context the average opinion among users H computed according to Dz. Clearly, 2) will only be good assumption if the distribution Di is representative of the target users, which may or may not be the case in practice. We call attention to another issue: when we use (1) to derive reward function, the accompanying assumption ( implies that we are neglecting individual differences in the preferences of the population H. Formally, we are ignoring the variance of the variable z =(H) == ply > y' xr, H).
Py > y/ z)) = EH-DA [P(y > y xz H)].
Preference modelling as a game. Let us focus on the simple problem of choosing which of two possible responses and y should follow a context z. An intuitive way to understand the deleterious effect of making this decision based on 2 is to think of it as a game. At each round, the player picks either y or y/. Then, a user h ~ Di is sampled and a reward is drawn from D2 v... Ji if was selected, and from D" .uch otherwise. Clearly, the best one can do in this game is to always pick y HH) > 0.5, and always pick y otherwise. This results in a expected reward of max (E))), We are proposing to change the game in a way that we only choose between and 3/ after the user has been revealed. In this case the optimal strategy is to pick y if 5(h) > 0.5, and y' otherwise. This increases the expected reward to E (max(2(H),
(H)).
((H)) 2 max max (E:(H), 1
User-aware reward model learning. As the game example above suggests, our strategy will be to model Pyy > y x, h) instead of Pyy > s xx). We will use data similar to that used in conventional RLHF, with small extra requirement: tuples in the dataset have to be labelled with (non-identifying) rater IDs. We have S == {[(he, Fiy. Vij. Kky 2zg) 1- I where hy is effectively an index indicating which rater determined 2ay based on Zay. Biy and Klzz for j 1,,,..., n. This is not a strong requirement: we are just surfacing a piece of information used to generate Sp-
Given S, we can define an augmented set of classifiers C == (H X' y22 - RR} which are now also a function of users h E H. Let 72 C H be the set of human raters h who provided feedback for the generation of S. We distinguish between two types of generalisation. In intra-user generalisation one is interested in minimising E E (H, X, Y, Y"), H) E H.. That is, although we are able to predict preferences over contexts x and responses y beyond the training data, we are unable to extrapolate to users not in the set of raters H. In inter-user generalisation one is interested in minimising Lp == Ep [f(cc H,, X, Y. Y'), Z).. That is, we able to predict the preferences of any user in H. over the entire set X' x ) Our focus is on inter-user generalisation.
The current practice of not distinguishing between users accomplishes neither intra- nor inter-user generalisation, for the expectation in T.D runs over H but the loss itself is agnostic to users. This is akin to postulating single user whose preferences coincide with the average preference in H. (cf 2).
3

---

Theoretical analysis. As discussed, our goal is to minimise Lp, but we only have access to Ls. We now analyse how well the latter tracks the former. Specifically, we will show how the difference [ID(c)) - [c)(c) behaves with respect to the defining characteristics of the learning problem.
This is non-standard learning theory analysis because the data in S is not i.i.d. First, we need to guarantee that the loss is bounded, so we replace the log(-) appearing in ① with a counterpart defined as lóg(-)= == max max(log(-)- fmin)/ where fmi is large enough in magnitude to render the capping of the logarithm inconsequential: this approach is related to the truncation sometimes used in the learning theory of classification [.. With this change T is guaranteed to lie in [0, 1]. We then define
v(e.) = XX Y, Y"), Z)). with (H, x, Y, Y', ) ~ D, where V(.) stands for variance.
Proposition 1. For any C C, > 0, O, and 8 E (0, 11. we have with probability least 1 8,
LD((c) - Ls(c) S
3m 9 + g² + 18gm
1
[ + (EIG HI)
(3)
where g = ln(2/5).
The proofs of our theoretical results are in Appendix A We will refer to the bound on the right-hand
side of (3) as In line with similar results in the literature, is O(In(1 (1))) as 6 -> 0+ [51]. Perhaps more surprising is the fact that, as the number of examples per rater n tends to oo. e approaches a constant. On closer inspection this makes intuitive sense: with fixed number of raters m, there is an irreducible error that cannot be fully eliminated even if we collect an infinite number of examples n per rater. In contrast, - 0 as with the literature, and also makes sense, as we can sample the same rater h over and over (and thus even with single
oo, regardless of n, a rate of O(1/vm). This is consistent
example per rater we will eventually have perfect estimation).
Another interesting observation for the discussion herein is the dependency of on the terms E[) and (([e(e_ H)). Intuitively, [[V(Ec) H)) measures how much the loss varies per user, on average, while HH) measures how much the average loss varies across users. If we keep all other variables fixed in (3). we have that c x /E[V(E, [H) + V(E(e))
Clearly, we want to make both terms as small as possible. This sheds light on the discussion above: while increasing the number of examples per rater n weakens the effect of (())) on the estimate Ls(c), it does not change the effect of (E(e])))) which can only be mitigated with larger number of raters m.
The discussion above invites the question of how to best allocate a budget of k = nm training examples. The right-hand side of our bound (3) is minimised when n = 1 and we should sample as many raters from H as training examples (with replacement). While it may not always be fcasible to have many raters providing only few examples each, understanding the trade-offs elicited by our theoretical results may guide the allocation of resources in practice.
k, which suggests
The bound in 3 shows how the empirical loss Ls deviates from the true generalisation loss LD for a single classifier e C. Since we are usually interested in finding a classifier that minimises LD. ideally we would be able to bound (c) - Ls(c)] over the entire set C. This is what we set out to do next. Let e" = arg min [c((c) and let č* == arg min (s(e). That is, c* the true minimiser of LD in C and 2* is the classifier we obtain by minimising the empirical loss Lg. We will bound the difference (D()") - LD(c").
First define &c == larg supeec V(C.). We will also need the concept of covering number for C. Given E 1], let C. S C be the smallest set of classifiers such that VeecilrEc.
[e(e(h, x. v)) e(c x, V,,,
a.
In words, Ca is the smallest set that covers C with respect to E: for every C, there is at least one d' E Ca whose loss is within a distance of a from that of c. We use |Cal as a measure of the complexity "capacity" of C. It follows from the definition that there exists a E [0, 1] such that |Cal < xo (for example, |Ca=1l = 1).
Proposition 2. For any m, n > 0 and any 8 (0, 1}, we have with probability at least 8 that
92 + 18mga 2 +(CK(/))))))++( + 2a
LD(²*) S TTD(c") inf
ga + $1
3m
where 9a == In (2|Ca 1/8).

---

Expression 4 is similar to B.. the main difference being its dependency on [Cal, our measure of the complexity of C. Fortunately the bound grows with [n([C,, 1). which means that we can afford to substantially increase the "size" of C without incurring too high cost in terms of the number of raters and the number n of training examples per rater needed. The precise value of [Cal] will depend
) | ||]]) (see Proposition 3 in Appendix A.2). It is possible derive bounds tighter than (4) if one is willing to make assumptions about C, use measures of complexity other than Cal. or both. See Proposition 4 in Appendix A.2 for an example in which combining logistic regression models and the notion of Rademacher complexity leads to improved variance dependence.
on a and the set C. We can show that |C,| < (1/xx)
The best possible generalisation error [D(c") will depend on two factors: how well the classifiers in C can model different users and how heterogenous the population H. is in terms of preferences. Consider a set Co whose classifiers cannot distinguish between users. We will construct more general set C of classifiers that can differentiate between users by imposing that, for any e E C, -(h,-) Co (in words, this means that any classifier in C reduces to a classifier in Co if we fix the user h). We will also make sure that all classifiers in Co also belong to C, so we have Co C C. It follows from the latter assumption that, by definition, |Cal 2 (CCo)al and V(Ec) 2 V(fc.).
Let us now analyse two scenarios. In the extreme scenario where all users in H have the same preference, it is clear that the best classifier in C' also belongs to Co. so considering the former instead of the latter will not decrease [c(c"). Using the fact that in this scenario ((E((c, |H] = 0, it is easy show that replacing Co with C cannot decrease, and will in general increase, the right-hand side of (4) That is, we may need more training examples to achieve the same generalisation error. This formalises the intuition that, when users agree with each other, having a smaller classifier set Co whose classifiers are agnostic to users is often advantageous.
Next turn our attention to the more realistic scenario where users do disagree with each other. In
this case considering C' instead of Co can yield a lower optimal population loss LD(c*), ). as illustrated by the game example following Equation (2) Interestingly, because in this scenario (EEle,, not necessarily zero, and in fact can be quite large, it may be the case that V(ETec H]) < V(Elfc, (H)
is
(f,,))HH) as V(ee) 2 V(fc,)). This means that, for large enough n, replacing Co with C can in fact decrease the effect of the loss variance on the bound (4)
(which implies that [[v(ec) H)]
The overall bound will not necessarily get smaller, though, as |Ca| will show up instead of (Co)all Regardless of its effect on the bound (or, more generally, on the resulting sample complexity), adopting set C of classifiers that distinguish between users may be highly beneficial when users disagree with each other, as the decrease in the optimal generalisation error (c()") can be substantial.
4 Reward-feature models
The theoretical results presented in the previous section apply to any classifier set C; we now discuss specific ways of defining this set and argue for one architecture in particular.
As discussed in Section 2 the classifiers og function like a "wrapper" around the reward functions re- which are the objects we are really interested in. We advocate the use of reward models with two disjoint sets of parameters: a common vector of parameters O E Rs that is shared among all users h E H. including those h & H. and parameters 0₁ E Rd that are specific to user h (this is closely related to the concept of "adapters" [50 30
We will call ro.e, adaptive reward models.
The partitioning of the parameters of adaptive reward models induces a division of their training procedure itself. The data in S can be used to learn O and 0j for a rater h E H. However, to learn the parameters On associated with a user not in S, h H H. one needs additional data containing Wh We will refer to the use
feedback provided by h, Sr. = (1x,, Vs. 1/s x:) })}a with of S to learn and 0 as training; learning On using S₁, will be referred to as adaptation.
We assume we have abundant data for training in S, but adaptation must take place with as few additional training examples as possible in S₁, (that is, n >> ń). Because of this, and also because we want the adaptation of the model to a specific user h 7 \ H to be quick, we generally want to have e >> d-in words, we want the number of shared parameters to be much larger than the number of parameters that are specific to a given individual. Another, slightly more practical reason to have a large number of shared parameters is that this allows for a distributed learning architecture in which the learning of can make use of a centralised, powerful computational infrastructure, while each 0, can be learned "locally" using less resources.
5

---

Reward features. We now discuss an architecture for r, that has particularly nice properties. We define reward features as a function (zz, y) X x y + Rd. We then assume that the reward for individual h is given by
(5) where Wa E Rd and (:--) denotes inner product. Following the steps taken in Section we define a parametric form for rh that reflects (5). We first define a parameterised function Φg : X x y + Rd with 0 E RR. Note that @ is the set of shared parameters defined above; the parameters 0₁ associated with specific h H. are simply a vector Wa E R Putting it all together, our parameterised model is ro, (1, v) (do (z, y), w.... We call this architecture a reward-feature model (RFM).
Th (z, y) = (0 v). wh),
Training. We now derive an appropriate optimisation objective for RFM's training. Let W Riiixd be a matrix formed by stacking [7Z] vectors Wn- Let co w (h. x, y. v.y') == (00 (2) & (2, wi), where wi, is the row of W corresponding to h. Plugging c into ①.. we get
E(co (...w/l (h, x, y. v/) v.,. 2) = -2 2ogo 8) -)- 00 )),, w (o)=)()logg(aa)) (() D') 3).
As before, if we minimise the empirical loss Ls induced by (6), we will automatically be maximising the likelihood C( WSS), which a proxy for L( W).
RFM's training yields a feature function Φo X' x y + Rd and a matrix W E R|A| Each dimension of Φo. (Φo),,, can be interpreted as criterion used by humans to express their preferences. The i-th element of W Wig represents how much rater h values (or does not value) criterion (do);- Importantly, learning @g does not depend on raters being able to articulate the criteria underlying their preferences; the only thing that is required from them is to rank pairs of candidate responses.
The question arises as to how well the features Do will be able to capture the preferences of the entire population H. We need Φh to have enough representational capacity-a proxy for which is the number of parameters e-and its dimension d to be sufficient to capture the preferences in S. However, when and d are too large, an RFM may effectively become [H] independent reward models. We usually do not want that. Instead, we want Φa to capture features that are common the raters in H. since these should also reflect the preferences of the users in H. more generally. This means that e and d have to be appropriately set or controlled through regularisation.
Adaptation. We are interested in using features de learned during training to specialise ra,w, to users beyond the raters h E H.. The problem of adapting To, to an unseen user can be formulated as small modification of training in which the loss (6) is minimised with respect
new set of coefficients w while the features O are held fixed. Formally, we define a classifier set parameterised by w, Cw (h, xr y, y/' = (0) (ar, y) - 2 (I. y/), w). plug into (6). and minimise Ls, with respect to w. Adaptation is particularly simple with an RFM: since 0 is frozen, optimising w comes down to logistic regression, which is well understood convex optimization problem whose sample complexity and references therein).
has been characterised for different scenarios (see, for example, 33 43 31
Connection with the theory. We now discuss how to connect RFM's two-stage learning process with the theory developed in Section 3 First note that the upper bound in (4) directly applies to RFM's intra-user generalisation loss, for in this case adaptation is not necessary. To apply version of the bound (4) to training and adaptation, we must see RFM as a function of only. We can accomplish this by folding adaptation into the functioning of the model. When faced with new tuple (h, x. y²), we create a dataset S₁ on-the-fly by sampling á examples from Dd, and then make Wh arg min Lss, (Cw). That is, Wi is longer a free parameter of the model. If we use the model in this way during training, we can use (4) to bound its post-adaption generalisation error.
If we are concerned with adaptation only, 4 directly applies greatly simplifies the bound but obscures some of the underlying insights. Because RFM's adaptation convex optimisation problem, there are known generalisation bounds that scale nicely, avoiding polynomial dependence on factors such as | and [>] (see for example [ Theorem 6. I).
special case when |H| 1. which
Why not other adaptive reward model architectures? As discussed, RFMs are but one instan- tiation of a more general class of adaptive reward models ro.0,- We now lay out few arguments supporting our choice of the RFM architecture in particular.

---

First, as discussed, the fact that RFM's architecture is linear in w gives rise to convex adaptation problem. This should not be overlooked: having model able to reliably adapt to new users in a low-data regime may be crucial [S]]. Second, and relatedly, RFM's simple architecture also makes it easier to improve the entire training pipeline, as one can resort to well established linear algebra techniques [52]. Third, the fact that the model's output is linear combination of the features Ø9 should make it easier to interpret their contributions (note that &g itself can be an arbitrarily complex non-linear functions of the inputs) (38). Fourth, it should be easy to add new, possibly handcrafted, features that represent agreed upon metrics like safety, helpfulness, and factuality 3 57 21).
But perhaps the main reason to adopt RFMs is the fact that they allow for an easy adaptation of the downstream LLM (or, more generally, policy). Recall that ultimately we are interested in using ro,w, to steer the behaviour of policy. There are methods in the literature specifically designed to synthesise a policy that performs well under a linear combination of features whose coefficients are only provided at deployment time 4 29 22 7 33 49 53 14 11]. In our case, this means that, after adapting to a user using a few examples, we can hand the resulting W over to one of these methods and immediately obtain a policy (or LLM) that is specialised to the corresponding reward.
Limitations. There is an inherent tension between training and adaptation: in general, the simpler the latter, the more demanding the former becomes. RFM sits one extreme of this spectrum, with an adaptation as simple as it can be at the expense of a potentially complex training. While this aligns with current L.LM practices-with ample data and compute for offline training and scarcer resources for adaptation-, it might not be the ideal trade-off in other contexts, or even for LLMs if resource availability shifts in the future.
5 Experiments
We now present experiments illustrating our findings. Due to space constraints we only describe the crucial aspects of our experimental setup; for further details and additional results, see Appendix B
We adopted UltraFeedback, a dataset carefully curated to ensure the quality and diversity of the responses [18]. This is a relatively large dataset for this type of study: the version we adopted has a training set with 60. 829 examples and test set with 985 examples [ On the solution side, we used Google DeepMind s (26) Gemma 1.1 2B model to implement both a baseline and XFM@The baseline is a reward model that neither distinguishes between raters nor performs adaptation, as is common practice. We implemented RFM using the same Gemma model, with the final layer replaced by d counterparts corresponding to the features Φg. Both models were trained using gradient descent to minimise their corresponding losses, starting from the pre-trained parameters of Gemma.
The goal of our experiments is twofold: to illustrate the theoretical results in Section 3annd to assess RFM's performance compared to existing baselines. To study both rigorously, we must be able to vary the number of raters as well as their heterogeneity in terms of preferences. We accomplished this by defining systematic way of generating synthetic raters with different characteristies. We created 13 features {0, y) capturing different aspects of a context-response pair: from superficial, casy-to-compute features, like length or the number of adverbs in y, to more semantic features such as the number of words in that are synonyms (or antonyms) of words in x' (see Appendix B.] for details). In contrast with features generated by LLMs, even our most nuanced features can be unequivocally and efficiently computed. They are also devoid of inherent valence.
We defined user as vector E 1-1,1)13 that is, user either likes (+1) dislikes (-1) each one of the features di. This gives rise to a space H with 218 users. Given an example (I. y, vo) and user w, determine the corresponding preference
1 [((((z, v), w) > (Φ(x, v)),"") w)), where [ is the indicator function. We defined a distribution D* parameterised by a single scalar p E [0, 1] indicating the probability that each &; is liked independently of the others. That is, we sample users h ~ Dz by drawing each whi from a Bernoulli with mean For each experiment, we sampled m raters to rank the training set and 500 held-out users to assess the inter-user generalisation of the models on the test set (this number is considerably larger than in previous studies [44] [3] 52)).
It is worth emphasising that throughout our experiments RFM did not have access to the real features & underlying the data. Instead, it learned features Φb parameterised by 0 E R using 6 The notation RFM(d) indicates that a d-dimensional db Rd was learned.
huggingfacc bhuggingface.coddattevvvlddadd www.eeawwweeeoalllllleeaieiieeee baapok_iinaaddddeeeeee hwggingface..o/ggoggoooooo segeginggaee.oggooollleemmmmme 1.1--bb-it
7

---

0.8
Test accuracy
0.7
0.6
Baseline RFM(0)
RFM(22)
0.5
RFM(128)
2
2a
24 2
2
0.5
0.6
0.7
0.8
0.9
40
Number of raters (m)
Proforanca homogeneity
Number of examples for adaptation (6)
(a) p = 0.5, ñ 30
(b) 60, ñ 30
(c) p 0.7, 60
Figure 1: Accuracy in predicting the preferences of 500 held-out users on the test set after adaptation
(estimate of inter-user generalisation). Error bars are 99% confidence intervals over runs.
Empirical analysis. Figure shows the performances of the baseline and RFM as we vary the number of training raters, the parameter p underlying Dz,, the number á of adaptation examples. and the number features learned by RFM. Note how RFM behaves as predicted by our theoretical results, with performance improving with the number of raters m. Also as predicted by the theory, the performance both the baseline and RFM improve with the homogeneity p of the users' preferences homogeneous as p - 1).
(Dx has maximum entropy when p 0.5, so preferences get However, RFM's performance much more robust to changes in with almost no degradation when sufficient number of features d 2 32 are learned. Also note how RFM can adapt well to new user using as few as î = 30 examples, and for d E {32, 128} performance keeps increasing with ń (albeit only slightly). This means that, by ranking only 30 examples, a user can replace generic reward model reflecting the average opinion of the population H. with a model specialised to them.
Modulating the LLM's output. Next we assess how well RFM's good performance transfers to the scenario where it is used to steer the behaviour of an LLM. We use best-of-n over up to n = 40 responses to each context æ in UltaFeedbackk s
100
test set [55] The responses were generated by Google Deep- Mind|s s 27 Gemma 2 9B and Gemma 2 27B (20 responses each). For each context æ in the test set, scored all 40 re- sponses using the baseline and RFM(32) adapted with ń = 30 examples. Next, we selected the best-of-n response context x according to the baseline and RFM, and declared either a winning model selected responses computed by user w. shoow results when best-of-n applied with increasing n. We highlight that the 40 candidate responses used come from distribution that different from the one used for training. Yet, RFM consis- tently outperforms the baseline, and the fraction of times its selected response is preferred grows with n.
80
60
Win
20
draw based on the actual score of their
Number of responses n for best-of-n
Figure 2: Relative accuracy in pre- dicting the preferences of 500 held- users using 30 examples.
Comparisons. We now compare RFM with three types of adaptive counterparts. The first one is linear model that adapts to user h by fine-tuning the final layer of the (trained) baseline through gradient descent on the corresponding dataset Sn. This similar to RFM, except that the baseline features being linearly combined were learned without raters h being distinguished (and thus this is a clear way to assess whether doing so is indeed beneficial). The second comparison involves the non-linear architecture proposed by Park et al. [42]. We partitioned the shared parameters = [0,, 02] and learned 0, together with the vectors W,. This was implemented by freezing the backbone Gemma model (0)) and representing as multilayer perceptron (MLP) with 3 hidden layers containing 32 units each (02; details in Appendix (3.2). The third comparison is with models that perform "in-context" adaptation. We implemented these using two prominent LLMs,, Google DeepMind's 's [28] Gemini 1.5 Pro and OpenAl's [40] GPT-4o. To assess the LLMs' prediction accuracy for held-out user h, we provided ñ 10 training examples ranked by together with the test example to be ranked (the precise prompt can be found in Appendix B.2) For reference, we also show the "zero-shot performance of Gemini, obtained with 0.

---

shows shows the results of the two LLMs side-by-side with those obtained by the baseline, the linear baseline,
the non-linear baselines, and RFM(32)- the last three also adapted using 10 examples. The linear base- line performs on par with its non-adaptive counterpart, which suggests that features learned in a user-agnostic way are not capable of capturing the variance of the population H. The non-linear baselines perform poorly, probably because they have too many parameters to be trained with only = 10 examples. Surprisingly, the LLMs also seem to be unable capture the users' pref- erences, with their performance reducing to chance (see further discussion in Appendix [..2 RFM correctly predicts the users' preferences around 70% of the time.
Test accuracy
baseline
(FM)32)
0.5 0.5
Non- Inear a (3 loyers)
inear layers)
GPT-4o
Gemni 1.5 Pro
Gemini 1.5 Pro tarm-shot)
Model
Figure 3: Accuracy in predicting the prefer- ences of 500 held-out users using ń 10 examples for adaptation. Error bars are 99% confidence intervals over 5 runs.
We also compared RFM Poddar al. S 44 variatonal preference learning (VPL), described in Section Even though in principle VPL allows for inter-user generalisation, Poddar et al. have only assessed VPL's intra-user generalisation. They report an estimate of this type of generalisation on the UltraFeedback dataset, using the features provided with it, of 61.49%. We tried to reproduce their experimental protocol closely possible, as explained in Appendix [.2 The resulting estimate of RFM(32)'s s intra-user generalisation is 61.61%, which is on par with VPLs.
Modelling groups of real users. In the previous experiments we had access to the features & underlying the raters' preferences; although this is useful to investigate the models' performance under different conditions, in more realistic scenario would only have the training examples in the dataset S. To simulate this situation, in this section we use reward models as the users h.
We performed experiments using two datasets: UltraFeedback, as before, and also Zollo et al. 65 Bersonall..M Each example in the Personall.L.M dataset has been scored by 10 reward models. To mirror this setup with UltraFeedback, all its test examples. We provide a detailed account our experimental setup in Appendix B.3
used publicly-aves avviialeee reward models to score and rank
All the reward models considered were trained with human preference data, and hence they reflect the opinion of groups of real people. As these models do not distinguish between raters raters-much like the baseline-, they reflect an average over preferences, and thus tend to "agree" considerably with each other. To avoid such overlapping, which may render the distinction of raters unnecessary. we filtered out all the examples in the training and test sets of both datasets in which two or fewer raters disagreed with the majority. We then carried out a "leave-one-out" cross-validation composed of rounds in which - of the models played the role of the raters H and the remaining model played the role of the held-out user (k 8 for UltraFeedback and k = 10 for PersonalLLM).
Results are shown in Figure We compare RFM with the non- adaativee and linear baselines used in the previous experiments. RFM's performance either matches or significantly surpasses that of the baselines in most cases, suggesting that it can be useful in real scenarios. As the reward models in this experiment reflect the preferences of aggregated real users, some of the heterogeneity of the population's 's preferences has probably been smoothed out in the training of these models. We conjecture that if the reward models were replaced by the real users underlying these models, RFM's advantage over the two baselines would be greater still.
3.5
Baseine
Linwar baseine
0.9
Saseline
Linear baseline
RFM11281
0.8
0.8
That accurar
0.7
lest accureeo
0.7
0.6
Held-out
Held-out
(a) UltraFeedback dataset
(b) PersonalLLM dataset
Figure 4: Accuracy in predicting the preferences of a held-out reward model using á = 50 examples for adaptation. Error bars are 99% confidence intervals.

---

6
Related work
We distinguish between two categories of adaptive reward models. In the first we have methods whose goal is to compute a single policy representing some form of compromise across users [22 12 54 16 19 61.. These methods take individual idiosyncrasies into account primarily to build more accurate models, and hence they do not necessarily provide mechanism for adapting to new users.
In the second category we have methods designed support the downstream specialisation of
g) Among works probably the closest to ours. They both concurrently propose the same architecture as RFM, though with different names and slightly different emphases. While we analyse the learning problem- how training and adaptation change as the dataset S changes-, [Shenteld Baaange-,Shenteld et al. focus on how to solve the resulting optimisation problem: how to go about training and adaptation once we have committed to a specific dataset S. [Shenfeld et al. also show how to leverage RFM's simple architecture to improve several aspects of the training pipeline, most notably stable initialisation scheme via singular value decomposition of the data matrix and more efficient way of selecting adaptation examples based on active learning. [Bose et al. provide a particularly clear exposition of the subject and well-executed empirical evaluation that includes comparisons and datasets not considered here. We see these two concurrent works as highly complementary to our own, as they collectively provide mutually reinforcing theoretical arguments and empirical evidence in favour of the proposed architecture.
policy to a Specific user, like RFMs 44
63
24
36
34
13
25
35 42
52
these, [Shenfeld al.]s [52] and Bose a
Poddar et al. s 441 variatonal preference learning (VPL) encodes examples previously ranked by a user into a latent variable, and then conditions the reward model and the policy on that variable (we compare VPL with RFM in Section 5). Zhao et al. (63 also encode pairs of ranked responses, but instead of resorting to variational techniques they use an LLM to do in-context inference of the rewards. Similar methods encode different types of information about users, like demographic data, either in isolation or together with examples of ranked pairs [24, 36 34). Chen et al. 13 propose to replace the Bradley- Herry model with Coombs's [17 ideal point model. The resulting method is similar to ours, albeit slightly more complex. [o et al. (25) approach is also similar to ours, but
they use features computed by LLMs with handcrafted prompts.
We redirect the reader to Appendix C for an in-depth discussion of some of the works cited above.
7 Conclusion
We have formalised and analysed the problem of learning a reward model that can adapt to users. To the best of our knowledge, this is the first time this problem is rigorously studied under the assumptions considered. We have derived a PAC bound that elicits the dependency of the approximation error on the number of training examples and raters. Our analysis provides formal framework for assessing the trade-offs involved in the collection of preference data and the use of an adaptive reward model.
We have also introduced RFM, reward- model architecture specifically designed for fast adaptation to new users. RFM can be trained using pairwise response comparisons provided by humans. This results in set of reward features that can be linearly combined to represent user, even if their preferences are not reflected in the training data. Such an adaptation process can be formulated as a simple logistic regression- a well-understood convex classification problem. We showed how RFM can be personalised to an unknown user using a few dozen pairs of examples ranked by them. We have presented experiments showing how RFM compares favourably with a non-adaptive baseline, which is today's prevailing practice, and also several adaptive counterparts.
RFM can be readily combined with zero-shot RL methods that construct policy on-the-fly which performs well under a linear combination of features. In the context of LLMs, this means that by responding to a few questions& user can have an otherwise generic model specialised to their taste. Although we have focused on the use of RFM in the context of L.LMs, it is also applicable to other modalities beyond language, like images, sound, and video. In fact, RFM can be used as a reward function in any scenario that can be formalised as an RLHF problem, which includes not only more general state and action spaces but also multi-step interactions of the policy with the environment.
10

---

Acknowledgements
We would like to thank Tom Schaul, Benjamin Van Roy, and Dan Andrei Calian for their insightful comments and useful feedback. We are also thankful to Bernardo Ávila Pires for assistance with one of the experiments and to Canfer Akbulut and Arianna Manzini for contributing to the discussion regarding the potential broader impact of our work. Finally, we would like to thank the anonymous reviewers for their comments and suggestions.
References
[1] A. Avital, K. Efremenko, A. Kontorovich, D. Toplin, and B. Waggoner. Non-parametric binary regression in metric spaces with KL loss. arXiv, 2020.
[2] M. G. Azar, Z. D. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello. A general theoretical paradigm understand learning from human preferences. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), 2024.
[3] Y. Bai, Jones, K. Ndousse, Askell, Chen, DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, N. Joseph, S. Kadavath, J. Kernion, T. Conerly, S. El-Showk, N. Elhage, Z. Hatfiel-- Dodds, D. Hernandez, T. Hume, S. Johnston,
Kravec, L. Lovitt, N. Nanda, C. Olsson, D. Amodei, T. Brown, J. Clark, S. McCandlish, C. Olah, B. Mann, and J. Kaplan. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv, 2022.
[4] A. Barreto, W. Dabney, R. Munos, J. Hunt, T. Schaul, H. van Hasselt, and D. Silver. Successor features for transfer in reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
[5] A. Barreto, D. Borsa, J. Quan, T. Schaul, D. Silver, M. Hessel, D. Mankowitz, A. Zidek, and R. Munos. Transfer deep reinforcement learning using successor features and generalised policy improvement. In Proceedings of the International Conference Machine Learning (ICML), 2018.
[6] A. Barreto, S. Hou, D. Borsa, D. Silver, and D. Precup. Fast reinforcement learning with generalized policy updates. Proceedings of the National Academy of Sciences, 2020.
[7] D. Borsa, A. Barreto, J. Quan, D. Mankowitz, H. van Hasselt, R. Munos, D. Silver, and
T. Schaul. Universal successor features approximators. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
[8] A. Bose, 7. Xiong, Y. Chi, S. S. Du, L.. Xiao, and M. Fazel. LoRe: Personalizing LLMss via low-rank reward modeling. In Proceedings of the Conference on Language Modeling (COLM), 2025.
[9] S. Boucheron, G. Lugosi, and P. Massart. Concentration Inequalities: A Nonasymptotic Theory of Independence. Oxford University Press, 2013.
[10] R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: The method of paired comparisons. Biometrika, 1952.
[11] Carleton, Mukherjee, S. Shakkottai, and D. Kalathil. MAVIS: Multi-objective alignment via value-guided inference-time search. arXiv, 2025.
[12] S. Chakraborty, J. Qiu, H. Yuan, A. Koppel, F. Huang, D. Manocha, A. S. Bedi, and M. Wang. MaxMin-RLHF: Towards equitable alignment of large language models with diverse human preferences. In Proceedings of the International Conference on Machine Learning (ICML), 2024.
[13] D. Chen, Y. Chen, A. Rege, and R. K. Vinayak. Modeling the plurality of human preferences via ideal points. ICMI. 2024 Workshop on Theoretical Foundations of Foundation Models, 2024.
11

---

[14] R. Chen, X. Zhang, M. Luo, W. Chai, and Z. Liu. PAD: Personalized alignment of LLMs at decoding-ting time. In Proceedings of the International Conference on Learning Representations (ICLR)) 2025.
[15] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems (NeurIPS). 2017.
[16] V. Conitzer, R. Freedman, J. Heitzig, W. H. Holliday, B. M. Jacobs, N. Lambert, M. Mossé, E. Pacuit, S. Russell, H. Schoelkopf, et al. Social choice should guide Al alignment in dealing with diverse human feedback. In Proceedings of the International Conference on Machine Learning (ICML), 2024.
[17] C. H. Coombs. Psychological scaling without a unit of measurement. Psychological Review, 1950.
[18] G. Cui, L. Yuan, N. Ding. G. Yao, W. Zhu, Y. Ni, G. Xie, Z. Liu, and M. Sun. UlraFeedback: Boosting language models with high-quality feedback. In Proceedings of the International Conference on Machine Learning (ICML), 2024.
[19] J. Dai, X. Pan, R. Sun, J. Ji, X. Xu, M. Liu, Y. Wang, and Y. Yang. Safe RLHF: Safe reinforcement learning from human feedback. In Proceedings of the International Conference on Learning Representations (ICLR), 2024.
[20] H. Dong, W. Xiong, D. Goyal, R. Pan, S. Diao. J. Zhang, K. Shum, and T. Zhang. RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment. Transactions on Machine Learning Research (TMLR), 2023.
[21] N. Dorka. Quantile regression for distributional reward models in RLHF, arXiv, 2024.
[22] V. Dumoulin, D. D. Johnson, P. S. Castro, H. Larochelle, and Y. Dauphin. A density estimation perspective on learning from pairwise human preferences. Transactions on Machine Learning Research (TMLR), 2024.
[23] Z. C. et al. I2 technical report. arXiv, 2024.
[24] E. Fleisig, R. Abebe, and D. Klein. When the majority is wrong: Modeling annotator disagree- ment for subjective tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023.
[25] D. Go, T. Korbak, G. Kruszewski, J. Rozen, and M. Dymetman. Compositional preference models for aligning LMs. In Proceedings of the International Conference on Learning Repre- sentations (ICLR), 2024.
[26] Google DeepMind. Gemma: Open models based on Gemini research and technology. arXiv, 2024.
[27] Google DeepMind. Gemma 2: Improving open language models at a practical size. arXiv, 2024.
[28] Google DeepMind, Gemini Team, Gemini: A family of highly capable multimodal models. arXiv, 2023. [Online; accessed in May 2025].
[29] T. Haarnoja, V. Pong, A. Zhou, M. Dalal, P. Abbeel, and S. Levine. Composable deep rein- forcement learning for robotic manipulation. In International Conference Roboties and Automation (ICRA), 2018.
[30] N. Houlsby, A. Giurgiu, S. Jastrzebski. B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. At- tariyan, and S. Gelly. Parameter-efffeientt transfer learning for NLP. In Proceedings of the International Conference on Machine Learning (ICML), 2019.
[31] D. Hsu and A. Mazumdar. On the sample complexity of parameter estimation in logistic regression with normal design. In Proceedings of the Annual Conference on Learning Theory (COLT), 2024.
12

---

[32] J. Hunt, A. Barreto, T. Lillicrap, and N. Heess. Composing entropic policies using divergence correction. In Proceedings of the International Conference on Machine Learning (ICML), 2019.
[33] J. Jang, S. Kim, B. Y. Lin, Y. Wang, J. Hessel, L. Zettlemoyer, H. Hajishirzi, Y. Choi, and P. Ammanabrolu. Personalized soups: Personalized large language model alignment via post- hoc parameter merging. In NeurIPS Workshop on Adaptive Foundation Models, 2024.
[34] J. Kim and Y. Yang. Few-shot personalization of LLMs with mis-aligned responses. Proceedings of the Conference of the Association for Computational Linguistics (ACL), 2025.
[35] Y. Lee, J. Williams, H. Marklund, A. Sharma, E. Mitchell, A. Singh, and C. Finn. Test-time alignment via hypothesis reweighting, arXiv, 2024.
[36] X. Li, Z. C. Lipton, and I.. Leqi. Personalized language modeling from personalized human feedback. arXiv, 2024.
[37] P. M. Long. On the sample complexity of PAC learning half-spaces against the uniform distribution. IEEE Transactions on Neural Networks, 1995.
[38] C. Molnar. Interpretable Machine Learning. 2025. URL httphttt////chisstopppo gwt.im......uu...... io/
interpretable-nl-boon
[39] OpenAI. GPT-4 technical report. arXiv, 2024.
[40] OpenAI. GPT-4o system card. arxiv, 2024. [Online; accessed in May 2025].
[41] L. Ouyannggn J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[42] C. Park, M. Liu, D. Kong, K. Zhang, and A. Ozdaglar. RLHF from heterogeneous feedback via personalization and preference aggregation. In ICML Workshop on Aligning Reinforcement Learning Experimentalists and Theorists, 2024.
[43] Y. Plan, R. Vershynin, and E. Yudovina. High-dimensional estimation with geometric constraints. Information and Inference: A Journal of the IMA, 2016.
[44] S. Poddar, Y. Wan, H. Ivison, A. Gupta, and N. Jaques. Personalizing reinforcement learning from human feedback with variational preference learning. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
[45] C. Poth, H. Sterz, I. Paul, S. Purkayastha, L. Engländer, T. Imhof, I. Vulić, S. Ruder, I. Gurevych, and J. Pfeiffer. Adapters: A unified library for parfmeter-efficient and modular transfer learning, In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP): System Demonstrations, 2023.
[46] A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. Improving language understanding by generative pre-training. 2018
[47] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2024.
[48] P. Ramachandran, P. Liu, and Q. V. Le. Unsupervised pretraining for sequence to sequence learning. arXiv, 2016.
[49] A. Rame, G. Couairon, C. Dancette, J.-B. Gaya, M. Shukor, L.. Soulier, and M. Cord. Rewarded soups: towards pareto-optimal aligmment by interpolating weights fine-tuned on diverse rewards. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
[50] S.-A. Rebuffi, H. Bilen, and A. Vedaldi. Efficient parametrization of multi-domain deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018
13

---

[51] S. Shalev-Shwartz and S. Ben-David. Understanding Machine Learning From Theory to Algorithms. Cambridge University Press, 2014.
[52] I. Shenfeld, F. Faltings, P. Agrawal, and A. Pacchiano. Language model personalization via reward factorization. In Proceedings of the Conference on Language Modeling (COLM), 2025.
[53] R. Shi, Y. Chen, Y. Hu, A. Liu, H. Hajishirzi, N. A. Smith, and S. S. Du. Decoding- time language model alignment with multiple objectives. In Advances in Neural Information Processing Systems (NeurIPS), 2025.
[54] A. Siththaranjan, C. Laidlaw, and D. Hadfield-Menell. Distributional preference learning: Understanding and accounting for hidden context in RLHF. In Proceedings of the International Conference on Learning Representations (ICLR), 2024.
[55] N. Stiennon, Ouyang, Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and Christiano. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[56] R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press, 2018.
[57] H. Wang, W. Xiong, T. Xie, H. Zhao, and T. Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts In Findings of the Association for Computational Linguistics (EMNLP), 2024.
[58] J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. Finetumed language models are zero-shot learners. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
[59] W. Xiong, H. Dong, C. Ye, Z. Wang, H. Zhong, H. Ji, N. Jiang, and T. Zhang. Iterative preference learning from human feedback: Bridging theory and practice for RLHF under KL-constraint. In Proceedings of the International Conference on Machine Learning (ICML), 2024.
[60] R. Yang, R. Ding. Y. Lin, H. Zhang, and T. Zhang. Regularizing hidden states enables learning generalizable reward model for LLMs. In Advances in Neural Information Processing Systems (NeurlPS), 2024.
[61] B. Yao, Z. Cai, Y.-.. Chuang, S. Yang, M. Jiang, D. Yang, and J. Hu. No preference left behind: Group distributional preference optimization. In Proceedings of the International Conference on Learning Representations (ICLR), 2024.
[62] L. Yuan, G. Cui, H. Wang, N. Ding, X. Wang. J. Deng, B. Shan, H. Chen, R. Xie, Y. Lin, Z. Liu, B. Zhou, H. Peng, 7. Liu, and M. Sun. Advancing L.M reasoning generalists with preference trees. In Proceedings of the International Conference on Learning Representations (ICIR), 2025.
[63] S. Zhao, J. Dang, and A. Grover. Group preference optimization: Few-shot alignment of large language models. In Proceedings of the International Conference on Learning Representations (ICLR), 2024.
[64] H. Zhong, Z. Deng, W. J. Su, Z. S. Wu, and L. Zhang. Provable multi-party reinforcement learning with diverse human feedback. arXiv, 2024.
[65] T. Zollo, A. Siah, N. Ye, L. Li, and H. Namkoong. PersonalLLM: Tailoring LLMs to individual preferences. In Proceedings of the International Conference on Learning Representations (ICLR), 2025.
14

---

A Proofs and additional theoretical results
In this section we prove the theoretical results in the main paper and present two complementary results. To simplify the notation, we define K. X' y and use k to refer to tuples (æ, y, v²).
A.1 Proofs the theoretical results in the main paper
We restate the results presented in the main paper and prove them.
Proposition 1. For any E C, m > O, >> O, and 5 E (0, 1], we have with probability at least 1
that
L + ((IC))))))))))))))))))))))))))))
(D(c)- - Ls(c)
g + g² + 18gm
3m
where g = In(2/5).
Proof. Letk" E K and z" z", and define
knlc,,,k,," (c,hkki,
(7)
We can write
- Ls(C)] = (D(c) 1 clc, his kig, mzs) Lo(C) 1 Σ lle, hi,
Let L = (c, H, K )n) with H ~ Dr and km, nn K"
(D (D. Since we know that
LL ELan1 < 1 and E(1 mm Σ, L = LD(c), we can write, using Bernstein's 's inequality,
me2
((IL(c)- Ls(c)) > c) S 2xxp
(8)
2V(LCM)
Based on 77. the law of total variance, and the law of large numbers, we can write
V(L E[v(LSMM)) [n ((E[L=M)))) H ~ DH
E[V(l(c, H. K, Z)) V(EE((, H. K,K, H.
-EV((e(,, H. K, H)) + V(E[e(c, H,K, H, Z ~ D.
(9)
E(((E)AA))
If we let = K, 2))141) and V(E[e.)H) HI, can
rewrite 8 as
mE²
((TT(() Ts(C) > 5
(10)
[[V( + 2 +
3
To get the desired bound we only need to follow standard arguments in derivations of this type 9 for
example], which detail here for completeness. Equating the right-hand side of (10) ō, we have
me²
2exp
(11)
E[V(L.IH) 2V(E(C,(HT) 2
7
15

---

Dividing both sides of 111 by 2 and taking the natural logarithm (as 6/2 > 0), we get
me²
In
2
2 E[V(L H)) 20(E H))
72
me²
In
(12)
2
E[((C]HH) 2V(E(() +
n
2
2
we let In
and
E[W( H)] +
and rearrange the terms, we can rewrite
7
as quadratic equation in
2
m1
2
ab 0,
3
whose solutions are
a + v va² + 9mab
c=
3m
Since we are using to upper bound LD((c) - Ls(c)), we want the largest of the two solutions. We
have a >> 0 and b O, a² 9mab
Thus, we take the solution with the positive sign.
Proposition 2. For any m. n > 0 and any 8 E (0, 11, we have with probubility at least 1 - ô that
- +
LD(ē*) < LD(c") + iiff
9a + V 2 + 188mga
+ 2a
3m
where 9a 1n(2(Cal/0).
be the set of a-nets over C according to dg, that ((cc), k(((
Proof. Define the semi-metric de(c, c') == max.eH,ie...ed
(k) Let A A= {C' | C C, de aa.
Let C. E arg min minc Clearly, this implies that
Veecdeec, [T((c) LD()) < a and [s(e)) I.s(é) < a.
Note that [Cal is the covering number NCC, de, a) 51 Analogously ( write, for ė Ca:
mE2
(((L Ls(0)) > >) < exx
2V(Le.) + =
Define V(Ec) sup, V(Lan). Recalling that C. C, it follows that
mc²
Viec L(st)())_ss() e) S 2exp
2V((c) +
If we apply the same argument in p to V(fc), we can write
me²
VeEc P ([LD(C) Ls(e) c) < exxp
22 H))
Applying the union bound over C.. we get
m
P (Jeec. (LD(E) - Ls(e) > e) S 2/Cal exp
- [E(V(ec 2V(E HJ) +
2
(13)
16

---

If we equate the right-hann hand side I3 to å and isolate e, as done in the proof of Proposition
we get
1 3m
Col
2/Caa 2 + In
(2|Ca] 8
(LLVCE)
Ea
+ 18m In
+ +((((()))))
n
(14)
where we used the subscript in e to note its dependency on a. We have already shown that with probability at least 1 - 8, we have that Veeca LD((c) Ls(e) I Ea. Given C, can pick E Ca such that de(c, 2) S a. Then,
[Lp(c) Ls(c)l
(D(e) Ls(() Ls(e) Ls(c)l
v [c(c Ls(²) + ( Ls(c)
< LD(e) Ls(²)) +a
[L((c) Lp(() + Lp()) - Ls(c)) +
< (D((c) Lo(()) + (LD(e) - Ls(8) +a
- (é( - Ls(E) + 2a.
(15)
Thus, say that, with probability at least 1 5, we have V'Ec (LD(c) Ls(c)) S ca 2a.
Since a was defined arbitrarily, we can write
VEEC (D(()) Ls(c) s inf + 2a).
(16)
Based on 16 we can write
(D( Ls(ö*) + inf(ra + 20) & [(((')) + inf(ca + 2a) s ID(C") + 2inf(ca + 20.)
a
A.2 Additional theoretical results
We now present additional theoretical results that complement those in the main paper. We start with result upper bounding [Ca the measure of complexity used in Proposition 2
Proposition 3. Let a E 10. Then, the smallest set C. S such that
Veecderec VnEH,KEK,EEZ |1(c(h, k), (h,,),, S
has size most (HII))
Proof. First, there exists finite subset M C [0, 1] of size a such that
Yue.o. vl a; 2 concretely, we may take y = - 2a _. and let M ( 2g : (l,....4)}
Now, fix a classifier e: 7 + R. We will specify classifier è E MHx with
(((c(h,)), 2) (()(h,)),, <a
(17)
for all (h, 2) E x Z. To do so, consider tuple (h, E H. If c(,, k) < 0.5.
we set o(h, k) such that E(č(h, k)), 0) -M is the closest value M to E(c(h, k), particular
within distance a. Then, since the derivative of log is positive and decreasing, we also have that (E(ā(h, k), 1) k), 0) < a. in ((c(h, k), 0) 2 0.5, we set o(h, k) so that E(č(h, k), 1) e M is the closest value in M to E(c(h, k), 1), and the conclusion follows similarly. Thus, we have exhibited with the property described (17). and is guaranteed to lie set size at most (HIAA) as required.
The analysis in Proposition]] is based on the concept of covering numbers of function spaces, and Proposition 3 provides a simple upper bound for the covering number C., is also possible to derive results based on other measures of complexity of function spaces. We provide one illustrative example here, based on the notion of Rademacher complexity. This new result also assumes that the classifiers in C are logistic regression models.
17

---

Proposition 4. Under the assumptions of Section 3 consider a hypothesis class C comprising logistic regression models over concatenated embeddings of user. prompt, and responses. with weights bounded in 1.² norm by W, and embedding T.2 norms bounded by 1. Let ct be the optimal model in this class for the population distribution DD and let č be the optimiser of the empirical loss. Then, with probability at least 5, we have
LD((²)= & LD(C*)+
+
2W
+ SV
g
1
+ V / + 18gm (EE((e-))H) V((( (E(()-1EII)
vm
2m
3m
where g log(6/8).
Proof. First, note that the classification loss we are concerned with a sum of non-i.id. terms:
= m Σ Σー Σ
Ls(c)=
e(c(he, Zays Vigs fky)) bz)).
j=1
so we cannot immediately apply the classical methods Rademacher complexity analysis (see, for example, Chapter 26 of [51]).
However, our data is i..d. at the level of the indices i, can begin by applying the framework of Rademacher complexity bounds at this level. We broadly follow the proof structure of [1 Theorem 26.5].
First, we have that
sup(Lo(c) Ls(c)) cEC
viewed as function of S. satisfies the bounded-different condition required for McDiarmid's in- equality, from which it follows that this quantity concentrates around its expectation; with probability at least 1 - 8, we have
sup(Lo(c) - [.((c)) S Es sup(Ip(c) - -ss(()))))s((()) + log (2/0) 2m
aEC
LoEC
The key supporting result [51] Lemma 26.2] then allows to relate the expected quantity the right-hand side to the Rademacher complexity of our hypothesis class. Concretely, we have
Fg
sup(I.(c) Iss(e)) LEEC
s 5 ( o So)),
where
m ( E(c(hn (clhh)ku)))))) : :ce
CoCos'
(18)
and R denotes Rademacher complexity of the input set.
Now, the bounded-differences inequality required to employ McDiarmid's inequality applies to the Rademacher complexity as a function of 5' too, R(E o S'), so that we may obtain with probability 1 8:
| (og(2/5)
[R( CC R(foCo
2m
Putting all these parts together, yields
log (4/8)
sup(Lo(c) Ls(c)) < 2)
(19)
IEC
2m
with probability at least 1 8. Finally, denoting c" the optimiser of the true loss, and čt the optimiser of the empirical loss, we have Lo(3)) - LD(c") = (LD(²") - Ls(ū")) + (Ls(E)) - Ls(c")) + (Ls(c") Lo(c"))
< (LD(e') - Ls(ū*)) + (Ls(e*) Lp(c")).
18

---

The first term is bounded by the inequality in Equation (9 The second term may be bounded by Bernstein's s inequality: note that this deviates slightly from the approach set forward in [5]] Chapter 26], but we can make use of Proposition here. This ultimately results in a bound of the form:
[TD(') T.D('') +
9
g² + 18gm
[1)]
2m
3m
where y = log(6/)).
Lastly, we derive an explicit form for the Rademacher complexity of our predictor, making use of the model class assumptions introduced in the statement of the result. To bound the empirical Rademacher complexity R(E C S), we can make standard argument using several manipulations based on the Rademacher calculus, as described [5] Chapter 26]. First, by the sum property, we focus on single summand per dimension. Next, by the contraction lemma, and the fact that the Sgearithm-sigmoll composition is I-Lipschitz, we may remove these elements from the under consideration, revealing the linear function class that we hypothesise. With the assumptions weight norms and embeddings made in the statement, we then obtaín an overall Rademacher complexity of W/ym. This results in an overall bound of:
LD(č*) S LD(c")
vm 3V
2W
1 3m
9 +
g2
18gm EVV(e- V(E [T)
2m
required.
B Details of the experiments and additional empirical results
We tried to keep our experimental setup as simple as possible to provide a realistic estimate of out-of-the-box performance. In particular, did not carry out extensive search over network architectures or hyper-parameters, keeping most them at sensible defaults from the outset. We also did specialise the training procedure RFM's architecture, using standard gradient descent to minimise the training and adaptation losses.
As mentioned in the main text, we used Google DeepMind 26 Gemma 1.1 2B to implement RFM and the baselines. The maximum context and response lengths were set lo = ly = 1, 525 tokens. Training and adaptation were carried out using gradient descent with a learning rate of 10-5, We performed random 90%-10% split of the training set and used the error in the smaller subset (a validation set) as a critcrion to select the model to undergo adaptation.
B.1 Empirical analysis
Problem setup. The following 13 features were used:
(2, y): the length of y.
(2, y): the average sentence length in y.
(2 y): the average word length in y.
(2 y): the proportion of characters in that are vowels.
(z y): the proportion of characters in that are punctuation symbols.
(1, y): the proportion of transitions between words in y that are alliterations.
(1 y): the proportion of words in y that adjectives.
(a w): the proportion of words in y that are adverbs.
(g y): the proportion of words in y that are verbs.
Φ10 y): the proportion words in that are nouns.
dn (z, y): the proportion of words that are synonyms of one of the words in æ.
(12 y): the proportion of words in that antonyms of of the words in
(1, y): the proportion of words in that also appear in
19

---

All the features were normalised to fall in the interval [0, 1] and then centered around their median value so that positive values represent values above the median (and vice versa). Our features were designed to capture potentially conflicting subjective criteria, yet they remain inherently neutral (i.e,, possessing no inherent valence). These features also vary in how they access inputs I: and y. which consequently influences their anticipated learning difficulty. We identify three categories of features. "Structural" features (e.g., ø1, 02. Φ3) treat x and y as raw strings devoid meaning and are thus expected to be straightforward to learn. In contrast, "syntactic" features (e.g., 07, Øg, Φg) analyze the grammatical role of words a sentence, and their dependence on inter-word relationships makes them more challenging to learn. Finally, "semantic" features (e.g., @11, 612), which rely the meaning of words, are likely the most difficult to learn.
As discussed in Section 5 we defined a user as a vector w and user w, the corresponding preference was determined
1-1,1)", Given an example (I, y, v.)
1f( v).
(0(x,,'), w))
(20)
where [ is the indicator function and (Ø, (z. y) y) are the features described above.
We defined a distribution over H. Dx. parameterised by single parameter determining P(w: for each i = 1, 2, ..., 13 independently. That is, in order to sample a user h from Dy. for each sample Z Bernoulli(p) and set Whi 2Z 1. The parameter of the
i 1, 2, .... 13 distribution Di allows us to control how homogeneous in terms of preferences groups sampled from it tend be. The entropy of Du peaks
0.5, resulting in the maximum degree of disagreement. illlstrates the effect of
As + 0 or p + 1, the preferences become more homogeneous. on the homogeneity of the users' preferences.
Palrwisa disagnemert fraction
0.50
0.45
0.40
0.20
0.5
0.6
0.7
0.8
0.9
Preference homogeneity (p)
Figure 5: Pairwise user disagreement as a function of p. For each value of p E (0.5, 0.75, 0.875, 0.9375), we sampled 100 users from DH and computed the fraction of the exam- ples in the UltrFeedback training set on which they disagreed.
We used of raters H of different sizes: m E {20, 40, 60,80, 100). Each run used different set H with raters sampled from Dz as explained above. We also sampled set of 500 held-out users per experiment using the same procedure. The held-out users were used to assess the inter-user generalisation of the models.
Unless otherwise noted, the default values for the parameters used in the experiments were: m = 60 raters, preference homogeneity level p 0.7, and á 30 examples used for adaptation.
Training and adaptation. Training was carried out for 6, 000 parameter updates with a batch size of 32. This means that the training procedure went over the entire UltraFeedback training set approximately three times. Each time the example (I;, Mss v/. was encountered. new rater h was sampled uniformly at random from H and the preference z; was determined through (20) with the corresponding Wi. For the experiments shown in Figure Ta specifically--aa m of raters we needed to make sure that each rater was trained with roughly the same number of examples n. So, extended training proportionally to ( the validation and test errors along training in this experiment).
a varying number
To perform the adaptation, we sampled ń E (10, 30, 50, 70, 90) examples from UltraFeedback's training set uniformly at random. Analogously training, each time the example (I,, Di, v/. was encountered user h was sampled uniformly random from the set of 500 held-out users, with
20

---

the corresponding preference 2 determined through 20 We carried out total of 6, 000 parameter updates with a batch of 32 for all 500 held-out users combined (so, only around 384 updates per user, in expectation).
For each experiment, carried out training followed by adaptation 5 times. All the numbers reported are averages over the corresponding 5 runs. The metric we report for adaptation in Figures [ and 3 is the inter-user test accuracy, an estimate of the fraction of examples in the test set correctly classified by the models, per user. To compute it, we went over the test set 50 times, always assigning to each example user sampled uniformly at random from the set of 500 held-out users. This means that each user is evaluated in approximately 100 examples, in expectation. The accuracy reported in Figures | and 3 the average number of correctly classified examples over the 50 passes over the test set.
Additional results. Figures and 7 show the baseline and RFM's 's accuracy on the test and validation sets during training. In Figure 6 we see the effect of varying the number of raters m, while in Figure 7 we see the effect of varying the preference homogeneity parameter p. The values reported are the intra-user validation accuracy and the intra-user fest accuracy. Their computation is analogous to that of the inter-user test accuracy explained above, with held-out users replaced by raters and the test set replaced by the validation set when applicable. Note that, in contrast with the adaptation error shown in Figure 1 the training error tends to go up with the number of raters This makes sense, since each rater corresponds to a classification problem being solved in parallel.
Besciino
RPMIL20
1000
20093
V100 13:00 50003 20030
X033
20009 1500
125
19:30
1000
20000
Figure 6: Validation and test accuracies as function of update steps and number m of training raters during the training phase (estimate of intra- user generalisation). The number of parameter updates is proportional to m to ensure that all raters see roughly the same number n of training examples. Error bars are 99% confidence intervals over 5 runs.
Figure Ic shows the performance of the baseline and RFM when using á E {10, 30, 50, 70, 90} examples for adaptation. Although these are small numbers from learning perspective, one may ask what happens under even more stringent conditions. To answer this question, we extended the RFM(32) results in Figure Ic to include test accuracies with á {1,3, 5). The results are shown in Table together with R3M(22)'s results from Figure Tc
As expected, RFM's performance improves monotonically with the number of examples á used for adaptation. Note that even the results with a single example remain above random chance (though significantly lower than the results with â > 10).
We point out that worse performance with very few adaptation examples is expected, for it reflects the intrinsic difficulty of the learning problem. Given that each user's preferences are derived from a combination of 13 features, many different combinations of those 13 features may explain the preference behind a few training examples, and multiple training examples may be necessary to disambiguate. reassuring to see that RFM is still able to capture some of the problem structure under these extreme conditions, and that its performance monotonically increases with ñ.
21

---

0.00
0.00
0.00
2000
2009 SICE
3006 100
200K 2000 3000 4000 2000
SOR
Figure 7: Validation and test accuracies as a function of update steps and preference heterogeneity y during the training phase (estimate of intra-user generalisation). Error bars are 99% confidence intervals over runs.
Table I: Accuracy in predicting the preferences of 500 held-out users UltraFeedback's test set after adaptation (cf. Figure 1.. The range of values shown are 99% confidence intervals over runs.
Test accuracy
0.5481 + 0.0217
3
0.5749 + 0.0109
5
0.5891 + 0.0056
10
0.7053 = 0.0182
30
0.7657 = 0.0790
50
0.7770 = 0.0333
70
0.7835 = 0.0217
0.7900 = 0.0172
In Figure 1b we show the effect of varying the level of homogeneity of the distribution Dz from which raters and held-out between the distribution used to sample raters and the real distribution underlying users. To simulate this scenario, we ran an experiment which held-out users were sampled from fixed distribution with p = 0.65 while raters were sampled from distributions with varying Results are shown in Figure The discrepancy between the rater and held-out user distributions has a negative impact on both RFM's s and the baseline's performance, as expected. However, the negative impact on the baseline is much more severe, and, contrast with RFM, it increases as p + 0.5.
sampled. In some real scenarios, there may be discrepancy
B.2 Comparisons
Comparison with the linear baseline. The linear baseline was adapted on top of the trained baseline using gradient descent with a learning rate of 10-5 (the same used for RFM). We froze all the parameters of the baseline trained with Figure [ except for the last layer. Then, we replaced the last layer with 500 linear layers, one for each held-out users h, and trained them using the data in the corresponding S₁. Note that this comes down to performing 500 logistic regressions in parallel on top of the frozen baseline's 's user-agnostic features (whose dimension is 1024). Adaptation followed the exact same protocol adopted for RFM, with 6, 000 parameter updates using a batch size of 32 shared among all users.
60 raters and preference homogeneity p 0.7 (c.f.
As an aside, note that using the protocol above to train all billion parameters of the baseline 500 times would be infeasible. This illustrates our point in Section advocating small number d of adaptable parameters.
22

---

0.8
Test accuracy
0.7
0.6
Baseline RFM(8)
RFM(32)
0.5
RFM(128)
0.5
0.6
0.7
0.8
0.9
Preference homogeneity (p)
Figure 8: Accuracy in predicting the preferences of 500 held-out users on the test set after adaptation (estimate of inter-user generalisation). Training was carried out with m = 60 raters and = 30 examples were used for adaptation. The homogeneity parameter was fixed 500 held-out users, and varied for the training raters as shown on the a-axis. Error bars are 99% confidence intervals over 5 runs.
0.65 for the
Comparison with the non-linear baseline. Park et al. 42 and Zhong et al. 64 propose principled (and similar) methods adapt a reward model to individual users. Since the former slightly more general, we compared RFM against it.
Using our terminology, we can describe Park et al. intermediate between training and adaptation. If we partition the shared parameters 0 = [0,, 02).
42 method as having single learning phase that
the method consists in keeping 01 fixed while 0₂ is learned together with the vectors wa associated with users. We note that, unlike RFM, this approach requires that the vectors Wi, are simultancously learned for all users h H we interested in adapting the model to.
Following Park et al. 42 experiments, we froze the backbone Gemma model (01), represented & as multilayer perceptron (MLP) with 3 or 5 hidden layers containing 32 units each (02), and learned the MLP together with the vectors Wh. make sure 0 was initialised with reasonable values, we ran our training phase with 0,, O2. and W being learned together, where the rows of W correspond to the raters h
H. (ef. Equation (6).
As shown in Figure 3 the non-linear baselines did not perform well. We conjectured that this may be due an excessive number of parameters to be learned with á 10 examples, we re-ran the experiment using ń 100. Results are shown in together with the other results from Figure Although the non-linear baselines perform slightly better with larger Å, they are still outperformed by non-adaptive and linear baselines, and considerably outperformed by RFM.
Table 2: Detailed comparison of RFM against linear and non- non-linear baselines (cf. Figure 3). The values shown the accuracy in predicting the preferences of 500 held-out users using examples for adaptation, together with 99% confidence intervals over runs.
Model
Test accuracy
Baseline
10
0.6258 = 0.0246
Linear baseline
0.6285 = 0.0176
Non-linear & (3 layers)
10
0.5543 + 0.0381
100
0.5706 + 0.0207
Non-linear & (5 layers)
10
0.5237 + 0.0273
100
0.5620 = 0.0452
RFM(32)
10
0. = 0.0182
An alternative to the non-linear architecture proposed by Park et 42 would be to have separate MLP per user (that is, replace with 0x). This architecture would probably required even more examples to be adapted, since the MLPs' parameters 0,, would be trained with the data Sj, only.
Comparison with in-context methods. We present the prompt used to evaluate the in-context capabilities of Gemini Pro and GPT-4o Figure It has three main parts: (i) the system instructions, delimited by [System] and [End of system]; (ii) sequence of 10 previous user
23

---

ratings (whose formatting is described in the system instructions); and (iii) the prompt, first response, and second response be assessed by the LLM. To avoid positional biases, we have the LLM assess the first and second responses both in their original order and in reversed order and average over the correctness of both L.LM outputs.
For each example in Ultraeeddacc'ss 's test set, we sample one user h uniformly at random from the set of 500 held-out users. We then draw from the training set 10 examples of responses previously ranked by h. The previously ranked responses are inserted in part (ii) of the prompt template, and the test example itself is inserted in part (iii) of the prompt template. We then compare the predicted comparison outcome against user h's ranking for that test example.
We also evaluate Gemini's zero-shot agreement with the held-out users (Gemini (zero-shot)) to help assess whether adding previously ranked responses as context makes a difference in terms of performance. In that setting, we omit the paragraph starting with "We will provide few examples" in part (i) of the prompt template and remove part (ii) altogether.
The fact that the LLMs' performance essentially reduces to chance in Figure 3 is somewhat surprising. We hypothesise the explanation is twofold: 1) despite the information being available in the prompt, the LLMs do not take advantage of the previous ratings to inform their decisions and instead revert to making a "judgement call" based on their existing alignment: and 2) this alignment is somewhat orthogonal to the features {ø, J==1 we used (described in Appendix B. Preliminary experiments with the features provided with the UltraFeedback dataset corroborate this hypothesis. The UltraFeed- back dataset comes with four features computed using OpenAl]s (39 GPT-4: helpfulness, honesty, instruction-follonn following, and truthfulness. Each example in the dataset has a score between and 5 associated with each feature. We defined user whose preferences were fully determined by the helpfulness features and reran the experiment described above. Results are in Table and demonstrate that 1) just like with our main in-context comparison experiment shown in Figure 3 the effect of 10-shot in-context prompting is negligible in comparison to zero-shot prompting, and 2) the LLMs perform considerably above chance when trying to predict preferences induced by a feature computed itself by an LLM.
Table Accuracy in predicting the preferences of a held-out user induced by the feature helpfulness provided with the UltraFeddback dataset. The three models used ń = 10 examples for adaptation.
Model
Test accuracy
Gemini 1.5 Pro (zero-shot)
0.6341
Gemini 1.5 Pro (in-context, 10-shot) GPT-4o (in-context, 10-shot)
0.6199
0.6392
We considered using the UltraFeedback features in our experiments, but for a more rigorous study we needed features that could be computed unequivocally and efficiently, and thus easily extrapolated beyond the UltraFeedback dataset. This was essential for the experiments with best-of-n shown in Figure 2 for example, in which we had to compute the features associated with examples (I., y, y²) not in the UltraFeedback dataset.
Comparison with VPL. Poddar et al. [44] evaluation protocol for VPL allows to measure intra- user generalization, but not inter-user generalization. This is because the preferences used for evaluation are obtained from the same four raters that were used to train the model. Those raters' preferences are derived from the four UltraFeedback features mentioned above: helpfulness,, honesty, and truthfulness, with each rater focusing exclusively on a single feature. In terms of evaluating intra-user generalization, VPI relies on an episodic protocol: for every rater hyg and for every tuple (z, y, y', 2) in the rater's test set, the authors simulate a new adaptation problem by drawing between two to eight other tuples sampled at random from the rater's test set to provide as context for the inference network to make a prediction. This means that the rater's 's identity h2 is never explicitly revealed to the model, but only contextually through the two to eight other tuples. In contrast, our evaluation methodology is more akin to transfer learning evaluation: for each rater, we adapt RFM on a held-out set of examples. The same held-out set used for all test examples, that is, we learn once using the held-out data and then use it to process all the test tuples (I, y, v/ 2).
instruction-ollowwie
24

---

[System]
[End of user question] Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user question displayed below.
rating starts with the tag [ser question] followed by [End user question]. After that, we have the tag [The Start Model A's ansuer] followed by the first response. The first response ends with the tag [The End of Model A's answer]. We then have the tag [The Start of Model B's answer] followed by the second response. The second response ends with the tag [The End of Model B's answer].
and the tag
We will provide few examples of previous ratings to help you understand the task. The example ratings will have the structure above followed the tag [Verdict], the verdict ("[[A]]" if assistant A is better, and "[[B]]" if assistant better), the tag [End verdict].
We are interested in your evaluation of the last two responses. Begin your evaluation by comparing these two responses and provide short explanation. Do not possible. After providing
favor certain names the assistants. Be as objective your explanation, output your final verdict by strictly following this format:
"[[A]]" if assistant A is better, and "[[]]]" if assistant B is better.
[End of system]
[User question]
<...>
[End of user question]
[The Start Model A's answer]
<...>
[The End of Model A's answer]
[The Start of Model B's answer] <...>
[The End of Model B's answer] [Verdict]
<...>
[End verdict]
<...>
[User question]
<...>
[The Start of Model A's answer]
<...>
[The End of Model A's answer]
[The Start of Model B answer] <...>
[The End Model B's answer]
Figure 9: The prompt template used for in-context evaluation. When evaluating Gemini's zero-shot capabilities we omit the paragraph starting with "We will provide a few examples" in the prompt.
25

---

The VPL authors provide the exact data used for evaluation on UltraFeedback For simplicity, we adapt our evaluation protocol as to get an unbiased estimate of the episodic metric used to evaluate VPI. We hold out 250 examples from VPL's training set to sample adaptation sets. After RFM has been trained on the four Utraaeedback raters derived from UltraFeedback features, we loop over raters, number á (2.3, 4, 5, 6, 7, 8) of examples used for adaptation, and 5 random seeds, each time drawing î examples from the held-out set for that rater, discarding the pre-trained set of weights for that rater, and learning new on the sampled adaptation set. We then aggregate the accuracies measured on the test set across adaptation set sizes, raters, and random seeds (see Figure TO for von-aggregated results). As result, the examples used for adaptation for each individual test example do not correspond exactly to the ones used by VPL (and are shared across test examples), but they are equally disjoint from the training set and their distribution is i.i.d. with respect to the distribution VPI. samples from.
We obtain an averaged test accuracy of 61.61 % against VPL's 61.49%. Given the methodological caveats outlined above, the conclusion we draw that, despite its simplicity, RFM achieves an intra-user generalization performance comparable with VPL's. We again highlight that this notion of generalisation is distinct from inter-user generalisation, which the main one we are targeting in this work, but for which VPL does not provide metrics to compare against.
nater h
Rater
Raner hs
Rater S
gor
0.
3 Number moumples adaptation
Nambbrr csurples used adugsation
Number cnumblca
Number exampria adaptat on
acaptatiin
Figure 10: Test accuracies for RFM(32) on the UltraFeeebbak-dderiveee UF-P-4 task introduced by Poddar et al. 44, broken down by rater and number of examples used for adaptation. The error bars represent 99% confidence intervals over randomly-sampled adaptation sets.
B.3 Modelling groups of real users
In the experiments described in Appendices B.1 and B.2 we had control over the definition of the training raters and held-out users. In the experiments with reward models acting users we removed this assumption. As mentioned in Section 5 we performed our experiments with two datasets:
UltraFeedback and PeM. We now describe each in turn.
For the experiments with UltraFeedback, shown in Figure 4a we used the following 8 publicly-
available reward models to emulate raters and users:
OpenAssiistanteewwardde htwps/iiatttt.aaaaoommodeeeeeeea
Bequeasdaa__RM-Missten 20 59
Ray2333_ G (60).
Ray2333_reward-moodde 20
[60].
Srvveaadda_RRMM-GGemmmaa
Beeemlm_imeemln2----rrwwww [23]
openbmb Beenbmb__uruu-Tee RM-7b 662.
All the models above are available on the Hugging Face website For each reward model ra and each
example (I;; yi, vf) in the training and test sets, we defined z (rx (Fi, yi) > ra (a, }- performed leave-one-old cross validation using the 8 resulting raters, as explained in Section 5 As
5
We
before, training was carried out for 6, 000 parameter updates with a batch size of 32, We performed
hhggingffce.cc co
The resulting data is available at bhl.genggccccooodasssrooggigiien
26

---

2 training runs followed by adaptation runs each, totalling 10 runs. Figure 11 shows the detailed results obtained in our experiments with the Ulrraeedbbckk dataset (Figure is slice of this figure with the number of adaptation examples fixed at á 50).
Bascline
RFM1128)
Held-out out reward model ]
Held-out reward model
Held reward model
Held-out reward
Text acturay
0.6
Hold- reward 5
Held-od out reward model
Hold out reward model
Test accurayy
0.66
50
70
30 5.0
10 Number of excamples used for adaptation
90
50 70
90
Number of excmmpes used for adaptation
Number of eocamppees used for adaptation
Number of examples used for adap adaptation tation
Figure 11: Accuracy in predicting the preferences of held-out Freward-moded users" on test set after adaptation (estimate of inter-user generalisation). Error bars are 99% confidence intervals over 10 runs (5 adaptation runs on top of 2 training runs).
We now describe our experiments with the PersonalLLM dataset, whose results are shown in Figure 4b) Each context in this dataset has 8 responses, and each response has been scored by 10 reward models. For our experiments, we picked the first two responses to each context and used them as our pair (v, y/). We then proceeded exactly as in the experiments with the Utrraeeddacck dataset, except that we performed 5 runs each involving training followed by adaptation.
C Expanded discussion of related work
In this section we present a more detailed discussion on the works we consider to be more directly relevant to ours.
Shenfeld et al. [52 concurrently propose the same architecture RFM, but under different name: "PReF", for personalisation via reward factorization. They also approach the subject from a different angle: while we study the learning problem from higher level of abstraction, and then specialise our results to RFM, they focus on how to exploit RFM's simple architecture to improve several aspects of the training pipeline. Among these, two contributions stand out: a stable initialisation through singular value decomposition and an efficient way of selecting examples for adaptation via active learning. Their theoretical results regard the latter. Shenfeld et al. (52) also present thorough empirical evaluation of their method, including ablation study of their proposed improvements, comparisons with Chen et al. [3] and Poddar al.]s approaches (discussed below), and small (but interesting) experiment involving human users.
the model
Bose et al. 8 also concurrently propose an architecture very similar to RFM, this time under the name LoRe (for "low-rank reward modelling"). The LoRe architecture closely resembles RFM's, in that both express individual reward functions as linear combinations of learned reward features (or "reward basis" in LoRe's terminology). A small, but potentially relevant difference is that their vector of coefficients is distribution (that is, Ed -i= W; = 1 and 0 < w; for all i 1, 1.2..., d). This slightly restricts the expressiveness of the model. Bose et al. [8] do not present theoretical results, but their empirical evaluation of the proposed architecture is extensive. They make some of the same comparisons as us namely: baseline, linear baseline, and VPL-and and also compare the proposed architecture with that of Chen et al. [13], discussed below.
Zhong et al. (64 and Park et al. [42 also propose methods that are reminiscent of RFM. Both papers put more emphasis on the theoretical analysis than on empirical results. Zhong et do not present experiments, while Park et al. only present simple empirical evaluations. We briefly describe one instantiation of Park et al. approach in Section 5 and Appendix B.2 and also compare it with RFM.
27

---

25 compositional preference model (CPM) expresses the learned reward function as a linear combination of features computed from the context and response. However, unlike RFM- which learns features using ( the CPM approach uses handcrafted ordinal features computed by L.Ms using pre-specified prompts.
Go et al.Js
Chen et al. 13 propose to replace the Bradley-Tenn -eery model with Coombs]s 177 ideal point model. This model posits that people make assessments of a given object based on the distance between the representation of the object in a latent space and an "ideal reference point" in the same space. While the inner product between RFM's Φ9 y) and Wi, does not strictly qualify as a distance function, we can think of Wn, as being akin to an "ideal point" for user h against which (1 y) is compared. Chen et al. [13] propose two different instantiations of the ideal point model -Model and Model B 8-which differ from RFM in few aspects. Like RFM's Φo (2, y), Model A computes features from the context and response jointly, but the ideal point constructed as a convex combination of "prototypical" ideal points and the comparison is made using the Euclidean distance. Model B defines an ideal point that depends on the context only, also as convex combination of prototypical ideal points. Unlike RFM, Model B computes response features separately and compares them to the context ideal point using the cosine similarity. Both Model and Model B constrain the user-specific parameters to lie a simplex whose number of vertices becomes an extra hyper-parameter (in addition to the dimension d of the ideal points). This presents an additional optimisation challenge and, as the authors themselves point out, does not generalise users whose ideal point would fall outside of the convex hull.
Poddar et al. 44 variational preference learning (VPL. works by encoding tuples {ari, Vi. VL. 2; i= associated with user into a latent variable, and then conditioning the reward model (and policy) on the latent variable. Zhao et al. [63 cast the problem of personalised preference learning as a few-shot learning problem and tackle it with in-context meta-lcarning. Their group preference optimization (GPO) approach trains transformer to predict target preferences for given homogeneous group of users from a context set of preferences from the same group. Like RFM, both works learn from a context set of preferences from a new user (or group of users). Unlike RFM-which learns through optimizing W both works amortize the process using a neural network (namely, VPL's encoder and GPO's transformer).
Li et al. 36 propose a personalised RLHF (P-RLHF) framework in which learnable user model computes an embedding for each user as the concatenation of an implicit embedding (that depends on the user's unique identifier) and an optional explicit embedding (that depends on textual information about the user). The user embedding is then used to condition base LLM through soft prompting. New users are accommodated by using a generic implicit embedding and computing an explicit embedding based on the new user's 'ser' textual information (if available).
Dumoulin et al. [22] present results on "rater misspecification". in which a model trained on the preferences of single rater which favors two characteristics (e.g., short generations or long generations, but not middling generations) behaves very differently from a model trained on the preferences of two raters, each of which favor one characteristic (short generations for one rater, long generations for the other). The authors frame the issue from the perspective of misspecified generative process for pairwise preferences and show that under correct assumptions (namely, the preference dataset is mixture of individual raters' preferences) it is possible to accurately capture multiple raters' preferences. In practice, the solution the authors propose (explicitly introducing as many models as there are raters) works in synthetic settings but was never shown to work at scale on larger problems. It assumes that one knows the number of raters in the preference dataset, but it does not assume that each individual preference tuple is annotated with the rater ID. The solution also does not consider how one would accommodate new users.
Chakraborty et al. 12 propose to learn a user-specific reward function by using an expectation- maximisation algorithm to define mixture of preference distributions. To learn single policy that better represent diverse human preferences, the authors propose a MaxMin alignment objective inspired by the Egalitarian principle in social choice theory.
D Broader impact
Beyond an improved user experience, adaptive reward models have great potential for positive impact. For example, they may allow for the inclusion of minority opinions and the representation of diverse
28

---

viewpoints in model outputs. However, as with most technological advances, this does not come without risks. If not implemented with ethical implications in mind, the specialisation of LLMs to user preferences may result in models behaving in undesirable ways. It may also reinforce existing points of view through sycophantic behaviour, contributing to the polarisation of opinions and the creation of "echo chambers".
There are ways to anticipate and mitigate these undesirable outcomes on the methodological, technical, and societal fronts. Methodologically, it is important to define rubrics for data collection and curation that clearly distinguish between genuine disagreement on subjective matters and denial of facts or deviations from prevalent ethical norms and scientific consensus. From a technical standpoint, the fact that sser-specillised reward model will be used to modulate the outputs of LLM renders the LLM itself a safeguard mechanism. For example, the best-of-n approach used in this paper re-ram re-ranks the L.LM's outputs based on the adapted reward model. Consequently, if the L.MM generates factually correct and ethical text, the re-ranking primarily prioritises or de de-prioritises aspects of a subjective lakue the style of the prose, for instance. Finally, on societal level, the personalisation of LLMs should be part of, and would directly benefit from, the wider ongoing discussion regarding the deployment of this new technology.
On the positive side, adaptive reward models and the consequent personalisation of LLMs offer great opportunity for the inclusion of diverse points of view into model outputs. Current reward models reflect the average preferences of the target population, which excludes under-represented or "outlier" preferences. In using specialised reward models adapt LLM outputs individuals, we create systems that can more accurately and reliably reflect the perspectives of users who hold minority views, potentially empowering them, together with everyone else, to participate more fully in social debate.
29

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All claims in the abstract are supported by experiments (Section and Ap- pendix B or proofs (Appendix A
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: There is a paragraph in Section (titled Limitations) in which the limitations the proposed model are explicitly discussed. Paragraph Training of the same section discusses RFM's requirement to select proper number of dimensions d for Φgg.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
30

---

Answer: [Yes]
Justification: Propositions I 2 3 and 4 are proved in Appendix A All the assumptions
underlying these results clearly stated in the text.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Complete experiment details are provided in Section 5 and Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., . a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
31

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: While the Gemma 1.1 2B [26] model used in our work is open-sourced, the specific training scripts used for our data preparation and model training not included in this release because they are deeply integrated with the proprietary infrastructure used to carry out experiments, making them unsuitable for public release.
However, we have taken the following steps ensure transparency and facilitate the understanding and potential reproduction of our work:
Data sources and preparation: we use two publicly available datasets, UltraFeed- back [18] and PersonalLLM [65], and provide detailed description of how we gener- ated synthetic users' preferences to function as labels (Section and Appendix (B)
For the experiments using reward models as users, we used publicly available models and provide the full references in Appendix B.3 We also made the resulting labelled data publicly available on the Hugging Face website, detailed in Appendix B.3
Experimental setup: We have provided comprehensive details of all experimental setups, including model configurations, hyperparameters, and the training procedures in Section 5 and Appendix
While the exact training scripts cannot be shared due to their infrastructural dependencies, the provided information on the open-source model (Gemma 2B), detailed dataset descriptions, and thorough explanation of the training configurations should allow the research community to understand our methodology and replicate the core findings.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https/cccppcccgguuieeeeeee
pondesuummissionoooiooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Section 5aas out the datasets used for the experiment, the train/test split, and how synthetic users are constructed and their preferences are sampled. Further details are provided in Appendix
Guidelines:
32

---

The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report enror bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Unless otherwise noted, test accuracies are reported the average over 5 independent runs (that is, with 5 different seeds) and are accompanied by 99% confidence interval.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma eгror bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the classifier of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: We focus the statistical (rather than computational) properties of the proposed approach. That is, both in our theoretical results and in our experiments we are mostly concerned with the methods' sample complexity. All the techniques use have well-understood demands in terms of compute and memory.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
33

---

9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:e///eerrrss.../ooboll////i/
Answer: [Yes]
Justification:
The research conducted in the paper conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification: We have thorough discussion on the potential broader impact of this work in Appendix D
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
34

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The paper properly credits the creators of all the assets used: Gemma 1.1 2B [26]. Gemma 2 9B [27] Gemma 2 27B [27]. the UltraFeedback dataset [88, the
PersonalLLM dataset [65). the 8 publicly available reward models (Appendix B.3), Gemini
1.5 Pro 128], and GPT-4o 40
All assets were used accordance with their respective
licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi phcore thcode. com/ datasete
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: As detailed in Appendix B.3 released dataset with the examples in the UltraFeedback dataset scored by 8 publicly available reward models. The data has been made available the Hugging Face website with detailed accompanying documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
35

---

The paper does not present crowdsourcing experiments or research with human subjects.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not perform experiments on human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: The paper introduces a reward model architecture that can be implemented using an L.M.. In our experiments we used Gemma 1.1 2B (26 to implement the proposed
model. The usage of Gemma described details in Section 5 and Appendix No LLMss
were used in the writing of the paper itself.
Guidelines:
The answer NA means that the core method development in this research does not involve L.MMs any important, original, non-standard components.
Please refer to LLM policy lhtt::://heuriiss.c//offeeeeeeee for what should or should not be described.
36


Reviews:
[{"review_id": "3wbSaBVtcm", "reviewer": "Reviewer_mJwX", "summary": "The authors formalize the problem of user-specialized reward modeling for RLHF in settings with heterogeneous human preferences, deriving a PAC bound that highlights the dependence of approximation error on both data quantity and rater diversity. They propose an adaptive reward model where individual preferences are represented as a linear combination of shared reward features, enabling fast personalization to new users with few examples.", "questions": "Can you conceptually compare your approach with LoRe (https://arxiv.org/pdf/2504.14439)? It appears to be highly relevant to your work.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n- Overall idea: The paper addresses a key limitation in current RLHF for LLMs by introducing user-specialised reward models that adapt to individual preferences, including those of users not present in the training data. The idea that individual preferences can be represented as a linear combination of shared reward features is both intuitive and practically implementable.\n- Reproducibility: Although training scripts are not released due to proprietary constraints, the paper provides comprehensive details on experimental setups, model configurations, hyperparameters, and adaptation procedures, making reproduction of the main results feasible. It would be helpful to improve the reproducibility if the authors can release the rater annotations.\n- Paper presentation: The paper is clearly written and well-structured, presenting complex theoretical analyses and empirical methods in a logical progression. The formalization of the problem and the derivation of PAC bounds are particularly clear, facilitating understanding of both the motivation and the solution.\n\n**Weaknesses**\n- The experimental evaluation relies primarily on the UltraFeedback dataset with synthetic raters, which, while offering high experimental control, limits the demonstration of the model’s effectiveness on diverse datasets. The empirical claims would be strengthened if the authors can show the results on the estabished preference datasets for personlization like personalLLM (https://arxiv.org/abs/2409.20296).", "was_revised": true, "final_justification": "With the new results added during the rebuttal, I believe this paper will make a positive contribution to our community, and I am raising my score to 5.", "rebuttal": "Thank you for your comments, and in particular for calling our attention to both the PersonalLLM dataset and the LoRe work! Following your suggestion, **we carried out an additional experiment using the PersonalLLM dataset**. We also like your suggestion to make our rater annotations available, and will follow it upon the release of the paper.\n\n\n&nbsp;\n\n---\n\n> **1. The empirical claims would be strengthened if the authors can show the results on the estabished preference datasets for personlization like personalLLM.** \n\n&nbsp;\n\nThanks again for pointing us to the PersonalLLM dataset, it is indeed a great asset in the study of adaptive reward models. Leveraging the fact that the examples in PersonalLLM have been scored by reward models, we conducted an experiment analogous to the one described in the subsection “Modelling groups of real users” of our paper’s Section 5.\n\nWe now briefly describe the details of the experiment and show the results. Each prompt in the PersonalLLM dataset has 8 responses, and each response has been scored by 10 reward models. For our experiments, we picked the first two responses to each prompt $x$, and used them as our pair $(y, y’)$. We then proceeded as in the experiments described in the section “Modelling groups of real users”: we filtered out all the examples in the training and test sets in which $2$ or fewer raters disagreed with the majority, and then performed leave-one-out cross-validation, holding out each model in turn as a user while using the remaining 9 models as training raters. \n\nBelow we show the results obtained by the baseline, the linear baseline (as described on line 326), and RFM(128). Each row shows the models’ accuracy in predicting the preferences of the held-out reward model (and corresponding 99% confidence intervals over $5$ runs).\n\n&nbsp;\n\n| Held-out model | Baseline | Linear baseline | RFM(128) |\n| :--- | :--------------- | :--------------- | :--------------- |\n| **0** | 0.3660 +- 0.0000 | 0.6534 +- 0.0013 | **0.7118 +- 0.0015** |\n| **1** | 0.8378 +- 0.0015 | 0.8438 +- 0.0005 | **0.8604 +- 0.0015** |\n| **2** | 0.8252 +- 0.0005 | 0.8252 +- 0.0005 | **0.8260 +- 0.0023** |\n| **3** | 0.8270 +- 0.0000 | 0.8444 +- 0.0006 | **0.8680 +- 0.0016** |\n| **4** | 0.8374 +- 0.0006 | 0.8260 +- 0.0008 | **0.8594 +- 0.0017** |\n| **5** | 0.7998 +- 0.0005 | **0.8190 +- 0.0000** | 0.8160 +- 0.0008 |\n| **6** | 0.8446 +- 0.0006 | 0.8428 +- 0.0005 | **0.8536 +- 0.0006** |\n| **7** | 0.6148 +- 0.0005 | 0.6060 +- 0.0000 | **0.6630 +- 0.0028** |\n| **8** | 0.6280 +- 0.0000 | **0.6422 +- 0.0005** | 0.6350 +- 0.0008 |\n| **9** | 0.6280 +- 0.0000 | 0.6022 +- 0.0005 | **0.6446 +- 0.0050** |\n\n&nbsp;\n\nAs one can see, results are similar to those shown in Figure 2(c). We first focus on the comparison between RFM and the baseline, which represents the current practice. Our model outperforms the baseline in all cases. The baseline performs well in predicting some of the held-out model’s preferences, which probably means that these specific models “agree” with the majority. In these cases RFM still outperforms the baseline, although the difference is small. However, for other reward models (notably model number 0) the baseline performs poorly, likely as a consequence of the fact that these models disagree with the majority. In such cases RFM significantly outperforms the baseline. As noted in our previous experiments with real preference data, RFM can work as a form of “safety net” to make sure that minority preferences are also represented.\n\nBecause the linear baseline can adapt to new users (albeit using user-agnostic features), it performs better than the baseline in most cases. Still, RFM outperforms the linear baseline in 8 out of 10 held-out models. \n\nWe note that the reward models playing the role of users in this experiment in fact reflect the preferences of aggregated real users. This means that some of the heterogeneity of the real population has probably been washed away in the training of these models. We conjecture that if the reward models were replaced by the real users underlying these models, RFM’s advantage over the two baselines would be greater still.\n\nWe plan to add the results above to the paper in a figure analogous to Figure 2(c); please let us know if you have any further suggestions.\n\nAs a final note, we observe that, in addition to carrying out the experiments above, we have also conducted experiments with additional baselines, as described in Response 6 to Reviewer LWzi.\n\n\n&nbsp;\n\n> **2. Can you conceptually compare your approach with LoRe?**\n\n&nbsp;\n\nThank you for calling our attention to the LoRe paper, we were not aware of it. One of the reasons we might have missed it is that it only became available on arXiv on 20 April, 2025 –that is, less than a month before NeurIPS’ deadline. As a kind reminder of NeurIPS’ policy, _“Papers appearing online after March 1st, 2025 are generally considered concurrent to NeurIPS submissions. Authors are not expected to compare to those.”_\n\nHowever, the reviewer is asking us to make a conceptual comparison, which seems fair. The comparison can be made at different levels of abstraction. From a strictly mechanistic perspective, the LoRe architecture is very close to ours, in that both our approaches express individual reward functions as linear combinations of learned reward features (or \"reward basis\" in LoRe's terminology). A small, but potentially relevant difference is that their vector of coefficients $w$ is a distribution (that is, $0 \\le w_i \\le 1$ and $\\sum_i w_i = 1$). This restricts the expressiveness of the model. \n\nIn terms of collecting empirical evidence in favour of the proposed architecture, Bose et al. make some of the same comparisons we do, namely: baseline, linear baseline, and VPL. They also compare LoRe with Chen et. al’s [10] _pluralist alignment_ (PAL), which we do not. On the other hand, they do not compare their method against the in-context LLM solutions, as we do in Figure 2(c), or with the non-linear features added following a suggestion by Reviewer LWzi (see Response 6 to that reviewer). They also use 3 datasets (PersonalLLM, Reddit TLDR, and PRISM), all of them considerably smaller than the UltraFeedback dataset we adopted (as discussed in Response 1 above, we have now also conducted experiments using PersonalLLM). Finally, Bose et al. do not investigate how LoRe’s performance transfers to the steering of the downstream LLM. As shown in Figure 2(a), we do so using best-of-$N$.\n\nAt a more analytical level, Bose et al. do not present theoretical results. As discussed in Response 3 to Reviewer DjQY, our theoretical analysis sheds light on the learning of the features $\\phi$. Specifically, we have shown how the associated generalisation error depends on the number of raters $m$, the number of examples per rater $n$, and the variance terms $\\mathbb{E}[\\mathbb{V}(\\ell_c |H )]$ and $\\mathbb{V}(\\mathbb{E}[\\ell_c |H])]$. That is, we have quantified how fast our approximation approaches the best possible approximation in its class. \n\nBesides providing a solid theoretical ground for RFM (and LoRe), our results also help to frame the learning of $\\phi$ in the more rigorous terms of empirical risk minimisation. This in turn informs us how to think about the use of our model in practice. For example, we make it explicit an assumption that is usually only implicit in previous work: users and raters come from the same distribution. We also show in Figure 4 what happens when this is not the case.\n\nAll that said, Bose et al.’s paper is a solid, well-executed piece of research. We see their concurrent work and ours as corroborating each other in building theoretical arguments and empirical evidence in favour of the proposed architecture. We will add a discussion on LoRe to our Related Work section as concurrent work.\n\n&nbsp;\n\n> **3. It would be helpful to improve the reproducibility if the authors can release the rater annotations.**\n\n&nbsp;\n\nThis is a great idea! We will release the annotations along with the camera-ready version of the paper.\n\n---\n\n&nbsp;\n\nWe thank the reviewer again for their comments and suggestions! Please let us know if any further clarifications are needed, and also if you have any further suggestions."}, {"review_id": "jQoKWCVc0g", "reviewer": "Reviewer_LWzi", "summary": "The paper highlights a key limitation in the current RLHF paradigm which is the assumption of homogeneous human preferences since users often disagree especially on subjective tasks hence learning a single, averaged reward model may misrepresent many individuals. To mitigate the above, the paper proposes an efficient framework for adaptive and personalized reward learning via learning shared features across users and with user-specific features for personalization. The paper provides strong theoretical guarantees with PAC bounds on the generalization error and empirical results showing benefit over standard (non-personalized) reward models.", "questions": "See Strengths And Weaknesses", "limitations": "See Strengths And Weaknesses", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "One of the key strengths of the paper lies in a rigorous theoretic analysis with PAC-style bounds that explicitly model inter-user and intra-user variance in reward modeling. This shows how the generalization error depends separately on the number of users, number of examples per user and variance across user-preferences. One key interesting concept that increasing data per user has diminishing returns unless the number of users also increases. The proposed approach generalizes to new users with as few as 10–30 labeled comparisons and thus efficient personalization is enabled without any need for gradient-based fine-tuning of the full model. Empirical results are nice which shows that the approach outperforms single utility, non adaptive, in-context based baselines.\n\nAlthough the paper provides a principled approach, however the proposed method still assumes linear parametrization which may be overly restrictive in practice and fail to capture complex user-specific reward structures. Additionally, the feature $\\phi$ requires futher assumptions? While adapting to new user preferences, which is a key component to the approach, how can we ensure $\\phi$ will generalize to them ? The paper does not address what assumptions on user preference distributions or feature richness are required to guarantee that $\\phi$ will generalize well to unseen users. \n\nThe paper provides strong theoretical guarantees, however the PAC bounds are standard and the paper doesn't explain why or what are the exact challenges or unique components in the current theoretical results for the specific problem? The bounds use covering number, but do not consider Rademacher complexity or VC-dimension, which could provide sharper insights or tighter sample complexity results.\n\nThe term Var(E[l_c|H)] plays an extreme crucial role in characterizing the diversity, which play a critical role in quantifying preference heterogeneity. However, the interpretation of these terms is only briefly discussed, and a more detailed explanation is essential. In particular, how these terms scale with respect to population size, user diversity, or feature misalignment should be discussed to better understand when the proposed method will be effective.\n\nFinally, the experimental section needs comparison with more SoTA baselines with multi-rewards which is currently missing. It will be helpful to compare and contrast with [1], [2] which are related and highlight the key benefits over the above methods. This would help position the proposed approach more clearly in relation to existing work and highlight its empirical advantages or trade-offs.\n\n\n[1]. Provable Multi-Party Reinforcement Learning with Diverse Human Feedback\n[2]. RLHF from heterogeneous feedback via personalization and preference aggregation", "was_revised": true, "final_justification": "I have read the rebuttal and most of my concerns are addressed.", "rebuttal": "Thank you for your thoughtful comments! Following your suggestions, **we've derived an additional theoretical result based on the Rademacher complexity and carried out an extra experiment featuring the SOTA comparisons you recommended**. We are confident these additions significantly enhance the paper's quality.\n\n&nbsp;\n\n---\n\n>  **1. [...] the proposed method still assumes linear parametrization which may be overly restrictive in practice and fail to capture complex user-specific reward structures.** \n\n&nbsp;\n\nAlthough the RFM approximation is linear in the features, the features themselves are arbitrary, potentially highly non-linear, functions of the inputs. They are also learned based on the preferences of raters sampled from the same distribution future users will come from. This means that, with enough parameters and features, we can approximate the users’ preferences at any desired level of accuracy. In principle, if we have one feature per user and a large approximator, we effectively have a separate reward model per user. In practice, we only need the users’ preferences to lie on a lower-dimensional manifold, and have enough raters and data in the training set to capture the geometry of this manifold.\n\nThat said, we acknowledge that a non-linear dependency on the features could in some cases more easily capture user-specific reward structures (in the sense that we would need fewer user-specific parameters). We do think however that the benefits provided by a linear dependency on the features largely outweigh this potential drawback. These benefits are discussed on line 249, and summarised here:\n\n1. The adaptation problem is convex;\n\n2. We can use robust linear algebra techniques to improve the adaptation [46];\n\n3. The features (and their contributions) are interpretable;\n\n4. We can add handcrafted features; \n\n5. Importantly: **we can resort to known RL techniques to synthesise a personalised LLM based on the vector of coefficients $w$, without any learning involved and with performance guarantees [4–7, 28 , 43 , 11]**. With a non-linear dependency on the features, one has to either fine-tune the LLM with RL for each user, which is impractical, or resort to methods without performance guarantees.\n\nPlease also see Response 2 to Reviewer DjQY.\n\n\n&nbsp;\n\n> **2. Additionally, the feature  requires futher assumptions? While adapting to new user preferences, which is a key component to the approach, how can we ensure  will generalize to them ?**\n\n&nbsp;\n\nWe can think of each user as a separate classification problem over the same input space. Under this view, we have to learn features such that all labellings of the inputs induced by the users are a linear classification. In other words, users’ preferences lie in a $d$-dimensional manifold, and our goal is to learn such a manifold. As long as we have an approximator with at least $d$ features and enough capacity, we should be able to learn the manifold given enough raters $m$ and examples per rater $n$, as indicated by our theoretical results (modulo standard optimisation issues). We will add this explanation to the paper. \n\nPlease see Response 3 to Reviewer 4gUj for a related discussion.\n\n&nbsp;\n\n> **3. …the PAC bounds are standard and the paper doesn't explain why or what are the exact challenges or unique components in the current theoretical results for the specific problem?**\n\n&nbsp;\n\nOne challenging aspect of our theoretical analysis, mentioned on lines 44 and 126, is the fact that the data is not i.i.d. Another interesting aspect is the use of Bernstein’s concentration inequality instead of the more commonly used Hoeffding's counterpart. This was necessary to surface the dependency of the generalisation error on the inter- and intra-user loss variances, as noted in your review.\n\nPlease see Response 7 to Reviewer DjQY for an extended discussion.\n\n&nbsp;\n\n> **4. The bounds use covering number, but do not consider Rademacher complexity or VC-dimension, which could provide sharper insights or tighter sample complexity results.**\n\n&nbsp;\n\nThe main goal of our theoretical analysis is to show the qualitative dependency of the generalisation error on $n$, $m$, and the inter- and intra-user loss variances. Instead of making structural assumptions to get the tightest possible bound, we chose to present a general result that illustrates the relevant trends. \n\nThat said, the point raised by the reviewer is interesting. It is indeed possible to obtain stronger results in our setup when assuming additional structure on the hypothesis class. We briefly indicate how Rademacher complexity bounds, as the reviewer suggests, can be used to obtain such bounds when the classifier is logistic regression over concatenated embeddings of user, prompt, and responses.\n\nThis bound can be obtained by following a standard Rademacher generalisation bound argument (e.g. as in Chapter 26 of [45]) at the level of user data, which relies on McDiarmid's inequality, and then applying Bernstein's inequality later in the argument to reveal a dependence on the $n$ parameter, as well as inter- and intra-user loss variances. Such an argument yields the following bound:\n\n$$\n    L_\\mathcal{D}(c^\\*) \\leq L_\\mathcal{D}(\\tilde{c}^\\*) + \\frac{2W}{\\sqrt{m}} + 3 \\sqrt{\\frac{g}{2m}} + \\frac{1}{3m}\\Bigg[ g + \\sqrt{g^2 + 18gm\\big( \\frac{1}{n} \\mathbb{E}[\\mathbb{V}(\\ell_c |H )] + \\mathbb{V}(\\mathbb{E}[\\ell_c |H])] \\big)} \\Bigg],\n$$\n\nwhere $g=\\log(6/\\delta)$. Here, we assume the concatenation of embeddings is guaranteed to have norm at most 1, and that we consider fitting weight vectors in the logistic regression of norm at most W, as is common in the analysis of such regression models. \n\nWe will add the above result to the paper; thank you for your suggestion!\n\n&nbsp;\n\n> **5. The term Var(E[l_c|H)] plays an extreme crucial role [...] In particular, how these terms scale with respect to population size, user diversity, or feature misalignment should be discussed to better understand when the proposed method will be effective.**\n\n&nbsp;\n\nThank you for the question; we agree that the term $\\mathbb{V}(\\mathbb{E}[\\ell_c |H])$ plays a crucial role. We will add a more detailed discussion. Here's a brief summary:\n\n$\\mathbb{V}(\\mathbb{E}[\\ell_c |H])$ depends both on the diversity of the population $\\mathcal{H}$ and the capacity of the classifier $c$ to capture it. Therefore:\n\n- **User diversity**: for a fixed classifier $c$,  $\\mathbb{V}(\\mathbb{E}[\\ell_c |H])$ should never decrease with user diversity, and it will in general increase. \n\n- **Feature misalignment**: For a fixed population $\\mathcal{H}$,  $\\mathbb{V}(\\mathbb{E}[\\ell_c |H])$ should never decrease with more misaligned features, and it will in general increase. \n\n- **Population size**:  $\\mathbb{V}(\\mathbb{E}[\\ell_c |H])$ does not necessarily increase with the population size (if all users $h$ have the same preferences, this term is zero regardless of $|\\mathcal{H}|$). In practice, we expect $\\mathcal{H}$ to generally become more heterogeneous with its size, which likely yields an increase in  $\\mathbb{V}(\\mathbb{E}[\\ell_c |H])$, as discussed.\n\n&nbsp;\n\n> **6. It will be helpful to compare and contrast with [i]. Provable Multi-Party Reinforcement Learning with Diverse Human Feedback [ii]. RLHF from heterogeneous feedback via personalization and preference aggregation**\n\n&nbsp;\n\nWe have carried out a comparison between RFM and [ii]. We chose [ii] because their method is slightly more general than [i]. Using our terminology, we can describe [ii] as having a single learning phase that is intermediate between training and adaptation. If we partition the shared parameters $\\theta = [\\theta_1, \\theta_2]$, the method consists in keeping $\\theta_1$ fixed while $\\theta_2$ is learned together with the vectors $w_h$ associated with users $h \\in \\mathcal{H}$. \n\nFollowing the experiments in [ii], we achieved this by freezing the backbone Gemma model ($\\theta_1$), representing $\\phi$ as a multilayer perceptron (MLP) with $k$ layers ($\\theta_2$), and learning it together with $w_h$ (to make sure $\\theta_2$ was initialised with reasonable values, we re-ran our training phase with both $\\theta_1$ and $\\theta_2$ being learned). We then repeated the experiments shown in Figure 2(b) using this architecture. The results for $\\phi$ using $k=3$ and $k=5$ hidden layers with 32 units each can be seen in the table below. For reference, we also included the other results in Figure 2(b). \n\n&nbsp;\n\n| Model                     | $\\hat{n}$ | Test accuracy +- 99% confidence interval |\n| :------------------------ | :-------- | :-------- |\n| Baseline                  | 10        | 0.6258 +- 0.0246                         |\n| Linear baseline           | 10        | 0.6285 +- 0.0176                         |\n| Non-linear $\\phi$ (3 layers) | 10        | 0.5543 +- 0.0381                         |\n|                           | 100       | 0.5706 +- 0.0207                         |\n| Non-linear $\\phi$ (5 layers) | 10        | 0.5237 +- 0.0273                         |\n|                           | 100       | 0.5620 +- 0.0452                         |\n| RFM(32)                   | 10        | 0.7053 +- 0.0182                         |\n\n\n\n&nbsp;\n\nThe models with non-linear $\\phi$ do not perform well. We conjectured that this may be due to the number of parameters to be learned, so we re-ran the experiment using $\\hat{n} = 100$ examples rather than $10$. Although the results improved slightly, they are still below the baselines, and considerably below RFM.\n\nWe will include these results in Figure 2(b). \n\nAs a final note, we highlight the fact that we have comparisons with _variational preference learning_ (VPL) [38] –arguably also a SOTA method–, as described in the subsection “Comparisons” of Section 5. \n\nPlease see Response 6 to Reviewer DjQY for a related discussion.\n\n---\n\n&nbsp;\n\nThank you again for your review! Please let us know if there are any further concerns to be addressed."}, {"review_id": "q1vvYBRYO2", "reviewer": "Reviewer_4gUj", "summary": "This paper considers a more general scenario of Reinforcement Learning from Human Feedback (RLHF), addressing the potential user preference heterogeneity that is usually ignored in the conventional RLHF. This paper provides theoretical justifications showing that the estimate bias depends not only on the number of examples but also on the number of different raters. A new reward-model architecture, Reward-Feature Model (RFM), is introduced for personalized RLHF. RFM separates reward function’s parameters into shared ones and user-specific ones, enabling fast adaptation to new users by fine-tuning only user-specific parameters. Experiments with LLMs demonstrate RFM’s superiority over non-adaptive baselines, in-context personalization methods on the test accuracy in predicting user preferences.", "questions": "1. In real-world scenarios, users may not be able to provide 30 (or more)  high-quality samples. Have you tested performance under more extreme few-shot conditions (e.g., $m \\leq 5$) and how does the model perform?\n2. The bound established in Proposition 2 appears to exhibit strong dependence on the dimensions of $\\mathcal{H}$, $\\mathcal{X}$, and $\\mathcal{Y}$. Is this bound tight or is it possible to further improve the dependence on dimensions?", "limitations": "See weaknesses.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. This paper rigorously shows that how the estimate error depends on variance induced by user preference heterogeneity, and it can not be fully eliminated by only increasing the number of examples, which sheds the light on the necessity for considering user-specific reward models. \n2. The proposed reward model, RFM, represent individual reward as a linear combination of  shared reward features, enabling data-efficient adaptation to new users with minimal feedback, which is well-suited for real-world scenarios where only limited preference data may be available for new users.\n3. Empirical results show that RFM outperforms the non-adaptive baseline and demonstrates particularly strong advantages in scenarios with higher user heterogeneity. It also shows that RFM can rapidly adapt to new users only with a few examples. Experiments with LLM verify that RFM provides a good reward function for LLM to produce better response compares to the baseline. Comparisons based on the real-world dataset are also provided.\n\nWeakness\n1. Although Figure 1 demonstrate the model's effectiveness particularly under high user heterogeneity, it is worth noting that the synthetic data is constructed precisely following the RFM framework. It remains unclear whether the results showed in Figure 1 would generalize to preference data that does not conform to this modeling assumption.\n2. The experimental validation with real human preference data only includes comparisons between RFM and the baseline model, omitting performance benchmarks against other adaptive methods.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thoughtful comments and suggestions! We **have run an additional experiment to address your question regarding extreme few-shot adaptation conditions**. Following your comment, **we have also added an adaptive baseline to one of our experiments with real human preference data**. \n\n&nbsp;\n\n\n---\n\n> **1. In real-world scenarios, users may not be able to provide 30 (or more) high-quality samples. Have you tested performance under more extreme few-shot conditions (e.g., m <= 5) and how does the model perform?**\n\n&nbsp;\n\nIn the context of LLMs, providing a sample for adaptation means choosing between two candidate responses to a given prompt. The collection of sample responses does not have to happen all at once; for example, a system can present the user with alternative responses at reasonable intervals. Under these conditions, collecting 30 samples does not seem infeasible.\n\n\nThat said, the reviewer is right to point out that in some circumstances 30 samples may be too many; also, regardless of the practical use of the model, understanding its behaviour under extreme few-shot conditions is a relevant scientific question. To investigate this question, we extended the RFM(32) results in Figure 1(c) to include test accuracies in 1-, 3-, and 5-shot settings. The results are shown in the table below. RFM(32)'s results in Figure 1(c) are also shown for reference.\n\n\n&nbsp;\n\n| $\\hat{n}$ | Test accuracy +- 99% confidence interval|\n|---|---|\n| 1 | 0.5481 +- 0.0217 |\n| 3 | 0.5749 +- 0.0109 |\n| 5 | 0.5891 +- 0.0056 |\n| 10 | 0.7053 +- 0.0182 |\n| 30 | 0.7657 +- 0.079 |\n| 50 | 0.7770 +- 0.0333 |\n| 70 | 0.7835 +- 0.0217 |\n| 90 | 0.7900 +- 0.0172 |\n\n\n&nbsp;\n\nAs expected, the model performance improves monotonically as the number of examples $\\hat{n}$ used for adaptation increases. Note that even the results with a single example ($\\hat{n}=1$) remain above random chance (though significantly lower than the 30-shot test accuracy).\n\n\nWe point out that worse performance with very few adaptation examples is expected, for it reflects the intrinsic difficulty of the learning problem. Given that each user's preferences are derived from a combination of 13 features, many different combinations of those 13 features may explain the preference behind a few training examples, and multiple training examples may be necessary to disambiguate. It is reassuring to see that RFM is still able to capture some of the problem structure under these extreme conditions, and that its performance monotonically increases with $\\hat{n}$.\n\nWe plan to add these results to the paper in case future readers are also curious about RFM's performance under extreme few-shot conditions; please let us know if you have further suggestions.\n\n&nbsp;\n\n> **2. The bound established in Proposition 2 appears to exhibit strong dependence on the dimensions of H, X, and Y. Is this bound tight or is it possible to further improve the dependence on dimensions?**\n\n&nbsp;\n\nBy assuming structure on either the hypothesis class or data distribution, tighter generalisation bounds can be obtained through e.g. Rademacher complexities; please see our Response 4 to Reviewer LWzi for how this looks in practice. Our aim with the bounds as displayed in the paper is to illustrate the form of generalisation bounds we obtain with our non-iid data, in particular its dependency on the number of raters $m$, the number of examples per rater $n$, and the inter- and intra-user loss variance terms in these bounds ($\\mathbb{V}(\\mathbb{E}[\\ell_c |H])]$ and $\\mathbb{E}[\\mathbb{V}(\\ell_c |H )]$). Instead of making assumptions about the hypothesis class or data distribution to derive the tightest possible bound, we chose to present a more general result that clearly shows the dependence of the generalisation error on the factors above.\n\nPlease see Response 4 to Reviewer LWzi for an example of how our bound can be specialised using the Rademacher complexity.\n\n\n&nbsp;\n\n> **3. Although Figure 1 demonstrate the model's effectiveness particularly under high user heterogeneity, it is worth noting that the synthetic data is constructed precisely following the RFM framework. It remains unclear whether the results showed in Figure 1 would generalize to preference data that does not conform to this modeling assumption.**\n\n\n&nbsp;\n\nWe call attention to the results shown in Figure 2(c), whose data was generated by real (aggregated) human preferences. Furthermore, following a suggestion by Reviewer mJwX, we have carried out additional experiments with another dataset, PersonalLLM, which also contains data based on real human preferences (see Response 1 to Reviewer mJwX). In both cases RFM considerably outperformed the baseline, which represents the current practice.\n\n\nWe used features to generate the synthetic data mostly as a strategy to easily change the number of users and the heterogeneity of their preferences (the same strategy is suggested in [i]; see their equation 1). Although strictly speaking it is true that the resulting synthetic data conforms with the modelling assumption underlying our model, we highlight that RFM never had access to the “real” features.\n\n\nFinally, we argue that the modelling assumption underlying RFM should hold in many real scenarios. We can think of each user as inducing a different classification problem over the same inputs (that is, users only change the labelling of the inputs). In the limit, if we have at least as many features as users, and enough representational capacity, we effectively have a separate classifier per user. More realistically, we expect there to exist a $d$-dimensional manifold over which the classification problems induced by all the users are (approximately) linear. If this is the case, and as long as we have at least $d$ features and enough capacity, we will eventually learn such a manifold given enough raters $m$ and examples per rater $n$, as shown by our theoretical results (modulo the usual optimisation difficulties associated with deep learning). \n\n\nFor related discussions, please also see Responses 2 and 4 to Reviewer DjQY and Responses 1 and 2 to Reviewer LWzi.\n\n\n[i]. PersonalLLM: Tailoring LLMs to Individual Preferences. Thomas P. Zollo, Andrew Wei Tung Siah, Naimeng Ye, Ang Li, Hongseok Namkoong.\n\n\n&nbsp;\n\n\n> **4. The experimental validation with real human preference data only includes comparisons between RFM and the baseline model, omitting performance benchmarks against other adaptive methods.**\n\n\n&nbsp;\n\n\nWe have emphasised the comparison with the baseline in our experiments because it represents the current practice. But we agree that including the results from other adaptive models in experiments with real human preference data would provide a more comprehensive picture.\n\nTo address this, we leveraged the new experiment we conducted on the PersonalLLM dataset, following a suggestion by Reviewer mJwX, which uses real human preference data. In addition to comparing RFM(128) with the baseline, we also compared it with the linear baseline (as described on line 326), **which is the adaptive model that performed best in our comparisons (cf. Figure 2(b) and Response 6 to Reviewer LWzi)**. The results are shown below:\n\n&nbsp;\n\n| Held-out model | Baseline | Linear baseline | RFM(128) |\n| :--- | :--------------- | :--------------- | :--------------- |\n| **0** | 0.3660 +- 0.0000 | 0.6534 +- 0.0013 | **0.7118 +- 0.0015** |\n| **1** | 0.8378 +- 0.0015 | 0.8438 +- 0.0005 | **0.8604 +- 0.0015** |\n| **2** | 0.8252 +- 0.0005 | 0.8252 +- 0.0005 | **0.8260 +- 0.0023** |\n| **3** | 0.8270 +- 0.0000 | 0.8444 +- 0.0006 | **0.8680 +- 0.0016** |\n| **4** | 0.8374 +- 0.0006 | 0.8260 +- 0.0008 | **0.8594 +- 0.0017** |\n| **5** | 0.7998 +- 0.0005 | **0.8190 +- 0.0000** | 0.8160 +- 0.0008 |\n| **6** | 0.8446 +- 0.0006 | 0.8428 +- 0.0005 | **0.8536 +- 0.0006** |\n| **7** | 0.6148 +- 0.0005 | 0.6060 +- 0.0000 | **0.6630 +- 0.0028** |\n| **8** | 0.6280 +- 0.0000 | **0.6422 +- 0.0005** | 0.6350 +- 0.0008 |\n| **9** | 0.6280 +- 0.0000 | 0.6022 +- 0.0005 | **0.6446 +- 0.0050** |\n\n&nbsp;\n\n**RFM outperforms the adaptive linear baseline in 8 out of 10 cases**. We conjecture that the results above are an underestimate of RFM’s performance in a real scenario. This conjecture stems from the observation that the reward models playing the role of users in this experiment reflect the preferences of aggregated real users. This means that some of the heterogeneity of the real population has probably been washed away in the training of these models. If the reward models were replaced by the real users underlying these models, RFM’s advantage over the two baselines would likely be greater still.\n\nWe plan to add the results above to the paper in a figure analogous to Figure 2(c); please let us know if you have any further suggestions.\n\nPlease see Response 1 to Reviewer mJwX for further details on the experiments above.\n\n\n\n---\n\n&nbsp;\n\nWe thank the reviewer again for their thoughtful review! Please let us know if you have any further suggestions to improve the paper."}, {"review_id": "qULNpP8CH4", "reviewer": "Reviewer_DjQY", "summary": "The authors study the learning of personalized reward models in contexts with high disagreement. They first analyze the reward learning problem using empirical risk minimization and provide a theoretical analysis of the generalization bound. Then, the authors propose to represent the reward model as the inner product between the shared user features and the user-specific feature. The reward model can be trained using empirical risk minimization on the pair-wise ranking data. When adapting to new users, only the new user-embedding vector needs to be updated with the shared user features fixed. The experiment is carried out on a synthetic dataset.", "questions": "Please refer to the Strengths And Weaknesses for questions. I also have the following question:\n\n1. The authors mentioned the data in S is not i.i.d in line 126 on Page 3, could the authors explain how the proof addresses this challenge?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.\tThis paper studied an important problem of reward personalization, which can have a broad impact on LLM finetuning.\n\n2.\tTo my knowledge, the theoretical analysis of reward personalization from the principle of empirical risk minimization is novel.\n\n3.\tThe proposed personalized reward model can adapt to new users.\n\nWeaknesses:\n\n1.\tOne major concern is the scalability of the proposed reward model. In the proposed model, each user has a user-specific feature vector w_h. However, in practice, there can be a large number of users, which brings the scaling challenge.\n\n2.\tAnother major concern is the capacity of the linear feature reward model. Considering the complexity of real users’ preferences, it is challenging to model using a linear combination of reward features.\n\n3.\tThe connection between the theoretical analysis and the proposed reward-feature model is not tight. The theoretical analysis demonstrates that personalization is helpful to obtain a better bound. However, the importance of personalization has been widely recognized by the LLM personalization literature, also discussed in Section 6.\n\n4.\tThe synthetic experiment is not convincing in demonstrating the superiority of the proposed reward-feature model. For example, in the synthetic dataset, there are only $2^{13}$ users and 13 features can be far from real user preferences. Given the concern about the capacity of the proposed reward, the authors need to use a more realistic dataset for evaluation.\n\n5.\tAs the number of reward features is a key parameter in the proposed reward feature model, it’s better to evaluate the sensitivity of this hyperparameter.\n\n6.\tThe baselines are weak. Considering many works on LLM personalization, more recent baselines are needed for comparison.", "was_revised": true, "final_justification": "The authors' rebuttal addresses some of the concerns. However, my concerns about weakness points 3,4 and 6 still exist.", "rebuttal": "Thank you very much for your thoughtful comments and detailed review! **We believe your review will be a great asset to accompany the paper, as it has discussions of some important aspects of our work in an extended (and slightly more informal) way.**\n\n\n&nbsp;\n\n---\n\n> **1. In the proposed model, each user has a user-specific feature vector w_h. However, in practice, there can be a large number of users, which brings the scaling challenge.**\n\n\n&nbsp;\n\n\nAny model that has user-dependent parameters will face this challenge. With a single vector of coefficients per user, RFM is arguably at the most favorable end of this spectrum. As an illustration, in our case we used a reward model with 2 billion parameters, which is small for current standards. Using 128 features per user, we would need more than 15.6 million users to double the size of the model. \n\n\nImportantly, one of the reasons to split the set of parameters in two is that this allows for a distributed architecture in which user-specific parameters can be stored independently from the shared parameters (see discussion on line 196). For example, while the shared parameters $\\theta$ should probably be stored in a fast-access, centralised memory, the vector $w_h$ could be delegated to a secondary memory and only be fetched on demand. Note that many of the current large-scale personalised software relies on similar architectures, with users described by local parameters that are often much more complex than a vector of coefficients. \n\n\n\n&nbsp;\n\n> **2. Considering the complexity of real users’ preferences, it is challenging to model using a linear combination of reward features.**\n\n\n&nbsp;\n\n\nAlthough the RFM approximation is linear in the features, the features themselves are arbitrary, potentially highly non-linear, functions of the inputs learned based on the preferences of raters sampled from the same distribution as future users. Therefore, with enough parameters and features, we can approximate the users’ preferences at any desired level of accuracy. In principle, if we have one feature per user and a large enough approximator, we effectively have a separate reward model per user. In practice, we only need the users’ preferences to lie on a $d$-dimensional manifold, where $d \\ll |\\mathcal{H}|$. As long as we have an approximator with at least $d$ features and enough capacity, we should be able to learn the geometry of this manifold given enough raters $m$ and examples per rater $n$, as indicated by our theoretical results (modulo optimisation issues). \n\nA non-linear dependency on the features could in some cases more easily capture user-specific reward structures. However, having a non-linear dependency on the features introduces complications. Most notably, it is unclear how to obtain a personalised LLM without having to go through the fine-tuning RL process for each user, which is unfeasible (unless one is willing to resort to methods without performance guarantees). In contrast, with an approximation that is linear in the features, one can use known RL techniques to synthesise a personalised LLM **without any learning involved and with performance guarantees [4–7, 28 , 43 , 11]**. A non-linear dependency on the features also gives rise to a non-convex adaptation, which usually requires more data and comes with weaker guarantees. See Response 6 to Reviewer LWzi for an illustration.\n\nPlease refer to Responses 1 and 2 to Reviewer LWzi for related discussions.\n\n\n&nbsp;\n\n\n> **3. The connection between the theoretical analysis and the proposed reward-feature model is not tight. The theoretical analysis demonstrates that personalization is helpful to obtain a better bound. However, the importance of personalization has been widely recognized by the LLM personalization literature.**\n\n\n&nbsp;\n\n\nFirst, we would like to say that, although it is indeed true that the benefits of personalisation have been acknowledged in the LLM literature, the arguments are often at an intuitive (and sometimes empirical) level. We believe that rigorously quantifying these benefits is an important contribution to the ongoing research.\n\nOur theoretical results are indeed not specific to RFM. Note though that, as explained in the subsection “Connection with the theory” of Section 4, we can specialise them to provide guarantees on RFM _after adaptation_ (that is, on the final user). We accomplish that by folding in the adaptation process into the model’s mechanics, effectively removing the vector of coefficients $w$ from the set of parameters. Although we propose this “trick” mostly as an analytical device, it can in fact be used in practice, precisely because RFM’s architecture is linear in the features: the calculation of $w$ comes down to a convex optimisation that can be quickly solved. In contrast, an architecture non-linear in the features would require folding a non-convex optimisation into the mechanics of the model, which in general is not feasible.\n\n\n&nbsp;\n\n\n>  **4. [...] in the synthetic dataset, there are only  users and 13 features can be far from real user preferences. [...]the authors need to use a more realistic dataset for evaluation.**\n\n\n&nbsp;\n\n\nWe highlight that we *do* have experiments with more realistic datasets. Please see results in Figure 2(c), whose data was generated by real (aggregated) human preferences. Furthermore, following a suggestion by Reviewer mJwX, we have carried out additional experiments with another dataset, PersonalLLM, which also contains data based on real human preferences (see Response 1 to Reviewer mJwX). In both cases RFM considerably outperformed the baseline, which represents the current practice.\n\n\nAlthough we agree that 13 features are probably not sufficient to capture human preferences in its full generality, we highlight that the resulting problem is far from trivial. To illustrate this, note that the in-context LLM solutions were unable to capture these features (cf. Figure 2(b)), even though they can capture simpler features (cf. Figure 8 and discussion on line 743).\n\nPlease also see Response 3 to Reviewer 4gUj.\n\n\n\n&nbsp;\n\n\n> **5. As the number of reward features is a key parameter in the proposed reward feature model, it’s better to evaluate the sensitivity of this hyperparameter.**\n\n\n&nbsp;\n\n\nWe note that we did vary the number $d$ of reward features in the experiments shown in Figure 1. Specifically, we tried $d  = 8, 32, 128$. As shown in the figure, $8$ features were not enough to fully capture the structure of the problem. Increasing $d$ to $32$ resulted in a significant boost in performance, but cranking it further up to $128$ did not help much. This makes sense, since in this case we know that the classification problem lies in a $13$-dimensional manifold. In all the experiments we ran, we noticed that, although sometimes increasing the number $d$ of features did not result in improved performance, it rarely (if ever) degraded the results. \n\n\n&nbsp;\n\n\n> **6. The baselines are weak. Considering many works on LLM personalization, more recent baselines are needed for comparison.**\n\n\n&nbsp;\n\n\nFirst, to make sure it does not go unnoticed, we highlight that we do have comparisons with _variational preference learning_ (VPL) [38], arguably one of the most prominent works in this area. The results we obtained with RFM were essentially on par with VPL’s, although our method is arguably much simpler (see Response 1 to Reviewer LWzi and Response 2 to Reviewer DjQY).\n\n\nFollowing a suggestion by Reviewer LWzi, we also carried out additional comparisons with the method presented in [ii], which uses non-linear features. Despite our honest attempts at improving the method’s performance, results are considerably worse than those of RFM. This provides further evidence in favour of having a simple, linear architecture on top of non-linear, learned features.\n\n\nPlease also see Response 6 to Reviewer LWzi.\n\n[ii]. RLHF from heterogeneous feedback via personalization and preference aggregation.\n\n\n&nbsp;\n\n\n> **7. The authors mentioned the data in S is not i.i.d in line 126 on Page 3, could the authors explain how the proof addresses this challenge?**\n\n\n&nbsp;\n\n\n\nThank you for the opportunity to discuss our theoretical results in a more informal way! \n\n\nIn our setup, having iid data would mean first sampling a user $h \\sim \\mathcal{D}\\_{\\mathcal{H}}$, then sampling a context $x \\sim \\mathcal{D}^h_{\\mathcal{X}}$, then sampling two responses $y, y’ \\sim (\\mathcal{D}^{h, x}\\_{\\mathcal{Y}})^2$, and finally sampling a preference $z \\sim \\mathcal{D}^{h, x, y, y’}\\_{\\mathcal{Z}}$. As explained on line 76, what we have instead is the sampling of $n$ tuples $t_h \\coloneqq \\{(x\\_i, y\\_i, y’\\_i, z\\_i) \\}_{i=1}^n$ following each $h$. This data is non-iid, and so prevents directly applying classical results from learning theory. We circumvented this by changing perspective, and defining a new loss $\\ell_n$ which is a function of $(c, h, t_h)$ rather than $(c, h, x, y, y’, z)$ (see equation 7). So, even though random variables defined via the original loss $\\ell$ would not be  i.i.d., the variables $L^{c,n}$ we defined based on $\\ell_n$ are (line 581).\n\n\nThe challenge does not end there, however. Although we can apply the standard learning theory techniques to $L^{c, n}$, this will make $n$ “disappear” (that is, it will not appear in the bound). The way we elicited the dependency on $n$ was to do the following: \n\n\n- First, use Bernstein’s inequality (rather than the more commonly used Hoeffding’s counterpart) to expose the variance of $L^{c, n}$ (equation 8); \n\n\n- Then, use the law of total variance to decompose $\\mathbb{V}(L^{c, n})$ into $\\mathbb{E}[\\mathbb{V}(L^{c, n}|H)]$ and $\\mathbb{V}(\\mathbb{E}(L^{c, n}|H])$ (equation 9); \n\n\n- Finally, note that $\\mathbb{V}(L^{c, n}|H) = \\dfrac{1}{n}\\mathbb{V}(\\ell|H)$ (equation 9).\n\n\nWe hope this clarifies it!\n\n---\n\n\n&nbsp;\n\n\n\nThank you again! Let us know if any further clarifications are needed."}]

Author rebuttal:

Meta-review: This paper studies user-specialized reward modeling for RLHF under heterogeneous human preferences, deriving a PAC bound that links approximation error to data quantity and rater diversity. They propose an adaptive reward model that represents individual preferences as linear combinations of shared reward features, allowing rapid personalization to new users with minimal examples. It reached a consensus that the problem being considered is of practical significance, with solid theoretical analyses and experimental results. The manuscript is also well-written. There were some concerns regarding the representativeness of the datasets, comparison with existing closely-related baselines [1][2] (as suggested by Reviewer LWzi), and the tightness of the PAC bounds. These concerns have been adequately addressed by the authors' new experiments and bounds derived during the rebuttal. Overall, this paper clears the bar for acceptance at NeurIPS. I suggest the authors incorporate the feedback and the rebuttal (e.g., detailed and careful comparison with [1][2] and new results) in preparing the camera-ready version of the paper.


[1] Provable Multi-Party Reinforcement Learning with Diverse Human Feedback 

[2] RLHF from heterogeneous feedback via personalization and preference aggregation
Decision: Accept (poster)