Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
Authors: Sriyash Poddar
Yanming Wan
Hamish Ivison
Abhishek Gupta
Natasha Jaques
Venue: NeurIPS 2024 spotlight (2024)
Keywords: reinforcement learning from human feedback
preference based reward learning
pluralistic alignment
multimodal reward modeling
variational inference
Primary area: reinforcement_learning

Abstract: Reinforcement Learning from Human Feedback (RLHF) is a powerful paradigm for aligning foundation models to human values and preferences. However, current RLHF techniques cannot account for the naturally occurring differences in individual human preferences across a diverse population. When these differences arise, traditional RLHF frameworks simply average over them, leading to inaccurate rewards and poor performance for individual subgroups. To address the need for pluralistic alignment, we develop a class of multimodal RLHF methods. Our proposed techniques are based on a latent variable formulation - inferring a novel user-specific latent and learning reward models and policies conditioned on this latent without additional user-specific data. While conceptually simple, we show that in practice, this reward modeling requires careful algorithmic considerations around model architecture and reward scaling. To empirically validate our proposed technique, we first show that it can provide a way to combat underspecification in simulated control problems, inferring and optimizing user-specific reward functions. Next, we conduct experiments on pluralistic language datasets representing diverse user preferences and demonstrate improved reward function accuracy. We additionally show the benefits of this probabilistic framework in terms of measuring uncertainty, and actively learning user preferences. This work enables learning from diverse populations of users with divergent preferences, an important challenge that naturally occurs in problems from robot learning to foundation model alignment.

Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
Sriyash Poddar; Yanming Wan", Hamish Ivison, Abhishek Gupta', Natasha Jaques'
Paul G. Allen School of Computer Science and Engineering
University of Washington
Seattle, WA 98195
<sriyash, ymwan, hamishiv, abhgupta, nj>@cs. uj>@cs.washingge... wwshington. edu
weirdlabuu.ginn www.ww.tttuubbb.. io/vpl
Abstract
Reinforcement Learning from Human Feedback (RLHF) is powerful paradigm for aligning foundation models to human values and preferences. However, cur- rent RLHF techniques cannot account for the naturally occurring differences individual human preferences across diverse population. When these differences arise, traditional RLHF frameworks simply average over them, Icading to inac- curate rewards and poor performance for individual subgroups. To address the need for pluralistic alignment, we develop class multimodal RLHF methods. Our proposed techniques are based latent variable formulation inferring novel user-specific latent and learning reward models and policies conditioned on this latent without additional user-specific data. While conceptually simple, we show that in practice, this reward modeling requires careful algorithmic consid- erations around model architecture and reward scaling. To empirically validate our proposed technique, we first show that it can provide way to combat under- specification in simulated control problems, inferring and optimizing user-specific reward functions. Next, we conduct experiments on pluralistic language datasets representing diverse user preferences and demonstrate improved reward function accuracy. We additionally show the benefits of this probabilistic framework in terms of measuring uncertainty, and actively learning user preferences. This work enables learning from diverse populations of users with divergent preferences, an important challenge that naturally occurs in problems from robot learning foundation model alignment.
Introduction
Reinforcement learning from human feedback (RLHF) has become the predominant technique for aligning AI foundation models to human values. Across domains like natural language processing
(NLP) 22 robotics (62 [50]. RLHF is effective way improve the performance, accuracy, and
safety of AI models, by ensuring that they align with human preferences [4] 34 29
The question
then becomes: whose preferences? Current RLHF approaches rely on a prescriptive set of values Moreover, they typically assume that all end-users share the same of values. Given the concerning lack of diversity AI 222. it clear that this approach cannot account for the range of social, moral, and political values that inform preferences populations. Recent work has demonstrated that with current RLHF techniques if the majority population has a weak preference for an outcome that will severely disadvantage a minority group.
curated by a small set of AI researchers [39 60 2
Equal Contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

How does volcano erupt?
How does a volcamo enupt?
Explain protosytheese
grasiy?
BTL Reward Model
Encoder
VPL- Reward Model
prefers detais
Incernet Rowand
Homagenous
Ignores Misorities
Accurolc Rewaed
Pessonalied
Accounts for Minocilies
Current RLHF approaches population of In this example, users have diverging preferences over the level of detail provided in the responses from a large language model. Without additional context. BTL model considers both responses to be equally likely. In contrast, our method, VPL, is personalized approach to RLHE. Using few samples from a particular user. we infer the distribution over their distinct preferences. Based on this distribution, we condition the reward model more accurately predict rewards, and enable steering the resulting policy to personalize the specific user. This enables accounting for and serving the preferenees of under-represented groups which would otherwise be ignored by the standard BTL model
Figure 1:
52
incorrectly assume unimodal BTL reward model for diverse
the learned reward model will ignore the minority group's preferences
158 These issues necessitate ideally, we would democratize RLHF to account for wider variety of human values to better serve diverse population.
pluralistic alignment models to human preferences [59 44 (see Figure I
Current approaches to RLHF 52 use the Seadley-Terry-Lune (BTL) 15 model to learn a reward model that explains the human preferences. While the BTL model accounts for noisy preferences, RLHF typically applies this model under the 'unimodal" assumption that all human preferences are derived from a single utility function. This fails to capture scenarios where preferences diverge ie. are multi-modal due to fundamentally different utilities. For example, Figure shows case where one group of users prefers detailed responses, while another prefers concise ones. By performing maximum likelihood estimation under the unimodal BTL model, current methods learn reward function that averages these multi-modal preferences (akin to mode averaging in imitation learning [5I] As we show in our experimental results, this model misspecificatiio leads to reward models that are inaccurate, and the policies optimized these rewards fail accomplish tasks
per any of the distinct preferences (see Figures 6 3 Thus, vanilla RLHF methods
[52 17 are
insufficient for aligning AI systems to diverse human values.
In many applications, from large language models (LLMs) to assistive robotics 7, users are diverse, and Al systems must adapt the generated responses to user-specific preferences to successfully complete the task. Consider, for example, robot assistant putting away dishes user's kitchen: each individual has unique preferences for how the dishes in their kitchen are organized, potentially diverging from others' preferences. the context LLMs, failing to adapt to user-specific prefer- ences can make them unhelpful, unsafe, and vulnerable to jailbreak in the presence of conflicting objectives (2 (58. To build safe and performant foundation models serving a diverse population, we need methods that can explicitly account for and adapt to the inherent plurality of human preferences.
These insights suggest that human preferences are not derived from single utility function, but affected by unobserved, hidden user context [58] To accurately model individual utilities, RLHF should be able efficiently infer and adapt the user context. With this in mind, we formulate RLHF as latent variable problem. Building on techniques from variational inference [1], we propose a method- -Variationa Preference Learning (VPL)-for multi-modal reward modeling. Intuitively, given a few preference annotations from particular user, our approach uses variational encoder to infer a latent distribution over hidden context, and latent conditional reward model to accurately recover the true multi-modal preference distribution. We derive evidence lower bound (ELBO)

---

for latent-variable preference-biee based reward optimization. Our proposed algorithm, VPL, effectively learns a distribution of reward functions from large corpora of preferences from diverse users.
In developing practical training methods for such latent-conditioned reward models, we show that several complexities and technical considerations arise. A key problem is that binary comparisons inherently lack information regarding the scale of rewards. Under the BTL model (and correspond- ingly the VPL model), the preference label between two alternatives A and B can only provide information about тΛ - rB. So, the learned reward function may have vastly varying reward scales across individual users that adversely affect the optimization landscape of multi-user reinforcement learning [31 69 using these rewards. To mitigate this, we show how a simple pairwise classifica- tion scheme 61 [9] can appropriately bound and scale reward estimates within our latent variable framework, thereby enhancing the performance of downstream learned policies. The predicted latent distribution enables several additional capabilities.
Section we use our approach to learn latent-conditioned policies that can personalize to particular users at test time. Additionally, the latent variable reward models can measure uncertainty in the reward distribution [56]. So, in Section 4.2 users to minimize the number of labels they need to we use our approach to actively query 88 10 9
provide before we can adapt to their distinct preferences.
Overall, our work introduces latent variable framework for reward modeling that can encode and approximate the distribution of user preferences directly from binary comparisons, and steer the downstream policy to adapt to diverse user preferences. We conduct a broad range of experiments across three simulated robotics environments and two language tasks with conflicting user preferences. Our results show that in simulated domains, VPI. accurately models rewards and improves task performance and personalization. We are able to scale this method to many users, and use active learning to adapt efficiently to particular users with significantly fewer queries at test time. In the language domain, we are able to train multiple LLM-based reward models that learn a separable embedding space that can distinguish between users with divergent preferences. The resulting models outperform existing by RLHF approaches 10-25% 52 58) by more precisely predicting rewards that align with diverse users and objectives across multiple datasets.
2 Related Work
Reinforcement Learning from Human Feedback (RLHF): We focus on the problem of reinforce-
ment learning (RL) from binary human preferences using the Gradley-Terry-Luun
(BTL) model [15].
This has a rich history in the field of RL and robotics, often referred to as Preference-based RL (PbRL)
65
28
8 10.
We specifically build on the framework outlined in Christiano et al. [17 and
further expanded in recent works [52
2
74
37 58]. This has seen a wide range of applications to finetuning language models for alignment [52 74 60)
60
ranging from training robots [17
10
62
Our work, enabling RLHF with diverse preferences, is easily applicable to any preference-based learning method, including recent techniques
[55] 24
that circumvent reward modeling altogether.
RLHF under non-BTL models: Prior work has aimed to study non-BTL models of human behavior to account for human irrationality and uncertainty in preferences, 64]. However, our key argument in this work is less about human irrationality (i.e. inconsistency), and more about the divergence between potentially rational preferences for different labeling users and end users. In this sense, our work is complementary in that the latent variable model can easily be adapted to non-BTL- models well. In fact, we incorporate the technique proposed in Swamy et al. [61 to improve reward learning for downstream applications.
and preferences 12
43
42
37
61
or intransitive preferences
49
61
characterize similar challenges as VPL., they largely focus on exploring the socictal issues underpinning the need for personalization and introducing datasets with diverse annotations. Conitzer et al. [19] argue Social Choice Theory could provide insights into how to aggregate diverse preferences. But these works do not propose a technical method to achieve modeling diverse preferences.
Personalized RLHF: While some works (39
40
35
Several prior works have looked at trading off conflicting alignment objectives (such as remaining both helpful and harmless) through techniques like Pareto-optimal optimization [3] [6] or multi- objective RL [67 21]. For example, Jang et al. 33 treats personalization as a multi-objective RL problem, requiring explicit decomposition into different sub-rewards and learning independent reward models for different users. Further, Dai al. (2] introduces Safe RLHF, an approach that explicitly models the different objectives of helpfulness vs harmfulness. In contrast, our work does not aim to
3

---

reconcile the diversity but rather solve the model misspecification and learn reward models that can infer the context and specialize a particular user.
Some methods which deal with diverse underlying preferences (e.g. 14] aim to model disagreement in the data but typically require more information such as annotator demographies 226 4 Li et al. [6) learns to map user information to fixed set of representations for each user, and conditions the reward model on this embedding. Feng et al. [25) proposes a pluralistic alignment framework, using smaller LMs that are trained on community or user-specific data. In contrast, our method does not assume access to personal user information, but only a few preference annotations from each user.
Some works have explored personalization outside of the context of reward learning. Zhao et al. 72 uses in-context learning with few samples from particular user group, learn preferences over discrete answers to a question, but does not focus on reward learning with a BTL loss over long context prompt and responses, which is our focus. In addition, our method learns latent distribution over user types to enable capabilities like active learning 71 and latent-conditioned policy learning.
Dumoulin et al. [23] explores annotator-mode misspecification and uses synthetic examples to show how RLHF fails in the presence of two users with different preferences, but they do not propose scalable approach for resolving this issue. The closest work to ours is Distributional Preference Learning (DPL) [58], which aims to account for hidden context in RLHF. While the motivation is similar, the techniques proposed in DPL include modeling only the mean and variance of the reward distribution across users, or particular categorical upproximation to the reward distribution. As such, DPL can capture uncertainty in the inferred reward distribution, but cannot accurately predict the reward for a particular user. In contrast, VPI. explicitly models a user-specific latent variable 2, and learns 2-conditioned, user-specific reward models, enabling us accurately reconstruct divergent reward functions for different users. As we will show in Section this leads to significantly improved empirical reward prediction results. Unlike DPL, VPL takes variational approach to user modeling and allows for reward interpretability, active learning, and model specialization to users at test time.
3 Technical Preliminaries
In this work, we build on the framework of preference-based reward learning (often referred to as RLHF) [ In particular, we will consider reward learning methods based on the Bradley- Tecry-Luce (BTL) choice model [15]. RLHF has two key phases: (1) inferring a reward function from human- numan--rovidee provided labels of ordinal preferences; (2) using reinforcement learning (RL) to train decision- making policy that maximizes the rewards inferred in step (1). We will instantiate this framework abstractly, which can then be specialized to both LLMS and robotics.
We define Markov decision process (MDP) 533 M (S.ATT, 7o Po). with state space S, action space A, transition dynamics T, discount factor 7 and initial state distribution po. Notably, we do not have access to underlying reward function, but instead have annotators who rank pairs of states 8 and sn. We assume annotators have an unknown reward function r(s) that informs their preference labels. More formally, annotator (h E H) takes a pair of states S and sB, and returns an ordinal preference i.c. = (s > $B). according to r(s) [15] 23]; where is the space of all possible annotators. Given dataset of annotated preferences D S/.
) RLHF procedure learns a reward function r using a maximum likelihood objective (MLE) on the preferences, where the likelihood: Py 8 8B) can be defined using the BTL model:
1(s s3))) i== typical
(alaa))
( = SA, 8p) = Pa (y = 0 A sp) = Po(S > sp) cra(sa) + (o((en)
Note, that while in this case, the ordinal preferences are on states SA, SB, this can be generalized to trajectories or snippets [1 52). Finally, the recovered reward function can then be used learn (or finetune) a policy To (a(ss) that can act to maximize the expected sum of approximated rewards in
the environment using standard RL algorithms [37 30 i.e. To arg max E |E, (rrr((s)) We
To
note that while the BTL model accounts for some IID noise in preferences through the probabilistic formulation 17 3 23), it does not account for hidden-context and divergent human preferences (58). and it does not allow the underlying reward models and policies to be personalized to specific users.

---

4 VPL: Incorporating Latent Context into Preference-Based Learning
The standard BTL formulation described in Section based on the assumption that all annotators E H share a single underlying reward function ( This does not hold in practice with a diverse range of annotators. To model diverse, pluralistic preferences, we frame multi-modal reward learning as latent variable problem, where the latent variable represents the hidden context affecting the underlying reward function (and thereby the preferences) of an annotator h H: for instance, it could be representative of their lived experience, or the style of response/policy that they would like the agent to perform. Following this, the fatent-conditional reward r (s 2) is a function of latent 2 along with state and preference likelihoods can be expressed with latent-conditional BTL model:
Po = 8A, Po(SA > 8B 12) (ro(Sa.0) + ero(sa.2)
The maximum likelihood objective for this model from a dataset of preference labels (with multiple E [log po(y 8A, =))(=) is in-
annotators), max E [logPo(y (y 8A) Sn.$g.y~D
& $A.$g.1--D
tractable due marginalization over the unobserved latent variable z. To tackle this, we can introduce a variational posterior approximation (2 {( B. y") {)), conditional on multiple annotations ((s ,',,'' h(s's, }/) provided by the same annotator 1 This results in a corresponding evidence lower bound (ELBO), L(o, v)), for the intractable marginal log Po(y 8A, 8n):
E R
E smaulell (t
[log SA.88. =)] DRa(no(= ((sh o |x(t))
(3)
(4% MM hiA DD (S4.90.3 MS4--)--
This objective samples a user h from the given annotators and set of annotations from this particular user {(s), A sp, = h { for inferring the latent variable z through the posterior (=(= {(sA. s'. ya ))))) Given the posterior, this objective involves optimizing two terms: a
maximum preference likelihood objective loggp(y SA, SB, =)) using the contextual BTL model and regularization term (Dxl (1 ((5 8.: y² (1)) p(=))) against a prior p(=). Intuitively this objective encodes set of user- proviprovdded annotations {( s/. )' into a latent distribution
using the encoder ge., and then learns a latent-conditional reward function r(s, z) that best explains the annotated preference data. As the variational encoder do generates latent distribution, this formulation further enables uncertainty estimation and active learning, as shown in Section 4.2 We
describe details further in Appendix B.3]
We emphasize that the only additional requirement of this objective, compared to standard RLHF [17], is set of annotated pairs from the same annotator, and this information is easily available when annotators are queried in batches 100. We refer to Algorithm for further details on implementation.
Personalized, latent-conditioned policies. The learned reward models ra(s, 2) can be used to train personalized, user-specific policies. During training, we learn a latent-conditioned policy To[-1s, 2) that maximizes the rewards ro(s, z) for different values of z. This allows the policy to adapt to diverse user preferences encoded in the latent space. We can use any standard (RL) [57]
algorithm
30
41) to optimize the latent-conditional reward maximization objective: we
arg max
E. (r 2)]. For this, we sample z from the prior p(z) during training and
xo.=Ep(=)
learn the policy (-1s,, 2), to maximize the corresponding latent-conditioned reward ra(ss 2).
At test-time, we infer a specific user's (Mhet) latent context z by posterior inference using set of labeled preference queries {(*'A s y = M (s'A s'g) )) i.e z ~ q0(= | {(55)) 8/g y )')):">) We then deploy the personalized policy Te(-) =) conditioned on the inferred = for that user. We outline the complete algorithms for policy training and deployment in Appendix
4.1 Scaled Rewards for Multi-Tash Learning
In practice optimizing latent-conditioned reward functions learned with the VPL objective poses unique challenges. The pairwise preferences used to train the reward model in Section 4 do not have
e Having multiple annotations are important here be able accurately infer the user's latent vector
5

---

information about the scale of rewards, but only their relative ordering. As a simple illustration, if we have a pair of states S. where the users prefer S i.c. 3 > sB, two different reward functions: r(*x)= = 100, r(*n) = 50 or r r**A) = 50, r(sn) 0 have the same likelihood under the BTL model. Empirically, we observe that this poses problems for optimizing Equation [3] different values of the latent variable 2 generate reward functions of vastly different scales. This is an issue for several reasons: 1) varying reward scales adversely affect the landscape of multi-user policy optimization (often observed in multi-task RL) BI] and 2) it is challenging to identify states where user preferences diverge across the population as differently scaled rewards cannot be directly compared.
To address this issue, we experiment with several different techniques for scaling the learned reward functions (see Appendix (..)) Our key insight in solving this challenge lies in the observation that while raw rewards from BTL are not scaled equally actoss 2, probabilities from the preference likelihood model p(y 8A; $B, 2) are appropriately scaled. This suggests that an effective solution to the reward scaling issue is to replace the raw rewards from the BTL model (r(s, =)) with likelihoods suggested by the pairwise preference likelihood model p(y S,, 8B, 2). In particular, a natural choice of scaled rewards for state A is the expected likelihood that A is "preferred" to all other states (or a sampled set of states) 8B observed in the data - -T (5,, = Ipo(y = 1 SA, SB; =)]. Since #nES
these are probabilities, normalized in the [0. 1] range, the scaling of rewards is consistent across latents z. Note that these expected likelihood rewards can easily be obtained from the objective in Equation 3 since we train a antent-conditional preference classifier. While proposed from a very different perspective, we note the similarity of this reward scaling approach to recent work [61 49) in particular, Self-Play Preference Optimization (SPO) [61], which was originally proposed to address the issue of intransitive preferences. Similar to [61 we assume that the oracle providing preference labels is Non-Markovian Due this similarity use VPL-SPO to indicate this approach of preference- based reward scaling throughout our experiments (See Algorithm
4.2 Active Learning of Preferences to Minimize Latent Uncertainty
A natural question that arises for test time deployment of the latent-conditioned policies is how to obtain the set of state pairs {(s) (1s4,.8))))) { to be annotated with preference labels and used for posterior inference, some are more informative than others. Certain states (where preferences vary across annotators) are particularly informative in accurately inferring the 2 which should be used for policy deployment [als, z). In VPL, the probabilistic modeling of the variational encoder naturally allows for active selection of the most informative query set based on maximal information gain, following prior work 8 IO [5). Here the provision of preference labels {v') will provide the maximum information about the latent distribution (and indirectly, the user preferences). This active query selection procedure can be expressed as the following optimization problem, maximizing the mutual information between the labels and the latent distribution.
(z {(s), A p) y² }
1). Not all query sets {( s s)g)) are made equal;
{( A >p + arg max I (2; ( I ge ((s) $)));
The posterior qu is multivariate Gaussian, and assuming a uniform distribution over the set of labels, (2 {( M allows for closed form solution for mutual information I. To solve the maximization objective, we chose the query set (s ) with the maximum information gain, across samples from the preference dataset. Finally, we elicit user labels on this maximal query set, infer the latent, and condition the policy on this latent at deployment.
5 Scaling VPL for Reward Learning in Large Language Models (LLMs)
VPL can be used train pluralistic reward models for LLMs, accounting for diverse human prefer- ences and values. Here we discuss the key details that are essential to scale our method to LLMs. The architecture of our LLM reward model is shown in Figure 2 Unlike prior work which attempted to insert summary embedding layers into LLMs (see e.g. 18] we find that we can successfully com- press user preference information into a concise, probabilistic embedding vector & without sacrificing reward model performance. Further details and hyperparameters are discussed in Appendix B
'There are some differences in setup with SPO likelihoods being computed against on-policy samples, while VPL-SPO likelihoods are computed against fixed offline dataset of comparator states.

---

epochs to fit the Prompt and Response Embeddings. Since using raw representations of
the prompt and responses can increase the context length significantly, we
LLM
LLM
LLM
use pre-trained L.LM to encode prompt and response pairs together [6] (to be consistent with previous notation, we assume a preferred state &
contains both prompt and response r), and e L.M (s*M))))) )). For
Eno
efficient training, we pre-compute and freeze the encoded embeddings.
Latent Encoder. Given a set of multiple encoded preference queries from the same user, {(e ef , 1i=1- we pass each through the same pair encoder to obtain h; = enc(ett elf). The latent representation is generated using self-attention layer over the set of encoded pairs, {h, J.M
Attention Layer
P. of (p.o)
Reward learning. Here, the representation eA'' of new state s4' is concatenated with a 2 sampled from the posterior distribution which is passed into an MLP to predict the rewards. The LLM is fine-tuned using low-rank adaptation (LoRA) [32]. and unlike typical RLHF settings, we find that need to train the reward model for 2 encoder and the reward model.
Figure VPL LLM ar- chitecture for reward leam- ing. The left and right parts denote the encoder ds and the reward model r(s, 2) re- spectively.
Data augmentation. As we scaled VPL to larger datasets with more users, we found that augmenting the training dataset with multiple con- text samples from the same user for each new data point is essential to learning an effective encoder. Formally, at training time, given a prompt and response pair '' s'B from particular user, generate M E {4, 8} duplicates of this labeled response with differ- ent contexts i.c. annotated prompt and response pairs (((s)) s/,, y²) ( 1))= where each context (f(s),, sn /² 1); is sampled from a user annotated subset of size K (K > N).
6 Experimental Evaluation on Simulated Control Tasks
In our experiments, we answer the following questions: (1) Can VPL accurately learn a multi-modal distribution of reward functions from a preference dataset labeled by diverse users? (2) Do the inferred latent user vectors enable learning a multi-task personalized policy? (3) Can leverage the posterior to actively query preferences for latent estimation? In this section, we show the benefits of VPL in multiple simulated control tasks and demonstrate that VPL able to capture multi-modality in the underlying reward functions, arising due to underspecification of annotator goals and preferences. In the following section, we ask whether VPI. scales up to RLHF for L.MMs.
Training and Evaluation Details:
We test our hypothesis across evaluation domains in two steps: 1) We train reward model on a dataset of preferences collected using diverse simulated humans; 2) We train policy using RL to maximize the learned rewards. For these experiments, we use Implicit Q-Learning
an offline RL algorithm that achieves strong performance on offline RL benchmarks (27). Using the learned reward function r (s 2), and the prior p(2). we label the reward- offline RL dataset D (s,, a,, 8141), by sampling latent 2 E p(z), and setting the reward as r*(()), or one-step look-ahead method, where = r=( (refer to Algorithm for complete method). We include the hyperparameters and the training details the Appendix B
Baselines: We compare our method against multiple baselines: 1) Oracle [47]: This is goal- conditioned offline RL method, that presents an oracle with access the true reward functions for all annotators. 2) BTL [17]: This is the standard RLHF method from [T7] 52 as a baseline, where the
reward model is approximated using the unimodal BTL function. 3) DPL (58): Following the work on accounting for hidden context in RLHF, we train a distributional reward model, using both the mean-varianee (Mean Var) and categorical (Categorical) approximation for the reward functions. 4)
VPL (Ours): We denote two versions of our method, VPL and VPL + SPO, discussed in Section
6.1 Tasks:
We evaluate our approach on three diverse simulated control tasks (Figure to demonstrate the effectiveness of VPL to learn latent conditioned policies.
7

---

Maze-Narn Aaazi-Navitttion is based on the D4RL benchmark [27] Here, users guide pointmass agent to goals marked with their preferred colors. The user's preferred color is underspecified. so the agent has to infer their preferences from few annotated comparisons and navigate to one of two/ten locations.
Cavens-Manipulattion requires the agent rearrange object table-top (akin rearranging dining table), based on user preferences. Built on the ravens benchmark [70), the agent controls a robot arm in continoous action space pick and place the box in one of the two preferred locations.
Habitat- Rearrange [68] tasks a mobile robot arm with relocating a bowl to user-specified location among five candidates (e.g., desk, kitchen, table). The robot is equipped with motion primitives for navigation and manipulation, and the problem involves reasoning over these locations. This simplified one-step reasoning task highlights the need for personalization in assistive robotics, as each person has distinct preferences for robots in their homes.
Habitat-Tide Tidy is simulated environment similar to TidyBot [66], where the robot infers placement locations for objects in a kitchen based on user preferences. Users may prefer sorting or cleaning objects by different attributes (e.g., function material), and the robot must adapt to these preferences. Like
the task involves reasoning over attributes for sorting based on existing preferences, using motion primitives for navigation and manipulation.
6.2 Can VPL capture multi-modal reward fuunctions from a dataset of diverse preferences?
We generate preferences using multi-modal reward func- tions across multiple didactic and simulated discrete and continuous control experiments, as shown in Figures 6 and I0 We see that the BTL baseline with an MLP model averages the different underlying rewards and learns an inaccurate reward model (Figure In the presence of majority. BTL converges the preferences ignoring the minority groups (See Figure IT While DPL [58] can recover the uncertainty in the reward models due underspecification. they have mechanism to recover the individual reward functions. As a result, the DPL, reward model estimates high variance rewards (see Figures 6 TO) for each particular user. In contrast, VPL infers the hid- den user context using the latent variable formulation and accurately recovers the multi-modal reward distribution.
(a) GT
(b) BTL
(c) VPL
Figure 3: Ground truth preferences (a) show that annotators prefer the robot nav- igate to two different goals. Unimodal BTL (b) averages over the two modes. VPL (c) accurately reconstructs diverse preferences, and learns z conditioned policies to reach either goal.
6.3 Do distributional reward functions enable learning a steerable multi-task policy?
Figure 4: Performance of a downstream policy on diverse control and reasoning tasks, using the rewards trained using different baselines. We report the mean and standard error over five seeds. Note: Habitat envs have one-step greedy policy so reward scaling and SPO+VPL are not required.
As discussed above, the baselines average the reward modes across the users and learn an inaccurate reward function. So, in Figure 4 the policy for the Maze-Navigation converges wrong goal, leading to poor performance and misalignment with all the underlying users. For the Ravens- Manipulation experiments, the baselines randomly choose goal location and do not adapt to the user preference at test time. Similarly, for the Habitar- Reattaatge tasks the baselines are unable to capture the diversity in user preferences bowl at the correct location. Finally, in the Habitat-Tice Tidy task, we observe that the baselines converge accuracy achieved when they ignore the diverse preferences of sorting the objects by the users.
the multiple locations and do not succeed in placing the
In contrast, the policies trained using VPI. outperform all the baselines in terms of the task success rate, according to the user's underlying reward function. For the navigation task, VPL correctly infers
8

---

the goal and the learned policy can navigate to the goals with high success rate, and the performance comparable goal-conditioned oracle. For Ravens-Mann Masrenss-appplaationnn VPL infers the user latent at test time and accurately places the box at the right location. Finally, VPI. is mostly able to correctly identify the preferred location for the bowl the user's home and has higher success rate than the baselines. We note that scaling the rewards via VPL. + SPO improves the performance of multi-task RL for optimizing diverse user preferences. In Habitat-Tidy, VPL is able to infer the user preferences and follow the preferred attributes while placing the objects. However, the accuracy of the robot is dependent on the context length and we discuss this in further detail in Appendix A.4
6.4 Can VPL enable active query selection for latent estimation?
In Section we present an objective to actively query users at test time to efficiently infer user preferences. Fig- ure showss that this technique leads to better performance of the learned policy across varying numbers of queries || This implies that the active learning objective 44 which maximizes information gain over the latent distribu- tion generates queries that are more discriminative and pro- vides a more informative posterior for user identification. This results in a more efficient adaptation of the down- stream policy the distinct user preferences, achieving the same performance with only half the queries. These methods can be potentially transferred LLMs query and identify user preferences with minimal questions.
1.0
Success Bate
Taanom Queries
Active Queries
0.5
Number Quortes
Figure 5: Active learning enables personal- izing policies to user preferences with fewer queries.
7 LLM Experiments
In this section, we ask: can VPL scale up to the pluralistic alignment of LLM-based reward models? We compare the reward modeling performance of our method against two baselines: the vanilla BTL model and DPL [58] We experiment with two LLMs: GPT2 [54] and Llama2-7B 633, and two pluralistic preference datasets.
7.1 Datasets for pluralistic LLM alignment
Prior RLHF works have focused mainly on unimodal BTL models, and as such there is lack of publicly available datasets containing annotated preferences with divergent objectives. To evaluate our method on capturing multi-modality in preferences for LLMs, we consider two benchmarks. First, we introduce a synthetic dataset, Pets, that directly represents multimodal preferences, and second, we augment the publicly available UltraFeedback (20] dataset.
Pets. Here, the dataset is generated reflect multi-modal user preferences, where each user has preference ranking over four kinds of animals (in this case cats, dogs, birds, and rabbits). To simulate a setting where users agree on some comparisons and disagree on others, we consider two users who agree on the best and worst pet and disagree on the middle pair of rankings over pets. Preferences here are divergent in certain cases (middle pets), and agree in other instances (best and worst pets), requiring multimodal preference modeling. We evaluate our approach on two versions of the dataset: Pets (Full), and Pets (Divergent) which contains only those prompt and response pairs where the users are divergent (i.e. they have conflicting preferences). For the contexts ((* 8' y² y²) J we randomly sample 1-4 other prompts and ranked responses from the same user.
UltraFeedback- To construct this dataset UF-P (where P stands for personalized), we use the fine- grained scores over different attributes available in the UltraFeedback (UF) 20) dataset construct different users a similar approach to prior work [58). We construct dataset with two users, UF-P-2, who prefer either helpfulness or honesty (hidden attribute), i.e. they generate preferences using the scores for their chosen attributes. To test the ability of VPL to model more users than has been previously attempted the literature, we create UF-P-4, which uses the fine-grained scores over all the four attributes in the UF dataset 220 to create dataset with four different users. Here, the users are divergent because given two responses, users following different objectives can have opposite preferences- some users can prefer the helpful response, while others prefer the honest response. In

---

Table We compare the accuracy of different reward models trained on two datasels. We report the mean and standard deviation of performance of GPT2-based models on three seeds, and one seed for Llama models.
GPT2
Llama2-7b
Pets (Divergent)
Pets (Full)
UF-P-2
UF-P-4
UF-P-2
BTL
52
63.27 + 0.57
94.92 + 0.00
49.84 = 0.14
53.48 + 0.03
47.17
DPL 58
70.62 = 1.13
95 - 0.00
49.57 = 0.42
52.92 + 0.06
49.51
VPL (Ours)
100 + 0.00
100 + 0.00
74.75 + 2.01
61.49 + 0.03
76.41
some cases, these responses are at odds. To ensure that successfully modeling the data requires fitting divergent preferences, we filtered out the responses where all users agree or are indecisive to primarily focus on multimodal preference modeling, and remove degenerate context queries that provide no information about the user distribution. However, in UF-P-4 the context can still contain queries where at least two users overlap. Thus, this provides a dataset to evaluate VPL in cases where different users agree on some responses, but not all of them. Finally, to generate the context {(8 s yz) ) for inferring latent distributions, for each prompt and response pair, we sample N different data points from smaller subset of size K from the dataset (K' = 100 for GPT2 and 16 for Llama2). For deployed LLM system, this is analogous to having a known set of survey questions from which the user must answer subset of 2-8 questions to personalize the model's behavior to their needs.
7.2 Does VPL help to make LLM reward models more pluralistically aligned?
In Table 1 we see that VPL is able to learn a more accurate reward model across all the datasets, capturing the multi-modality in the language preference data. This indicates that VPL can infer the latent representation of the user's preferences = from a few annotated samples, and successfully adapt the reward model. In contrast, the baselines-includingg the BTL model typically used in widely deployed RLHF [52] models-are unable to the datasets because they do not account for divergent preferences. Because the datasets are imbalanced, the baselines random guessing by fitting only the preferences of the majority group, and thus, the performance on Pets (Full) appears high, even though the basclines fail to adapt to the divergent preferences.
sometimes perform better than
8 Conclusion
In this work, we presented VPL, a technique for pluralistic alignment of preference- based RLHF models via variational inference. We show that VPL can capture diverse preferences and be used for steerable personalized model learning while capturing uncertainty in preferences. We discussed practi- cal considerations for scaling VPL for LLMs and policy learning and showed results across simulated control problems and LLM-based RLHF, significantly outperforming current RLHF techniques.
Limitations and Future Work. A key limitation of this work is that as yet, realistic preference datasets containing the opinions of diverse users do not yet exist at scale. This limitation necessitated creating our synthetic preference datasets. Although this was also the approach taken in prior work on personalized RLHF (c.g. [58] (3)). ). an important direction for future work will be to apply VPL to more realistic preference data from diverse groups of users. Further, our current experiments on the UltraFeedback dataset assume that when adapting to a new user, we could ask them for preferences over samples from a fixed set of survey questions. In the future, would be useful to relax this assumption and apply VPL to preferences obtained naturally during conversation with the user.
We believe VPL could offer promising safety benefits in addition modeling diverse user preferences. Uncertainty detection may help prevent jailbreak attacks caused by conflicting rewards [58]. By capturing uncertainty in the distribution of user preferences, VPL could enhance safety by stopping or refusing to respond when uncertainty cannot be sufficiently reduced [36].
Acknowledgement
We would like to thank members of the WEIRD and Social RI. for the valuable discussions and feedback on the project and manuscript. This project was partly supported by NSF grant no. 22123 10.
10

---

References
[1] R. Akrour, M. Schoenauer, and M. Sebag, April: Active preference learning-based reinforvement learning. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2012. Bristol, UK, September 24-28, 2012. Proceedings, Part II 23, pages 116-13 111--31.550 Springer, 2012.
[2] Y. Bai, S. Kadavath, S. Kundu, A. Askell, Kernion, A. Jones, A. Chen, A. Goldie, A. Mirho- seini, C. McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprini arXiv:2212.0807 2022.
[3] A. Bajesy, D. P. Losey, M. K. O'Malley, and A. D. Dragan. Learning from physical human corrections, feature at time. In Proceedings of the 2018 ACM/IEEE International Conference on Human-Robo Interaction, pages 141-149, 2018.
[4] C. Baumler, A. Sotnikova, and H. Daumé III. Which examples should be multiply annotated? Rogers, Boyd-Graber, and N. Okazaki,
active learning when annotators may disagree. editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 10352- 10371, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/ 222023.findigg--21
URL htttss://aclanthottet
org/2023. findings--ae acl. 658
[5] P. BehnamGhader, V. Adlakha, M. Mosbach, D. Bahdanau, N. Chapados, and S. Reddy. LLM2 Large language models are secretly powerful text encoders. arXiv preprint, 2024.
URL hRtps:/:://artt
org/ abs/ 2404. 05961
[6] K. Bhatia, A. Narayan, C. M. De Sa, and C. Ré. Tart: A plug-and-play transformer module for task-agnostic reasoning. Advances in Neural Information Processing Systems, 36:9751-9788, 2023.
[7] T. Bhattacharjee, E. Gordon, al. more autonomy always better? exploring preferences of users with mobility impairments in robot-assisted feeding. In Proceedings of the 2020 ACM/IEEE International Conference on
Interaction, 2020.
[8] E. Biyik and D. Sadigh. Batch active preference- baseed learning of reward functions. In 2nd Annual Conference Robot Learning, CoRL 2018, Zürich, Switzerland, 29-31 October 2018, Proceedings, volume 87 of Proceedings of Machine Learning Research, pages 519-528. PMLR, 2018.
[9] E. Bıyık, M. Palan, N. C. Landolfi, D. Losey, and D. Sadigh. Asking easy questions: user- user-friendly approach to active reward learning. arXiv preprint arXiv: 2019.
[10] E. Biyik, D. P. Losey, M. Palan, N. C. Landolfi, G. Shevehuk, and D. Sadigh. Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences. The International Journal of Robotics Research, 41(1):45-61 67, 2022.
[11] D. Blei, A. Kucukelbir, and J. McAuliffe. Variational inference: A review for statisticians. arXiv preprint 1601.0
2016.
[12] A. Bobu, D. R. R. Scobee, J. F. Fisac, S. S. Sastry, and A. D. Dragan. LESS is more: Rethinking probabilistic models of human behavior. In T. Belpaeme, E. Young, H. Gunes, and L. D. Riek, editors, HRI '20: ACM/IEEE International Conference on Human-Roke
Interaction, Cambridge, United Kingdom, March 23-26. 2020, pages 429 437, ACM, 2020. doi: 10.1145/ 3319502. 3374081 1. HtRp:ttts:/// //doi. 10t.://.//./.././/1.111 00145/3319502.333 337441
[13] R. Boldi, L.. Ding, L. Spector, and S. Niekum. Pareto-optimal learning from preferences with hidden context. ArXiv, abs/2406. 15599, 2024. URI. httph:ttp//// http://aacpeeetmiicccccooooeoeee org/
CorpusID:27070322
[14] A. Bose, M. Curmei, D. L. Jiang, J. Morgenstern, S. Dean, L. Ratliff, and M. Fazel. Initializing services in interactive ml systems for diverse users, 2023. URL httpts:///xxxii. org/ abs/
2312. 11846]
[15] R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324, 1952. doi: 10.2307/2334029,
[16] S. Chakraborty, J. Qiu, H. Yuan, A. Koppel, F. Huang, D. Manocha, A. S. Bedi, and M. Wang. Maxmin-rlhf: Towards equitable alignment of large language models with diverse human preferences, arXiv preprint arXiv:2402.08925, 2024.

---

[17] P. F. Christiano, J. Leike, T. Brown, M. Martic. S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In Neural Information Processing Systems (NeurIPS), 2017.
[18] G. Cideron, S. Girgin, A. Raichuk, O. Pietquin, O. Bachem, and L. Hussenot. vec2text with round-trip translations. urXiv preprini arXiv:2209.0677 2022.
[19] V. Conitzer, R. Freedman, J. Heitzig, W. H. Holliday, B. M. Jacobs, N. Lambert, M. Mossé,
E. Pacuit, S. Russell, H. Schoelkopf, et al. Social choice for aligmment: Dealing with diverse human feedback. arXiv preprint arXiv:2404. 10271, 2024.
[20] G. Cui, L. Yuan, N. Ding, G. Yao, W. Zhu, Y. Ni, G. Xie, Z. Liu, and M. Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377, 2023.
[21] J. Dai, X. Pan, R. Sun, J. Ji, X. Xu, M. Liu, Y. Wang, and Y. Yang. Safe rlhf: Safe reinforcement learning from human feedback. arXiv preprint arXiv:2310.122 127733,, 2023.
[22] H. Daume and K. Heller. Neurips 2018 demographics and inclusion survey; Summary of responses. Advances in Neural Information Processing Systems, 2018.
[23] V. Dumoulin, D. D. Johnson, S. Castro, H. Larochelle, and Y. Dauphin. A density estimation perspective on learning from pairwise human preferences. ArXiv, 0/0s//311.14111 2023. URL https:://pp... https:///ap..semattcccccte www.ssemmmmoooolaaaeeee
[24] K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela. Kto: Model alignment as
prospect theoretic optimization. arXiv preprint arXiv:2402.01. 01 3066, 2024.
[25] S. Feng, T. Sorensen, Y. Liu, J. R. Fisher, C. Y. Park, Y. Choi, and Y. Tsvetkov. Modular pluralism: Pluralistic alignment via multi-llm collaboration. ArXiv, abs/2406. 15951, 2024. URL.
https://api. eeeemattiscchn lar. org/Corpus I 270703169
[26] E. Fleisig, R. Abebe, and D. Klein. When the majority is wrong: Modeling annotator disagree- ment for subjective tasks. In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6715-6726, Singapore, Dec. 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023. emnlp-main.411 URL htw.jittts//accaaatttooo
org.:org//// emnlp-main. 415
[27] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven
reinforcement learning. arXiv preprint arXiv:2004.077 072199, 2020.
[28] Fürnkranz, E. Hüllermeier, W. Cheng, and S.-H. Park. Preference-based reinforcement
learning: formal framework and a policy iteration algorithm. Machine learning, 89: 19:123-156. 2012.
[29] I. Gabriel. Artificial intelligence, values, and alignment. Minds and Machines, 30(3):11-443) Sept. 2020. ISSN 1572-8641. doi: 10. 1007/s 1023-020-00539992222 URL/ttpp:/ doi. org/ 10. 1000//010200----
[30] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. J. G. Dy and A. Krause, ed- itors, Proceedings of the 35th International Conference on Machine Learning. ICMI. 2018, Stockholmsmäissan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Ma- chine Learning Research, pages 1856-1865. PMLR, 2018. URL [ttp::tt://rr press//80/haarnojaaa .tml
mlr
[31] M. Hessel, H. Soyer, I.. Espeholt, W. Czarnecki, S. Schmitt, and H. Van Hasselt. Multi-task deep reinforcement learning with popart. Proceedings of the AAAI Conference on Artificial Intelligence, 2019.
[32] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 09685, 2021.
[33] J. Jang, S. Kim, B. Y. Lin, Y. Wang, J. Hessel, I.. Zettlemoyer, H. Hajishirzi, Y. Choi, and P. Ammanabrolu. Personalized soups: Personalized large language model alignment via post- hoc parameter merging. arXiv preprint arXiv:2310. / 1564, 2023.
[34] Ji, T. Qiu, B. Chen, B. Zhang, H. Lou, K. Wang, Y. Duan, Z. He, J. Zhou, Z. Zhang, F. Zeng, K. Y. Ng. J. Dai, X. Pan, A. Oaraa,, Y. Lei, H. Xu, B. Tse, J. Fu, S. McAleer, Y. Yang. Y. Wang, S.-C. Zhu, Y. Guo, and W. Gao. Ai alignment: A comprehensive survey, 2023.
[35] J. Ji, M. Liu, J. Dai, X. Pan, C. Zhang, C. Bian, B. Chen, R. Sun, Y. Wang, and Y. Yang. Beaver- tails: Towards improved safety alignment of Ilm via a human-preference dataset, Advances in Neural Information Processing Systems, 36, 2024.
12

---

[36] K. Kang, E. Wallace, C. Tomlin, A. Kumar, and S. Levine. Unfamiliar finetuning examples control how language models hallucinate. arXiv preprint arXivv2403.05612, 2024.
[37] C. Kim, J. Park, J. Shin, H. Lee, P. Abbeel, and K. Lee. Preference transformer: Modeling human preferences using transformers for RL. In International Conference on Learning Representations, 2023. htttp::////
net/forum'id-Poot
[38] D. P. Kingma and M. Welling, Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
[39] H. R. Kirk, B. Vidgen, Röuger, and S. A. Hale. Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback. arXiv preprint arXiv:2303. 05453, 2023.
[40] H. R. Kirk, A. Whitefield, Röttger, A. Bean, K. Margatina, J. Ciro, R. Mosquera, M. Bartolo, A. Williams, H. He, et al. The prism alignment project: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models. arXiv preprint arXiv:2404.16019 2024.
[41] Kostrikov, Nair, and Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representutions, 2022. URL htttps://openreviee net/forum? 1d=682292.WFF
[42] C. Laidlaw and A. Dragan. The boltzmann policy distribution: Accounting for systematic suboptimality in human models. arXiv preprint arXiv:2204. 10759, 2022.
[43] C. Laidlaw and S. Russell. Uncertain decisions facilitate better preference learning. Advances in Neural Information Processing Systems,
1-5500005588,3 2021.
[44] N. Lambert, K. Gilbert, and T. O. Zick. The history and risks of reinforcement learning and hu- man feedback. 2023. URL https: htt///pp.ssaaanncccccoclllleeeee
org CorpusID 264405813
[45] J. Leike, D. Krueger, T. Everitt, M. Martic, V. Maini, and S. Legg. Scalable agent alignment via reward modeling: a research direction. arXiv preprint /rXiv=1//1.00771, 2018.
[46] X. Li, Z. C. Lipton, and L. Leqi. Personalized language modeling wwwffrrrrmoooorrraaro personalized human feed- back. ArXiv, abs/2402.05133,, 2024. UtLpttts:://ae http:///ppppeeeeaniccccoooooooo www.stiimmcooooooooooo hatpaaaiiiiiicclaaaooror/oruuuu orr//oorp//CDDC
267547503
[47] M. Liu, M. Zhu, and W. Zhang. Goal-conditioned reinforcement learning: Problems and solutions. arXiv preprint arXiv:2201.0220 2022.
[48] N. Muennighoff. Sgpt: Gpt sentence embeddings for semantic search. arXiv preprint arXiv:2202.08900 08904, 2022.
[49] R. Munos, M. Valko, D. Calandriello, M. G. Azar, M. Rowland, Z. D. Guo, Y. Tang, M. Geist, T. Mesnard, A. Michi, et al. Nash learning from human feedback. arXiv preprint
arXiv:2312.00886 2023.
[50] V. Myers, E. BIyık, and D. Sadigh. Active reward learning from online preferences, 2023.
[51] Osa, J. Pajarinen, G. Neumann, A. Bagnell, P. Abbeel, and J. Peters. An algorithmic perspective on imitation learning. Found. Trends Robotics, 7(1-2):1-179, 2018. doi: 10.1561/ 2300000053. htLpttps::
org/ 10. /561/2300000053,
[52] I.. Ouyang, J. Wu, X. Jiang. D. Almeida, C. I.. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe. Training language models follow instructions with human feedback, 2022.
[53] M. L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, 2014.
[54] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. arXiv preprint, 2019.
[55] R. Rafailov, A. Sharma, F. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
[56) X. Ran, M. Xu, L.. Mei, Q Xu, and Q. Liu. Detecting out-of-distribation samples via variational auto-encoder with reliable uncertainty estimation. Neural Networks, 145:199-208 2022.
13

---

[57] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.063477 2017. /ttp://a//rr 1707 06347
[58] A. Siththaranjan, C. Laidlaw, and D. Hadfield-Menell. Distributional preference learning: Understanding and accounting for hidden context in rlhf. arXiv preprint arXiv:2312.08332 2023.
[59] T. Sorensen, J. Moore. J. Fisher, M. Gordon, N. Mireshghallah, C. M. Rytting, A. Ye, L. Jiang, X. Lu, N. Dziri, et al. A roadmap to pluralistic alignment. arXiv preprint arXiv:2402.05070, 2024.
[60] N. Stiennon, I.. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. Advances Neural Information Processing Systems, 33:3008-380 -0000, 2020.
[61] G. Swamy, C. Dann, R. Kidambi, Z. S. Wu, and A. Agarwal. A minimaximalist approach to reinforcement learning from human feedback. arXiv preprint arXiv:2401.04000 2024.
[62] M. Torne, M. Balsells, Z. Wang, S. Desai, T. Chen, P. Agrawal, and A. Gupta. Breadcrumbs to the goal: Goal-conditioned exploration from human-in-thee-oop feedhack, 2023.
[63] H. Touvron, Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. I.lama 2: Open foundation and fine-tuned chat models. arXiv preprint wVXi:22307009288 2023.
[64] Y. Wang. Q. Liu, and Jin. rlhf more difficult than standard rI? a theoretical perspective. In
A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages wew.iis..e.ip.p... cc/ wwwppppppa.pee files/ patp///ppp//p///////e//
Curran Associates.
Inc., 2023. URL httpp:: 00166227566996fff661444466666666 Pwwpeaddddddeee-ee---ee-e----eee Poneeee-eeeeee---eeeeeeeeeeee pdf
[65] A. Wilson, A. Fern, and P. Tadepalli. A bayesian approach for policy learning from trajectory preference queries. Advances in neural information processing systems, 25, 2012.
[66] J. Wu, R. Antonova, A. Kan, M. Lepert, A. Zeng, S. Song, J. Bohg. S. Rusinkiewicz, and
T. Funkhouser. Tidybot: Personalized robot assistance with large language models. Autonomous Robots, 2023.
[67] Z. Wu, Y. Hu, W. Shi, N. Dziri, A. Suhr, Ammanabrolu, N. A. Smith, M. Ostendorf, and H. Hajishirzi. Fine-grained human feedback gives better rewards for language model train- ing. ArXiv, ftbs/2306.01993, 2023. URL httpts:/pppiiii semanti wrg. I 259064099
[68] P. Xavi, U. Eric, A. Szot, M. D. Cote, R. Partsey, J. Yang, R. Desai, A. W. Clegg, M. Hlavac, T. Min, T. Gervet, V. Vondrus, V.-P. Berges, J. Turner, Maksymets, Z. Kira, M. Kalakrishnan, J. Malik, D. S. Chaplot, U. Jain, D. Batra, A. Rai, and R. Mottaghi. Habitat 3.0: A co-habitat for humans, avatars and robots, 2023.
[69] T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33:5824-5836, 2020.
[70] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, al. Transporter networks: Rearranging the visual world for robotic manipulation. In Conference on Robot Learning, pages 726-747. PMLR, 2021.
[71] S. Zhang, D. Yu, H. Sharma, 7. Yang, Wang, H. Hassan, and Wang- Self-exploring language models: Active preference elicitation for online alignment. ArXiv, 200/22055.11333 2024. https:tt:::///
ht //pp1. http:///ppp.sseniisseeeeeee www.ssecscaaaalrrr....... httpaaaaieeestcaaoooooooooooppaa 277095505
[72] Zhao, J. Dang, and Grover. Group preference optimization: Few-shot alignment of large language models. arXiv preprint arXiv:2310.1152. 2023.
[73] Z. Zhou, Liu, C. Yang, J. Shao, Y. Liu, X. Yue, W. Ouyang, and Y. Qiao. Be- yond one-preference-for-alll Multi-objective direct preference optimization. arXiv preprint arxiv:2310.033703 2023.
[74] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irv- ing. Fine-tuning language models from human preferences. arXiv preprint Siti::9909.055000 2019. URLhtttps / //axxvi.vvvv.///////a../aaaa org/ wwwp:r.rgggbb....ee httg/aab/1999... 08593
14

---

A Additional Experiments
A.1 Didactic example toy reward learning problem.
To more earefully understand the behavior of VPL empirieal. we construct a didactic example 23 shown in Figure 6 In this problem. let us consider mixture of M different annotators providing preferences, where each annotator has reward function specified by N (14, o,) that they use to assign binary preferences. Mathematically, sample the preferences from mixture of Gaussians:
(n())))
1 0.V2T
p(sA > 8B eri(+a) + eti(s)) where e
Multi-annotator preferences are simulated by sampling an annotator from this mixture distribution and then assigning binary preferences according to the chosen reward function. We train VPL as described in Section to recover the underlying distribution over reward functions. As expected in Figure [6] standard RLHF [[7] averages over the different modes since it can only represent single reward function. While prior work in accounting for hidden context in RLHF (DPL (58]) learn the uncertainty in the reward functions due hidden context, it is able to accurately disambiguate different modes. Meanwhile, VPL is able to infer the underlying context using the approximate posterior qu and the recover the individual reward modes through the latent-conditional reward function rs,, 2).
Reward Functions min)
User Preferenccs, pla,
BTL and DPL fall recover the tue mult-mods rewards
VPE movers the tlcunndriingg
-17
Figure 6: Didactic experiments comparing standard BTL 17 DPL 58 and VPL (Ours). Four Gaussian reward functions generate different binary preference data. The traditional BTL approach (17] averages the different modes, and DPL (58) captures the uncertainty in the rewards due to the multi-modality but cannot accurately predict the true modes. VPL (ours) infers the hidden latent described in Section 4and recovers the individual distribution of reward functions.
A.2 Does scaling rewards help improve performance?
To avoid the problem of high variance rewards (Section we com- pare the performance of VPI-no norm with VPL + SPO. We further compare against two normalizing schemes: VPL-batchnorm where the rewards for each latent is normalized by the mean rewards r(s.2)
6/)
across a set of state samples i.e. (((8,)) 3r Exes "(8'2j) and
VPLmax norm where all the rewards in the offline dataset are nor- malized by the maximum reward for any latent.
022
discuss the problem of generating scaled rewards from latent variable-hased reward models and compare the perfor- mance across multiple baselines discussed above. As shown in Figure 7 the batch norm scaling generates highly biased estimates of the rewards, which catastrophic for the method. However, VPL methods have decent performance at test-time, but are an unprinci- pled approach the scaling problem. Our SPO VPL presents general method for estimating normalized rewards. Thus, in Figure we can see that our method outperforms baseline approaches in terms of success rate. The baselines have unscaled biased estimate of the multi-modal rewards leading to sub-optimal performance. For the avees-manipulationn environment, the dataset doesn't contain sub-optimal
In Section 4.
SPO
V00
Vil
Figure 7: Comparing scaling methods on Maze-Navigation.
trajectories, VPL (with max norm) performs comparably.
15

---

1.0
0.5
0.8
0.4
Success Rate
0.6
Stoces Rate
0.4
0.2
0.1
Oracle
VPL (Ours)
VPL. (Cur)
DMt -attgooiil
VPL SPO (Ours)
DPL. MennVar
(a) VPL scales to Maze-Navigation task with ten modes of user preferences. BTL expectedly aver- ages the modes and fails learn. We also see the benefits of scaling rewards across this domain as well, where VPL +SPO performs better than VPL.
(b) We compare the performance of baselines and VPL on
environment with 100 users. VPL can scale to much larger set of diverse users, complementing the roal-world ca- pabilities shown in Table [and Figure
Figure 8: Comparison of VPL's scalability on different tasks and user bases.
A.3 Does VPL scale with the number of diverse users?
In order to test the effectiveness of VPL in scaling to control problem with larger modes of underlying preferences, we create task with ten underlying locations that the users could prefer. So, the challenge here to disambiguate the user preference among a larger space of possible goals and condition the policy to navigate successfully to the goal. Figure Sa saswwww that method is able to navigate to the individual goals with higher success rate, whereas the baseline DPL model 117 collapses to single user mode. This demonstrates the benefits of scaling VPL to setting with a large population of diverse users. To test the method at a larger scale, we increase the number of users in the Rearrane tasks 100. It is combinatorial problem as the users provide rankings over five locations, so all possible users/orderings are 5!. We observe in Figure Bhb VPL significantly outperforms the baselines in inferring the user preference, and steering the robot policy accordingly.
A.4 How does context length affect VPL?
In Habitat-TidyBot, the robot's task is to relocate an object in the kitchen (fork, knife, spoon, bowl, pitcher) to specific location based on prefer- ences. The user can prefer to arrange objects ac- cording to the object function (i.e. kitchenware tableware), or material (metal or plastic). The agent has to query the location of some of the objects, in- fer the user type, and arrange the requested object accordingly. Fig shows that the baselines converge to the correct location for the objects agreed upon by the different users, but converge one mode for the divergent choices. Meanwhile, VPL infers and adapts user preferences and aligns with humans perfectly. However, one caveat to this is the context length i.e. the number of queries to each user. In Figure [ we show that as the query length increases VPL can iden- tify users with higher accuracy and achieve higher performance. This happens because certain queries are uninformative about the user preferences (such as things users agree on), and thus, generates a high variance posterior. As result, longer context length increases the probability of useful queries,
1.0
Success Rate
0.5
0
Number of Queries N
Figure In the Habitat-TidyBot tasks, the agent has to relocate objects in the house based on the user's preferences. We show that the accuracy of user modeling and choos- ing the right location for the object increases with the number of queries the agent makes to infer the latent distribution.
16

---

which enables low variance posterior inference and improved alignment during decision-making In Section 4.2 we also show how to use an active learning approach to generate queries based on max information gain objective. For LLMs, in Figure 13 we show that higher context length also provides robustness to noise in the annotated queries.
A.5 Visualizing Learned Rewards and Embeddings
We visualize the rewards generated by the baseline and the latent conditioned rewards models on the diverse domains described in Section G We observe that VPL reconstructs the multi-modal reward functions, based on the inferred latent distribution. Figure 3 shows that the BTL models averages the reward over the user-preferred goals, while VPL accurately reconstructs the individual user-specific rewards. Figure shows that for optimal trajectories sol vilning the task, VPL can accurately match the ground rewards for the two modes. At the same time, DPL [58] predicts
variance rewards for both cases due to the inherent multi-modality. Finally, in Figure [I] we see that the majority of users consider the desk to be the preferred location of the bowl, and standard BTL models converge to the majority population. Mcanwhile, VPL can generate user-specific rewards, satisfying all the user groups.
Goal 0
Goal
1.0
1.0
0.0
0.0
15
20
10
15
20
Oracie
VPL DPL
Figure 10: In the Ravens- Manton-aManilliiiionnn task, we compare the predicted rewards for states 8y along timesteps in oracle trajectories to either of the goals the user prefers. VPL (Ours) can learn the individual reward functions for the two different (closely matching the ground truth rewards for both users) leading to performant policies (see Figure 4h, while DPL. 58 learns high variance reward function due to the multi-modality.
6::
C:3
111
0.35
8.35
[:43
0.1
E/330
9.35
Twe?
E.27
Lwee
Dwrrd
02
Cwek
375
05
111
0,30
6.12
6.11
tatttaat wid.
2006 catteio0i
d004 cothe bolas wb
(a) Ground truth
(b) BTL
(c) Our method
Figure 11: In the Habitat- Habitat-Rearrange task, annotators have rankings, i.e. preferences the different locations
they want the robot place their bowl their home. Accordingly, (a) shows the rewards associated with particular location ("column") for each annotator ("row"), We see that majority of the users rank the desk as the most preferred location. Consequently, in (b), unimodal BTL converges to this majority preference ignoring other users. However, in (c) see that VPL accurately reconstructs diverse preferences and aligns to all five
17

---

A.6
Analysing the Latent Space
Users preferring honesty Users preferring helpfulness
20
10
-10-
The encoder clusters users in the latent space.
-20
-40
-60 -40
-20
Figure 12: We train GPT2-based VPI. on the UltraFeedback- dataset. In this plot, we visualize the T-SNE features of the latent distribution 2 produced by the encoder de set of annotated prompts and responses {s'A s/ y ) from the two users in the dataset. We see that the encoder clusters the users in the latent space, allowing the decoder to personalize the reward models according to multiple objectives preferred by the diverse users belonging to cluster.
In Figure we show that the VPL-Encoder effectively learns latent space with clusters that
correspond to the user types in the dataset. Previous work has shown that attempting to compress information within an LLM into single bottleneck embedding layer can hinder performance [18] However, using the architectural design of VPL as well as user-context data augmentation, VPL able to learn compressed user representation that accurately separates users according their preferences, from only a few preference labels.
A.7 Does VPL scale to real-world settings with larger and noisy users?
A key assumption in approach is that context questions accurately represent individual users with- noise in the underlying dataset. To test VPL's robustness to noisy context labels at test time, we injected noise by flipping the questions answered by each user and evaluated the trained model's accuracy predicting rewards. This experiment can help us evaluate how well the model would generalize to new users that have similar preferences those experi- enced during training, but may not answer questions in exactly the same way. Figure 13 illustrates that VPL is able to outperform prior work even when 25% of preference labels are flipped at test time, Notably, we observed that longer context lengths result in more accurate reward modeling, even with higher noise in- jection. This is because the encoder can generate more accurate inferences the latent distribution when provided with more information through a larger number of context questions and response labels.
Reward Accuracy
0.6
0.5
Iit
Length
Figure 13: Reward accuracy with varying levels of noisy labels introduced test time.
We note further that when 50% the preference labels are flipped, the context preference queries provide no information about the user, and the performance of VPL is equivalent to the baseline BTL model. This mirrors additional findings presented Appendix A.8] which demonstrate that when VPI. is trained on unimodal preference language dataset, it gives equivalent performance to BTL., Essentially, when extra preference data is available for a user, VPI. can personalize the reward model effectively and attain higher performance. But without that information, performs just as well the
18

---

default BTL model. These findings, as shown in Table Figure 13 and the Appendix, demonstrate VPL's effectiveness in handling multi modaityy and noise in large-scale preference datasets, without compromising performance even when no additional preference information is available. This capability suggests the potential for VPI. to contribute to the development of next-generation LLMss that are more personalized, inclusive, and efficient.
A.8 Does VPL work under uni-modal settings?
To test that the introduction of a variational framework does not decrease performance in settings where all users have singlc/aligned preferences, we run an experiment on the UF-P-4 dataset, where we considered the preferences of single user (preferring the model to be "honest" over all other attributes) analyze the single-modal case suggested. The standard BTL model gives a 77.04% eval accuracy while our VPL model gives a 77.16% eval accuracy. Our model matches the baseline performance, indicating no drop in performance when using VPL compared traditional RLHF over an unimodal dataset.
A.9 Social Impact
This work has a clear social impact when deployed on user-facing systems like LLMs or household robots. In pluralistic alignment, we assume that some differences in user preferences reflect divergent but equally valid perspectives on which moral or cultural values an LLM should align to; for example, individuals from one culture may hold collectivist moral values, while another culture is more individualist. Both value systems should be respected, and as such L.Mss should be able to recognize and adapt to the values held by particular user. However, the personalized model could potentially either be sycophantic or align with adversarial users, which is undesirable. This raises very interesting questions, such as: At what point should the LLM embrace more universal of values? How can we detect when such point has occurred in particular conversation? The probabilistic framework of the user distribution could allow us to identify low probability or rare behavior, and also the distributional nature of reward functions can help us point out responses where the users are divergent (maybe signifying disagreement). Additionally, model could flexibly switch between adhering to the user's personal preferences and conforming to more universal perspective on topics where could be biased, or sensitive to jailbreak (2). Taking inspiration from Constitutional Al (2), we can allow a system designer to specify the topics for which the L.M should not engage in user personalization. Overall, this presents an exciting future research direction toward building safe personalized LLMs.
B
Implementation Details
B.1 Task Details
We evaluate our methods on three simulated control environments.
Naviinavvi This task adapted from the "maze2d-medium- environment from the D4RI. benchmark (27). The observation space is the position and velocity of the robot (Pz. Py: P=), and the pointmass is controlled using torque control. In this environment, point mass doesn't it have access to a goal, and diverse users guide the agent to (two or ten) different locations in the maze, marked with their preferred colors. The users label the preferences over two states based on the shortest path to the goal from each state, i.e, the user prefers states closer to their preferred color location. The offline dataset for IQL is collected using the waypoint controllers provided in the D4RL benchmark. For each episode, the agent is spawned at random location in the maze, interacts with random and navigates to the goal based on the learned reward model and corresponding policy trained on the offline dataset. The oracle reward function is the optimal Q-value of the state, generated using dynamic programming solution, which is available in D4RI..
Ravens- Manesns-anipllaiionn This task is adapted from the ravens benchmark 70 space is the 3-D position of the object, the 3-D position of the end effector, and the grasp state of the object, i.e., (eeay ekgg 26 Pai Py: Pz, grasp). The agent commands absolute positions for the 3-DOF robot arm in end-effector space i.c. (ee'g ecy, ee')). This setup resembles how robot arm would
The observation
19

---

infer user preferences to organize a dining table. The users prefer two different locations for the box spawned at a random location at the beginning of each episode. To collect offline data, we use motion planning oracle with some added noise, which tries to pick the box and place it randomly at one of the two locations. The oracle reward function is as follows:
100
if goal_aa goal_dist < 0.05 and not grasped
reward 1
if goal_ dist < 0.05 and grasped
100
2 + dist) exp( poo(-rrpperr_iss]
if not goal_dist < 0.05 and grasped
if not goal_ dist < 0.05 and not grasped
Wlbitat-Reerrangg This is task based on the Meta Habitat simulator 668 Here, the objective for the Mobile Manipulator to pick a bowl and place it at the user's preferred location in the home. However, the exact location is underspecified and needs to be inferred from the user-annotated preferences. The robot uses a motion primitive to navigate and place the object at five possible locations ('desk', 'room', "dining', , "coffee table', 'sofa'). This problem is reduced to discrete one-step problem, where the robot has to reason about the best possible location to put the bowl. For ranking the states, the users are generated by randomly choosing five random orderings of the locations, each corresponding to an individual user. At test time, the agent is greedy and chooses the location with the maximum inferred reward from the learned reward model.
Sabbitat-TidyBo This task is based on Meta Habitat (68 and inspired from the TidyBot task [66]. In this environment, there are 5 objects in the kitchen (spoon, knife, plate, bowl, and spatula). Each user has their preferences for sorting the objects according to particular attributes (material such as metal, plastic function i.e. tableware cooking ware). The robot observes queries the user for the existing location of a subset of objects, then rearranges the misplaced object according to the inferred user preference and greedily selects the goal with the higher reward. The baseline here would converge to sort the objects according to one attribute only, while VPL. would infer the latent, and choose the correct location for the given object.
B.2
LLM Preference Learning Dataset Descriptions
Pets This dataset is generated synthetically to specifically study the ability of models to perform divergent preference modeling. The goal here is to choose between different types of pets. For each animal, including bird, cat, dog, and rabbit, we use GPT-4 to generate 100 sentences that describe these kinds of pets. Then we define as to have mix of contexts where users agree and disagree, we construct a preference ordering where both groups like birds the most and rabbits the least. One group of users prefers dogs to cats while another group disagrees and prefers cats to dogs. That is to say, among all 6 comparisons between two kinds of pets, only one pair (dogs versus cats) leads to divergent opinions, while the users agree on other comparisons (birds better than dogs, dogs better than rabbits and so on). This tests the ability of the preference models to capture multimodality, even when the users do agree on some set of preferences.
user groups based their preference order over the pets. So
We then construct the Pets dataset by clustering the prompt and ranked responses according to the
group of preferences that they align with. To generate the Pets dataset, we randomly sample a pair of different pets as well as two corresponding sentences, and then label them be "chosen" and "rejected" according to the preference of either "dog group" or "cat group". The prompt is fixed to be "Human: Please talk about one kind of pet." After all the chosen/rejected pairs are generated, randomly sample 1-4 pairs of responses from the same user that are comparing dogs and cats for use as the context the variational encoder. These are informative since a user's choice over these contexts can clearly express the user's preference group. In this way, we can generate a "full" Pets dataset, and based on that we filter out a "divergent" split that only contains controversial data points (comparing dogs and cats). This dataset is meant as a didactic test for language modeling capabilities, but scalability further tested in the following section with the UltraFeedback-P dataset.
Here we show an example data point for Pets.
Prompt: "Human: Please talk about one kind of pets."
Response A: "Cats communicate through vocalizations." (Rejected)
20

---

Response B: "Birds exhibit complex social behaviors within flocks." (Chosen)
Contexts: I"chosen": "Cats have a preference for certain types of litter.", "rejected": "Dogs enjoy exploring their surroundings."" "J. ["chosen":"Caa have preferred scratching sub- strate.", "rejeected"" "Dogs have a unique personality."]
UltraFeedback-P The original UltraFeedback dataset (0 contains 64,000 pairs of responses, where each response evaluated across four dimensions: helpfulness, honesty, instruction following. and truthfulness. Instead of using the averaged scores, we leverage the fine-grained scores to create a dataset with multi-modal, divergent user preferences. We assume that each user ranks responses based on the score for only one attribute. For example, one user might prioritize helpfulness, while another values honesty the most. Consequently, the same response pair can receive divergent labels from different users depending on their chosen attribute. Following prior work (2 [58. we only consider the first two attributes: helpfulness and honesty, to create the dataset UF-P-2. Next, we filter out response pairs where the two users agrec are indecisive, since we care about multimodal reward modeling in this work, leaving approximately 4,000 prompt and response pairs labeled by each user. From this dataset, we randomly sample smaller subset of K data points, which serves as our context sample set to help identify the latents corresponding to a particular user. For each prompt and pair of responses, we get samples from the context set to characterize the particular user providing responses. These samples, along with the prompt, responses (with preferences over which response is chosen), form complete data point (context prompt responses). This data point is then used to infer the latent distribution and personalized reward for that specific user. Additionally, we introduce a version of the dataset UF-P-4, where we consider all four attributes as distinct users. In the filtering step, we first filter out the response pairs where all of the four users agree on. Then within the data subset of each user, we filter the response pairs where that particular user gives equal ratings to the two responses. It results in dataset of approximately 7, 7,500 prompt and response pairs labeled by each user. The way we generate the context is similar UF-P-2, except that we randomly pick a number from 2 to 8 to be the number of samples to form the context. Note that in UF-P-4, still possible that a question from the context set cannot help with distinguish between two users, because we only filter out the data points that are agreed by all users. This presents large, diverse and challenging benchmark for pluralistic alignment, created from the existing available open-source datasets [20).
B.3 Implementation Details
While the methods described in the methods section only learn the reward model and the latent encoder, We assume that our prior is a multi-variate Gaussian with mean fr and covariance = diag(oo")) where μl, a E Rd; where is the dimension of the latent embedding. In all experiments, they are initialized from standard Gaussian. However, in our control experiments, observed that using learned Gaussian i.c. setting μ and a to learnable parameters under the ELBO objective in Eq. improved performance and stability during training.
Learned Prior:
also use prior p(2) as is common in variational inference methods.
LLM Embeddings. In our experiments, we use the embedding from the last token as our encoding for the prompt+response input. However, we also tried 11m2vec (5) and weighted pooling mecha- nism 48). However, we find that using the last token embedding as inputs to the encoder and for predicting the rewards performs the best.
Active Learning Complexity. In our active inference technique, we use sampling- based method inspired by [8] to generate the active queries for the model. Given a dataset of D queries (s), { we sample S query batches of size Q, where Q number of annotations per batch we get from a user (total possible combinations are K Cxx) Here, Q E [2, 8), so we need to perform O(S Q) passes over the model with batch size 2 ~ [4, 256). Furthermore, this process only needs be performed once after the model is trained to obtain the most discriminative set of queries for the given model. Finally, whenever a new user interacts with the system, we need to get labels on the actively inferred queries (usually 2-4) but do not require any additional passes over the query dataset. In our experiments (Figure 5), we show that using active learning allows the model to achieve comparable performance with fewer queries (~~ 2), as compared to randomly sampled larger (~ 8) queries.
21

---

B.4 Hyperparameters
Table Hyperparameters for learning reward models using VPL. We sweep over these values and report the best results on 5 seeds.
Hyperparameter
Value
Encoder and Decoder Architecture
MLP
Hidden layers
2 layers of width 256
Optimizer
Adam (Kingma & Ba, 2015)
Learning rate
3,0000 x 10
Latent dimension
{8, 16, 32}
B
Cosine annealing between and every 25% steps
VAE Prior
Multi-variate gaussian with learnable parameters μ4, a E R
Context set of queries ||
2,4,8,16
Comparison set size (for SPO + VPL)
1000
Number of annotated sets
5000 (Maze), 10000(Ravens)
Table 3: Hyperparameters for IQL. We use the same parameters across all experiments.
Hyperparameter
Value
Architecture
MLP
Hidden layers
2 layers of width 256 (4 layers of width 1024 for users 5)
Optimizer
Adam (Kingma & Ba, 2015)
Learning rate
3.0000 10-4
Discount
0.99
Expectile
0.9
Temperature
10
Dataset size
4M steps (Navigation), 5K trajectories (Manipulation)
Table 4: Hyperparameters for LLM experiments
Hyperparameter
Value
Pair Encoder Architecture
2 layer MLP with LeakyReLU
Hidden Dimension
512 (GPT2), 1024 (Llama2-7b)
Latent Dimension
512 (GPT2), 1024 (Llama2-7b)
Learning rate
1.0000 x 10
Learning rate scheduler
Cosine with 3% warmup steps
Context set size N
8
Full context sampling set K
100 (GPT2), 16 (Llama2-77)
Batch size
32 (GPT2), 512 (Llama)
Optimizer
Adith weight decay = 0.001) 0.0001 (for Pets), 0.0 (for UltraFeedback-P)
B
Computational Resources
2 x RTX4090, 4 x A100
C
Algorithms
22

---

Algorithm Learning Multimodal Reward Functions using VPL
Require: Preference Data {(s s/s. "ig.y')
Require: Encoder F., Reward Model R. prior p(z)
while not done do
2: Sample batch B ~ D
3: Compute μ.B 0B B(B)
4:
Sample ((B,BOB)
5:
Append 2 to B: [(SA, 8B, y)} - ([55||,,8|(3) v)}
6:
Compute rewards: Tea R() and re 2()
7:
Compute reconstruction loss: Lieson cross cntropy - 7on)),)
8:
Compute KL-loss: LKL = DKDKKLW(BBB. EB) p(z))
9:
Compute total loss: Liotal = Crewon + LKL
10:
Update E. and R by optimizing Liotl
11: end while
Algorithm 2 Policy Optimization using IQL and VPL
Require: Offline Dataset (TI, .
Require: Reward Model (3)
Require: Prior p(2)
Require: Policy [[ass, 2)
for each trajectory Ti {(se, ae, (141)) in D do
2: Sample p(=)
3: for each state 8; in t; do
4: Compute reward ri (81
# Alternatively, = ro(8)41,2)
5: Update dataset with (3t, 2)
6: end for
7: end for
8: Train policy (aass,)) z) using IQL
Algorithm 3 Policy Optimization using IQL and SPO VPL (Note the changes from Algorithm
Require: Offline Dataset (TI, .2....)
Require: Preference Model Po(SA, 8B,2)
Require: Prior p(z)
Require: Policy =)
Require: Comparison set = {51, 82, ...5..
Sampled randomly from offline dataset
1: for each trajectory T; {(St, ae ( in D do
2: Sample ~ p(z)
3: for each state in T; do
4: Compute reward rt Te
5: Update dataset with (st,Te,
6: end for
7: end for
8: Train policy (a s, 2) using IQL
23

---

1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
We highlight the claims and contributions in the abstract and the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We highlight the limitations our approach the Conclusion.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA].
Justification:
We do not claim any substantial theoretical results that requires proof.
Guidelines:
24

---

The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We describe all the algorithms, datasets, models and hyperparameters in detail in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model limited in some way (e.g., registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
25

---

Justification:
We do not provide immediate access to the data and code, but will do so in the future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (nttps://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide all details in the main paper and the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report standard error across multiple seeds for all results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
26

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We indicate the compute resources used in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http::///urrisscueoo http://sc.cccuulleccccccccclleea
Answer:
[Yes]
Justification:
We confirm the code of ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the impact of the work in the concluding section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
27

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We use publicly available datasets and models, and the synthetic toy dataset generated has no risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided relcasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite all the datasets, models and other assets used in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwpsspipeaattteeeeee thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
28

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We introduce new dataset, and provide detailed explanation of the dataset in the Appendix.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details the dataset/code/mmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: We dont use real humans crowd sourcing as subjects for any of our experi- ments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our experiments do not require an IRB.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "adk8AsKyKD", "reviewer": "Reviewer_PuPi", "summary": "This paper introduces a new framework for preference learning which tailors to user-preferences. Human feedback with Variational Preference Learning (VPL) learns a latent reward / preference for each user at the test-time. They furthur show potential application of techniques from active learning and uncertainty estimation to the framework.", "questions": "Suggestions: \n\n- Having a single-modal dataset, such as summarization, would help further ground the work and increase the experimental depth. The expectation is that we shouldn't see many benefits when using VPL compared to traditional RLHF.", "limitations": "Limitations have been specified by the authors.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Paper is generally well-written and clear\n- The research problem is very well motivated -- personalization to preferences that go beyond a universal notion of a single preference function. \n- Method itself is novel and well-formulated to match the problem.", "weaknesses": "- My main concern lies in evaluation outside of designed control environments. To truly test the quality of a Reward Model (RM), one needs to show the downstream policy performance benefiting from improvements in the RM. These results are not present currenlty in section 7. The issue of reward variance as discussed in Section 4.1, may require further design decisions coupled with the optimization algorithm (PPO, REINFORCE, RLOO, online contrastive losses like DPO, etc.). Previous work has studied the general issue of gradient variance in RLHF (which is directly related to reward variance through the REINFORCE estimator) which suggest that this may not be an issue [1]. \n\n\n[1] Ahmadian et. al. \"Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s feedback and are glad that they agree with the motivation of the problem and the novelty of the approach. We address the concerns raised as follows:\n\n> **“Having a single-modal dataset [...] would help further ground the work and increase the experimental depth.”**\n\nThis is an interesting analysis and we thank the reviewer for suggesting this. We ran additional experiments on the UltraFeedback dataset, where we considered the preferences of a single user (preferring the model to be “honest” over all other attributes) to analyze the single-modal case as suggested. **The standard BTL model gives a 77.04% eval accuracy while our VPL model gives a 77.16% eval accuracy. Our model matches the baseline performance, indicating that there is no drop in performance when using VPL compared to traditional RLHF over an unimodal dataset.**\n\n> **“evaluation outside of designed control environments.”**\n\nWe present experiments on learning policies using VPL reward modeling in simulated control domains, which we have expanded and made more realistic in the rebuttal PDF. We acknowledge that our language experiments are focused on preference modeling, but we primarily follow and compare to prior work, which also focuses on preference modeling from diverse datasets alone [1, 2], without including experiments for LLM fine-tuning. We believe that preference modeling alone is interesting, as modeling diverse users may also provide increased interpretability in highlighting potential reasons for preferences [3, Sec. 2]. We believe that further exploring how to best apply VPL to downstream tasks and larger, noisier datasets is an interesting and exciting avenue for future research. Additionally, Prior work has shown that improving RM performance can yield improved downstream performance, both when used in RLHF training [4, 5, 7] and when used in best-of-N settings [5, 6].\n\n--\n\n\n\n[1] Siththaranjan et al. (2023). Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF.\n\n[2] Zhao et al. (2023). Group Preference Optimization: Few-Shot Alignment of Large Language Models.\n\n[3] Sorensen et al. (2024). A Roadmap to Pluralistic Alignment.\n\n[4] Shen et al. (2023). The Trickle-down Impact of Reward (In-)consistency on RLHF.\n\n[5] Gao et al. (2022). Scaling Laws for Reward Model Overoptimization.\n\n[6] Ivison et al. (2024). Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback.\n\n[7] Meng et al. (2024). SimPO: Simple Preference Optimization with a Reference-Free Reward.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VH1fsRzeaP", "reviewer": "Reviewer_KQbQ", "summary": "Instead of learning a unimodal reward model as in standard RLHF, this work aims to learn a reward that covers a diverse range of preferences. It assumes that user preferences are not explicitly given, such as through verbal descriptions in the prompt/instruction. Instead, preferences are implicitly provided through rankings among candidate responses. Technically, it uses variational inference to learn an encoding that characterizes any user’s preferences. Accordingly, the policy model is conditioned on the learned preference latent code, making the generation steerable. The authors also discuss how to select the most representative set of response pairs for each user.", "questions": "See the Weakness section.", "limitations": "See the Weakness section.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow.\n- The study is addressing a crucial problem. The setup is realistic, as users may not always want to explicitly state their preferences.", "weaknesses": "1. It is unclear how many labels are needed to accurately learn a user preference or profile encoder. There should have been experiments evaluating (a) how encoder’s performance improves as the # of user samples increases; and (b) how well the encoding generalizes — can it encode unseen user profiles with high fidelity, and can it extrapolate and interpolate?\n\n2. The claim of being the first work that learns latent-code conditioned reward is not correct. [1] and [2] below also learn multi-modal reward.\n    - [1] Guan, Lin, Karthik Valmeekam, and Subbarao Kambhampati. \"Relative behavioral attributes: Filling the gap between symbolic goal specification and reward learning from human preferences.\" ICLR 2023\n    - [2] Wu, Zeqiu, et al. \"Fine-grained human feedback gives better rewards for language model training.\" NeurIPS 2023\n\n3. The setup of the LLM experiment is quite simple -- the number of attributes or dimensions of preferences is very limited. While the feasibility of attribute-conditioned reward modeling has been demonstrated in previous works [1,2], this work doesn't significantly extend beyond them in terms of scalability.\n\n4. One missing piece in the experiment is an analysis of the latent code's negative impact on the language model. An important feature of language models, especially LLMs, is their versatility. They are not supposed to only answer questions related to pets. One question that needs to be answered is whether conditioning on a pets-related latent code would lead to catastrophic forgetting or distortions in responses to other tasks/questions unrelated to pets.\n\n5. I understand that Section 4.2 discusses the strategy for selecting the most representative state-pair sets during deployment. However, this process seems quite costly as it requires multiple full passes through the dataset. A complexity analysis should have been included.\n\n6. Real-world user profiles and preference data are often unbalanced. The imbalance may affect the approach's effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the useful feedback. We appreciate that the reviewer recognizes the importance of the problem, and the realistic setup introduced to study it. The reviewer raised useful questions about the effectiveness of the user encoder with varying labels, the generalization performance, and the scale of experiments. Overall, the concerns raised are very interesting and we provide additional experiments to demonstrate the usefulness of our method in real-world settings. We respond to all the concerns individually below:\n\n> **[1] and [2] below also learn multi-modal reward**\n\nThank you for pointing us to these references. While these methods present conditional reward models, we highlight the specific differences between our work and the references. We will update our related section to contrast our contribution with these works as follows:\n\n[1] presents an approach to **learning a reward model conditioned on a feature vector** that represents the preferences for the desired behavior. Further, in [1] the users provide feedback on the individual features of a trajectory. However, in our work, we do not assume access to such explicit features. Instead, we propose a method to encode and predict latent distributions directly from binary preferences in an unsupervised setting. **Moreover, [1] does not include experiments on language models and only validates the method on simulated locomotion experiments, while we scale our approach to state-of-the-art preference datasets for LLMs, along with diverse and realistic simulated experiments.**\n\n[2] introduces a method to use dense rewards over multiple specific attributes to align LLMs. **This treats the problem of pluralistic alignment as a multi-objective RL problem** i.e. learn different reward models for each user or objective individually. Whereas in our work we aim to encode and model the multi-modal user preferences, and then condition the same model i.e. steer it to align with the particular user.\n\n> **“this work doesn't significantly extend beyond them in terms of scalability.”**\n\nIn addition to the differences highlighted above, we also scaled our experiments to demonstrate VPL. Following this, we include five additional experiments in the rebuttal PDF:\n1. In Table AM:1, **we scale the language experiments to the UltraFeedback dataset with four users.** In these new experiments, we go beyond the prior work [5] that includes two users (harmlessness and helpfulness), and include all four attributes in the UltraFeedback dataset. By using all available attributes in UF as distinct users, we have created a challenging benchmark for pluralistic alignment from existing real RLHF language datasets.\n2. Our method is more general and focuses on inferring the (potentially unknown number of) different preferences without explicit user information and learning a latent conditioned model to adapt to the particular end user. Meanwhile, [1] uses explicit features for modeling different users or goals, and [2] learns different reward models for the different features. **The probabilistic nature of VPL enables us to do active learning as well (Figure 5 in original submission), which makes this a more efficient model for diverse preferences.**\n3. In Figure AM: 1, we expand our robotics experiments to additional tasks in the Habitat simulator.  We include a new Habitat-Tidy task inspired by TidyBot [7], that presents a realistic setting for diverse users in the real world. \n4. To test VPL with a large number of users, in Figure AM: 4, we demonstrate **that it outperforms the baselines in the Habitat-Rearrange task (when scaled to ~ 100 users)**.\n5. We also show that VPL can **generalize to new users** at test-time i.e. adapt the model to align with unseen goals in Figure AM: 3.\n\nOverall, the additional experiments and key differences highlight that our proposed approach presents a scalable and general method for pluralistic alignment over the included references. In summary,  [1,2] focuses on learning personalized policies or reward models given the explicit classes or users. Further, these methods are tested only over language or control experiments. In our work, **we present an initial step towards learning a probabilistic framework to predict the latent user distribution directly from binary comparisons for language models and learn personalized policies that are tested in dynamic and realistic control settings.**\n\n\n> **“how encoder’s performance improves as the # of user samples increases”**\n    \nWe present additional experiments, where we vary the number of labels at test-time, for LLMs (in Figure AM:2) and control experiments (in Figure 5 in the original submission). **It shows that the performance of the model improves with the number of labels, reaching peak performance at 8 labels**.  However, **actively selecting the queries could allow the model to perform similarly with fewer queries** (see Section 4.3 and Figure 5). Furthermore, using more labels makes the model robust to small noise levels as observed in Figure AM:2.\n    \n> **“how well the encoding generalizes”**\n    \nTo test the generalization of VPL to new users, we include experiments in Figure AM:3, where the reward model and policy are trained on users preferring 10 different goals in the maze, while at test-time the agent interacts with **users preferring 5 unseen goals that are sampled in-distribution**. We compare the performance to an oracle baseline, that uses the exact goal information during training and test time, to analyze how well the encoder can infer the distribution over unseen users. **In Figure AM:3, we observe that VPL significantly outperforms the baselines, and has performance at par with the oracle baseline.** This demonstrates that VPL can interpolate between different users, even slightly outperforming the oracle that is trained on only the set of 10 goals, while VPL uses the prior to optimize the policy over the set of all possible in-distribution goals.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X0yAzqyjIa", "reviewer": "Reviewer_Z9tQ", "summary": "The primary objective of the paper is to design a multi-modal RLHF strategy to align diverse preferences with a latent variable model. The latent variable represents the users/topics and the reward model conditioned on the latent variable is learned for each user preference. The empirical results support the hypothesis on simulated control problems and pluralistic language datasets.", "questions": "1. Can the authors motivate a practical scenario of the setup, where this setup can make sense?\n2. It's not very clear on the specific nature of the prior and posterior being used for each of the tasks. \n3. Also it is done on majorly simulated tasks and environments. Hence, it is important to show the performance in more realistic environments and benchmarks.", "limitations": "Check above", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Aligning to diverse preference with variational inference is one of the most natural ideas and the work provides an interetsing step in that direction.\n2. The approach also provides a method to actively learn user preferences leveraging the posterior uncertainty.\n3. The empirical performance and ablation shows that learning under the probabilistic framework is able to precise multimodal reward model.", "weaknesses": "1. The setup is not extremely practical. For ex: If I am the LLM company when a new user comes to the system and asks a question via the prompt? Then, instead of answering this, we will be doing some active learning to efficiently identify the user right? Since the posterior q(z|y_1, y_2) needs to be estimated for that? Is there any other way to do that or we need to incorporate the active learning strategy every time for each new user/group?\n2. The experimental setup is restricted to a simulated environment and tasks. It is crucial to understand the efficacy of the approach when scaled to more realistic environments and tasks [1, 2, 3]\n3. The work lacks comparison with several baselines on multi-objective RL or aligning with diverse preferences. Please provide a detailed comparison with them, not necessarily through experiments but at least a detailed discussion will be crucial. [1, 2, 3, 4, 5]\n\n\nReferences : \n1. MaxMin-RLHF: Towards an equitable alignment of large language models with diverse human preferences\n\n2. RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation\n\n3. Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration\n\n4. Pareto-Optimal Learning from Preferences with Hidden Context\n\n5. PERSONALIZED SOUPS: PERSONALIZED LARGE LANGUAGE MODEL ALIGNMENT VIA POST-HOC PARAMETER MERGING", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for providing useful feedback regarding our work. We appreciate that the reviewer believes that our introduced approach is natural towards aligning models to diverse preferences. The reviewer brings up useful concerns regarding the scale of the experiments, demonstrating the effectiveness of VPL in realistic benchmarks and drawing bridges to practical scenarios for this problem and solution. We address these questions by providing additional experiments, and outline those and other responses below.\n\n> **“The experimental setup is restricted to a simulated environment and tasks.”**\n   \nThank you for suggesting that we scale our experiments to more realistic tasks and environments. Following this, we include five additional experiments in the rebuttal PDF:\n1. In Table AM:1, **we scale the language experiments to UltraFeedback dataset with four users,** which is comparable in scale and diversity to state-of-the-art prior work on preference modeling [6]. In these new experiments, we go beyond the prior work [6] that includes two users (harmlessness and helpfulness), and include all four attributes in the UltraFeedback dataset. By using all available attributes in UF as distinct users, we have created a challenging benchmark for pluralistic alignment from the best available RLHF language datasets.\n2. In Figure AM: 1, **we expand our robotics experiments to additional tasks in the Habitat simulator.**  We include a new Habitat-Tidy task inspired by TidyBot [7], that presents **a realistic setting for diverse users in the real world.** \n3. To test VPL with a large number of users, in Figure AM: 4, we demonstrate that it outperforms the baselines in the Habitat-Rearrange task (when scaled to ~ 100 users).\n4. We also show that VPL is able to generalize to new users at test-time i.e. adapt the model to align with unseen goals in Figure AM: 3.\n5. In Figure AM:2, we show that this learned model is robust to noise or inconsistency in user context labels when modeling diverse user preferences.\n\nOverall, the additional experiments and analysis, along with the existing results cover a diverse number of realistic tasks with increasing scale and diversity. \n\n> **Can the authors motivate a practical scenario of the setup, where this setup can make sense?**\n\nConsider real-world tasks such as autonomous household robotics. **Each person has strong individual preferences for things like how a user arranges things in their home [7], which must be modeled by a robot to effectively assist in cleaning the home (such as one person may prefer storing shirts in the drawer, while another may prefer them on the shelf). In assistive robotics for the disabled** [10], different users have different physical constraints and preferences regarding feeding methods and food choices. To succeed at these tasks, the robot must efficiently infer and adapt to each user's unique preferences. Our simulated control experiments, as well as the additional experiments shown in Figure AM:1 in the rebuttal PDF, test whether VPL is a promising method for this type of real-world robotics task.\n\nLLMs must interact with a large, diverse, global population of users. Within this population, individuals from one culture may hold collectivist moral values, while another culture is more individualist [8]. Or in certain cultures, it is morally acceptable to eat a certain type of fish while pregnant, but in some others, it is a taboo. **Both of these value systems should be respected, and as such LLMs should be able to recognize and adapt to the values held by a particular user.**\n\nIn our experiments, the UltraFeedback dataset contains multiple attributes that users may prefer, such as harmlessness and helpfulness [9]. E.g. if the prompt is “teach me how to dive into a river” - users can prefer the models to be either helpful (i.e. provide the steps to learn diving) or harmless (i.e. warn the user that it is dangerous and provide no instructions). Here, the LLMs should infer and personalize to the values and preferences of each user, and VPL presents a step in that direction.\n\nIn summary, these directions present some practical cases where an AI model has to personalize and cater to multiple users. And, **our method presents a practical approach to this problem where each user answers a few questions to personalize the downstream model.**\n\n> **“The setup is not extremely practical. For ex: If I am the LLM company when a new user [...] any other way to do that or we need to incorporate the active learning strategy every time for each new user/group?”**\n    \nWe thank the reviewer for bringing up this concern. We would like to present an important clarification that the **active learning procedure needs to be performed only once after the model has been trained** to generate the most discriminative queries i.e. using active learning we get a set of N query pairs. Beyond active learning, we can **instead select random queries** (potentially leading to inefficient performance). For each new user, we get labels over the N pairs, which enables us to estimate the posterior $q(z | (s^i_1, s^i_2, y^i)_{i=1}^N)$. **Our experiments show we get accurate performance in as little as 2 test-time queries** (see Figure AM:2 in the rebuttal PDF and Figure 5 in the original submission). However, if we do not have access to any test-time queries for a new user, we have access to the prior $p(z) \\sim \\mathcal{N}(\\mu, \\sigma^T \\mathrm{I})$, and **can sample from the mode of the prior,** where the model would respond according to the majority group (as in existing preference learning methods). Thus with no additional information, VPL is as performant as a vanilla RLHF model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XWEGdS3pSB", "reviewer": "Reviewer_hKQW", "summary": "This paper approaches the challenge of pluralistic alignment, which arises when the preferences among humans diverge across a population. The work identifies that current preference modeling assumptions do not account for multi-modal reward distributions, which is often the case for pluralistic preferences. Thus, they end up averaging modes and generating inaccurate reward models. To address that, the paper proposes a latent variable model that explicitly models different users and applies variational inference techniques to infer the user latent that conditions the reward model and the optimized policy. The paper also proposes scaling rewards following the likelihood from the model, as a way to ensure all latents follow the same scale, which is necessary for latent-conditioned policy optimization. Finally, the paper evaluates the proposed model in a set of control and language environments, demonstrating gains over prior preference modeling choices.", "questions": "- How does VPL handle irrational users? (i.e., users whose preferences are inconsistent across context/responses)\n- Do the LLM experiments leverage the active inference from Section 4.2?\n- Why isn’t VPL-SPO shown in Table 1?", "limitations": "- I believe the experimental setup does not support any claims of scalability with LLMs and it is unclear whether the proposed method would work in a scenario with multiple users and several variables affecting preferences, which is often the case in the preference datasets.\n- There are some concerns in terms of social impact that are not discussed in Section A.5. A well-performant latent model results in a very personalized language policy, which can potentially lead to two main problems: first, the model being biased/presenting sycophancy [3] to satisfy user beliefs; second, the model reinforcing users with antisocial/unethical/criminal preferences, which often presents preferences that are very far from the rest of the society. Both scenarios are potentially harmful and would be interesting to discuss ways to mitigate them.\n\n\n**References**:\n\n[1] Siththaranjan et. al. Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF, 2023.\n\n[2] Ng et. al. Policy Invariance under Reward Transformations: Theory and Applications to Reward Shaping. ICML, 1999.\n\n[3] Sharma et. al. Towards Understanding Sycophancy in Language Models, 2023.\n\n\n# Post-Rebuttal\n\nPlease refer to my comment under the rebuttal message. Based on that, I am increasing my scores and recommending acceptance.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper approaches pluralistic alignment, a challenge that is mostly always overlooked by the RLHF community, which implicitly assumes that all humans present the same preferences and values. Therefore, the work is very relevant;\n\n- The main merit of the paper is to identify that current modeling assumptions, while assuming the same preference over humans, fall short when modeling diverse preferences. The paper brings very didactic cases that explicitly show such failure and the consequences in the optimized policy (e.g., Figure 3).\n\n- The introduction of the latent variable modeling for preference modeling in RLHF is interesting and novel, and a natural extension of the prior work [1].", "weaknesses": "- The major concern is on the evaluation setup. Although the paper brings a diverse set of experiments, they are all setups where the hidden context is composed of very few variables with simple distributions. This is acceptable to show the failure of previous methods, but not sufficient to claim scalability in the LLM setup for the proposed method. In more detail:\n    - The “Pets” dataset is too simple and does not even require language. From the example in the Appendix B.2, it is possible to ignore the prompt and extract the pet variable from the responses and context. The problem then boils down to fitting a categorical classifier over four classes, where the input is the pet in response A and B, and the same for the context. Given the amount of variables and the dataset size, it is questionable if variational inference is required here - perhaps a simple linear model or MLP can learn directly from the context.\n    - The “Ultrafeedback-P” indeed contains much richer prompts and responses, and the hidden variables (helpfuness, honesty) are not straightforward to extract from the responses. However, the experiment assumes only two users, and there are only two hidden variables as well. Again, this raises concerns if variational inference is required here - perhaps a base LLM can already separate these users in the feature space, but there is no such a baseline in the work.\n        - Interestingly, the paper highlights the requirement of filtering out the context where the users agree, which simplifies the task, as it is easier to identify users with extreme disagreement. It would be important to justify why that is necessary and if it is not a limitation of the proposed method.\n    - The paper is motivated by personalized RLHF and provides modeling for latent-conditioned policies, but in the LLM experiments there is only preference modeling and no RLHF/alignment. This time of experiment is necessary to claim that VPL scales up for pluralistic alignment.\n    - Some methodology details are missing: how the prior is learned (the training objective, data); how humans are simulated in the control environments (L279); how the accuracy of the reward model in Table 1 is defined; the description and details of the environment in used in Appendix A.3;\n- Another crucial concern is that the proposed reward scaling technique is not equivalent to the learned latent reward in the sense of resulting in the same optimal policies (it is not a policy invariant reward shaping [2]). A minimal example that illustrate this is the following MDP: \n    - Imagine an MDP with 11 states (s_0 to s_10). Initial state s_0. There are two actions, a_1, a_2. If the policy takes a_1, it goes to s_1 and receives r_1 = 1000. If it takes a_2, it goes to s_2 and receives r_2 = 100. From s_3, regardless of the action taken, the agent goes to s_1 -> s_3 -> s_5 -> s_7 -> s_9 (terminal), and does not receive rewards in any of these states. Similarly, from s_2, it goes to s_2 -> s_4 -> s_6 -> s_8 -> s_10 (terminal), but receives reward +100 at each state. Assuming no discount in the return, we have V(s_1) = 1000 and V(s_2) = 500, thus the optimal policy should choose a_1. However, if you consider the reward scaling proposed in this work and assume a uniform distribution over states to compute the expectation in L211, then V(s_1) ~= 17/9 and V(s_2) ~= 30/9, which is a different optimal policy.\n    - The reward shaping seems to work empirically at least for the Maze Navigation (Ravens-Manipulation is slightly worse and it is unclear in the LLM experiments if the VPL version leverages the scaling or not). It is important to state this caveat, since reward scaling implicitly assumes policy invariance under reward transformation.\n- One minor concern is regarding the data requirements during test-time inference. During deployment, VPL would require interacting with a user to collect a few labels before performing inference. In the LLM experiment with GPT2, it looks like it requires 100 samples, so it is questionable if this is sample-efficient enough for interacting with humans in the real world. Is this experiment using the active inference from Section 4.2?\n\n\n**Summary**: I believe the paper brings a contribution while presenting the failure of previous methods in pluralist alignment, and showing how they fail. Furthermore, the proposed methodology is interesting and seems to work well in small cases. Nevertheless, the presented empirical evidence does not support the scalability claims in LLMs, as the datasets are too simple and even raises questions whether very simple models could solve it. Furthermore, the reward scaling technique proposed is not well grounded theoretically and leads to different optimal policies. There are also some potential societal consequences that are not discussed. Therefore, I do not support acceptance in the current form of the paper, but. I am open to changing my score in case of the raised concerns being addressed or any misunderstanding clarified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We appreciate that you found the method interesting and agree that pluralistic alignment is a challenge that is mostly overlooked by the RLHF community.  We have conducted 5 additional experiments to demonstrate scalability and include the results in the rebuttal PDF, and address your concerns below. \n\n\n> **“The major concern is on the evaluation setup. [...] not sufficient to claim scalability in the LLM setup for the proposed method.”**\n\nIn our LLM experiments, we adopt the widely used UltraFeedback dataset to expose pluralistic alignment. The original dataset has 4 attributes along which fine-grained preferences are generated, and in the original submission, we used two of these attributes to indicate diverse users (similar to prior work [2]). **We present additional experiments in Table AM:1 where we compare VPL and the baselines, in a scaled and more diverse setting where all four attributes are present in the preference dataset.** Effectively, in these experiments, **we go beyond the prior work** [2] that includes two users (harmlessness and helpfulness), and include all four attributes in the UltraFeedback dataset. This dataset presents a standard benchmark, which is similar to past works in preference modeling with diverse users [2, 3], as well as state-of-the-art preference modeling works that use either UF or HH-RLHF [4,5,6]. By using all available attributes in UF as distinct users, **we have created a challenging benchmark for pluralistic alignment from the best available real RLHF language datasets.**\n\nThe reviewer suggested : “Although the paper brings a diverse [...] hidden context is composed of very few variables with simple distributions”. So, to demonstrate VPL in the presence of more users, we include new experiments with a large number of users and hidden variables, in Figure AM:2. We supplement the Maze and Habitat experiments with 10 and 5 users (Figures 8 and, 4 respectively), **with new Habitat-Rearrange experiments, where we have ~ 100 users**. Each user has a ranking over five different locations for storing objects in their home. **Here, the space of users and variables is combinatorial (as total possible orderings are 5!)**. In Figure AM:4, we observe that VPL infers the user distribution and adapts the downstream model, **demonstrating its effectiveness with a large user population**. Additionally, in Figure AM:3, we test the generalization of VPL to unseen users at test time and show that it can effectively infer and follow unseen goals at test time, outperforming the baselines.\n\n> **“Another crucial concern is that the proposed reward scaling technique is not equivalent to the learned latent reward [...]”**\n\nWe appreciate the reviewer for taking the time to craft this example. To test the given MDP under the scaling from SPO [1], and standard version (BTL) of preference learning [13], we simulated the MDP you suggested and report the value function of states $s_1, s_2$.\n\n|                   | $V(s_1)$ | $V(s_2)$ |\n|-------------------|----------|----------|\n| Ground Truth      | 1000     | 500      |\n| Markovian BTL     | -18.87   | 22.70    |\n| Markovian SPO     | 17/9     | 30/9     |\n| Non-Markovian BTL | 36.46    | -33.36   |\n| Non-Markovian SPO | 35/9     | 10/9     |\n\n**Under a Markovian oracle, both methods fail to generate the correct value function i.e. $V(s_1) < V(s_2)$.** This points to a larger issue in preference learning, which all RLHF methods face and is not unique to our scaling. Essentially, in preference-based learning, we only have binary labels, and no notion of reward scale; so we assume how the scale of rewards affects the preferences (e.g. BTL or SPO). Consequently, under the assumed scale, a cumulative reward estimate (i.e. the value function) might not have the same ordering as the ground truth, which is what we observe in the above table.\n\nWe adopt a common assumption in preference learning, particularly in [1] that inspired the scaling: the oracle providing preference labels is non-Markovian (i.e. it prefers all states in a trajectory with higher returns over those in a lower return one). **As a result, in the given MDP, the states $s_1, s_3, s_5, s_7, s_9$ would be preferred over the states $s_2, s_4, s_6, s_8, s_{10}$. This generates the accurate value functions and policy i.e. $V(s_1) > V(s_2)$, and the policy under this scaling is invariant.**\n\n> **“data requirements during test-time inference [...] it looks like it requires 100 samples, so it is questionable if this is sample-efficient enough for interacting with humans in the real world”**\n\nWe would like to clarify the training and evaluation setup here:  **VPL uses only a random sample of 2-8 questions from the subset of 100 questions in the LM experiments to predict the latent distribution**, which provides us with a sample efficient way to infer the user distribution at test-time, as opposed to using all 100 samples.\n\n\n> **“How does VPL handle irrational users? (i.e., users whose preferences are inconsistent across context/responses)”**\n\nIn Figure AM:2 of the rebuttal PDF, we include experiments with varying number of labels queried from each user at test time. We also add noise to the context i.e. we flip the labels with a certain probability to simulate inconsistent and noisy users. **The performance degrades with increasing noise, but VPL still outperforms the baseline at 25% noise level.  With more labels, VPL performs better at higher noise levels,** but at 50% noise, the context provides equal information for all user types. Consequently, it fails to identify the particular user and the performance coincides with the baseline (that has no mechanism for personalization). This shows that VPL is able to handle irrational users i.e. users whose preferences are inconsistent across context/responses.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
[Disclaimer: All Figures and Tables referred to as **AM:X** are in the **A**dditional **M**aterial.]

We thank the reviewers for their careful reading and constructive feedback. We appreciate that all reviewers recognize the importance and relevance of the problem of pluralistic alignment in preference learning. All reviewers agree that our realistic setup and experiments show the failure of the baselines, and our proposed method is an interesting and novel approach to the described problem. 

Reviewers (hkQw, Z9tQ, KQbQ) had concerns about the scale of experiments for LM preference modeling and evaluation of our method in diverse environments. To address these concerns, we have conducted 7 additional experiments; the results of 5 are included in the rebuttal PDF, and additional results requested by reviewers hKQW and PuPi are included in the response to those reviewers. Below, we provide further context on the 5 experiments in the rebuttal PDF.  

> **Scaling language model results to state-of-the-art preference datasets. (Reviewers hKQW, Z9tQ, KQbQ)**

In Table AM:1, we present additional results while scaling VPL to a version of the UltraFeedback dataset, where we consider users preferring all four diverse attributes. Here, as in prior work [1], we treat the rating attributes (harmless, helpful, etc.) as distinct users. **Going beyond prior work([1] considers only two users), we scale to 4 users, and test whether we can steer the reward model to personalize to the particular user.** As reviewer hkQw mentioned, this further presents a **larger and more complicated dataset** (that is comparable in scale to prior works[1,2,3,4]) to infer the user type and predict the downstream rewards. At the time of writing, the standard and best available RLHF datasets used in prior work on preference modeling are HH-RLHF [5] and UF [6]. By using all available attributes in UF as distinct users, we have created a challenging and scaled benchmark for pluralistic alignment from existing real RLHF language datasets.

In Table AM:1, we observe that VPL outperforms all the baselines in terms of accuracy, leading to a 10% improvement. VPL performs comparably to an Oracle baseline with explicit user information. **This highlights the ability of VPL to scale to larger datasets with more users, and richer prompts and responses.**

> **Scaling to dynamic and realistic home environments. (Reviewers Z9tQ, KQbQ)**

To apply VPL to a set of complex control environments, and practical settings for robots interacting with diverse users, **we include additional experiments in the Meta Habitat Simulator[7].** Inspired by TidyBot [8], we include experiments in a realistic home environment, where the robot has to pick and place objects around the house, personalizing the cleanup based on user preferences (particularly, storing kitchen goods based on their attributes i.e. putting plastic items together and metal items together, or do it for tableware/kitchenware). VPL is able to infer the user’s preferences and steer the robot to the desired behavior with a 30% higher accuracy. Meanwhile, the baselines collapse to a single solution that doesn't cater to the diverse users, as shown in Figure AM:1. **We believe this additional experiment illustrates both the need for personalized preference learning for applications like autonomous household robotics and the fact that VPL is a promising method for achieving it at scale.**

> **Scaling to many (∼100) users. (Reviewers hKQW, KQbQ)**

Currently, the UltraFeedback dataset with four users provides us with one of the largest and most diverse datasets for aligning models, to the best of our knowledge. So, to test the effectiveness of our approach in the presence of a larger number of users (not bounded by datasets), in Figure AM:4, **we include simulated experiments on a new Habitat-Rearrange task, where we have ~ 100 users.** Each user has a ranking over five different locations preferred for storing objects in their home. Here, the space of users and variables is combinatorial (as the total possible orderings are 5!). **This presents a challenging benchmark with a much larger number of users and hidden variables, and in Figure AM:4, we show that VPL consistently outperforms the baselines by ~ 30%** in aligning to user-preferred locations for the rearrangement task.

> **Effect of noise and number of queries from users at test-time. (Reviewers hKQW, KQbQ)**

In Figure AM: 2, we evaluate VPL in the presence of irrational users i.e. how does VPL perform when the users are noisy or inconsistent across their preferences? To simulate this, we progressively add more noise to the context i.e. flip the preference labels in the context with increasing probability. We also test the effect of the number of labels provided by a user, on the downstream performance.

We make the following observations:
1. The performance of **the model degrades with increasing noise, however, VPL still significantly outperforms the baselines** with noisy and inconsistent preference labels from the users.
2. At 50% noise the context labels provide equal information to the encoder about all users. Consequently, it fails to identify the particular user and the performance coincides with the baseline (that has no mechanism for personalization).
3. **Eliciting more preference labels from users generates better latent estimates and improves the accuracy of the preference model.** 
4. Additionally, it also makes the performance robust at high levels of noise. Overall, **VPL provides reasonable performance with as low as ~ 2 queries,** and we report the best performance at ~ 8 test-time queries.

Overall, these results show that VPL is able to "handle irrational users (i.e., users whose preferences are inconsistent across context/responses)”, as requested by reviewer hKQW. Further, this also shows that VPL works is test-time data efficient (i.e. shows reasonable performance with 2 user queries).

Meta-review: This paper proposes a new approach to pluralistic alignment, a very important and topical problem. I agree with Reviewers hKQW and Z9tQ that using latent variable modeling is a natural approach; I'd even say it feels like the "right" approach. Unfortunately, there wasn't much substantiave back-and-forth during the discussion period, but the conversation with Reviewer hKQW seemed fruitful, and I would also encourage the authors to incorporate their responses into the camera-ready. Reviewer KQbQ's concerns do actually appear to be well-addressed, and Reviewer Z9tQ's remaining concerns don't seem like deal breakers. The proposed active learning approach is clearly effective, and I think it's both clever and well-explained. The fact that the experiments cover both simulated control tasks and LLMs is already impressive and shows the generality of the approach. Overall, this is an excellent paper that I expect to be of great interest to many NeurIPS attendees.
Decision: Accept (spotlight)