Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning

Authors: sriyash  poddar
yanming  wan
hamish  ivison
abhishek  gupta
natasha  jaques
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 02:09:35

Abstract: Reinforcement Learning from Human Feedback (RLHF) is a powerful paradigm for aligning foundation models to human values and preferences. However, current RLHF techniques cannot account for the naturally occurring differences in individual human preferences across a diverse population. When these differences arise, traditional RLHF frameworks simply average over them, leading to inaccurate rewards and poor performance for individual subgroups. To address the need for pluralistic alignment, we develop a class of multimodal RLHF methods. Our proposed techniques are based on a latent variable formulation - inferring a novel user-specific latent and learning reward models and policies conditioned on this latent without additional user-specific data. While conceptually simple, we show that in practice, this reward modeling requires careful algorithmic considerations around model architecture and reward scaling. To empirically validate our proposed technique, we first show that it can provide a way to combat underspecification in simulated control problems, inferring and optimizing user-specific reward functions. Next, we conduct experiments on pluralistic language datasets representing diverse user preferences and demonstrate improved reward function accuracy. We additionally show the benefits of this probabilistic framework in terms of measuring uncertainty, and actively learning user preferences. This work enables learning from diverse populations of users with divergent preferences, an important challenge that naturally occurs in problems from robot learning to foundation model alignment.

Main paper: 
 introduction: 
 Reinforcement learning from human feedback (RLHF) has become the predominant technique for aligning AI foundation models to human values. Across domains like natural language processing (NLP) [52] to robotics [62, 50] , RLHF is an effective way to improve the performance, accuracy, and safety of AI models, by ensuring that they align with human preferences [45, 34, 29] The question then becomes: whose preferences? Current RLHF approaches rely on a prescriptive set of values curated by a small set of AI researchers [39, 60, 2] . Moreover, they typically assume that all end-users share the same set of values. Given the concerning lack of diversity in AI [22] , it is clear that this approach cannot account for the range of social, moral, and political values that inform preferences in populations. Recent work has demonstrated that with current RLHF techniques if the majority population has a weak preference for an outcome that will severely disadvantage a minority group, Figure 1 : Current RLHF approaches [52] incorrectly assume a unimodal BTL reward model for a diverse population of users. In this example, users have diverging preferences over the level of detail provided in the responses from a large language model. Without additional context, the BTL model considers both responses to be equally likely. In contrast, our method, VPL, is a personalized approach to RLHF. Using a few samples from a particular user, we infer the distribution over their distinct preferences. Based on this distribution, we condition the reward model to more accurately predict rewards, and enable steering the resulting policy to personalize to the specific user. This enables accounting for and serving the preferences of under-represented groups which would otherwise be ignored by the standard BTL model [52] .the learned reward model will ignore the minority group's preferences [58] . These issues necessitate pluralistic alignment of models to human preferences [59, 44] (see Figure 1 ); ideally, we would democratize RLHF to account for a wider variety of human values to better serve a diverse population.Current approaches to RLHF [52] use the Bradley-Terry-Luce (BTL) [15] model to learn a reward model that explains the human preferences. While the BTL model accounts for noisy preferences, RLHF typically applies this model under the 'unimodal' assumption that all human preferences are derived from a single utility function. This fails to capture scenarios where preferences divergei.e. are multi-modal-due to fundamentally different utilities. For example, Figure 1 shows a case where one group of users prefers detailed responses, while another prefers concise ones. By performing maximum likelihood estimation under the unimodal BTL model, current methods learn a reward function that averages these multi-modal preferences (akin to mode averaging in imitation learning [51] . As we show in our experimental results, this model misspecification leads to reward models that are inaccurate, and the policies optimized on these rewards fail to accomplish tasks per any of the distinct preferences (see Figures 6, 3 ). Thus, vanilla RLHF methods [52, 17] are insufficient for aligning AI systems to diverse human values.In many applications, from large language models (LLMs) to assistive robotics [7] , users are diverse, and AI systems must adapt the generated responses to user-specific preferences to successfully complete the task. Consider, for example, a robot assistant putting away dishes in a user's kitchen: each individual has unique preferences for how the dishes in their kitchen are organized, potentially diverging from others' preferences. In the context of LLMs, failing to adapt to user-specific preferences can make them unhelpful, unsafe, and vulnerable to jailbreak in the presence of conflicting objectives [2, 58] . To build safe and performant foundation models serving a diverse population, we need methods that can explicitly account for and adapt to the inherent plurality of human preferences.These insights suggest that human preferences are not derived from a single utility function, but are affected by unobserved, hidden user context [58] . To accurately model individual utilities, RLHF should be able to efficiently infer and adapt to the user context. With this in mind, we formulate RLHF as a latent variable problem. Building on techniques from variational inference [11, 38] , we propose a method-Variational Preference Learning (VPL)-for multi-modal reward modeling. Intuitively, given a few preference annotations from a particular user, our approach uses a variational encoder to infer a latent distribution over hidden context, and a latent conditional reward model to accurately recover the true multi-modal preference distribution. We derive an evidence lower bound (ELBO) for latent-variable preference-based reward optimization. Our proposed algorithm, VPL, effectively learns a distribution of reward functions from large corpora of preferences from diverse users.In developing practical training methods for such latent-conditioned reward models, we show that several complexities and technical considerations arise. A key problem is that binary comparisons inherently lack information regarding the scale of rewards. Under the BTL model (and correspondingly the VPL model), the preference label between two alternatives A and B can only provide information about r A -r B . So, the learned reward function may have vastly varying reward scales across individual users that adversely affect the optimization landscape of multi-user reinforcement learning [31, 69] using these rewards. To mitigate this, we show how a simple pairwise classification scheme [61, 49] can appropriately bound and scale reward estimates within our latent variable framework, thereby enhancing the performance of downstream learned policies. The predicted latent distribution enables several additional capabilities. In Section 4 we use our approach to learn latent-conditioned policies that can personalize to particular users at test time. Additionally, the latent variable reward models can measure uncertainty in the reward distribution [56] . So, in Section 4.2, we use our approach to actively query [8, 10, 9] users to minimize the number of labels they need to provide before we can adapt to their distinct preferences.Overall, our work introduces a latent variable framework for reward modeling that can encode and approximate the distribution of user preferences directly from binary comparisons, and steer the downstream policy to adapt to diverse user preferences. We conduct a broad range of experiments across three simulated robotics environments and two language tasks with conflicting user preferences. Our results show that in simulated domains, VPL accurately models rewards and improves task performance and personalization. We are able to scale this method to many users, and use active learning to adapt efficiently to particular users with significantly fewer queries at test time. In the language domain, we are able to train multiple LLM-based reward models that learn a separable embedding space that can distinguish between users with divergent preferences. The resulting models outperform existing by RLHF approaches 10-25% [52, 58] by more precisely predicting rewards that align with diverse users and objectives across multiple datasets. 
related work: 
 Reinforcement Learning from Human Feedback (RLHF): We focus on the problem of reinforcement learning (RL) from binary human preferences using the Bradley-Terry-Luce (BTL) model [15] . This has a rich history in the field of RL and robotics, often referred to as Preference-based RL (PbRL) [65, 28, 1, 8, 10] . We specifically build on the framework outlined in Christiano et al. [17] and further expanded in recent works [52, 2, 74, 60, 37, 58] . This has seen a wide range of applications ranging from training robots [17, 10, 62] to finetuning language models for alignment [52, 74, 60] . Our work, enabling RLHF with diverse preferences, is easily applicable to any preference-based learning method, including recent techniques [55, 24] that circumvent reward modeling altogether.RLHF under non-BTL models: Prior work has aimed to study non-BTL models of human behavior and preferences [12, 43, 42, 37, 61] to account for human irrationality and uncertainty in preferences, or intransitive preferences [49, 61, 64] . However, our key argument in this work is less about human irrationality (i.e. inconsistency), and more about the divergence between potentially rational preferences for different labeling users and end users. In this sense, our work is complementary in that the latent variable model can easily be adapted to non-BTL models as well. In fact, we incorporate the technique proposed in Swamy et al. [61] to improve reward learning for downstream applications.Personalized RLHF: While some works [39, 40, 35] characterize similar challenges as VPL, they largely focus on exploring the societal issues underpinning the need for personalization and introducing datasets with diverse annotations. Conitzer et al. [19] argue Social Choice Theory could provide insights into how to aggregate diverse preferences. But these works do not propose a technical method to achieve modeling diverse preferences.Several prior works have looked at trading off conflicting alignment objectives (such as remaining both helpful and harmless) through techniques like Pareto-optimal optimization [13, 16] or multiobjective RL [67, 21] . For example, Jang et al. [33] treats personalization as a multi-objective RL problem, requiring explicit decomposition into different sub-rewards and learning independent reward models for different users. Further, Dai et al. [21] introduces Safe RLHF, an approach that explicitly models the different objectives of helpfulness vs harmfulness. In contrast, our work does not aim to reconcile the diversity but rather solve the model misspecification and learn reward models that can infer the context and specialize to a particular user. Some methods which deal with diverse underlying preferences (e.g. [14] ) aim to model disagreement in the data but typically require more information such as annotator demographics [26, 4] . Li et al. [46] learns to map user information to a fixed set of representations for each user, and conditions the reward model on this embedding. Feng et al. [25] proposes a pluralistic alignment framework, using smaller LMs that are trained on community or user-specific data. In contrast, our method does not assume access to personal user information, but only a few preference annotations from each user. Some works have explored personalization outside of the context of reward learning. Zhao et al. [72] uses in-context learning with few samples from a particular user group, to learn preferences over discrete answers to a question, but does not focus on reward learning with a BTL loss over long context prompt and responses, which is our focus. In addition, our method learns a latent distribution over user types to enable capabilities like active learning [71] and latent-conditioned policy learning.Dumoulin et al. [23] explores annotator-model misspecification and uses synthetic examples to show how RLHF fails in the presence of two users with different preferences, but they do not propose a scalable approach for resolving this issue. The closest work to ours is Distributional Preference Learning (DPL) [58] , which aims to account for hidden context in RLHF. While the motivation is similar, the techniques proposed in DPL include modeling only the mean and variance of the reward distribution across users, or a particular categorical approximation to the reward distribution. As such, DPL can capture uncertainty in the inferred reward distribution, but cannot accurately predict the reward for a particular user. In contrast, VPL explicitly models a user-specific latent variable z, and learns z-conditioned, user-specific reward models, enabling us to accurately reconstruct divergent reward functions for different users. As we will show in Section 6, this leads to significantly improved empirical reward prediction results. Unlike DPL, VPL takes a variational approach to user modeling and allows for reward interpretability, active learning, and model specialization to users at test time. 
technical preliminaries: 
 In this work, we build on the framework of preference-based reward learning (often referred to as RLHF) [17, 1] . In particular, we will consider reward learning methods based on the Bradley-Terry-Luce (BTL) choice model [15] . RLHF has two key phases: (1) inferring a reward function from human-provided labels of ordinal preferences; (2) using reinforcement learning (RL) to train a decision-making policy that maximizes the rewards inferred in step (1) . We will instantiate this framework abstractly, which can then be specialized to both LLMs and robotics.We define a Markov decision process (MDP) [53] M = (S, A, T , γ, ρ 0 ), with state space S, action space A, transition dynamics T , discount factor γ and initial state distribution ρ 0 . Notably, we do not have access to an underlying reward function, but instead have annotators who rank pairs of states s A and s B . We assume annotators have an unknown reward function r(s) that informs their preference labels. More formally, an annotator (h ∈ H) takes a pair of states s A and s B , and returns an ordinal preference i.e. y = 1(s A ≻ s B ), according to r(s) [15, 23] ; where H is the space of all possible annotators. Given a dataset of annotated preferencesD = {(s i A , s i B , y = 1(s i A ≻ s i B ))} N i=1, a typical RLHF procedure learns a reward function r ϕ (s) using a maximum likelihood objective (MLE) on the preferences, where the likelihood: p ϕ (y | s A , s B ) can be defined using the BTL model:EQUATIONNote, that while in this case, the ordinal preferences are on states s A , s B , this can be generalized to trajectories or snippets [17, 52] . Finally, the recovered reward function can then be used to learn (or finetune) a policy π θ (a|s) that can act to maximize the expected sum of approximated rewards in the environment using standard RL algorithms [57, 30, 41] i.e. π θ = arg maxθ E π θ [ t γ t r ϕ (s t )].We note that while the BTL model accounts for some IID noise in preferences through the probabilistic formulation [17, 3, 23] , it does not account for hidden-context and divergent human preferences [58] , and it does not allow the underlying reward models and policies to be personalized to specific users.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] . Justification: We do not claim any substantial theoretical results that requires a proof. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
vpl: incorporating latent context into preference-based learning: 
 The standard BTL formulation described in Section 3 is based on the assumption that all annotators h ∈ H share a single underlying reward function r ϕ (s). This does not hold in practice with a diverse range of annotators. To model diverse, pluralistic preferences, we frame multi-modal reward learning as a latent variable problem, where the latent variable z represents the hidden context affecting the underlying reward function (and thereby the preferences) of an annotator h ∈ H; for instance, it could be representative of their lived experience, or the style of response/policy that they would like the agent to perform. Following this, the latent-conditional reward r ϕ (s, z) is a function of latent z along with state s, and preference likelihoods can be expressed with a latent-conditional BTL model:EQUATIONThe maximum likelihood objective for this model from a dataset of preference labels (with multiple annotators), maxϕ E s A ,s B ,y∼D [log p ϕ (y | s A , s B )] = E s A ,s B ,y∼D log p ϕ (y | s A , s B , z)p(z)dz is in-tractable due to marginalization over the unobserved latent variable z. To tackle this, we can introduce a variational posterior approximationq ψ (z | {(s i A , s i B , y i )} N i=1 ), conditional on multiple annotations {(s i A , s i B , y i = h(s i A , s i B )} N i=1provided by the same annotator h2 . This results in a corresponding evidence lower bound (ELBO), L(ϕ, ψ), for the intractable marginal log p ϕ (y | s A , s B ):EQUATIONThis objective samples a user h ∼ H from the given annotators and a set of annotations from this particular user {(s i A , s i B , y i = h(s i A , s i B )} N i=1 for inferring the latent variable z through the posteriorq ψ (z | {(s i A , s i B , y i )} N i=1). Given the posterior, this objective involves optimizing two terms: a maximum preference likelihood objective (log p ϕ (y | s A , s B , z)) using the contextual BTL model and a regularization term (D KL (q ψ (z | {(s i A , s i B , y i )} N i=1 )) ∥ p(z))) against a prior p(z). Intuitively this objective encodes a set of user-provided annotations {(s i A , s i B , y i )} N i=1 into a latent distribution using the encoder q ψ , and then learns a latent-conditional reward function r(s, z) that best explains the annotated preference data. As the variational encoder q ψ generates a latent distribution, this formulation further enables uncertainty estimation and active learning, as shown in Section 4.2. We describe details further in Appendix B.3.We emphasize that the only additional requirement of this objective, compared to standard RLHF [17] , is a set of annotated pairs from the same annotator, and this information is easily available when annotators are queried in batches [10] . We refer to Algorithm 1 for further details on implementation.Personalized, latent-conditioned policies. The learned reward models r ϕ (s, z) can be used to train personalized, user-specific policies. During training, we learn a latent-conditioned policy π θ (•|s, z) that maximizes the rewards r ϕ (s, z) for different values of z. This allows the policy to adapt to diverse user preferences encoded in the latent space. We can use any standard (RL) algorithm [57, 30, 41] to optimize the latent-conditional reward maximization objective:π θ = arg max θ E π θ ,z∈p(z) [ t γ t r ϕ (s t , z)].For this, we sample z from the prior p(z) during training and learn the policy π θ (•|s, z), to maximize the corresponding latent-conditioned reward r ϕ (s, z).At test-time, we infer a specific user's (h test ) latent context z by posterior inference using a set of labeled preference queries {(s i A , s i B , y i = h test (s i A , s i B ))} N i=1 i.e z ∼ q ψ (z | {(s i A , s i B , y i )} N i=1). We then deploy the personalized policy π θ (•|s, z) conditioned on the inferred z for that user. We outline the complete algorithms for policy training and deployment in Appendix C.In practice, optimizing latent-conditioned reward functions learned with the VPL objective poses unique challenges. The pairwise preferences used to train the reward model in Section 4 do not have information about the scale of rewards, but only their relative ordering. As a simple illustration, if we have a pair of states s A , s B , where the users prefer s A i.e. s A ≻ s B , two different reward functions: r(s A ) = 100, r(s B ) = 50 or r(s A ) = 50, r(s B ) = 0 have the same likelihood under the BTL model. Empirically, we observe that this poses problems for optimizing Equation 3; different values of the latent variable z generate reward functions of vastly different scales. This is an issue for several reasons: 1) varying reward scales adversely affect the landscape of multi-user policy optimization (often observed in multi-task RL) [31] , and 2) it is challenging to identify states where user preferences diverge across the population as differently scaled rewards cannot be directly compared.To address this issue, we experiment with several different techniques for scaling the learned reward functions (see Appendix A.2). Our key insight in solving this challenge lies in the observation that while raw rewards from BTL are not scaled equally across z, probabilities from the preference likelihood model p(y | s A , s B , z) are appropriately scaled. This suggests that an effective solution to the reward scaling issue is to replace the raw rewards from the BTL model (r(s, z)) with likelihoods suggested by the pairwise preference likelihood model p(y | s A , s B , z). In particular, a natural choice of scaled rewards for state s A is the expected likelihood that s A is "preferred" to all other states (or a sampled set of states) s B observed in the data -r ϕ (sA , z) = E s B ∈S [p ϕ (y = 1 | s A , s B , z)].Since these are probabilities, normalized in the [0, 1] range, the scaling of rewards is consistent across latents z. Note that these expected likelihood rewards can easily be obtained from the objective in Equation 3 since we train a latent-conditional preference classifier. While proposed from a very different perspective, we note the similarity of this reward scaling approach to recent work [61, 49] , in particular, Self-Play Preference Optimization (SPO) [61] , which was originally proposed to address the issue of intransitive preferences. Similar to [61] we assume that the oracle providing preference labels is Non-Markovian. Due to this similarity3 , we use VPL-SPO to indicate this approach of preference-based reward scaling throughout our experiments (See Algorithm 3).A natural question that arises for test time deployment of the latent-conditioned policies is how to obtain the set of state pairs {(s i A , s i B )} N i=1 to be annotated with preference labels and used for posterior inference,z ∼ q ψ (z | {(s i A , s i B , y i )} N i=1). Not all query sets {(s i A , s i B )} N i=1 are made equal; some are more informative than others. Certain states (where preferences vary across annotators) are particularly informative in accurately inferring the z which should be used for policy deployment π(a|s, z). In VPL, the probabilistic modeling of the variational encoder naturally allows for active selection of the most informative query set based on maximal information gain, following prior work [8, 10, 50] . Here the provision of preference labels {y i } N i=1 will provide the maximum information about the latent distribution (and indirectly, the user preferences). This active query selection procedure can be expressed as the following optimization problem, maximizing the mutual information between the labels and the latent distribution.EQUATIONThe posterior q ψ is a multivariate Gaussian, and assuming a uniform distribution over the set of labels,q ψ (z | {(s i A , s i B )} N i=1) allows for closed form solution for mutual information I. To solve the maximization objective, we chose the query set (s i A , s i B ) N i=1 with the maximum information gain, across samples from the preference dataset. Finally, we elicit user labels on this maximal query set, infer the latent, and condition the policy on this latent at deployment.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We describe all the algorithms, datasets, models and hyperparameters in detail in the paper. 
scaling vpl for reward learning in large language models (llms): 
 VPL can be used to train pluralistic reward models for LLMs, accounting for diverse human preferences and values. Here we discuss the key details that are essential to scale our method to LLMs. The architecture of our LLM reward model is shown in Figure 2 . Unlike prior work which attempted to insert summary embedding layers into LLMs (see e.g. [18] ), we find that we can successfully compress user preference information into a concise, probabilistic embedding vector z without sacrificing reward model performance. Further details and hyperparameters are discussed in Appendix B. Prompt and Response Embeddings. Since using raw representations of the prompt and responses can increase the context length significantly, we use a pre-trained LLM to encode prompt and response pairs together [6] (to be consistent with previous notation, we assume a preferred state s A contains both a prompt and response [x, r], and e A = LLM (s A )). For efficient training, we pre-compute and freeze the encoded embeddings.Latent Encoder. Given a set of multiple encoded preference queries from the same user, {(e A i , e B i , y i )} N i=1 , we pass each through the same pair encoder to obtain h i = enc(e A i , e B i ). The latent representation z is generated using self-attention layer over the set of encoded pairs, {h i } N i=1 . Reward learning. Here, the representation e A ′ of a new state s A ′ is concatenated with a z sampled from the posterior distribution which is passed into an MLP to predict the rewards. The LLM is fine-tuned using low-rank adaptation (LoRA) [32] , and unlike typical RLHF settings, we find that we need to train the reward model for ≥ 1 epochs to fit the encoder and the reward model.Data augmentation. As we scaled VPL to larger datasets with more users, we found that augmenting the training dataset with multiple context samples from the same user for each new data point is essential to learning an effective encoder. Formally, at training time, given a prompt and response pair s ′ A , s ′ B from a particular user, we generate M ∈ {4, 8} duplicates of this labeled response with different contexts i.e. annotated prompt and response pairs ({(s i A , s i B , y i )} N i=1 ) M j=1 ; where each context ({(s i A , s i B , y i )} N i=1 ) j is sampled from a user annotated subset of size K (K > N ).Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We do not provide immediate access to the data and code, but will do so in the future.Guidelines:• The answer NA means that paper does not include experiments requiring code. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss the impact of the work in the concluding section. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experimental evaluation on simulated control tasks: 
 In our experiments, we answer the following questions: (1) Can VPL accurately learn a multi-modal distribution of reward functions from a preference dataset labeled by diverse users? (2) Do the inferred latent user vectors enable learning a multi-task personalized policy? (3) Can we leverage the posterior to actively query preferences for latent estimation? In this section, we show the benefits of VPL in multiple simulated control tasks and demonstrate that VPL is able to capture multi-modality in the underlying reward functions, arising due to underspecification of annotator goals and preferences.In the following section, we ask whether VPL scales up to RLHF for LLMs.Training and Evaluation Details: We test our hypothesis across evaluation domains in two steps: 1) We train a reward model on a dataset of preferences collected using diverse simulated humans;2) We train a policy using RL to maximize the learned rewards. For these experiments, we use Implicit Q-Learning [41] , an offline RL algorithm that achieves strong performance on offline RL benchmarks [27] . Using the learned reward function r ϕ (s, z), and the prior p(z), we label the reward-free offline RL dataset D = (s t , a t , s t+1 ), by sampling a latent z ∈ p(z), and setting the reward as r t = r ϕ (s t ), or a one-step look-ahead method, where r t = r ϕ (s t+1 ) (refer to Algorithm 2 for complete method) . We include the hyperparameters and the training details in the Appendix BBaselines: We compare our method against multiple baselines: 1) Oracle [47] : This is a goalconditioned offline RL method, that presents an oracle with access to the true reward functions for all annotators. 2) BTL [17] : This is the standard RLHF method from [17, 52] as a baseline, where the reward model is approximated using the unimodal BTL function. 3) DPL [58] : Following the work on accounting for hidden context in RLHF, we train a distributional reward model, using both the mean-variance (MeanVar) and categorical (Categorical) approximation for the reward functions. 4) VPL (Ours): We denote two versions of our method, VPL and VPL + SPO, discussed in Section 4.We evaluate our approach on three diverse simulated control tasks (Figure 4 ) to demonstrate the effectiveness of VPL to learn latent conditioned policies.Maze-Navigation is based on the D4RL benchmark [27] . Here, users guide a pointmass agent to goals marked with their preferred colors. The user's preferred color is underspecified, so the agent has to infer their preferences from a few annotated comparisons and navigate to one of two/ten locations.Ravens-Manipulation requires the agent to rearrange an object on a table-top (akin to rearranging a dining table), based on user preferences. Built on the ravens benchmark [70] , the agent controls a robot arm in a continuous action space to pick and place the box in one of the two preferred locations.Habitat-Rearrange [68] tasks a mobile robot arm with relocating a bowl to a user-specified location among five candidates (e.g., desk, kitchen, table ). The robot is equipped with motion primitives for navigation and manipulation, and the problem involves reasoning over these locations. This simplified one-step reasoning task highlights the need for personalization in assistive robotics, as each person has distinct preferences for robots in their homes.Habitat-Tidy is a simulated environment similar to TidyBot [66] , where the robot infers placement locations for objects in a kitchen based on user preferences. Users may prefer sorting or cleaning objects by different attributes (e.g., function or material), and the robot must adapt to these preferences. Like Habitat-Rearrange, the task involves reasoning over attributes for sorting based on existing preferences, using motion primitives for navigation and manipulation.6.2 Can VPL capture multi-modal reward functions from a dataset of diverse preferences? We generate preferences using multi-modal reward functions across multiple didactic and simulated discrete and continuous control experiments, as shown in Figures 6, 3 , 4, 11 and 10. We see that the BTL baseline with an MLP model averages the different underlying rewards and learns an inaccurate reward model (Figure 3b ). In the presence of a majority, BTL converges to the preferences ignoring the minority groups (See Figure 11 ). While DPL [58] can recover the uncertainty in the reward models due to underspecification, they have no mechanism to recover the individual reward functions. As a result, the DPL reward model estimates high variance rewards (see Figures 6, 10 ) for each particular user. In contrast, VPL infers the hidden user context using the latent variable formulation and accurately recovers the multi-modal reward distribution.6.3 Do distributional reward functions enable learning a steerable multi-task policy?Figure 4 : Performance of a downstream policy on diverse control and reasoning tasks, using the rewards trained using different baselines. We report the mean and standard error over five seeds. Note: Habitat envs have a one-step greedy policy so reward scaling and SPO+VPL are not required.As discussed above, the baselines average the reward modes across the users and learn an inaccurate reward function. So, in Figure 4 the policy for the Maze-Navigation converges to a wrong goal, leading to poor performance and misalignment with all the underlying users. For the Ravens-Manipulation experiments, the baselines randomly choose a goal location and do not adapt to the user preference at test time. Similarly, for the Habitat-Rearrange tasks the baselines are unable to capture the diversity in user preferences over the multiple locations and do not succeed in placing the bowl at the correct location. Finally, in the Habitat-Tidy task, we observe that the baselines converge to an accuracy achieved when they ignore the diverse preferences of sorting the objects by the users.In contrast, the policies trained using VPL outperform all the baselines in terms of the task success rate, according to the user's underlying reward function. For the navigation task, VPL correctly infers the goal and the learned policy can navigate to the goals with a high success rate, and the performance is comparable to a goal-conditioned oracle. For Ravens-Manipulation, VPL infers the user latent at test time and accurately places the box at the right location. Finally, VPL is mostly able to correctly identify the preferred location for the bowl in the user's home and has a higher success rate than the baselines. We note that scaling the rewards via VPL + SPO improves the performance of multi-task RL for optimizing diverse user preferences. In Habitat-Tidy, VPL is able to infer the user preferences and follow the preferred attributes while placing the objects. However, the accuracy of the robot is dependent on the context length and we discuss this in further detail in Appendix A.4.6.4 Can VPL enable active query selection for latent estimation? In Section 4.2, we present an objective to actively query users at test time to efficiently infer user preferences. Figure 5 shows that this technique leads to better performance of the learned policy across varying numbers of queries ∥N ∥. This implies that the active learning objective 4 which maximizes information gain over the latent distribution generates queries that are more discriminative and provides a more informative posterior for user identification. This results in a more efficient adaptation of the downstream policy to the distinct user preferences, achieving the same performance with only half the queries. These methods can be potentially transferred to LLMs to query and identify user preferences with minimal questions. 
llm experiments: 
 In this section, we ask: can VPL scale up to the pluralistic alignment of LLM-based reward models?We compare the reward modeling performance of our method against two baselines: the vanilla BTL model and DPL [58] . We experiment with two LLMs: GPT2 [54] and Llama2-7B [63] , and two pluralistic preference datasets.Prior RLHF works have focused mainly on unimodal BTL models, and as such there is a lack of publicly available datasets containing annotated preferences with divergent objectives. To evaluate our method on capturing multi-modality in preferences for LLMs, we consider two benchmarks. First, we introduce a synthetic dataset, Pets, that directly represents multimodal preferences, and second, we augment the publicly available UltraFeedback [20] dataset.Pets. Here, the dataset is generated to reflect multi-modal user preferences, where each user has a preference ranking over four kinds of animals (in this case cats, dogs, birds, and rabbits). To simulate a setting where users agree on some comparisons and disagree on others, we consider two users who agree on the best and worst pet and disagree on the middle pair of rankings over pets. Preferences here are divergent in certain cases (middle pets), and agree in other instances (best and worst pets), requiring multimodal preference modeling. We evaluate our approach on two versions of the dataset: Pets (Full), and Pets (Divergent) which contains only those prompt and response pairs where the users are divergent (i.e. they have conflicting preferences). For the contexts {(s i A , s i B , y i )} N i=1 , we randomly sample 1-4 other prompts and ranked responses from the same user.UltraFeedback-P. To construct this dataset UF-P (where P stands for personalized), we use the finegrained scores over different attributes available in the UltraFeedback (UF) [20] dataset to construct different users -a similar approach to prior work [58] . We construct a dataset with two users, UF-P-2, who prefer either helpfulness or honesty (hidden attribute), i.e. they generate preferences using the scores for their chosen attributes. To test the ability of VPL to model more users than has been previously attempted in the literature, we create UF-P-4, which uses the fine-grained scores over all the four attributes in the UF dataset [20] to create a dataset with four different users. Here, the users are divergent because given two responses, users following different objectives can have opposite preferences -some users can prefer the helpful response, while others prefer the honest response. In Table 1 : We compare the accuracy of different reward models trained on the two datasets. We report the mean and standard deviation of performance of GPT2-based models on three seeds, and one seed for Llama models.Llama2 for inferring latent distributions, for each prompt and response pair, we sample N different data points from a smaller subset of size K from the dataset (K = 100 for GPT2 and 16 for Llama2). For a deployed LLM system, this is analogous to having a known set of survey questions from which the user must answer a subset of 2-8 questions to personalize the model's behavior to their needs.7.2 Does VPL help to make LLM reward models more pluralistically aligned?In Table 1 , we see that VPL is able to learn a more accurate reward model across all the datasets, capturing the multi-modality in the language preference data. This indicates that VPL can infer the latent representation of the user's preferences z from a few annotated samples, and successfully adapt the reward model. In contrast, the baselines-including the BTL model typically used in widely deployed RLHF [52] models-are unable to fit the datasets because they do not account for divergent preferences. Because the datasets are imbalanced, the baselines can sometimes perform better than random guessing by fitting only the preferences of the majority group, and thus, the performance on Pets (Full) appears high, even though the baselines fail to adapt to the divergent preferences. 
conclusion: 
 In this work, we presented VPL, a technique for pluralistic alignment of preference-based RLHF models via variational inference. We show that VPL can capture diverse preferences and be used for steerable personalized model learning while capturing uncertainty in preferences. We discussed practical considerations for scaling VPL for LLMs and policy learning and showed results across simulated control problems and LLM-based RLHF, significantly outperforming current RLHF techniques.Limitations and Future Work. A key limitation of this work is that as yet, realistic preference datasets containing the opinions of diverse users do not yet exist at scale. This limitation necessitated creating our synthetic preference datasets. Although this was also the approach taken in prior work on personalized RLHF (e.g. [58, 73] ), an important direction for future work will be to apply VPL to more realistic preference data from diverse groups of users. Further, our current experiments on the UltraFeedback dataset assume that when adapting to a new user, we could ask them for preferences over samples from a fixed set of survey questions. In the future, it would be useful to relax this assumption and apply VPL to preferences obtained naturally during a conversation with the user.We believe VPL could offer promising safety benefits in addition to modeling diverse user preferences. Uncertainty detection may help prevent jailbreak attacks caused by conflicting rewards [58] . By capturing uncertainty in the distribution of user preferences, VPL could enhance safety by stopping or refusing to respond when uncertainty cannot be sufficiently reduced [36] . 
acknowledgement: 
 We would like to thank members of the WEIRD and Social RL for the valuable discussions and feedback on the project and manuscript. This project was partly supported by NSF grant no. 2212310. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We use publicly available datasets and models, and the synthetic toy dataset generated has no risk for misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite all the datasets, models and other assets used in the paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We introduce a new dataset, and provide detailed explanation of the dataset in the Appendix. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Didactic example on a toy reward learning problem.To more carefully understand the behavior of VPL empirically, we construct a didactic example [23] as shown in Figure 6 . In this problem, let us consider a mixture of M different annotators providing preferences, where each annotator i has a reward function specified by N (µ i , σ i ) M i=1 that they use to assign binary preferences. Mathematically, we sample the preferences from a mixture of Gaussians:EQUATIONMulti-annotator preferences are simulated by sampling an annotator from this mixture distribution and then assigning binary preferences according to the chosen reward function. We train VPL as described in Section 4 to recover the underlying distribution over reward functions. As expected in Figure 6 , standard RLHF [17] averages over the different modes since it can only represent a single reward function. While prior work in accounting for hidden context in RLHF (DPL [58] ) can learn the uncertainty in the reward functions due to hidden context, it is not able to accurately disambiguate different modes. Meanwhile, VPL is able to infer the underlying context using the approximate posterior q ψ and the recover the individual reward modes through the latent-conditional reward function r(s, z).Figure 6 : Didactic experiments comparing standard BTL [17] , DPL [58] and VPL (Ours). Four Gaussian reward functions generate different binary preference data. The traditional BTL approach [17] averages the different modes, and DPL [58] captures the uncertainty in the rewards due to the multi-modality but cannot accurately predict the true modes. VPL (ours) infers the hidden latent as described in Section 4 and recovers the individual distribution of reward functions.A.2 Does scaling rewards help improve performance? To avoid the problem of high variance rewards (Section 4), we compare the performance of VPL no-norm with VPL + SPO. We further compare against two normalizing schemes: VPL batchnorm where the rewards for each latent is normalized by the mean rewards across a set of state samples i.e. r ′ (s, z) = r(s,z) z) , and VPLmax-norm where all the rewards in the offline dataset are normalized by the maximum reward for any latent.1 M s ′ ∈S r(s ′ ,In Section 4.1, we discuss the problem of generating scaled rewards from latent variable-based reward models and compare the performance across multiple baselines discussed above. As shown in Figure 7 , the batch norm scaling generates highly biased estimates of the rewards, which is catastrophic for the method. However, VPL methods have decent performance at test-time, but are an unprincipled approach to the scaling problem. Our SPO + VPL presents a general method for estimating normalized rewards. Thus, in Figure 7 we can see that our method outperforms the baseline approaches in terms of success rate. The baselines have an unscaled or a biased estimate of the multi-modal rewards leading to sub-optimal performance. For the ravens-manipulation environment, the dataset doesn't contain sub-optimal trajectories, VPL (with max norm) performs comparably.(a) VPL scales to Maze-Navigation task with ten modes of user preferences. BTL expectedly averages the modes and fails to learn. We also see the benefits of scaling rewards across this domain as well, where VPL +SPO performs better than VPL.(b) We compare the performance of baselines and VPL on a Habitat-Rearrange environment with 100 users. VPL can scale to a much larger set of diverse users, complementing the real-world capabilities shown in Table 1 and Figure 4 . In order to test the effectiveness of VPL in scaling to a control problem with larger modes of underlying preferences, we create a task with ten underlying locations that the users could prefer. So, the challenge here is to disambiguate the user preference among a larger space of possible goals and condition the policy to navigate successfully to the goal. Figure 8a shows that our method is able to navigate to the individual goals with a higher success rate, whereas the baseline DPL model [17] collapses to a single user mode. This demonstrates the benefits of scaling VPL to a setting with a large population of diverse users. To test the method at a larger scale, we increase the number of users in the Habitat-Rearrange tasks to 100. It is a combinatorial problem as the users provide rankings over five locations, so all possible users/orderings are 5!. We observe in Figure 8b that VPL significantly outperforms the baselines in inferring the user preference, and steering the robot policy accordingly. In Habitat-TidyBot, the robot's task is to relocate an object in the kitchen (fork, knife, spoon, bowl, pitcher) to a specific location based on user preferences. The user can prefer to arrange objects according to the object function (i.e. kitchenware or tableware), or material (metal or plastic). The agent has to query the location of some of the objects, infer the user type, and arrange the requested object accordingly. Fig 4 shows that the baselines converge to the correct location for the objects agreed upon by the different users, but converge to one mode for the divergent choices. Meanwhile, VPL infers and adapts to user preferences and aligns with humans perfectly. However, one caveat to this is the context length i.e. the number of queries to each user. In Figure 9 we show that as the query length increases VPL can identify users with higher accuracy and achieve higher performance. This happens because certain queries are uninformative about the user preferences (such as things users agree on), and thus, it generates a high variance posterior. As a result, longer context length increases the probability of useful queries, which enables low variance posterior inference and improved alignment during decision-making. In Section 4.2, we also show how to use an active learning approach to generate queries based on a max information gain objective. For LLMs, in Figure 13 , we show that a higher context length also provides robustness to noise in the annotated queries.We visualize the rewards generated by the baseline and the latent conditioned rewards models on the diverse domains described in Section 6. We observe that VPL reconstructs the multi-modal reward functions, based on the inferred latent distribution. Figure 3 shows that the BTL models averages the reward over the user-preferred goals, while VPL accurately reconstructs the individual user-specific rewards. Figure 10 shows that for optimal trajectories solving the task, VPL can accurately match the ground rewards for the two modes. At the same time, DPL [58] predicts high-variance rewards for both cases due to the inherent multi-modality. Finally, in Figure 11 we see that the majority of users consider the desk to be the preferred location of the bowl, and standard BTL models converge to the majority population. Meanwhile, VPL can generate user-specific rewards, satisfying all the user groups.Figure 10: In the Ravens-Manipulation task, we compare the predicted rewards for states s t along timesteps t in oracle trajectories to either of the goals the user prefers. VPL (Ours) can learn the individual reward functions for the two different (closely matching the ground truth rewards for both users) leading to more performant policies (see Figure 4 ), while DPL [58] learns a high variance reward function due to the multi-modality. they want the robot to place their bowl in their home. Accordingly, (a) shows the rewards associated with a particular location ("column") for each annotator ("row"). We see that a majority of the users rank the desk as the most preferred location. Consequently, in (b), unimodal BTL converges to this majority preference ignoring other users. However, in (c) we see that VPL accurately reconstructs diverse preferences and aligns to all five users.Figure 12 : We train GPT2-based VPL, on the UltraFeedback-P dataset. In this plot, we visualize the T-SNE features of the latent distribution z produced by the encoder q ψ on a set of annotated prompts and responses {s i A , s i B , y i } N i=1 from the two users in the dataset. We see that the encoder clusters the users in the latent space, allowing the decoder to personalize the reward models according to multiple objectives preferred by the diverse users belonging to a cluster.In Figure 12 , we show that the VPL-Encoder effectively learns a latent space with clusters that correspond to the user types in the dataset. Previous work has shown that attempting to compress information within an LLM into a single bottleneck embedding layer can hinder performance [18] . However, using the architectural design of VPL as well as user-context data augmentation, VPL is able to learn a compressed user representation that accurately separates users according to their preferences, from only a few preference labels.A.7 Does VPL scale to real-world settings with larger and noisy users? A key assumption in our approach is that context questions accurately represent individual users without noise in the underlying dataset. To test VPL's robustness to noisy context labels at test time, we injected noise by flipping the questions answered by each user and evaluated the trained model's accuracy in predicting rewards. This experiment can help us evaluate how well the model would generalize to new users that have similar preferences to those experienced during training, but may not answer questions in exactly the same way. Figure 13 illustrates that VPL is able to outperform prior work even when 25% of preference labels are flipped at test time. Notably, we observed that longer context lengths result in more accurate reward modeling, even with higher noise injection. This is because the encoder can generate more accurate inferences of the latent distribution when provided with more user information through a larger number of context questions and response labels.We note further that when 50% of the preference labels are flipped, the context preference queries provide no information about the user, and the performance of VPL is equivalent to the baseline BTL model. This mirrors the additional findings presented in Appendix A.8, which demonstrate that when VPL is trained on a unimodal preference language dataset, it gives equivalent performance to BTL. Essentially, when extra preference data is available for a user, VPL can personalize the reward model effectively and attain higher performance. But without that information, it performs just as well as the default BTL model. These findings, as shown in Table 1 ,Figure 13 , and the Appendix, demonstrate VPL's effectiveness in handling multi-modality and noise in large-scale preference datasets, without compromising performance even when no additional preference information is available. This capability suggests the potential for VPL to contribute to the development of next-generation LLMs that are more personalized, inclusive, and efficient.To test that the introduction of a variational framework does not decrease performance in settings where all users have single/aligned preferences, we run an experiment on the UF-P-4 dataset, where we considered the preferences of a single user (preferring the model to be "honest" over all other attributes) to analyze the single-modal case as suggested. The standard BTL model gives a 77.04% eval accuracy while our VPL model gives a 77.16% eval accuracy. Our model matches the baseline performance, indicating no drop in performance when using VPL compared to traditional RLHF over an unimodal dataset.This work has a clear social impact when deployed on user-facing systems like LLMs or household robots. In pluralistic alignment, we assume that some differences in user preferences reflect divergent but equally valid perspectives on which moral or cultural values an LLM should align to; for example, individuals from one culture may hold collectivist moral values, while another culture is more individualist. Both value systems should be respected, and as such LLMs should be able to recognize and adapt to the values held by a particular user. However, the personalized model could potentially either be sycophantic or align with adversarial users, which is undesirable. This raises very interesting questions, such as: At what point should the LLM embrace a more universal set of values? How can we detect when such a point has occurred in a particular conversation? The probabilistic framework of the user distribution could allow us to identify low probability or rare behavior, and also the distributional nature of reward functions can help us point out responses where the users are divergent (maybe signifying disagreement). Additionally, a model could flexibly switch between adhering to the user's personal preferences and conforming to a more universal perspective on topics where it could be biased, or is sensitive to jailbreak [2] . Taking inspiration from Constitutional AI [2] , we can allow a system designer to specify the topics for which the LLM should not engage in user personalization. Overall, this presents an exciting future research direction toward building safe personalized LLMs.We evaluate our methods on three simulated control environments.Maze-Navigation. This task is adapted from the "maze2d-medium-v2" environment from the D4RL benchmark [27] . The observation space is the position and velocity of the robot (p x , p y , p z ), and the pointmass is controlled using torque control. In this environment, point mass doesn't have access to a goal, and diverse users guide the agent to (two or ten) different locations in the maze, marked with their preferred colors. The users label the preferences over two states based on the shortest path to the goal from each state, i.e, the user prefers states closer to their preferred color location. The offline dataset for IQL is collected using the waypoint controllers provided in the D4RL benchmark. For each episode, the agent is spawned at a random location in the maze, interacts with a random user, and navigates to the goal based on the learned reward model and corresponding policy trained on the offline dataset. The oracle reward function is the optimal Q-value of the state, generated using a dynamic programming solution, which is available in D4RL.This task is adapted from the ravens benchmark [70] . The observation space is the 3-D position of the object, the 3-D position of the end effector, and the grasp state of the object, i.e., (ee x , ee y , ee z , p x , p y , p z , grasp). The agent commands absolute positions for the 3-DOF robot arm in end-effector space i.e. (ee ′x , ee ′ y , ee Habitat-Rearrange This is a task based on the Meta Habitat simulator [68] . Here, the objective for the Mobile Manipulator is to pick a bowl and place it at the user's preferred location in the home. However, the exact location is underspecified and needs to be inferred from the user-annotated preferences. The robot uses a motion primitive to navigate and place the object at five possible locations ('desk', 'room', 'dining', 'coffee table', 'sofa'). This problem is reduced to a discrete one-step problem, where the robot has to reason about the best possible location to put the bowl.For ranking the states, the users are generated by randomly choosing a five random orderings of the locations, each corresponding to an individual user. At test time, the agent is greedy and chooses the location with the maximum inferred reward from the learned reward model.Habitat-TidyBot This task is based on Meta Habitat [68] and inspired from the TidyBot task [66] . In this environment, there are 5 objects in the kitchen (spoon, knife, plate, bowl, and spatula). Each user has their preferences for sorting the objects according to particular attributes (material such as metal, plastic or function i.e. tableware or cooking ware). The robot observes or queries the user for the existing location of a subset of objects, then rearranges the misplaced object according to the inferred user preference and greedily selects the goal with the higher reward. The baseline here would converge to sort the objects according to one attribute only, while VPL would infer the latent, and choose the correct location for the given object.Pets This dataset is generated synthetically to specifically study the ability of models to perform divergent preference modeling. The goal here is to choose between different types of pets. For each animal, including bird, cat, dog, and rabbit, we use GPT-4 to generate 100 sentences that describe these kinds of pets. Then we define two user groups based on their preference order over the pets. So as to have mix of contexts where users agree and disagree, we construct a preference ordering where both groups like birds the most and rabbits the least. One group of users prefers dogs to cats while another group disagrees and prefers cats to dogs. That is to say, among all 6 comparisons between two kinds of pets, only one pair (dogs versus cats) leads to divergent opinions, while the users agree on other comparisons (birds better than dogs, dogs better than rabbits and so on). This tests the ability of the preference models to capture multimodality, even when the users do agree on some set of preferences.We then construct the Pets dataset by clustering the prompt and ranked responses according to the group of preferences that they align with. To generate the Pets dataset, we randomly sample a pair of different pets as well as two corresponding sentences, and then label them to be "chosen" and "rejected" according to the preference of either "dog group" or "cat group". The prompt is fixed to be "Human: Please talk about one kind of pet." After all the chosen/rejected pairs are generated, we randomly sample 1-4 pairs of responses from the same user that are comparing dogs and cats for use as the context to the variational encoder. These are informative since a user's choice over these contexts can clearly express the user's preference group. In this way, we can generate a "full" Pets dataset, and based on that we filter out a "divergent" split that only contains controversial data points (comparing dogs and cats). This dataset is meant as a didactic test for language modeling capabilities, but scalability is further tested in the following section with the UltraFeedback-P dataset.Here we show an example data point for Pets.• Prompt: "Human: Please talk about one kind of pets."• Response A: "Cats communicate through vocalizations." (Rejected)• Response B: "Birds exhibit complex social behaviors within flocks." (Chosen)• Contexts: ["chosen": "Cats have a preference for certain types of litter.","rejected":"Dogs enjoy exploring their surroundings."], ["chosen":"Cats have a preferred scratching substrate.","rejected": "Dogs have a unique personality."] UltraFeedback-P The original UltraFeedback dataset [20] contains 64,000 pairs of responses, where each response is evaluated across four dimensions: helpfulness, honesty, instruction following, and truthfulness. Instead of using the averaged scores, we leverage the fine-grained scores to create a dataset with multi-modal, divergent user preferences. We assume that each user ranks responses based on the score for only one attribute. For example, one user might prioritize helpfulness, while another values honesty the most. Consequently, the same response pair can receive divergent labels from different users depending on their chosen attribute. Following prior work [2, 58] , we only consider the first two attributes: helpfulness and honesty, to create the dataset UF-P-2. Next, we filter out response pairs where the two users agree or are indecisive, since we care about multimodal reward modeling in this work, leaving approximately 4,000 prompt and response pairs labeled by each user. From this dataset, we randomly sample a smaller subset of K data points, which serves as our context sample set to help identify the latents corresponding to a particular user. For each prompt and pair of responses, we get 8 samples from the context set to characterize the particular user providing responses. These samples, along with the prompt, responses (with preferences over which response is chosen), form a complete data point (context + prompt + responses). This data point is then used to infer the latent distribution and personalized reward for that specific user. Additionally, we introduce a version of the dataset UF-P-4, where we consider all four attributes as distinct users.In the filtering step, we first filter out the response pairs where all of the four users agree on. Then within the data subset of each user, we filter out the response pairs where that particular user gives equal ratings to the two responses. It results in a dataset of approximately 7,500 prompt and response pairs labeled by each user. The way we generate the context is similar to UF-P-2, except that we randomly pick a number from 2 to 8 to be the number of samples to form the context. Note that in UF-P-4, it is still possible that a question from the context set cannot help with distinguish between two users, because we only filter out the data points that are agreed by all users. This presents a large, diverse and challenging benchmark for pluralistic alignment, created from the existing available open-source datasets [20] .Learned Prior. While the methods described in the methods section only learn the reward model and the latent encoder, we can also use a prior p(z) as is common in variational inference methods. We assume that our prior is a multi-variate Gaussian with mean µ and covariance Σ = diag(σσ T ), where µ, σ ∈ R d ; where d is the dimension of the latent embedding. In all experiments, they are initialized from a standard Gaussian. However, in our control experiments, we observed that using a learned Gaussian i.e. setting µ and σ to learnable parameters under the ELBO objective in Eq. 3 improved performance and stability during training.LLM Embeddings. In our experiments, we use the embedding from the last token as our encoding for the prompt+response input. However, we also tried llm2vec [5] and a weighted pooling mechanism [48] . However, we find that using the last token embedding as inputs to the encoder and for predicting the rewards performs the best.Active Learning Complexity. In our active inference technique, we use a sampling-based method inspired by [8] to generate the active queries for the model. Given a dataset of D queries (s i A , s i B )|K| i=1 , we sample S query batches of size Q, where Q is number of annotations per batch we get from a user (total possible combinations are K C N ). Here, Q ∈ [2, 8] , so we need to perform O(S * Q) passes over the model with batch size 2 Q ∼ [4, 256] . Furthermore, this process only needs to be performed once after the model is trained to obtain the most discriminative set of queries for the given model. Finally, whenever a new user interacts with the system, we need to get labels on the actively inferred queries (usually 2-4) but do not require any additional passes over the query dataset. In our experiments (Figure 5 ), we show that using active learning allows the model to achieve comparable performance with fewer queries (∼ 2), as compared to randomly sampled larger (∼ 8) queries. Compute reconstruction loss: L recon = cross entropy(σ(r s A -r s B ), y)8:Compute KL-loss:L KL = β • D KL (N (µ B , B ) ∥ p(z)) 9:Compute total loss: L total = L recon + L KL Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.Having multiple annotations are important here to be able to accurately infer the user's latent vector zThere are some differences in setup with SPO likelihoods being computed against on-policy samples, while VPL-SPO likelihoods are computed against a fixed offline dataset of comparator states. 


Decision: This paper proposes a new approach to pluralistic alignment, a very important and topical problem. I agree with Reviewers hKQW and Z9tQ that using latent variable modeling is a natural approach; I'd even say it feels like the "right" approach. Unfortunately, there wasn't much substantiave back-and-forth during the discussion period, but the conversation with Reviewer hKQW seemed fruitful, and I would also encourage the authors to incorporate their responses into the camera-ready. Reviewer KQbQ's concerns do actually appear to be well-addressed, and Reviewer Z9tQ's remaining concerns don't seem like deal breakers. The proposed active learning approach is clearly effective, and I think it's both clever and well-explained. The fact that the experiments cover both simulated control tasks and LLMs is already impressive and shows the generality of the approach. Overall, this is an excellent paper that I expect to be of great interest to many NeurIPS attendees.

Reviews: review: paper_summary: paper_summary: This paper introduces a new framework for preference learning which tailors to user-preferences. Human feedback with Variational Preference Learning (VPL) learns a latent reward / preference for each user at the test-time. They furthur show potential application of techniques from active learning and uncertainty estimation to the framework.
strength_weakness: strength_weakness: - My main concern lies in evaluation outside of designed control environments. To truly test the quality of a Reward Model (RM), one needs to show the downstream policy performance benefiting from improvements in the RM. These results are not present currenlty in section 7. The issue of reward variance as discussed in Section 4.1, may require further design decisions coupled with the optimization algorithm (PPO, REINFORCE, RLOO, online contrastive losses like DPO, etc.). Previous work has studied the general issue of gradient variance in RLHF (which is directly related to reward variance through the REINFORCE estimator) which suggest that this may not be an issue [1]. 


[1] Ahmadian et. al. "Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs"
questions: questions: Suggestions: 

- Having a single-modal dataset, such as summarization, would help further ground the work and increase the experimental depth. The expectation is that we shouldn't see many benefits when using VPL compared to traditional RLHF.
limitations: limitations: Limitations have been specified by the authors.
paper_summary: paper_summary: Instead of learning a unimodal reward model as in standard RLHF, this work aims to learn a reward that covers a diverse range of preferences. It assumes that user preferences are not explicitly given, such as through verbal descriptions in the prompt/instruction. Instead, preferences are implicitly provided through rankings among candidate responses. Technically, it uses variational inference to learn an encoding that characterizes any user’s preferences. Accordingly, the policy model is conditioned on the learned preference latent code, making the generation steerable. The authors also discuss how to select the most representative set of response pairs for each user.
strength_weakness: strength_weakness: 1. It is unclear how many labels are needed to accurately learn a user preference or profile encoder. There should have been experiments evaluating (a) how encoder’s performance improves as the # of user samples increases; and (b) how well the encoding generalizes — can it encode unseen user profiles with high fidelity, and can it extrapolate and interpolate?

2. The claim of being the first work that learns latent-code conditioned reward is not correct. [1] and [2] below also learn multi-modal reward.
    - [1] Guan, Lin, Karthik Valmeekam, and Subbarao Kambhampati. "Relative behavioral attributes: Filling the gap between symbolic goal specification and reward learning from human preferences." ICLR 2023
    - [2] Wu, Zeqiu, et al. "Fine-grained human feedback gives better rewards for language model training." NeurIPS 2023

3. The setup of the LLM experiment is quite simple -- the number of attributes or dimensions of preferences is very limited. While the feasibility of attribute-conditioned reward modeling has been demonstrated in previous works [1,2], this work doesn't significantly extend beyond them in terms of scalability.

4. One missing piece in the experiment is an analysis of the latent code's negative impact on the language model. An important feature of language models, especially LLMs, is their versatility. They are not supposed to only answer questions related to pets. One question that needs to be answered is whether conditioning on a pets-related latent code would lead to catastrophic forgetting or distortions in responses to other tasks/questions unrelated to pets.

5. I understand that Section 4.2 discusses the strategy for selecting the most representative state-pair sets during deployment. However, this process seems quite costly as it requires multiple full passes through the dataset. A complexity analysis should have been included.

6. Real-world user profiles and preference data are often unbalanced. The imbalance may affect the approach's effectiveness.
questions: questions: See the Weakness section.
limitations: limitations: See the Weakness section.
paper_summary: paper_summary: The primary objective of the paper is to design a multi-modal RLHF strategy to align diverse preferences with a latent variable model. The latent variable represents the users/topics and the reward model conditioned on the latent variable is learned for each user preference. The empirical results support the hypothesis on simulated control problems and pluralistic language datasets.
strength_weakness: strength_weakness: 1. The setup is not extremely practical. For ex: If I am the LLM company when a new user comes to the system and asks a question via the prompt? Then, instead of answering this, we will be doing some active learning to efficiently identify the user right? Since the posterior q(z|y_1, y_2) needs to be estimated for that? Is there any other way to do that or we need to incorporate the active learning strategy every time for each new user/group?
2. The experimental setup is restricted to a simulated environment and tasks. It is crucial to understand the efficacy of the approach when scaled to more realistic environments and tasks [1, 2, 3]
3. The work lacks comparison with several baselines on multi-objective RL or aligning with diverse preferences. Please provide a detailed comparison with them, not necessarily through experiments but at least a detailed discussion will be crucial. [1, 2, 3, 4, 5]


References : 
1. MaxMin-RLHF: Towards an equitable alignment of large language models with diverse human preferences

2. RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation

3. Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration

4. Pareto-Optimal Learning from Preferences with Hidden Context

5. PERSONALIZED SOUPS: PERSONALIZED LARGE LANGUAGE MODEL ALIGNMENT VIA POST-HOC PARAMETER MERGING
questions: questions: 1. Can the authors motivate a practical scenario of the setup, where this setup can make sense?
2. It's not very clear on the specific nature of the prior and posterior being used for each of the tasks. 
3. Also it is done on majorly simulated tasks and environments. Hence, it is important to show the performance in more realistic environments and benchmarks.
limitations: limitations: Check above
paper_summary: paper_summary: This paper approaches the challenge of pluralistic alignment, which arises when the preferences among humans diverge across a population. The work identifies that current preference modeling assumptions do not account for multi-modal reward distributions, which is often the case for pluralistic preferences. Thus, they end up averaging modes and generating inaccurate reward models. To address that, the paper proposes a latent variable model that explicitly models different users and applies variational inference techniques to infer the user latent that conditions the reward model and the optimized policy. The paper also proposes scaling rewards following the likelihood from the model, as a way to ensure all latents follow the same scale, which is necessary for latent-conditioned policy optimization. Finally, the paper evaluates the proposed model in a set of control and language environments, demonstrating gains over prior preference modeling choices.
strength_weakness: strength_weakness: - The major concern is on the evaluation setup. Although the paper brings a diverse set of experiments, they are all setups where the hidden context is composed of very few variables with simple distributions. This is acceptable to show the failure of previous methods, but not sufficient to claim scalability in the LLM setup for the proposed method. In more detail:
    - The “Pets” dataset is too simple and does not even require language. From the example in the Appendix B.2, it is possible to ignore the prompt and extract the pet variable from the responses and context. The problem then boils down to fitting a categorical classifier over four classes, where the input is the pet in response A and B, and the same for the context. Given the amount of variables and the dataset size, it is questionable if variational inference is required here - perhaps a simple linear model or MLP can learn directly from the context.
    - The “Ultrafeedback-P” indeed contains much richer prompts and responses, and the hidden variables (helpfuness, honesty) are not straightforward to extract from the responses. However, the experiment assumes only two users, and there are only two hidden variables as well. Again, this raises concerns if variational inference is required here - perhaps a base LLM can already separate these users in the feature space, but there is no such a baseline in the work.
        - Interestingly, the paper highlights the requirement of filtering out the context where the users agree, which simplifies the task, as it is easier to identify users with extreme disagreement. It would be important to justify why that is necessary and if it is not a limitation of the proposed method.
    - The paper is motivated by personalized RLHF and provides modeling for latent-conditioned policies, but in the LLM experiments there is only preference modeling and no RLHF/alignment. This time of experiment is necessary to claim that VPL scales up for pluralistic alignment.
    - Some methodology details are missing: how the prior is learned (the training objective, data); how humans are simulated in the control environments (L279); how the accuracy of the reward model in Table 1 is defined; the description and details of the environment in used in Appendix A.3;
- Another crucial concern is that the proposed reward scaling technique is not equivalent to the learned latent reward in the sense of resulting in the same optimal policies (it is not a policy invariant reward shaping [2]). A minimal example that illustrate this is the following MDP: 
    - Imagine an MDP with 11 states (s_0 to s_10). Initial state s_0. There are two actions, a_1, a_2. If the policy takes a_1, it goes to s_1 and receives r_1 = 1000. If it takes a_2, it goes to s_2 and receives r_2 = 100. From s_3, regardless of the action taken, the agent goes to s_1 -> s_3 -> s_5 -> s_7 -> s_9 (terminal), and does not receive rewards in any of these states. Similarly, from s_2, it goes to s_2 -> s_4 -> s_6 -> s_8 -> s_10 (terminal), but receives reward +100 at each state. Assuming no discount in the return, we have V(s_1) = 1000 and V(s_2) = 500, thus the optimal policy should choose a_1. However, if you consider the reward scaling proposed in this work and assume a uniform distribution over states to compute the expectation in L211, then V(s_1) ~= 17/9 and V(s_2) ~= 30/9, which is a different optimal policy.
    - The reward shaping seems to work empirically at least for the Maze Navigation (Ravens-Manipulation is slightly worse and it is unclear in the LLM experiments if the VPL version leverages the scaling or not). It is important to state this caveat, since reward scaling implicitly assumes policy invariance under reward transformation.
- One minor concern is regarding the data requirements during test-time inference. During deployment, VPL would require interacting with a user to collect a few labels before performing inference. In the LLM experiment with GPT2, it looks like it requires 100 samples, so it is questionable if this is sample-efficient enough for interacting with humans in the real world. Is this experiment using the active inference from Section 4.2?


**Summary**: I believe the paper brings a contribution while presenting the failure of previous methods in pluralist alignment, and showing how they fail. Furthermore, the proposed methodology is interesting and seems to work well in small cases. Nevertheless, the presented empirical evidence does not support the scalability claims in LLMs, as the datasets are too simple and even raises questions whether very simple models could solve it. Furthermore, the reward scaling technique proposed is not well grounded theoretically and leads to different optimal policies. There are also some potential societal consequences that are not discussed. Therefore, I do not support acceptance in the current form of the paper, but. I am open to changing my score in case of the raised concerns being addressed or any misunderstanding clarified.
questions: questions: - How does VPL handle irrational users? (i.e., users whose preferences are inconsistent across context/responses)
- Do the LLM experiments leverage the active inference from Section 4.2?
- Why isn’t VPL-SPO shown in Table 1?
limitations: limitations: - I believe the experimental setup does not support any claims of scalability with LLMs and it is unclear whether the proposed method would work in a scenario with multiple users and several variables affecting preferences, which is often the case in the preference datasets.
- There are some concerns in terms of social impact that are not discussed in Section A.5. A well-performant latent model results in a very personalized language policy, which can potentially lead to two main problems: first, the model being biased/presenting sycophancy [3] to satisfy user beliefs; second, the model reinforcing users with antisocial/unethical/criminal preferences, which often presents preferences that are very far from the rest of the society. Both scenarios are potentially harmful and would be interesting to discuss ways to mitigate them.


**References**:

[1] Siththaranjan et. al. Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF, 2023.

[2] Ng et. al. Policy Invariance under Reward Transformations: Theory and Applications to Reward Shaping. ICML, 1999.

[3] Sharma et. al. Towards Understanding Sycophancy in Language Models, 2023.


# Post-Rebuttal

Please refer to my comment under the rebuttal message. Based on that, I am increasing my scores and recommending acceptance.

Comments: title: Response to Rebuttal by Authors
Continued Rebuttal
Continued Rebuttal
Continued Rebuttal
Continued Rebuttal
Continued: Author Rebuttal
comment: Thank you for engaging in discussion with us. We provide additional clarification regarding the motivation and practicality for VPL with LLMs. We also include a detailed explanation of the active learning approach and the prior/posterior structure (in addition to the outline in Section 4).


### ***LLM Motivation***

As we show in our new experiment for reviewer PuPi, by default our method gives the same reward modeling performance as the current standard BTL model if there is a single user in the dataset or if we have no additional information from the user. So, if we have no additional preference labels from a user, our technique will not hurt performance. **However, if we can obtain 2-8 annotations from a user about which response they prefer, we can personalize our reward model to their specific preferences and values, unlocking the benefits of pluralistic alignment of LLMs.**

### ***Active Learning***
We would like to clarify the active learning workflow in detail:
1. We have a set of queries  $(s^i_1, s^i_2)^K_{i=1}$, which is a pair of states (or responses to a given prompt, in case of LLMs). To create the training set, we sample a batch of queries of size $N$, and randomly ask one of the users to annotate it with their preferences ie. we get $\mathbf{S_j} = [(s^i_1, s^i_2, y^i)_{i=1}^N]_j$, where $N << K$.
2. We get multiple annotated batches from the diverse users to form the training set for the reward model i.e $D = (\mathbf{S_1}, \mathbf{S_2} \dots)$.
3. We train the reward model as indicated in Algorithm 1, and obtain the encoder $q(z | \mathbf{S_j})$  (i.e it takes as input a subset of queries $\mathbf{S_j}$) and the reward model R(s,z). The encoder output is $q(z | \mathbf{S_j})$, which is multi-variate gaussian distribution approximating the distribution over user preferences / groups or types.
4. ***Here, we start the active learning process***. Given $q$ and all the queries $(s^i_1, s^i_2)_{i=1}^K$, we generate multiple subsets of size N (total possible samples are $^N C_K$). For each given subset, we can find the information gain over the posterior in Step 3 and Eq. 4.
5. We choose the subset of N pairs $\textbf{S}_{active} = (s^i_1, s^i_2)_{i=1}^N$ with the max information gain. This is the set of questions that are most informative about the user type or distribution.
6. Finally,  **at evaluation for all incoming users, we ask them to annotate the same set of questions in $\textbf{S}_{active}$, and then provide it as input to the encoder to obtain the posterior over this user's preferences**.

**Therefore, the entire process of active learning has to be done only once after training to obtain $\mathbf{S}_{active}$. At eval time, we just need a users to provide labels for the same $N$ pairs to predict the posterior distribution for reward / policy conditioning.**


### ***Posterior and Prior:***
1. We use a gaussian prior and an MLP based posterior similar to the approach in a standard VAE [1].
2. **The prior is a standard multi-variate gaussian of dimension** $d$, where the mean $\mu=[0 \dots 0]^T$ and the covariance $diag(\sigma \sigma^T), \text{ where } \sigma=[1 \dots 1]^T$. Here, $\mu, \sigma \in \mathrm{R}^d$; $d$ is the size of the latent dimension.  In the robotics experiments, we set them to be learnable parameters by setting the requires_grad property = True in pytorch.
3. For, **the posterior in robotics and LLMs we predict two vectors** $\hat{\mu}, \hat{\sigma} \in \mathrm{R}^d$  **using the encoder q and the annotated pairs $\textbf{S}_j$ as input**. It generates the posterior, which is also a multi-variate gaussian with the mean $\hat{\mu}$, and the covariance $diag(\hat{\sigma} \hat{\sigma}^T)$
4. The encoder architecture in the robotics case is a simple MLP that takes in the annotated pairs $\mathbf{S_j} = [(s^i_1, s^i_2, y^i)_{i=1}^N]_j$ i.e. the input is of dimension (2S+1)*N, where S is state dimension, and N is the number of queries.
5. For the LLM encoder architecture, we lay it out in detail in Section 5 and Figure 2 of the paper. It is similar to the robotics case, with certain modifications to handle the high dimensional and complex LLM embeddings.
6. For robotics, we sweep over possible values $d$ in { $8,16,32$ }, and for LLMs, $d=512$. We include the detailed hyperparameters in Appendix B.4
7. Finally, to predict the reward or condition the policy on the user type/preference, we sample a vector $z$ from the predicted posterior, and augment the input to the reward model or the policy to generate personalised rewards / behavior.

[1] Kingma et al. (2013). Auto-Encoding Variational Bayes.

Thank you for the suggestions and we will make this workflow more clear in the paper.
I appreciate the authors’ efforts on addressing questions and bringing new empirical evidence for the work.

My major concern (about the empirical evidence not supporting the scalability claims) was properly addressed with the rebuttal experiments. There are now new experiments showing the method works for >100 users in Habitat Environment and an extension in the Ultrafeedback benchmark. The missing methodology details were also clarified in the rebuttal. I strongly recommend the authors to add these experiments and new clarification text for the camera-ready version, as it considerably improves the paper.

My third concern was also clarified. In terms of the second concern (reward scaling), the rebuttal acknowledges the limitation and links it to an underlying assumption on the oracle’s preference labeling process (non-Markovian). I believe this assumption is strong and somewhat simplistic (if the ground-truth labeler only looks at the trajectory’s return and not for particular states, then the problem modeling could be simplified to a bandit setting instead of the full sequential decision-making setup - as it is often the case of traditional LLM-based RLHF). I believe the camera-ready version should also explicit this limitation/underlying assumption as a clarification.

Overall, most of my crucial concerns were addressed and the paper largely benefited from the new content. Therefore, I am raising my score accordingly.

Other points (questions/limitations) were properly discussed as well. Again, I strongly recommend authors to incorporate this on the camera-ready version, particularly the discussion regarding the societal impact.


(For the next time, **please make sure to adhere to the rebuttal length restriction**).
Thanks for providing clarifications to my concerns. I agree with the motivation for robotics, although for the LLMs still its not very clear. So, to be clear once you have the N pairs, with which you perform the active learning. But, what does it mean that the active learning is done once is not clear? Can you explain this in more detail.
Also, can you please specify the prior and posterior used for the robotics and LLM case?
We would like to thank the reviewer for their response. We apologize for the response length and will be considerate of that in the future.  Could the reviewer please point us to specific questions to expand upon? This would help us address their concerns more effectively.
I thank the authors for their detailed response.

First, I would like to point out that the rebuttal exceeded the character limit by being posted as an Official Comment instead of as a Rebuttal. I hope the authors can better adhere to the conference policy and respect the time of reviewers.

While I don't think the rebuttal adequately addresses my concerns, I still find that the upsides of this work outweigh the downsides. Therefore, I would like to maintain my current positive rating.
> **"Methodology details"**

Thank you for pointing out the missing details. We will update them clearly in the manuscript, as follows:

1. We assume that our prior is a multi-variate Gaussian with mean $\mu$ and covariance $\Sigma=\text{diag}(\sigma\sigma^T)$, where $\mu, \sigma \in \mathrm{R}^d$. In all experiments, they are initialized from a standard Gaussian. However, in our control experiments, we observed that using a learned Gaussian i.e. setting $\mu$ and $\sigma$ to learnable parameters under the ELBO objective (Eq. 3) improved performance and stability during training.
2. The humans are simulated using Oracle reward functions that are included in Appendix B.1. We randomly sample a user type and query a batch of annotations from the given user to generate a single data point.
3. The accuracy of the reward model is "1" if the LM rewards model assigns a higher reward to the response preferred by the given user over the alternative response, and "0" otherwise. We report the average accuracy of the model over the eval set consisting of multiple prompt and response pairs, labeled by diverse users.
4. VPL-SPO is required as a part of policy optimization while currently, we focus on preference modeling for LLMs in this work. VPL and VPL-SPO will model the preferences with similar accuracy. The only difference would be the scale of rewards that affects policy optimization.


-- 

[1] Swamy et al. (2024). A Minimaximalist Approach to Reinforcement Learning from Human Feedback.

[2] Siththaranjan et al. (2023). Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF

[3] Zhao et al. (2023). Group Preference Optimization: Few-Shot Alignment of Large Language Models.

[4] Ivison et al. (2023). Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2.

[5] Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model.

[6] Cui et al. (2023). UltraFeedback: Boosting Language Models with Scaled AI Feedback.

[7] Sorensen et al. (2024). A Roadmap to Pluralistic Alignment.

[8] Shen et al. (2023). The Trickle-down Impact of Reward (In-)consistency on RLHF.

[9] Gao et al. (2022). Scaling Laws for Reward Model Overoptimization.

[10] Ivison et al. (2024). Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback.

[11] Meng et al. (2024). SimPO: Simple Preference Optimization with a Reference-Free Reward.

[12] Sharma et al. (2023). Towards Understanding Sycophancy in Language Models.

[13] Ouyang et al. (2022). Training language models to follow instructions with human feedback.

[14] Bai et al. (2022) Constitutional AI: Harmlessness from AI Feedback.
> **the model being biased/presenting sycophancy [12] to satisfy user beliefs; second, the model reinforcing users with antisocial/unethical/criminal preferences, which often presents preferences that are very far from the rest of the society.**

Thank you for bringing up this insight, and we will include the following discussion in the social impact statement.

In pluralistic alignment, we assume that some differences in user preferences reflect divergent but equally valid perspectives on which moral or cultural values an LLM should align to; for example, individuals from one culture may hold collectivist moral values, while another culture is more individualist. Both value systems should be respected, and as such LLMs should be able to recognize and adapt to the values held by a particular user. However, the personalized model could potentially either be sycophantic or align with adversarial users, which is undesirable. This raises very interesting questions, such as: At what point should the LLM embrace a more universal set of values? How can we detect when such a point has occurred in a particular conversation? The probabilistic framework of the user distribution could allow us to identify low probability or rare behavior, and also the distributional nature of reward functions can help us point out responses where the users are divergent (maybe signifying disagreement). Additionally, a model could flexibly switch between adhering to the user's personal preferences and conforming to a more universal perspective on topics where it could be biased, or is sensitive to jailbreak [14]. Taking inspiration from Constitutional AI [14], we can allow a system designer to specify the topics for which the LLM should not engage in user personalization. Overall, this presents an exciting future research direction toward building safe personalized LLMs.

> **Pets dataset is simple**

We acknowledge that **Pets dataset is a simple synthetic dataset**. Therefore, we present the improved capabilities of our method with additional experiments on a larger and more diverse UltraFeedback dataset (UF-P-4),  with four different users (In Table AM:1). The Pets dataset provides a **sanity check over VPL** to show that the model is able to adapt to multi-modal preferences in imbalanced language datasets.

> **“this raises concerns if variational inference is required here - perhaps a base LLM can already separate these users in the feature space, but there is no such a baseline in the work.”**

  We include an additional baseline in Table AM:1, "VPL+Ground Truth User", where we replace the predicted latent user vector by the ground truth one-hot user vector to adapt the model to different users at training and test-time. **We show that VPL provides comparable performance to this condition (63% vs 66%)**. Here, a key advantage of VPL is that **it does not assume access to explicit user types** but learns to encode and cluster users directly from preference labels, while achieving similar accuracy.

> **there is only preference modeling and no RLHF/alignment**

We present experiments on learning policies using VPL reward modeling in simulated control domains. We acknowledge that our language experiments are focused on preference modeling, but we primarily follow and compare to prior work, which also focuses on preference modeling from diverse datasets alone [2, 3], without including experiments for LLM fine-tuning. We believe that preference modeling alone is interesting, as modeling diverse users may also provide increased interpretability in highlighting potential reasons for preferences [7, Sec. 2]. We believe that further exploring how to best apply VPL to downstream tasks and larger, noisier datasets is an interesting and exciting avenue for future research. Additionally, prior work has shown that improving RM performance can yield improved downstream performance, both when used in RLHF training [8, 9, 11] and when used in best-of-N settings [9, 10].
    
> **“the requirement of filtering out the context where the users agree..”**
   
In the additional experiments with four users (Table AM: 1 of the rebuttal PDF), we filter out instances only where all users disagree. So, the context can still contain queries where at least two users overlap. Thus, **VPL works in cases where different users agree on some responses, but not all of them.**

We will add the following to the limitations sections of the paper: “In LLM preference modeling, VPL assumes that the queries used to generate context inputs for posterior inference contain some useful information to identify the users. In our work, we filter out the instances where all users agree i.e. avoiding degenerate contexts that provide no information about the users.”
> **“comparison with several baselines on multi-objective RL or aligning with diverse preferences. ”**
    
We thank the reviewer for providing additional references for our work.  We will incorporate the additional references in the related works section as follows:

[1, 4] aims at trading conflicting alignment among diverse users with different objectives through techniques like Pareto-optimal optimization or multi-objective RL. The goal of such methods is to optimize the reward model to maximize worst-case performance over the different groups. In contrast, **our work does not aim to optimize against the diversity but rather solve the model misspecification** and learn reward models that can infer the context and specialize to a particular user. This ensures the model can align to all user groups, rather than trade-off among them.

[2] introduces an approach that uses explicit clustering of human groups and learns individual reward functions for the different clusters. Our work instead relies on variational inference and latent conditioned rewards to infer and model diverse humans directly from the preference data.  [2] further introduces an additional method that assumes a single reward function for all the clusters, but **adopts a probabilistic approximation to the reward model, similar to DPL [6]. We include DPL as a baseline in all the LLM experiments** in Table 1 in the original submission and Table AM:1 in the rebuttal PDF, and show that our method outperforms DPL across multiple datasets.

We included a reference to [5] in L112 of the original submission.

[3] proposes a pluralistic alignment framework, **using smaller LMs that are trained on community or user-specific data**. Further, it uses responses from the smaller community LMs to adapt a larger LLM to provide responses covering all or just one specific user. Meanwhile, **our approach adopts an unsupervised method (no access to explicit user distributions)** to identify the latent and condition the preference model towards the specific user preferences.


> **prior and posterior structure**

Here, we provide additional details about the structure of the prior and posterior of our model. We assume that our prior is a multi-variate Gaussian with mean $\mu$ and covariance $\Sigma=\text{diag}(\sigma\sigma^T)$, where $\mu, \sigma \in \mathrm{R}^d$. In all experiments, they are initialized from a standard Gaussian. However, in our control experiments, we observed that using a learned Gaussian i.e. setting $\mu$ and $\sigma$ to learnable parameters under the ELBO objective improved performance and stability during training. The posterior is an MLP that takes in the annotated samples $\textbf{S} \sim (s^i_1, s^i_2, y^i)_{i=1}^N$ and predicts the latent distribution $\mathcal{N}(f(\textbf{S}), g(\textbf{S}))$.

> **Also it is done on majorly simulated tasks [...]**

Our work presents **an algorithmic solution to the problem of personalization** in RLHF. We present extensive experiments across simulated control and language experiments, which we believe present realistic setups and benchmarks. So, we believe real-robot experiments to be beyond the scope of this algorithmic paper and leave on future work the challenge of deploying this method to real-world robot systems.

 
 
--

[1] Chakraborty et al. (2024). MaxMin-RLHF: Towards an equitable alignment of large language models with diverse human preferences.

[2] Park et al. (2024). RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation.

[3] Feng et al. (2024). Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration.

[4] Boldi et al. (2024). Pareto-Optimal Learning from Preferences with Hidden Context

[5] Jang et al. (2024). Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging.

[6] Siththaranjan et al. (2023). Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF

[7] Wu et al. (2023). TidyBot: Personalized Robot Assistance with Large Language Models.

[8] Graham et al. (2013). Moral Foundations Theory: The Pragmatic Validity of Moral Pluralism.

[9] Cui et al. (2023). UltraFeedback: Boosting Language Models with Scaled AI Feedback.

[10] Bhattacharjee et al. (2020). Is More Autonomy Always Better? Exploring Preferences of Users with Mobility Impairments in Robot-assisted Feeding.
> **“This process seems quite costly as it requires multiple full passes through the dataset. A complexity analysis should have been included.”**
  
Thank you for this comment, we would like to clarify **that our approach doesn't require multiple passes over the entire dataset for each new user**. In our active inference techniques, we use a sampling-based method inspired by [3] to actively query the model. Given a dataset of D queries $(s^i_A, s^i_B)_{i=1}^{|D|}$, we sample $S$ query batches of size $Q$, where $Q$ is number of annotated samples per batch we get from a user. Here, $Q \in [2,8]$, **so we need to perform O(S * Q)** passes over the model with batch size $2^Q \sim [4, 256]$. Furthermore, this process only needs to be performed once after the model is trained to obtain the most discriminative set of queries for the given model. Finally, whenever a new user interacts with the system, we need to get labels on the actively inferred queries (usually 2-4) but do not require any additional passes over the query dataset.

> **“The imbalance may affect the approach's effectiveness.”**

Thank you for your attention to this detail. In our control and language experiments, particularly the control experiments and the pets dataset are imbalanced i.e. the preference dataset contains an unequal number of preferences from individual groups or users. As a result, the baselines can achieve > 50% accuracy, converging to the preferences of the majority user groups. So, **VPL works in the presence of imbalanced datasets.**
While recent works [7, 8] focus on achieving Pareto-optimal performance across the groups, VPL treats each user individually via latent conditioning and does not suffer from this problem. VPL is able to personalize to the minority groups as well (see Figure 11 in Appendix A).
    
> **One missing piece in the experiment is an analysis of the latent code's negative impact on the language model. An important feature of language models, especially LLMs, is their versatility. They are not supposed to only answer questions related to pets. One question that needs to be answered is whether conditioning on a pets-related latent code would lead to catastrophic forgetting or distortions in responses to other tasks/questions unrelated to pets.**


Thank you for raising this insightful question. VPL introduces a latent bottleneck during the reward learning process, but since the base architecture of the policy and the reward model does not change otherwise, we do not believe this should majorly affect the performance of the model. We will leave a thorough investigation of this question to future work, but will note this as a potential limitation in the limitations section.


--

[1] Guan et al. (2023). Relative behavioral attributes: Filling the gap between symbolic goal specification and reward learning from human preferences.

[2] Wu et al. (2023). Fine-grained human feedback gives better rewards for language model training.

[3] Sadigh et al. (2017). Active Preference-Based Learning of Reward Functions.

[4] Peng et al. (2024). Pragmatic Feature Preferences: Learning Reward-Relevant Preferences from Human Input.

[5] Siththaranjan et al. (2023). Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF.

[6] Ivison et al. (2023). Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2.

[7] Boldi et al. (2024). Pareto-Optimal Learning from Preferences with Hidden Context

[8] Chakraborty et al. (2024). MaxMin-RLHF: Towards an equitable alignment of large language models with diverse human preferences.
> **Generalization to unseen users.(Reviewers KQbQ)**

In Figure AM:3, we compare VPL against the baselines in a setting where the agent is trained on preferences from users preferring goals from a set of 10 locations in the maze, but at evaluation, **it interacts with users preferring goals from a set of 5 unseen locations** (sampled in distribution). We include a ground truth baseline, conditioned on the exact training and test locations (so explicit goal information). We show that VPL can interpolate between users and pursue the unseen goals better than the baselines, performing comparably to the ones with ground truth information. **This provides evidence that VPL could adapt and personalize AI models to unseen users.**


> **VPL performance in unimodal settings (Reviewers PuPi)**

An interesting concern by reviewer PuPi was to show the performance of VPL in settings with an unimodal preference dataset. We ran additional experiments on the UltraFeedback dataset, where we considered the preferences of a single user (preferring the model to be “honest” over all other attributes) to analyze the single-modal case as suggested. The standard BTL model gives a 77.04% eval accuracy while our VPL model gives a 77.16% eval accuracy. Our model matches the baseline performance, indicating that **there is no drop in performance when using VPL compared to traditional RLHF over an unimodal dataset**.

> **Better explanation of the active learning approach to VPL (Reviewers hKQW, KQbQ)**

Here, we include additional details to clear a misunderstanding regarding the active learning experiments. In our active inference technique, we use a sampling-based method inspired by [9] to generate the active queries for the model. Given a dataset of D queries $(s^i_A, s^i_B)_{i=1}^{|D|}$, we sample $S$ query batches of size $Q$, where $Q$ is number of annotations per batch we get from a user. Here, $Q \in [2,8]$, so we need to perform O(S * Q) passes over the model with batch size $2^Q \sim [4, 256]$. Furthermore, this process only needs to be **performed once after the model is trained to obtain the most discriminative set of queries for the given model.** Finally, whenever a new user interacts with the system, we need to get labels on the actively inferred queries (usually 2-4) but do not require any additional passes over the query dataset. In our experiments (Figure 5), we show that using active learning allows the model to achieve comparable performance with fewer queries (\~2), as compared to randomly sampled larger (\~8) queries.


--

[1] Siththaranjan et al. (2023). Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF

[2] Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model.

[3] Zhao et al. (2023). Group Preference Optimization: Few-Shot Alignment of Large Language Models.

[4] Conitzer et al. (2024). Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback.

[5] Bai et al. (2022). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback.

[6] Cui et al. (2023). UltraFeedback: Boosting Language Models with Scaled AI Feedback.

[7] Puig et al. (2023). Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots.

[8] Wu et al. (2023). TidyBot: Personalized Robot Assistance with Large Language Models.

[9] Sadigh et al. (2017). Active Preference-Based Learning of Reward Functions.