Personalized Language Modeling from Personalized Human Feedback

## Abstract

Abstract Reinforcement Learning from Human Feedback (RLHF) is commonly used to fine-tune large language models to better align with human preferences. However, the underlying premise of algorithms developed under this framework can be problematic when user preferences encoded in human feedback are diverse. In this work, we aim to address this problem by developing methods for building personalized language models. We first formally introduce the task of learning from personalized human feedback and explain why vanilla RLHF can be ineffective in this context. We then propose a general Personalized-RLHF (P-RLHF) framework, including a user model that maps user information to user representations and can flexibly encode our assumptions on user preferences. We develop new learning objectives to perform personalized Direct Preference Optimization that jointly learns a user model and a personalized language model. We demonstrate the efficacy of our proposed method through (1) a synthetic task where we fine-tune a GPT-J 6B model to align with users with conflicting preferences on generation length; and (2) an instruction following task where we fine-tune a Tulu-7B model to generate responses for users with diverse preferences on the style of responses. In both cases, our learned models can generate personalized responses that are better aligned with the preferences of individual users.

## 1 Introduction

Figure: Figure 1: Our Personalized RLHF framework. For training, user information ($u_{1},u_{2},u_{3}$) and preference data (e.g. the preferences for detailed or concise responses) are collected from each user. The user model maps the user information into user representations (user-specific embedding $e_{1},e_{2},e_{3}$ and generic $e_{0}$ for unknown user information), which are learned jointly with the language model. During language generation, for users seen during training, the responses tailored to their individual preferences are generated based on their own user representations ($e_{1},e_{2},e_{3}$) and the prompt ($x^{\prime}$), while for new users unseen during training, the responses are generated using the generic user representation ($e_{0}$) and the prompt ($x^{\prime}$).
Refer to caption: x1.png

Reinforcement Learning from Human Feedback (RLHF) is a widely adopted framework to align pre-trained large Language Models (LMs) [, , ] with human values [].
Current dominating RLHF approaches *implicitly* assume that all crowdsourced human feedback (and preference) comes from the same distribution [, , , ]\xledit. However,
human preferences are inherently diverse and subjective.
When determining a more preferable response, different people may share certain common criteria such as helpfulness, but disagree on other aspects such as length or style.
\xledit
The uniform preference distribution assumption made by current RLHF methods can potentially obscure the inter-user variations, and the LMs fine-tuned under such assumption will likely fail to generate responses that align with each individual user’s preference.
To prevent such misalignments, the diversities in human preferences need to be explicitly accounted for in RLHF [].

In addition to addressing misalignments, personalizing LM outputs based on individual preferences also presents a significant opportunity for democratizing LM utilization across various domains, such as fostering more engaging learning experiences by customizing the content for students in online education and reducing the administrative burden of medical professionals by tailoring summary reports for their respective priorities. To build such personalized systems, we need to account for the fact that human preferences are multifaceted and content-dependent, making it challenging to directly characterize preferences with predefined metrics or templates. A pertinent question thus arises: can we leverage historical user preferences, or more generally, behavioral data to identify the individual preferences of all the users who contributed to the dataset?

Personalization is a well studied topic in machine learning and recommendation systems [], which focuses on generating tailored recommendations or responses based on user profile (e.g. demographic or interests) or historical data of individual users. However, when personalizing LLM generations for different user preferences, it can be difficult to obtain the preference profiles as human preferences are multi-facted, dynamic, and nuanced, which are challenging or even impossible to describe using natural language.
In this paper, we introduce the setting for *Learning from Personalized Human Feedback* (Section 4), \xleditwhich proposes to utilize only the user identifier and the historical feedback data of each user to build personalized language models attuned to individual user preferences.
In personalization scenarios, the numeric user identifier is the most readily accessible form of user information, given that it can be automatically assigned by the backend system based on user login names or IP addresses and requires no additional input from users.
We begin with formalizing the shortcomings of vanilla RLHF (Section 3),
then move to proposing a general *personalized RLHF (P-RLHF)* framework.
Our proposed framework jointly learns a user model that captures individual user preference
and a language model for personalized language generation.
To instantiate our framework, we discuss how different assumptions on user preferences can influence the design of the user model (Section 5.1).
Under the P-RLHF framework, we develop new learning objectives for performing personalized Direct Preference Optimization (P-DPO, section 5.2).
\xleditOn synthetic generation and instruction following tasks using publicly available preference datasets,
we show that P-DPO
can effectively learn personalized LMs, improving the alignment between LM behavior and individual user preferences (Section 6).

## 2 Related Work

RLHF optimizes LMs as RL policies to generate responses aligned with human preferences, using reward models learned from human feedback.
The need for separate reward modeling and policy optimization makes RLHF a complex procedure and prone to instabilities during training.
Direct Preference Optimization (DPO) has emerged as an RL-free algorithm \xleditwhich directly fine-tunes the language model using the preference data,
significantly improving the training efficiency of RLHF [].
RLHF has been utilized to improve the LM performances on a variety of NLP tasks, including summarization [, ], question answering [, ], instruction following [] and improving helpfulness and harmlessness [, ]. While vanilla RLHF models the reward of a whole sequence using a scalar score by a single reward model, recent studies have imposed more sophisticated structures on reward learning, e.g., training separate reward models for different targets [], assigning fine-grained rewards to text segments [], \xleditor merging LMs fine-tuned with separate reward models on pre-defined preference dimensions for personalization [].
To address the diversity in the preference data, one research direction focuses on learning robust LM policies that perform consistently across different groups in the data, including enhancing the performance on challenging samples via group invariant learning [] and mitigating the performance discrepancies between different preference groups via distributionally
robust optimization (DRO) methods [].

Our work differs from previous RLHF approaches in three ways: \xledit(1) we model user-specific preferences instead of assuming that all users share the same preference over responses,
\xledit(2) our personalized LMs are learned directly using personalized feedback data, rather than requiring additional fine-grained learning signals or pre-defined preference dimensions, and (3) we focus on the personalization setting where the LMs are fine-tuned to best accommodate each individual user’s preferences, rather than learning a robust and equitable LM which reduces the performance discrepancies between user groups from a fairness point of view. Though the objective is inherently different, we demonstrate in Section 6 that our method significantly improves the performance on minority users when there is an imbalance in the preference distribution, with the personalization learning objective.
\xleditDue to space constraints, additional related work including crowdsourcing and conditional natural language generation are discussed in Appendix A.

## 3 Vanilla RLHF

We will briefly go over the vanilla RLHF pipeline including DPO
and reflect on their underlying assumptions on user preferences.
In vanilla RLHF, there are three steps [, , ]:
(1) obtain a supervised fine-tuned policy (denoted as $\pi^{\text{SFT}}$) using a demonstration dataset;
(2) learn a Reward Model (RM) using a preference dataset;
and (3) optimize the LM against the learned reward model using reinforcement learning (RL), e.g. via proximal policy optimization (PPO) [].
Uncovering a reparametrization of the optimal LM under the learned RM and the RL objective,
DPO directly learns the LM using a preference dataset [].

#### Vanilla Reward Modeling

The vanilla reward learner has access to a *preference* dataset $\mathcal{D}=\{(x_{i},y_{i,1},y_{i,2})\}_{i=1}^{n}$.
In each sample, $x_{i}$ is the prompt,
$y_{i,1}$ and $y_{i,2}$ are two generated texts such that $y_{i,1}$ is preferred over $y_{i,2}$ (i.e., $y_{i,1}\succ y_{i,2}$) under the prompt $x_{i}$.
A reward model that maps a tuple $(x,y)$ of prompt $x$ and generated text $y$ to a scalar is learned through the following optimization procedure:

$$ $\displaystyle r_{\text{vanilla}}\in\operatorname*{arg\,min}_{r}-\mathbb{E}_{x, y_{1},y_{2}\sim\mathcal{D}}[\log\sigma(r(x,y_{1})-r(x,y_{2}))],$ (1) $$

where $\sigma$ is the sigmoid function and the minimization is over all measurable functions.
As noted in Zhu et al. [], Rafailov et al. [], the underlying assumption for using (1) to learn the reward model $r_{\text{vanilla}}$ is that the user preferences follow
the Bradley-Terry (BT) model [].
In other words, the vanilla RM $r_{\text{vanilla}}$ is the maximum likelihood estimator on the dataset $\mathcal{D}$ under the assumption: for all prompt $x$ and generated texts $y_{1},y_{2}$, user preferences follow

$$ $\displaystyle\mathbb{P}(y_{1}\succ y_{2}|x)=$ $\displaystyle\frac{\exp\left(r(x,y_{1})\right)}{\exp\left(r(x,y_{1})\right)+ \exp\left(r(x,y_{2})\right)}=\sigma(r(x,y_{1})-r(x,y_{2})).$ $$

#### MDP for Vanilla Language Generation

In using the learned RM to perform LM fine-tuning,
the LM is modeled as a policy in a Markov Decision Process (MDP).
More specifically, the MDP is parameterized by $\langle\mathcal{S},\mathcal{V},r,P,T_{\text{max}}\rangle$ where the action space $\mathcal{V}$ is a finite vocabulary set.
At the beginning of each episode, a prompt $x$ is given as the initial state $s_{1}\in\mathcal{S}$.
At each time step $t\in[T_{\text{max}}]$,
the action $a_{t}\in\mathcal{V}$ is a token sampled from the LM policy $a_{t}\sim\pi(\cdot|s_{t})$.
The next state is defined as $s_{t+1}=(s_{t},a_{t})=(x,a_{1},\ldots,a_{t})$.
Thus, the transition probability $\mathbb{P}(s_{t+1}|s_{t},a_{t})=1$ if $s_{t+1}$ appends $a_{t}$ to $s_{t}$ and $\mathbb{P}(s_{t+1}|s_{t},a_{t})=0$ otherwise.
The episode ends at $T\leq T_{\text{max}}$ when an end-of-sequence token is generated or the maximum step $T_{\text{max}}$ is reached.
Given the prompt $s_{1}=x$,
the generated output sequence is $y=(a_{1},\ldots,a_{T})$.
The reward model assigns a scalar reward
at the end of an episode to the generated text $y$
by assigning $r_{t}=r_{\text{vanilla}}(x,y)$ when $t=T$ and $r_{t}=0$ for $t<T$.
Commonly,
the LM $\pi$ is learned through Proximal Policy Optimization (PPO) []:

$$ $\displaystyle\pi_{\text{vanilla}}$ $\displaystyle\in\operatorname*{arg\,max}_{\pi}\mathbb{E}_{x\sim\mathcal{D},y \sim\pi(\cdot|x)}[r_{\text{vanilla}}(x,y)]-\beta\text{KL}(\pi(\cdot|x),\pi^{ \text{SFT}}(\cdot|x)),$ $$

where KL is short-handed for the Kullback–Leibler divergence and $\beta>0$ is a tunable parameter controlling the deviance between $\pi_{\text{vanilla}}$ and $\pi^{\text{SFT}}$.

#### Vanilla DPO

As noted in Rafailov et al. [],
given any RM $r$, its corresponding optimal policy under the RL objective (3) can be written as

$$ $\displaystyle\pi(y|x)=\frac{1}{Z(x)}\pi^{\text{SFT}}(y|x)\text{exp}\left(\frac {r(x,y)}{\beta}\right),$ (2) $$

where $Z(x)$ is a generated-text-independent (or $y$-independent) normalizing factor. Plugging (2) into the reward objective (1), we obtain the following way of obtaining $\pi_{\text{vanilla}}$:

$$ $\displaystyle\pi_{\text{vanilla}}\in\operatorname*{arg\,min}_{\pi}-\mathbb{E}_ {x,y_{1},y_{2}\sim\mathcal{D}}\bigg{[}\log\sigma\bigg{(}\beta\log\frac{\pi(y_{ 1}|x)}{\pi^{\text{SFT}}(y_{1}|x)}-\beta\log\frac{\pi(y_{2}|x)}{\pi^{\text{SFT} }(y_{2}|x)}\bigg{)}\bigg{]},$ (3) $$

where $\mathcal{D}$ is the preference data given in (1).
Finally, under this reparametrization, the corresponding vanilla RM $r_{\text{vanilla}}$ can be written as
$r_{\text{vanilla}}(x,y)=\beta\log\frac{\pi_{\text{vanilla}}(y|x)}{\pi^{\text{
SFT}}(y|x)}+\beta\log Z(x).$
In the following, we reflect on the underlying assumption about user preferences
for vanilla RLHF.

### 3.1 Undesirable Assumption on Vanilla RLHF

We study the behavior and underlying assumption of $r_{\text{vanilla}}$ that is either learned explicitly through the reward modeling step (1) or implicitly through DPO (3).
We show that the corresponding assumption is particularly problematic when users have diverse and subjective preferences.
The proofs for this section are in Appendix B.

As in Ziegler et al. [](^1^11[https://huggingface.co/datasets/openai/summarize_from_feedback](https://huggingface.co/datasets/openai/summarize_from_feedback)), often times, the reward learner has access to identifier information $u\in\mathcal{U}$ of the user who provides their preferences (and annotations), in addition to the prompt and generated texts $(x,y_{1},y_{2})$.
In vanilla RLHF, while we make the explicit assumption that user preferences follow a BT model,
we often ignore the implicit assumption we make on *preference uniformity*:

###### Assumption 3.1 (Preference Uniformity) .

In vanilla reward modeling and DPO, the user preferences are assumed to be uniform, i.e., for all $u\in\mathcal{U}$,

$$ $\displaystyle\mathbb{P}(y_{1}\succ y_{2}|x,u)=\mathbb{P}(y_{1}\succ y_{2}|x).$ (4) $$

This assumption may be reasonable when our goal is to uncover certain preferences that are common across different users, concerning topics like factuality and safety.
In settings where user preferences are diverse (e.g., on styles of generated texts),
this assumption may be undesirable.
We showcase this by first analyzing how $r_{\text{vanilla}}$ behaves on the training dataset,
and then discussing general problems with the Preference Uniformity Assumption 3.1.

###### Lemma 3.2 .

[$r_{\text{vanilla}}$ is equivalent to majority voting]
For all $i\in[n]$, the estimated user preference under $r_{\text{vanilla}}$ is given by

$$ $\displaystyle\mathbb{P}(y_{i,1}\succ y_{i,2}|x_{i})=$ $\displaystyle\sigma(r_{\text{vanilla}}(x_{i},y_{i,1})-r_{\text{vanilla}}(x_{i} ,y_{i,2}))=\frac{\sum_{j\in[\mathcal{C}_{i}]}\mathbb{I}\{y_{j,1}=y_{i,1}\}}{| \mathcal{C}_{i}|},$ $$

where $\mathcal{C}_{i}=\{j\in[n]|x_{j}=x_{i},y_{j,1}=y_{i,1},y_{j,2}=y_{i,2}\}\cup\{j
\in[n]|x_{j}=x_{i},y_{j,1}=y_{i,2},y_{j,2}=y_{i,1}\}$
is the set of sample indices that share the same prompt and response pairs as $x_{i}$.

The above lemma, though straightforward,
showcases one of the fundamental problems with $r_{\text{vanilla}}$.
That is, it induces a majority voting regime where responses preferred by the majority are assumed to be preferred by all users.
In settings where subjectivity in preferences and diversity in opinions matter,
such a majority-voting scheme may silence the preferences of the minority communities.
Reflecting more on the Preference Uniformity Assumption (3.1),
we find that under this assumption, when there is a minority and a majority group that differ in their preferences,
the minority group will necessarily suffer more in the sense that their true preference $\mathbb{P}(y_{1}\succ y_{2}|x,u_{\text{minority}})$ deviates from the assumed uniform preference $\mathbb{P}(y_{1}\succ y_{2}|x)$ more than that for $\mathbb{P}(y_{1}\succ y_{2}|x,u_{\text{majority}})$.
In addition, this deviance increases as the size of the majority group increases.

###### Lemma 3.3 .

When $m=2$ and $\mathbb{P}(u_{\text{majority}})\geq\mathbb{P}(u_{\text{minority}})$, we have that
$|\mathbb{P}(y_{1}\succ y_{2}|x)-\mathbb{P}(y_{1}\succ y_{2}|x,u_{\text{
minority}})|>|\mathbb{P}(y_{1}\succ y_{2}|x)-\mathbb{P}(y_{1}\succ y_{2}|x,u_{
\text{majority}})|$.
In addition, as the majority group size increases, the minority group deviates from the assumed uniform preference more, i.e., $|\mathbb{P}(y_{1}\succ y_{2}|x)-\mathbb{P}(y_{1}\succ y_{2}|x,u_{\text{
minority}})|$ is monotonically increasing with respect to $\mathbb{P}(u_{\text{majority}})$.

Lemma 3.2 and 3.3 showcase that the RM $r_{\text{vanilla}}$, obtained under vanilla reward modeling (1) or vanilla DPO (3), may be unsuitable when user preferences are diverse.
In the following, we propose methods for Personalized DPO (P-DPO) to capture the diversity and subjectivity in human preferences (Section 5).
\lleditDue to space limits, we move discussions on Personalized Reward Modeling to Appendix E.2.
Below we first formally define the task of learning under personalized human feedback.

###### Assumption 3.1 (Preference Uniformity) .

###### Lemma 3.2 .

###### Lemma 3.3 .

## 4 Learning from Personalized Human Feedback: Problem Setup

We first formally define the learning setup when given a *personalized preference* dataset.
This includes specifying what personalized reward and language models are, and the corresponding MDP for personalized language generation.

A personalized human feedback (or preference) dataset $\mathcal{D}_{\text{p}}=\{(x_{i},y_{i,1},y_{i,2},u_{i})\}_{i=1}^{n}$ consists of $n$ samples where $u_{i}\in\mathcal{U}$ is the information of the user who annotates the data or provides the preferences, $x_{i}$ is the prompt,
$y_{i,1}$ and $y_{i,2}$ are two generated texts such that $y_{i,1}\succ y_{i,2}$.
We consider cases where the user information $u_{i}$ is the user id, i.e., $\mathcal{U}=\{0\}\cup[m]$ where $m$ is the maximum number of known user ids, and we set $u_{i}=0$ when the user id is unknown.(^2^22\lleditIn general, though the space of user information $\mathcal{U}$ can be of various forms, we focus on $\mathcal{U}$ being the set of user ids, as (1) it requires the least amount of personal information from a user, and (2) it is the most readily accessible and arguably most flexible form of user information, e.g., we are only able to find publicly available *human* annotated preference learning datasets where $\mathcal{U}$ are the user ids.) For more general cases, we use $u_{0}$ to denote empty user information.

The personalized reward learner is given the dataset $\mathcal{D}_{\text{p}}$ and tasked to learn a personalized RM that maps a tuple $(x,y,u)$ of prompt $x$, generated text $y$, and user information $u$ to a scalar $r_{\text{p}}(x,y,u)$, which is a user-specific reward of $(x,y)$ for user $u$. In language generation, the personalized LM takes in a prompt $x$ and the user information $u\in\mathcal{U}$ and outputs a sequence of tokens $a_{t}\sim\pi_{\text{p}}(\cdot|s_{t},u)$ where $a_{t}\in\mathcal{V}$, $s_{1}=x$ and $s_{t+1}=(s_{t},a_{t})$.
The LM $\pi_{\text{p}}(\cdot|s_{t},u)$ customizes its text generation based on user $u$’s personal preference.
When $u=0$, i.e., there is no user information, the LM generates a non-personalized response.
Similar to the MDP for vanilla language generation,
the MDP for personalized language generation only assigns a reward at the end of an episode using the personalized reward model.
That is, $r_{T}=r_{\text{p}}(x,y,u)$ where $y=(a_{1},\ldots,a_{T})$.

To obtain a personalized language generation policy $\pi_{\text{p}}$,
one may adapt the vanilla DPO approach (Section 5.2).
We provide details on this approach below and discuss how one may first learn a personalized reward model and then optimize the LM against it in Appendix E.2.

## 5 Learning from Personalized Human Feedback: Methodology

We first present our general personalized RLHF (P-RLHF) framework for developing personalized language models.
In P-RLHF, we either directly learn a personalized LM or first learn a personalized RM then build an LM using this RM.
We denote their corresponding methods P-DPO and P-RM respectively, and defer discussion on P-RM to the appendix.
When building personalized LMs, we start with a base LM, often times, $\pi^{\text{SFT}}$, and specify two additional components:

- •
a learnable User Model $f_{\text{P}}$ that extracts a user embedding (tensor) $e_{u}$ from the user identifier $u$. In other words, for all $u\in\mathcal{U}$, a user embedding is given by $e_{u}=f_{\text{P}}(u)$;
- •
an Aggregator function that combines the user embedding $e_{u}$ with the text embedding $e_{x}$. The text embedding is given by the base LM. For example, it can be the input embedding for the text. The aggregator outputs a combined embedding for the user and text to generate personalized language.

Below we first provide some examples of user models
and then discuss how the aggregator may be
specified in P-DPO (Section 5.2).
We will also present new learning objectives for learning the user and personalized language model
and connect P-DPO with vanilla DPO.

### 5.1 User Models

The structure of a user model $f_{\text{P}}$ encodes one’s *preference assumptions* on how different users’ preferences are related to each other.
As noted, $f_{\text{P}}$ may output a vector (or more broadly a tensor) $e_{u}$.
In the following, we illustrate how $f_{\text{P}}$ can be defined.

###### Example 1 (Uniform Preference) .

For all $u\in\mathcal{U}$, the user model $f_{\text{P}}(u)=e$ outputs the same embedding.

We note that this embedding $e$ can be an empty tensor.
This user model assumes that all users share the same embedding,
which is the underlying assumption of vanilla RLHF.

###### Example 2 (Individualized Preference) .

Let $\mathcal{U}=\{0\}\cup[m]$ be the set of user indices.
The user model outputs $f_{\text{P}}(0)=e_{0}$ for (unknown) users indexed by 0 0.
For all $u\in[m]$,
the user model outputs $f_{\text{P}}(u)=e_{0}+o_{u}$
where $o_{u}$ is a user-specific offset tensor.

This user model assumes that each user $u$ has their individualized preference offset $o_{u}$ while maintaining a component $e_{0}$ shared across users, as shown in Figure 4(a).
The common tensor $e_{0}$ can be understood as the commonality across user preferences concerning topics like factuality and safety.
When the individual user embedding $e_{0}$ and offsets $o_{u}$ are vectors,
one can implement this user model as an embedding table. We provide more details on this in Section 6.

###### Example 3 (Cluster-based Preference) .

Let $\mathcal{U}=\{0\}\cup[m]$ be the set of user indices.
For all $u\in\mathcal{U}$,
the user model output $f_{\text{P}}(u)=V\cdot w_{u}$ where $V\in\mathbb{R}^{K\times d}$ \lleditis an embedding table, $K$ is the number of clusters, and
$w_{u}\in\mathbb{R}^{K}$ is a weight vector for each user.

Inspired by the crowdsourcing literature [], we develop this clustering-based user model that assumes user embeddings (and hence preferences) span a common set of vectors given by $V$; each user embedding is a weighted combination of these vectors (Figure 4(b)).
In the special case where $w_{u}$’s are one-hot vectors and thus each user embedding $e_{u}$ is a row of $V$, user embeddings form clusters and hence the name cluster-based preference.
\lleditGiven a user model, to perform P-DPO,
we need ways to integrate the user embeddings with text embeddings and a new learning objective for learning the user model simultaneously with the language model, as illustrated next.

###### Example 1 (Uniform Preference) .

###### Example 2 (Individualized Preference) .

###### Example 3 (Cluster-based Preference) .

### 5.2 Personalized DPO for Personalized LM

Given the *learnable* user model $f_{\text{P}}$,
we have a user embedding $e_{u}\in\mathbb{R}^{T_{u}\times d}$ for each user $u\in\mathcal{U}$.
We integrate it into the personalized language model through soft prompting [].
In this case, the aggregator prepends $e_{u}$ to the input (text not positional) embedding $e_{x,y}\in\mathbb{R}^{T_{x,y}\times d}$ given by the base LM, where $T_{x,y}$ is the token length and $d$ is the token-wise embedding dimensionality.
The user embedding $e_{u}\in\mathbb{R}^{T_{u}\times d}$ is a tensor with $T_{u}$ being its corresponding user token length—a factor that controls the expressivity of user embeddings.

Given the personalized LM $\pi_{\text{P}}$ specified with the corresponding user model $f_{\text{P}}$ and aggregation mechanism (i.e., soft prompting),
we use the following learning objective in P-DPO:

$$ $\displaystyle\min_{\pi_{\text{P}}}-\mathbb{E}_{x,y_{1},y_{2},u\sim\mathcal{D}_ {\text{P}}}\bigg{[}\alpha\log\sigma\bigg{(}\beta\log\frac{\pi_{\text{P}}(y_{1} |x,u)}{\pi^{\text{SFT}}(y_{1}|x)}-\beta\log\frac{\pi_{\text{P}}(y_{2}|x,u)}{ \pi^{\text{SFT}}(y_{2}|x)}\bigg{)}$ $\displaystyle+(1-\alpha)\log\sigma\bigg{(}\beta\log\frac{\pi_{\text{P}}(y_{1}| x,u_{0})}{\pi^{\text{SFT}}(y_{1}|x)}-\beta\log\frac{\pi_{\text{P}}(y_{2}|x,u_{ 0})}{\pi^{\text{SFT}}(y_{2}|x)}\bigg{)}\bigg{]},$ $$

where $\beta>0$ controls the deviance of $\pi_{\text{P}}$ from the policy $\pi^{\text{SFT}}$.
Recall that $u_{0}$ indicates empty user information.
The loss can be viewed as a combination of a user-specific loss term that relies on explicit user identifier $u$ and a user-agnostic loss term that depends on $u_{0}$. The user-agnostic loss uses the same preference data but without any user identifier.
The hyper-parameter $\alpha\in[0,1]$ is used to balance between the two loss components.

###### Remark 5.1 .

When $\alpha=0$ and $f_{\text{P}}$ is the uniform preference-based user model (Example 1) that outputs an empty tensor,
P-DPO is the same as vanilla DPO.

###### Remark 5.1 .

## 6 Experiments

In this section, we empirically evaluate the effectiveness of P-DPO in aligning with individual user preferences. Our experiments are constructed on two open-ended text generation tasks (1) Generation with Synthetic Preferences: we use the Reddit TL;DR summarization dataset(^3^33[https://huggingface.co/datasets/openai/summarize_from_feedback](https://huggingface.co/datasets/openai/summarize_from_feedback)) curated by Stiennon et al. [], where the pairwise preferences over Reddit post summaries were crowdsourced from multiple workers, and a GPT-J 6B model [] supervised fine-tuned using the TRLX library(^4^44[https://huggingface.co/CarperAI/openai_summarize_tldr_sft](https://huggingface.co/CarperAI/openai_summarize_tldr_sft)) [] is used as the SFT. To better control the diversity of user preferences, we simulated individual worker preferences based on the summary length.
(2) Instruction Following under Different Preference Profiles: we use the Personalized-Soups dataset [], which includes pairwise feedback for responses to GPT4-Alpaca instructions [] under various user preference profiles, and Tulu-7B [], an instruction fine-tuned LLaMA-7B model, as the SFT.

### 6.1 Generation with Synthetic Preferences

In TL;DR dataset, each comparison includes a Reddit post $x$, two summaries $y_{1}$ and $y_{2}$, and the id of the worker who annotated it.
For proof of concept, we simulated two preference scenarios:
we randomly sampled $70\%$ of the workers and set them to prefer the longer response and set the rest $30\%$ of the workers to prefer the shorter one, making the preference for longer responses the majority group in the data.
To ensure effective learning of user preferences with sufficient data, we include the top $10$ workers with the highest annotation counts in the train split of the dataset for training, and these workers are denoted by ids from $1$ to $10$ for reference purposes.
For evaluation, we use all the data in the validation split, including comparisons annotated by both top $10$ and non-top $10$ workers.
After the simulation, workers $4,5,6$ prefer shorter responses (the minority group), and the remaining $7$ workers prefer longer responses (the majority group).
We experimented with user models that encode (1) individualized preference assumption (Example 2), or (2) cluster-based preference assumption with $K=5$ (Example 3).
We set $\alpha=0.5$ and $T_{u}=10$ in both cases.
More details about the dataset, the preprocessing steps, and the experiments can be found in Appendix C.1 and C.2.

As the learning objective of P-DPO can be viewed as deriving the optimal policy under an implicit reward function $r_{\text{P}}(x,y,u)=\beta\log\frac{\pi_{\text{P}}(y|x,u)}{\pi^{\text{SFT}}(y|x)}$, we first evaluate its performance using the accuracy of this implicit reward, i.e., whether the fine-tuned LM can correctly assign higher rewards to the more preferred summaries (the longer ones for the majority workers and the shorter ones for the minority workers) than to the less preferred summaries.
We report three accuracy-based metrics: (1) Accuracy-top: the pooled accuracy of all samples annotated by the top $10$ workers, (2) Accuracy-generic: the accuracy of comparisons annotated by unseen workers in the validation set, to measure how strong P-DPO will perform on new users with the generic user embedding $e_{0}$ learned from the data of seen users, and (3) Accuracy-average: the mean and standard error of the per-user accuracy of the top $10$ workers, divided into the majority group and the minority group.

Figure: Figure 2: Accuracies (Acc) of vanilla DPO and P-DPO models. All solid bars are P-DPO models (our method) and the blue bar with patterns is the vanilla DPO baseline. (a) The accuracies of top $10$ workers. (b) The accuracies of P-DPO models in the abalation study in Section 6.3 on top $10$ workers, where Ind stands for Individual. (c) The accuracies of top $40$ workers. The numeric results are provided in Appendix C.3.
Refer to caption: x2.png

The accuracies of the vanilla DPO model and the P-DPO models are shown in Figure 2 (a). Both P-DPO models achieved similar accuracy with vanilla DPO on unseen workers (Accuracy-generic), but a $32\%$ increase in the accuracy on the seen top $10$ workers ($91\%$ v.s. $59\%$ for Accuracy-top). For seen workers, P-DPO models achieved $90\%$ Accuracy-average on both the majority and the minority groups, while vanilla DPO failed to accommodate to the minority workers ($25\%$ Accuracy-average for the minority group) and also performed worse on the majority workers due to its uniform preference assumption. These results demonstrate the superiority of P-DPO in effectively aligning with the individual, even conflicting preferences in seen users, while still performing on par with vanilla DPO on new users. From the Accuracy-top curves shown in Figure 3 (b), we can see that the accuracies of both P-DPO models (the red and green lines) increased rapidly after training started and converged to optimal performance level before the end of one epoch, showcasing the learning efficiency of P-DPO.

Figure: Figure 3: (a) The number of words (mean and standard error) in the responses P-DPO with individualized preference generated for workers $1$ to $10$, compared to SFT(S), vanilla DPO (V) and P-DPO using generic user embedding (G). Workers $4,5,6$ are from the minority group (prefer shorter responses) and P-DPO only generated end-of-text tokens for them thus resulting in generation lengths of 0 0. (b) The Accuracy-top curves over training steps for the vanilla DPO and P-DPO models. (c) The Accuracy-generic curves over training steps for the vanilla DPO and P-DPO models.
Refer to caption: x3.png

In addition, we evaluated the generation capabilities of P-DPO by comparing the number of words in the responses generated for different workers, using $50$ prompts randomly sampled from the evaluation set. From Figure 3 (a), it is evident that P-DPO with individualized preference generated significantly longer responses for majority workers, while only generated the end-of-text token for minority workers, showing that it has fully captured the preference for short responses of the minority workers. Similar to the trend of Accuracy-generic, the lengths of responses P-DPO generated for new users using the generic user embeddings (the orange bar) are similar to those of vanilla DPO (the blue bar). The generations for two example prompts are provided in Appendix F.

### 6.2 Instruction Following under Different Preference Profiles

The Personalized-Soups (P-SOUPS) dataset [] includes pairwise feedback for responses to instructions in GPT-4 Alpaca []. The responses were sampled from Tulu-7B [] and the comparisons were annotated by GPT-4 using preference prompts on three dimensions including expertise, informativeness and style (denoted by P1, P2 and P3). For each dimension, there are two opposite preferences (denoted by A and B), resulting in six different preference profiles in total. For example, the preference prompts for P1A and P1B are ”Generate/Choose a response that can be easily understood by an elementary school student” and ”Generate/Choose a response that only a PhD Student in that specific field could understand”, representing two completely different preferences for the expertise level of the responses. In our experiments, we treat each individual preference profile as a unique user, i.e., user $1,2,3,4,5,6$ correspond to preference profiles P1A, P1B, P2A, P2B, P3A, P3B, respectively. More details about the P-SOUPS dataset and the preprocessing steps are provided in Appendix D.

Drawing from the strong performance of P-DPO in Section 6.1, we focused our experiment on P-DPO with individualized preference for P-SOUPS, with $\alpha=0.5$ and $T_{u}=10$. The performance of P-DPO is compared with three baseline methods:
(1) Tulu-7B SFT prompted with instructions without preference prompt,
(2) Tulu-7B fine-tuned via vanilla DPO using pairwise feedback without preference prompt in the input, and
(3) Tulu-7B prompted with instructions with preference prompt.
Note that (3) is a very strong baseline as it directly provides the LLM with the ground-truth preferences, which is often not available in real-world personalization scenarios.
For the training and evaluation of P-DPO, only instructions were provided to the LLM without the preference prompts. One advantage of P-DPO is that it can infer the individual user preferences from the feedback data and encode the preferences into the user representations, thus requiring no effort from the users to articulate their preferences, which can be challenging or even impossible when the preferences are subjective and nuanced. Following  [], we evaluate the performance by the pairwise win-rate between P-DPO and the baseline methods on the generations for $50$ instructions from the Koala evaluation [], using the same AlpacaFarm-based framework(^5^55[https://github.com/joeljang/RLPHF](https://github.com/joeljang/RLPHF)) [] for simulated evaluation by GPT-4.

**Table 1: The win-rates ($\%$) of P-DPO against three baseline methods, evaluated by GPT-4. “Pref” stands for “Preference Prompt”. The win-rates for each user is evaluated using their ground-truth preference prompt, while P-DPO does not have access to such preference prompts during training. For each baseline, the mean and standard error across all $6$ users are provided in the last column.**
| Baseline Method | User 1 | User 2 | User 3 | User 4 | User 5 | User 6 | Mean $\pm$ SE |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Tulu SFT w/o Pref | $91.67$ | $86.36$ | $100.00$ | $59.57$ | $96.00$ | $100.00$ | $88.93\pm 5.70$ |
| Tulu vanilla DPO | $95.92$ | $86.67$ | $100.00$ | $63.04$ | $100.00$ | $100.00$ | $90.94\pm 5.45$ |
| Tulu SFT w/ Pref | $73.47$ | $74.42$ | $90.48$ | $48.00$ | $59.09$ | $76.00$ | $70.24\pm 5.50$ |

The win-rates for each individual user are shown in Table 1. For baselines (1) and (2), the same generation was used for every user. While having no access to the preference prompts, P-DPO outperformed Tulu-7B SFT and the DPO fine-tuned Tulu-7B (baselines (1) and (2)) by having around $90\%$ win-rates on average, and for some preference profiles (e.g. user 3 and 6, prefer conciseness and unfriendly responses), the win-rates are $100\%$. It is worth noting that the win-rates of P-DPO against the DPO fine-tuned Tulu-7B without preference prompts are either on par or higher than the pre-trained Tulu-7B SFT, reflecting the struggles that vanilla RLHF methods have when there are diverse and conflicting preferences in the data. For baseline (3) with access to the ground-truth preference prompts, P-DPO achieved above $59\%$ win-rates on $5$ users out of $6$, and $70.24\%$ win-rate on average, further demonstrating the effectiveness of P-DPO in aligning with individual user preferences\lledit, as P-DPO requires only user indices while baseline (3) utilizes the ground truth preference prompt in generating personalized responses.

### 6.3 Ablation Study

To study the effect of P-DPO hyper-parameters ($T_{u}$, $\alpha$ and $K$ in cluster-based preference) and our design choice for individualized preference, we conducted an ablation study using the TL;DR dataset with the top $10$ workers on four additional configurations (1) individualized preference with $T_{u}=1$ and $\alpha=0.5$, (2) individualized preference with $T_{u}=10$ and $\alpha=1.0$, (3) individualized preference with $f_{\text{P}}(u)=o_{u}$ instead of $f_{\text{P}}(u)=e_{0}+o_{u}$, i.e., the generic user embeddings are not included in the individual user embeddings, with $T_{u}=10$ and $\alpha=0.5$, and (4) cluster-based preference with $K=2$, $T_{u}=10$, and $\alpha=0.5$.

The accuracies of the four additional configurations are shown in Figure 2 (b), compared with the vanilla DPO and the two P-DPO configurations presented in Section 6.1. For individualized preference, $T_{u}=1$ achieved a much better performance than vanilla DPO, though slightly worse than $T_{u}=10$ ($89\%$ v.s. $91\%$) when $\alpha$ is fixed. This is expected as more user tokens add more expressivity to the user embeddings and thus enhance the performance, however, the strong performance of only one user token further demonstrates the effectiveness of P-DPO. With $T_{u}$ fixed to $10$, $\alpha=1.0$ achieved slightly higher accuracy than $\alpha=0.5$ on seen users. However, we observed a wild fluctuation on Accuracy-generic for $\alpha=1.0$ compared to $\alpha=0.5$ as shown in Figure 3 (c), showing the necessity of the user-agnostic loss in learning a stable generic user representation which will then be applied for new users.
As in Figure 3 (b), the accuracy of P-DPO with individualized preference without $e_{0}$ did not grow as fast as its counterpart with $e_{0}$, showing the utility of the common preference component $e_{0}$ in facilitating the learning of individual preferences.
For cluster-based preference, $2$ clusters performed significantly worse than $5$ clusters, albeit still better than vanilla DPO, and the accuracy of cluster $K=2$ model also increased much more slowly than other P-DPO models (Figure 3 (b)). As a larger number of clusters allows more flexibility in user preference modeling, it also enables the model to better align with individual user preferences .

In personalization scenarios, the number of users often exceeds $10$. We experimented with the same two P-DPO configurations in Section 6.1 with the top $40$ workers. As shown in Figure 2 (c), P-DPO was still able to perform as competitively as in the $10$ workers setting on all the accuracy metrics.

## 7 Conclusions

To build personalized LMs, we propose P-RLHF—a personalized RLHF framework for working with preference data that contain user identifiers.
Our framework jointly learns a user model and a LM.
We propose (1) different designs of user models to capture structural preference assumptions; and
(2) new learning objectives for personalized personalized language modeling (P-DPO).
Empirically, our methods have effectively learned personalized LMs that generate responses better aligned with individual user preferences.
For future directions, it is important to study privacy related issues in building personalized language models.

## Appendix A Additional Related Work

#### Crowdsourcing

When collecting large sets of labeled data (like in the preference data collection phase of RLHF),
crowdsourcing is often adopted by first dispatching the unlabeled samples to multiple annotators and then estimating the ground-truth labels by aggregating the noisy annotations [, ].
The observed annotations are often modeled as the confused outputs for the hidden ground-truth labels and the confusion of each annotator is characterized by an individual confusion matrix [, , ].
Recent research has introduced novel methods to better capture real-world annotator behaviors.
For instance, Imamura et al. [] modeled the confusion matrices at a cluster level to capture the shared confusion patterns among annotators.
Inspired by the behavioral assumptions (on annotators) in crowdsourcing literature,
we design analogous strategies to model user preferences at the population, cluster, and individual levels through different user model structures.

#### Conditional Natural Language Generation

With the advent of autoregressive pre-trained LMs such as GPT-3 [] and PaLM [], natural language generation tasks are often performed via prompting or in-context learning approaches [, , , ]. To personalize language generations without re-training the LM, prompts with relevant historical data are used to align the LM outputs with user intents [] or opinions []. The methods most closely related to our work include prefix-tuning [] and soft-prompt learning [], which prepend task-specific continuous embeddings to the transformer layers or the embedded inputs to adapt the pre-trained LMs to specific downstream tasks. While the previous approaches learn task-specific embeddings from datasets with reference outputs, our approach instead focuses on the personalization setting by learning user-specific representations from preference datasets (instead of traditional text generation or labeling datasets).

## Appendix B Proofs in Section 3.1

See 3.2

###### Proof.

For all $i\in[n]$, denote $s_{i}=r_{\text{vanilla}}(x_{i},y_{i,1})-r_{\text{vanilla}}(x_{i},y_{i,2})$.
The first-order condition for (1) with respect to $s_{i}$ is given by:

$$ $\displaystyle\mathbb{I}\{j\in\mathcal{C}_{j}:y_{1,j}\succ y_{2,j}\}-\sum_{j\in \mathcal{C}_{j}:y_{1,j}\succ y_{2,j}}\sigma(s_{j})-\sum_{j\in\mathcal{C}_{j}:y _{2,j}\succ y_{1,j}}\sigma(s_{j})=0.$ $$

Re-arranging the terms gives the result.
∎

See 3.3

###### Proof.

We start with the decomposition:

$$ $\displaystyle\mathbb{P}(y_{1}\succ y_{2}|x)=\sum_{j\in[m]}\mathbb{P}(u_{j}) \mathbb{P}(y_{1}\succ y_{2}|x,u_{j}).$ $$

Using this decomposition, the deviance between the group-wise preference and the marginalized preference is given by

$$ $\displaystyle|\mathbb{P}(y_{1}\succ y_{2}|x)-\mathbb{P}(y_{1}\succ y_{2}|x,u_{ 1})|=|(1-\mathbb{P}(u_{1}))(\mathbb{P}(y_{1}\succ y_{2}|x,u_{2})-\mathbb{P}(y_ {1}\succ y_{2}|x,u_{1}))|.$ $$

Similarly, we obtain that

$$ $\displaystyle|\mathbb{P}(y_{1}\succ y_{2}|x)-\mathbb{P}(y_{1}\succ y_{2}|x,u_{ 2})|=|\mathbb{P}(u_{1})(\mathbb{P}(y_{1}\succ y_{2}|x,u_{1})-\mathbb{P}(y_{1} \succ y_{2}|x,u_{2}))|.$ $$

Let $\mathbb{P}(u_{1})=\mathbb{P}(u_{\text{majority}})$ and $\mathbb{P}(u_{2})=\mathbb{P}(u_{\text{minority}})$.
Since $\mathbb{P}(u_{1})\geq\mathbb{P}(u_{2})$, we obtain the result.
∎

###### Proof.

###### Proof.

## Appendix C Generation with Synthetic Preferences Experiment Details

### C.1 Reddit TL;DR summarization dataset

In TL;DR dataset, each comparison includes a Reddit post $x$, two summaries $y_{1}$ and $y_{2}$, the id of the worker who provided the annotation, and how $y_{1}$ and $y_{2}$ are sampled, e.g., from prior SFT or PPO checkpoints. As we do not have access to the SFT model used by Stiennon et al. [], we initialize the personalized LM in P-DPO using an open-source SFT(^6^66[https://huggingface.co/CarperAI/openai_summarize_tldr_sft](https://huggingface.co/CarperAI/openai_summarize_tldr_sft)). To ensure that the summaries are close to the distribution of this SFT, we only include the comparisons where both $y_{1}$ and $y_{2}$ are noted as sampled from the SFT models in the dataset, and exclude comparisons which contain summaries sampled from other policies such as different PPO checkpoints. In Sections 6.1 and 6.3, we used the comparisons annotated by the the top $10$ and top $40$ workers for preference simulation and P-DPO training. The statistics of the dataset are listed in Table 2.

**Table 2: Statistics of the TL;DR dataset. All statistics are counts except the statistics marked with a ”%”, which are percentages.**
| Statistics | Top $10$ Workers | Top $40$ Workers |
| --- | --- | --- |
| Majority workers | $7$ | $26$ |
| Minority workers | $3$ | $14$ |
| Train Comparisons | $23,299$ | $38,065$ |
| Train Comparisons from majority workers | $16,607$ | $25,821$ |
| Train Comparisons from majority workers % | $71.28\%$ | $67.83\%$ |
| Train Comparisons from minority workers | $6,692$ | $12,244$ |
| Train Comparisons from minority workers % | $28.72\%$ | $32.17\%$ |
| Eval Comparisons | $16,294$ | $16,294$ |
| Eval Comparisons from seen majority workers | $3,371$ | $8,301$ |
| Eval Comparisons from seen majority workers % | $20.69\%$ | $50.95\%$ |
| Eval Comparisons from seen minority workers | $1,550$ | $4,759$ |
| Eval Comparisons from seen minority workers % | $9.51\%$ | $29.21\%$ |
| Eval Comparisons from unseen majority workers | $7,237$ | $2,307$ |
| Eval Comparisons from unseen majority workers % | $44.42\%$ | $14.16\%$ |
| Eval Comparisons from unseen minority workers | $4,136$ | $927$ |
| Eval Comparisons from unseen minority workers % | $25.38\%$ | $5.69\%$ |

### C.2 P-DPO Experiment Details

All the LMs in P-DPO experiments are initialized to the open-source, GPT-6B based SFT(^7^77[https://huggingface.co/CarperAI/openai_summarize_tldr_sft](https://huggingface.co/CarperAI/openai_summarize_tldr_sft)). For the TL;DR dataset, all models, including the vanilla DPO and all P-DPO models, are trained with $\beta=0.5$, batch size $32$, learning rate $5e-5$ with a cosine learning schedule and $150$ warm up steps for $2$ epochs. We utilized LoRA [] for training, with LoRA $\alpha=16$, LoRA $r=8$ and LoRA dropout $0.05$. All models are trained with a PyTorch based, personalized DPO Trainer we develop by extending the DPO Trainer in the TRL library []. All of our experiments are run using 80G A100s or H100s.

### C.3 Additional Experiment Results

The numeric results for the accuracy metrics shown in Figure 2 are provided in Tables 3, 4, and 5.

**Table 3: The accuracy metrics of the vanilla DPO and the two P-DPO configurations in Sec 6.1, as shown in Figure 2 (a). All accuracies are in $\%$.**
| Model | Accuracy-top | Accuracy-generic | Accuracy-average<br>Majority | Accuracy-average<br>Minority |
| --- | --- | --- | --- | --- |
| Vanilla DPO | $58.91$ | $55.37$ | $74.82\pm 1.22$ | $25.10\pm 1.09$ |
| P-DPO Individual | $91.04$ | $55.34$ | $89.26\pm 0.57$ | $94.35\pm 0.28$ |
| P-DPO Cluster K=5 | $91.12$ | $54.55$ | $89.24\pm 0.74$ | $94.78\pm 0.18$ |

**Table 4: The accuracy metrics of the P-DPO configurations for top $10$ workers in the ablation study in Sec 6.3, as shown in Figure 2 (b). All accuracies are in $\%$.**
| Model | Accuracy-top | Accuracy-generic | Accuracy-average<br>Majority | Accuracy-average<br>Minority |
| --- | --- | --- | --- | --- |
| Individual $T_{u}=1$ | $88.78$ | $54.92$ | $85.92\pm 0.57$ | $94.15\pm 0.11$ |
| Individual $\alpha=1.0$ | $93.54$ | $54.87$ | $92.37\pm 0.51$ | $95.23\pm 0.08$ |
| Individual w/o $e_{0}$ | $88.88$ | $54.77$ | $87.13\pm 0.97$ | $91.96\pm 0.65$ |
| Cluster $K=2$ | $72.79$ | $55.01$ | $82.32\pm 2.02$ | $51.24\pm 9.30$ |

**Table 5: The accuracy metrics of the vanilla DPO and the same two P-DPO configurations described in Sec 6.1 for top $40$ workers, as shown in Figure 2 (c). All accuracies are in $\%$.**
| Model | Accuracy-top | Accuracy-generic | Accuracy-average<br>Majority | Accuracy-average<br>Minority |
| --- | --- | --- | --- | --- |
| Vanilla DPO | $54.91$ | $57.58$ | $67.96\pm 0.92$ | $30.61\pm 0.98$ |
| P-DPO Individual | $92.97$ | $57.85$ | $91.94\pm 0.50$ | $95.14\pm 0.40$ |
| P-DPO Cluster K=5 | $91.74$ | $56.77$ | $90.27\pm 0.56$ | $94.44\pm 0.69$ |

## Appendix D Instruction Following under Different Preference Profiles Experiment Details

### D.1 Personalized-Soups Dataset

The Personalized-Soups (P-SOUPS) dataset [] includes pairwise comparisons for responses to GPT-4 Alpaca instructions []. These responses, sampled from Tulu-7B [], were then annotated by GPT-4 across three distinct preference dimensions: expertise, informativeness, and style (referred to as P1, P2, and P3 respectively). Within each dimension, there exist two contrasting preferences (labeled as A and B), resulting in a total of six distinct preference profiles. We directly used the dataset provided in the Personalized-Soups github repository(^8^88[https://github.com/joeljang/RLPHF](https://github.com/joeljang/RLPHF)) and removed the duplicate comparisons for each preference profile. The preference prompts and the number of comparisons for each preference profile are shown in Table 6. In our experiments, we did a random split of $90\%/10\%$ for training and validation, and the validation set was used to monitor the same accuracy metrics as defined in Section 6.1

**Table 6: The preference prompts and the number of comparisons for each preference profile. The user ids are the user ids used in P-DPO experiments.**
| User<br>Id | Preference<br>Profile | Dimension | Preference Prompt | Number of<br>Comparisons |
| --- | --- | --- | --- | --- |
| 1 | P1A | Expertise | Generate/Choose a response that can be<br>easily understood by an elementary school student. | $8,959$ |
| 2 | P1B | Expertise | Generate/Choose a response that only<br>a PhD Student in that specific field could understand. | $9,069$ |
| 3 | P2A | Informativeness | Generate/Choose a response that is concise and<br>to the point, without being verbose. | $8,239$ |
| 4 | P2B | Informativeness | Generate/Choose a response that is very informative,<br>without missing any background information. | $8,626$ |
| 5 | P3A | Style | Generate/Choose a response that is friendly,<br>witty, funny, and humorous, like a close friend. | $9,356$ |
| 6 | P3B | Style | Generate/Choose a response (that answers)<br>in an unfriendly manner. | $9,222$ |

### D.2 P-DPO Experiment Details

All the LMs in P-DPO experiments are initialized to the Tulu-7B [] SFT. For the P-SOUPS dataset, all models, including the vanilla DPO and all P-DPO models, are trained with $\beta=0.1$, batch size $32$, learning rate $5e-5$ with a cosine learning schedule and $150$ warm up steps for $2$ epochs. We utilized LoRA [] for training, with LoRA $\alpha=32$, LoRA $r=8$ and LoRA dropout $0.1$. All models are trained with a PyTorch based, personalized DPO Trainer we develop by extending the DPO Trainer in the TRL library []. For the win-rate evaluation, we directly leveraged the evaluation code in the github repository of [].
All of our experiments are run using 80G A100s or H100s.

## Appendix E Additional Details for Section 5

### E.1 Graphical models for individualized and cluster-based preference assumptions

Figure: (a) Individualized.

### E.2 Personalized RM for Personalized LM

Given the *learnable* user model $f_{\text{P}}$,
we have a user embedding $e_{u}$ for each user $u\in\mathcal{U}$.
Our next task is to decide how we want to include it into the personalized RM $r_{\text{p}}(x,y,u)$.
We discuss two approaches: (1) use $e_{u}$ as a soft prompt;
or (2) when $e_{u}$ is a vector, use $e_{u}$ as a linear head.
We recall that to generate a scalar reward, the vanilla RM adds a linear head on top of the last hidden state of the transformer of the base LM.

In the case of soft prompting, the aggregator prepends $e_{u}$ to the input (text not positional) embedding $e_{x,y}\in\mathbb{R}^{T_{x,y}\times d}$ given by the base LM, where $T_{x,y}$ is the token length and $d$ is the token-wise embedding dimensionality.
The user embedding $e_{u}\in\mathbb{R}^{T_{u}\times d}$ is a tensor with $T_{u}$ being its corresponding user token length.
One factor that controls the expressivity of user embeddings is the size of their corresponding user token length $T_{u}$.
The rest of $r_{\text{P}}$ is similar to that of the vanilla one, i.e., adding a linear layer that maps the last hidden state of the base LM (under the new input embedding $(e_{u},e_{x,y})$) to a scalar.

In the case where $e_{u}$ is a linear head,
the aggregator function can be taken as an inner product between $e_{u}$ and the hidden state $e_{x,y}$ of the last transformer layer of the base LM, thus outputting a scalar reward value.
Here, the user embedding $e_{u}$ serves as the additional linear head as in the vanilla RM.

We utilize the user model $f_{\text{P}}$ and the user embedding aggregation mechanism to fully specify the parameterized personalized RM $r_{\text{P}}$.
To learn the RM (including the user model $f_{\text{P}}$), we use the following objective:

$$ $\displaystyle\min_{r_{\text{P}}}-\mathbb{E}_{x,y_{1},y_{2},u\sim\mathcal{D}_{ \text{P}}}\bigg{[}\alpha\log\sigma(r_{\text{P}}(x,y_{1},u)-r_{\text{P}}(x,y_{2 },u))+(1-\alpha)\log\sigma(r_{\text{P}}(x,y_{1},u_{0})-r_{\text{P}}(x,y_{2},u_ {0}))\bigg{]},$ $$

where $\alpha\in[0,1]$. Recall that $u_{0}$ indicates empty user information.
The loss can be viewed as a combination of a user-specific loss term that relies on explicit user identifier $u$ and a user-agnostic loss term that depends on $u_{0}$. The user-agnostic loss uses the same preference data but without any user identifier.
The hyper-parameter $\alpha$ is used to balance between the two loss components.

###### Remark E.1 .

We note that when $\alpha=0$ and $f_{\text{P}}$ is the uniform preference-based user model (Example 1), we can reduce P-RM to vanilla reward modeling by either (1) take the user embedding as a soft prompt and set $f_{\text{P}}$ to output an empty tensor; or (2) take the user embedding as a linear head and set $f_{\text{P}}$ to output a vector.

Given the personalized RM, one can adopt multiple strategies to generate personalized texts:
(1) Best-of-$N$: given an appropriate fine-tuned LM (either $\pi^{\text{SFT}}$ or an LM learned under the original RLHF pipeline), we can rank the $N$ sampled text using the personalized RM, ensuring the selected text is more attuned to the individual user’s preference;
(2) policy optimization: one can also directly optimize the LM policy with respect to the personalized RM.

###### Remark E.1 .

## Appendix F Generation Examples

The generations by the SFT, vanilla DPO, and P-DPO with individualized preference for one example prompt randomly chosen from the evaluation set of the TL;DR dataset are shown in Table LABEL:tab:tldr_generation_example_one.
We note that since in the synthetic preference dataset, the user preference is regarding the length of the response instead of the summarization itself,
the *optimal* behavior of the model should be based on the length of the generated summary instead of the summary quality itself.

We can see that for the minority users $4,5,6$ who prefer shorter summaries, P-DPO generated an empty response. Using the generic user embeddings, P-DPO generated responses with lengths similar to those of the generations by the vanilla DPO. For the majority users, P-DPO generated much longer responses. As the user preferences are only based on the response lengths, the generated responses are not guaranteed to be coherent linguistically (there may be duplicates in the responses).

**Table 7: The generations by the SFT, vanilla DPO, and P-DPO with individualized preference for an example prompt from the TL;DR dataset.**
| Prompt |
| --- |
| SUBREDDIT: r/AskReddit |
| TITLE: How to Pass my Geometry Final |
| POST: Hey guy’s I’m in a continuation school because I’m far behind on high school credits due to poor choices. Anyhow I have a final that will be the determining factor on if I pass the class or not, so this is really important to me. I asked the teacher for a study guide or some practice questions and she flat out refuses to give me anything to go off of. In the past when asking her for help she always directs me to another teacher instead of helping me herself. What can I do online that will help me study a general understanding of geometry and let me know what I know and what I need to work on for tomorrow’s final? |
| TL;DR: |
| Generation by SFT |
| I need to study for a geometry final and I need to know what I know and what I need to work on. |
| Generation by Vanilla DPO |
| I need to pass my geometry final tomorrow and I need to know what I know and what I need to study for the final. |
| Generation by P-DPO Using Generic User Embeddings |
| I need to pass a final tomorrow and I’m not sure what I need to study for it. I need to know what I know and what I need to work on. |
| Generation by P-DPO for User 1 |
| I need to pass a geometry final tomorrow and the teacher won’t give me any study guides or anything to go off of. What can I do online to help me study? I’m a high school junior. |
| Generation by P-DPO for User 2 |
| I need to pass a geometry final tomorrow and the teacher won’t give me any study guides or anything to go off of. What can I do online to help me study? (I’m on a school computer) |
| Generation by P-DPO for User 3 |
| I need to pass a geometry final tomorrow and the teacher won’t give me any study guides or anything to go off of. What can I do online to help me study? I’m a high school junior. |
| Generation by P-DPO for User 4 |
|  |
| Generation by P-DPO for User 5 |
|  |
| Generation by P-DPO for User 6 |
|  |
| Generation by P-DPO for User 7 |
| Teacher refuses to help me study for a final that will determine if I pass the class or not. What can I do online to help me study? (I’m a high school junior) |
| Generation by P-DPO for User 8 |
| I need to pass a final tomorrow and the teacher won’t give me any help online, what can I do to study for it? I’m a high school junior and I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a sophomore. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a junior. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a senior. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a sophomore. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a freshman. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a freshman. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a freshman. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a freshman. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a freshman. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a freshman. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a freshman. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a freshman. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a freshman. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a freshman. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a freshman. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a freshman. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a freshman. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a freshman. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a freshman. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a freshman. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a freshman. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a freshman. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a freshman. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a freshman. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a freshman. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a freshman. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a freshman. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a freshman. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a freshman. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a freshman. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a freshman. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a freshman. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a freshman. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a freshman. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a freshman. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a freshman. I’m taking geometry. I’m in a continuation school. I’m far behind. I’m a freshman. |
| Generation by P-DPO for User 9 |
| I need to pass a geometry final tomorrow and the teacher won’t give me any study guides or anything to go off of. What can I do online to help me study? I’m a high school junior. |
| Generation by P-DPO for User 10 |
| I need to pass a final that will determine if I pass the class or not, but the teacher refuses to give me any help. What can I do online to help me study? (I’m on a school computer) |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: * Does this work have released code and dataset?
* There are works have been done on personalized user preferences using RLHF. How does this one differ from them, and is there a comparison used as a baseline?
soundness: 3
strengths: * The structure of this article is well-organized and clearly articulated, making it easy for readers to follow the flow of ideas and concepts presented throughout the text. Each section is logically arranged, allowing for a seamless understanding of the material.

* The section on RLHF is detailed, with a clear progression from general RLHF to the specifically designed personalized DPO. This work considers different user preferences from multiple angles and granularities. It clearly explains the differences between P-RLHF and Vanilla DPO to clarify the motivation.

* The work uses the TL;DR dataset, which has Reddit posts, summaries, and worker IDs. It conducts an experiment where most workers (70%) prefer longer responses and a smaller group (30%) prefers shorter ones. The top 10 workers with the most annotations were included for training. After the experiment, some workers liked shorter responses while others liked longer ones. Overall, this shows a strong effort and good understanding of data analysis, which is impressive.

* There is a solid exploration of how personalized LLMs can adapt to different user preferences through P-DPO fine-tuning. The study found that the personalized model can generate longer or shorter responses based on users' implicit preferences, demonstrating its flexibility and adaptability. The experiment also compared the performance of P-DPO using generic user embeddings with traditional vanilla DPO, showing that the personalized model excels in understanding and responding to user needs, validating its effectiveness in practical applications.
summary: The paper introduces Personalized-RLHF (P-RLHF), an innovative framework designed to enhance personalized content generation in large language models (LLMs) by leveraging a lightweight user model. Its key contributions include:
Scalability: P-RLHF allows LLMs to efficiently generate personalized content while accommodating a growing user base.
Preference Handling: The framework effectively processes both explicit user preferences, provided as textual input, and implicit preferences derived from feedback data.
Reduced User Burden: It alleviates the requirement for users to fully articulate their preferences, addressing the challenges of obtaining detailed user input in practical applications.

Overall, this paper is easy to follow, but there are issues such as a lack of innovation, no released code, and an excess of symbols. However, overall, this is a ready piece of work.
weaknesses: * It seems that this article does not have released code. I'm not sure if I just couldn't find it, but without released code, reproducibility cannot be guaranteed.

* Like other RL works, the notation in this paper is too numerous and complex. Although the structure and presentation of the article are good, it somewhat hinders the readers' understanding. Perhaps a table to organize the notations could be helpful.

* There are concerns regarding the effectiveness of not training an additional reward model and relying solely on the user model.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Why include the user-identifier-agnostic loss term in P-DPO?
soundness: 2
strengths: 1. The issues related to DPO in personalized modeling are thoroughly discussed.
2. The presentation is easy to follow and well-structured.
3. In both synthetic and human evaluations, P-DPO significantly outperforms DPO.
summary: This paper addresses the important challenge of generalizing large language models (LLMs) to capture personalized user preferences. To achieve this, the authors develop several models that incorporate users' implicit preferences, along with a new learning objective aimed at optimizing the reference LLMs for personalization. They compare the proposed method, P-DPO, with the standard DPO in both synthetic and human evaluation settings. In both cases, P-DPO shows advantages. The paper is well-written, with a clearly motivated problem statement. The discussion on the limitations of existing RLHF methods is thorough and insightful.

However, the reviewer has the following concerns about this work.
1. The contribution appears limited. The primary innovation, the new learning objective P-DPO introduced in Section 4.4, facilitates personalization by incorporating user embeddings into the implicit reward model of DPO. However, this approach is not particularly novel, as the integration of user embeddings for personalization is extensively discussed in the recommendation literature [1].
2. The evaluation lacks comprehensiveness. The authors only consider DPO as a baseline and do not include recent work on personalized LLMs, such as [2] and [3]. Additional relevant studies can be found at https://github.com/HqWu-HITCS/Awesome-Personalized-LLM. Furthermore, the experiments overlook the recommendation task, which is a crucial application for personalized LLMs. In the human evaluation, the sample size is limited to at most 20 users and 76 samples, which may lead to biased results and unreliable conclusions.
3. The advantages of integrating both users' explicit and implicit embeddings are not well supported. The authors are suggested to include an ablation study to validate the importance of the learned implicit user embeddings.
4. The motivation for including the user-identifier-agnostic loss term in P-DPO is not clear to the reviewer. The advantage of this design is also not evident.

[1] SSE-PT: Sequential Recommendation Via Personalized Transformer
[2] Factual and Tailored Recommendation Endorsements using Language Models and Reinforcement Learning
[3] Personalized Large Language Models
weaknesses: For a detailed discussion of the reviewers' concerns, please refer to the summary.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. An explanation is requested for the occurrence of zero-length responses presented in Section 5.2 and Figure 3. Logically, even for users who prefer concise replies, it is expected that responses maintain a reasonable lower bound on length rather than resulting in either no content or excessively long responses. The current approach appears overly polarized and does not account for practical application scenarios where balanced response lengths are desired.
2. It is recommended that Figure 1, which outlines the proposed module design, be incorporated directly into the main body of the text. This integration would enhance the comprehensibility of the framework by ensuring that the figure is closely tied to the corresponding narrative, thereby making the overall framework easier to understand for readers.
3. I suggest the author add more nuanced analyses, rather than a strictly binary approach to the generated content, which are necessary to substantiate the effectiveness of the approach.
soundness: 2
strengths: 1. The paper presents the P-RLHF framework, which integrates both explicit and implicit user models to facilitate personalized learning from user feedback, thereby introducing a novel approach to the research on personalized LLMs. The introduction of P-DPO represents an innovative optimization method that effectively balances personalization and generalization for both known and unknown users. This approach demonstrates excellent scalability while addressing the personalized needs of multiple users, marking a significant advancement over traditional RLHF methods.
2. The overall structure of the paper is exceptionally clear, methodically introducing the background, identifying the problem, proposing solutions, and validating the experiments related to personalized RLHF. The logical progression facilitates easy comprehension for readers. Each experimental section is meticulously designed with clearly defined objectives, enabling readers to effortlessly track the innovative aspects of the methodology and their corresponding outcomes.
3. The response generation results provided in the paper offer a clear and intuitive demonstration of how different user preferences are handled in a personalized manner. The personalized LLMs showcased in the study hold significant practical value, and the paper successfully achieves personalized content generation on the utilized datasets. This effectively highlights the framework’s capability to cater to diverse user preferences in real-world applications.
summary: This paper introduces Personalized Reinforcement Learning from Human Feedback (P-RLHF), a novel framework designed to enhance the personalization capabilities of large language models (LLMs) by integrating both explicit and implicit user models. By leveraging user-specific information and feedback, P-RLHF effectively captures diverse and individualized user preferences, addressing the limitations of traditional RLHF methods that assume uniform user preferences.
The authors validate the efficacy of the P-RLHF framework through a series of comprehensive experiments, ranging from controlled synthetic settings to large-scale real-world datasets. These experiments demonstrate that P-RLHF not only successfully captures and adheres to individual user preferences but also maintains high performance and scalability in diverse application scenarios. The clear and logical presentation of the framework, coupled with intuitive response generation results, underscores the practical value and potential impact of personalized LLMs in real-world applications.
Overall, this paper makes substantial contributions to the field of personalized LLM by presenting a well-structured framework and a novel optimization method that together advances the state-of-the-art in personalized language model training.
weaknesses: 1. Zero-Length Responses in Experiment One: The occurrence of zero-length responses, which the authors justify as an expected outcome via mathematical proofs, raises questions about whether an LLM should indeed produce responses with no content. This result appears to stem from a polarized experimental design and methodology, which may not reflect practical application scenarios or user expectations. A more nuanced experimental setup could better balance realistic use cases with the need for distinct preference modeling.
2. Figure 1 Lacks Identification of Key Modules: While Figure 1 presents the overall framework, it does not clearly indicate critical components introduced by the authors, such as P-DPO. And there is no explanation of Figure 1 in the main text. This omission complicates the interpretation of the framework diagram, hindering the reader’s ability to intuitively grasp the structure and contributions of the proposed approach.
3. Insufficient Ablation Study: The ablation experiments, which appear only in the appendix, focus solely on parameter variations rather than examining which specific design choices contribute to the improved performance over baseline models. For example, it is unclear whether the personalized capabilities remain robust when only the implicit model is used without the explicit component. The paper lacks a thorough investigation into such aspects, leaving questions about the impact of each module unanswered.
4. Binary Nature of Evaluation Metrics: Both the preference modeling and experimental evaluation adopt a highly binary approach. In modeling preferences, options are presented as direct opposites (e.g., long vs. short responses), which likely contributes to the appearance of zero-length responses. Furthermore, evaluation is limited to win-rates relative to baseline models, which does not provide a clear view of the actual performance gains. Since the goal of personalization is to improve user satisfaction, more nuanced analyses, rather than a strictly binary approach to the generated content, are necessary to substantiate the effectiveness of the approach.
5. Lack of Consideration for Contextual Shifts in User Preferences: User preferences can be context-dependent, particularly when involving multidimensional combinations of preferences. In specific scenarios, user preferences may shift according to immediate needs or context. The reliance on a static dataset overlooks these potential dynamics, and the paper does not address how such variations might impact model performance.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. How does the model ensure that implicit feedback does not conflict with explicit user preferences, and how does it resolve such conflicts?
2. Can the authors provide more detailed information on the computational efficiency of P-RLHF? How does it compare in terms of time and memory consumption to other RLHF approaches?
3. How does the framework handle dynamic user preferences that may change over time, and how quickly can it adapt?
soundness: 3
strengths: - The paper addresses an important area of research—personalizing LLMs based on user feedback.
- The proposal to use both explicit and implicit feedback for personalization is valuable and well-motivated.
- The experiments cover several datasets, which helps validate the general applicability of the approach.
summary: This paper introduces a framework called Personalized Reinforcement Learning from Human Feedback (P-RLHF) aimed at personalizing large language models (LLMs) based on individual user preferences. The authors propose a lightweight user model that jointly learns user preferences (both explicit and implicit) and the LLM. The key claims are that this framework scales efficiently with increasing users, handles diverse feedback, and reduces the need for explicit user prompts.
weaknesses: - The paper does not sufficiently address how implicit feedback is effectively used or how it handles contradictory user preferences.
- The scalability claims, while central to the paper, are not backed up by detailed computational analysis or resource cost comparisons with other approaches.
- There is limited discussion on how this approach performs in real-world dynamic environments where user preferences might shift over time.
- The experimental results, while positive, are not sufficiently compelling to clearly demonstrate the advantage of P-RLHF over simpler personalization methods.

Meta-review: The paper introduces Personalized-RLHF (P-RLHF), an new framework designed to enhance personalized content generation in large language models (LLMs) by leveraging a lightweight user model. The proposed method allows the LLMs to efficiently generate personalized content, and processes both the explicit and implicit user preferences. The authors have performed experiments on real dataset to demonstrate the effectiveness of the proposed method. 

Overall, this paper is clearly written, and the authors have introduced some new ideas. However, for this work, the experimental analysis is not sufficient. Firstly, the authors do not compare the proposed method with existing personalized LLM methods. Secondly, how the proposed model can handle the both the user implicit and explicit preference is not well studied. How explicit preference and implicit preference will impact the performance of the proposed model should also be well studied. Besides showing some metric numbers, the authors are also recommended to present some case studies to demonstrate the effectiveness of the proposed method.

Decision: Reject