Venue: NAACL 2024

# Fine-Tuning Language Models with Reward Learning on Policy

 Hao Lang  Fei Huang  Yongbin Li 1

Alibaba Group

{hao.lang, f.huang, shuide.lyb}@alibaba-inc.com

Footnote 1: Corresponding author.

###### Abstract

Reinforcement learning from human feedback (RLHF) has emerged as an effective approach to aligning large language models (LLMs) to human preferences. RLHF contains three steps, i.e., human preference collecting, reward learning, and policy optimization, which are usually performed serially. Despite its popularity, however, (fixed) reward models may suffer from inaccurate off-distribution, since policy optimization continuously shifts LLMs' data distribution. Repeatedly collecting new preference data from the latest LLMs may alleviate this issue, which unfortunately makes the resulting system more complicated and difficult to optimize. In this paper, we propose reward learning on policy (RLP), an unsupervised framework that refines a reward model using policy samples to keep it on-distribution. Specifically, an unsupervised multi-view learning method is introduced to learn robust representations of policy samples. Meanwhile, a synthetic preference generation approach is developed to simulate high-quality preference data with policy outputs. Extensive experiments on three benchmark datasets show that RLP consistently outperforms the state-of-the-art. Our code is available at [https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/rlp](https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/rlp).

## 1 Introduction

Large language models (LLMs) Brown et al. (2020); Bommasani et al. (2021) have shown great promise in following open-ended user instructions Askell et al. (2021); Ouyang et al. (2022); Longpre et al. (2023). These capabilities are largely attributed to the fine-tuning of pretrained LLMs using Reinforcement Learning from Human Feedback (RLHF) Christiano et al. (2017); Bai et al. (2022), which is a prominent technique to align LLMs with human preferences and greatly enhance their usability and safety OpenAI (2023); Anthropic (2023); Google (2023).

A typical RLHF procedure is comprised of three interrelated steps: human preference collecting, reward learning, and policy optimization (Figure 1 top). The reward learning step fits a reward model to the preference data that elicits evaluations from humans. The policy optimization step uses reinforcement learning (RL) to fine-tune a language model to produce outputs assigned high reward.

In practice, the three key steps of RLHF are often performed serially Casper et al. (2023). Since policy optimization shifts the language model's data distribution during the RL phase, the (fixed) reward model will be inaccurate off-distribution which is trained on offline data Touvron et al. (2023). Hence, reward model accuracy can quickly degrade and in turn degenerate the policy that exploits differences between the inferred and true reward Gao et al. (2023).

The above issue can be mitigated by gathering new human preference data from an up-to-date version of policy Ziegler et al. (2019). However, the resulting system is significantly more complicated

Figure 1: Comparison of standard RLHF (top) and RLHF with reward learning on policies (bottom). Different from (top), which performs reward learning and policy optimization serially, we iteratively train one of the two models with the help of the other.



and difficult to optimize, involving iterations of data gathering, reward learning, and RL fine-tuning. Moreover, significant work is required to maintain high data quality over a long time in this setting.

In this paper, we show how to optimize a reward model against the policy to keep it on-distribution, without repeatedly collecting new human preference data. We propose _Reward Learning on Policy (RLP)_, a framework that refines a reward model using policy samples in an unsupervised manner. RLP first trains a reward model and a language model policy from scratch with standard RLHF methods, and then retrains the reward model when exposed to the sample distribution of the trained policy. Finally, RLP retains the policy on the re-trained reward model, which attempts to maintain an accurate reward for the latest policy.

Concretely, RLP uses policy samples to retrain the reward model via two methods: unsupervised multi-view learning (UML) and synthetic preference generation (SPG). RLP-UML constructs two views for an input by generating two responses from the policy [23], then optimizes a multi-view information bottleneck loss [17] when fitting the reward model to a dataset of human preferences. This training objective follows the information bottleneck principle [40] and helps learn robust representations of the policy's data distribution.

In addition, RLP-SPG simulates preferences on policy generations to supplement the human preference data. Rather than producing and scoring two outputs with LLMs as in Reinforcement Learning from AI Feedback (RAIIF) [5, 9], RLP-SPG generates a set of outputs for an instruction. In this way, RLP-SPG can quantify uncertainty and decide when to trust model predictions via measuring the size of the largest semantic equivalence cluster [8, 6]. Thus, RLP-SPG selectively generates pairwise preferences for instructions with low uncertainty [10], where the preferred output is sampled from the largest cluster of the output set and the non-preferred one is sampled from the rest clusters. This sampling scheme also conforms to the self-consistency assumption, i.e., the most consistent output is selected as the final prediction [14, 15].

Our main contributions are as follows:

* We propose Reward Learning on Policy (RLP), an unsupervised framework that refines a reward model using policy samples to keep it on-distribution for RLHF.
* We optimize a multi-view loss when retraining the reward model to learn representations of the policy's data distribution. We also simulate preferences with a set of policy outputs, which enables selective generation and high-quality data construction.
* Our experiments on three standard benchmark datasets show that RLP outperforms existing methods for learning from human feedback, including PPO-based RLHF.

## 2 Related Work

**Instruction tuning** is a procedure to fine-tune pretrained LLMs with instructions and human-written completions [11, 12], which increases the usability of LLMs [13]. Recently, **RLHF** has emerged as the central method for fine-tuning LLMs based on human preferences and further improves their downstream task performance and alignment with user intent [15]. Generally, RLHF methods first fit a reward model to human preferences, then fine-tune a language model to maximize the inferred reward using RL algorithms.

**Reward models** tend to be an imperfect estimate of the true reward due to misspecification [16] and misgeneralization [21], and imperfect in reward models leads to reward hacking [17]. Methods with reward ensemble [18] and diverse feedback [23] are proposed to tackle this issue. Our method retrains the reward model with policy samples to make it on-distribution and generalize to the policy's data distribution.

**Human feedback simulation** aims to generate additional synthetic preference data using weak human supervision and LLMs [5]. RLAIF approaches obtain pairwise preferences by scoring two outputs from a shared prompt [9], whereas RLCD generates outputs from two variants of a prompt [14]. Our method RLP-SPG is the first attempt to simulate human preferences using a set of outputs.

**Uncertainty quantification** provides confidence scores for generations of LLMs, helping users decide when to trust these generation results [6]. Supervised methods fine-tune 

the language model to predict the uncertainty (Kadavath et al., 2022; Lin et al., 2022), while unsupervised methods measure uncertainty by calculating semantic entropy or semantic dispersion amongst generated answers (Kuhn et al., 2023; Lin et al., 2023). In this work, we measure uncertainty to selectively generate preference data.

## 3 Preliminaries

We start by introducing the instruction following task (Ouyang et al., 2022; Bai et al., 2022a). Given user instructions \(x\in{\mathcal{X}}\) (e.g., "Generate a definition for artificial intelligence"), we aim to develop a model \(\pi_{\theta}\) that generates high-quality responses \(y\sim\pi_{\theta}(y|x)\) as judged by some latent reward model. In this study, we focus on RLHF for this task, due to its central role in instruction-following LLMs (Ouyang et al., 2022). RLHF usually consists of three steps: human preference collecting, reward modeling, and RL policy optimization (Dubois et al., 2024; Rafailov et al., 2024; Casper et al., 2023).

Step 0, SFT:RLHF generally begins with a pre-trained model, which is fine-tuned with supervised learning on instruction-following demonstrations (\(x,y\)), to produce a model \(\pi^{\mathrm{SFT}}(y|x)\).

Step 1, Human preference collecting:The first step is to produce pairs of responses \((y_{1},y_{2})\sim\pi^{\mathrm{SFT}}(y|x)\) for the instruction \(x\). These are then presented to humans who express preferences for each response, denoted as \(y_{w}\succ y_{l}\mid x\) where \(y_{w}\) and \(y_{l}\) denotes the preferred and non-preferred completion amongst \((y_{1},y_{2})\) respectively.

Step 2, Reward learning:The second step is to fit a reward model \(r_{\phi}(x,y)\) by minimizing the negative log-likelihood loss (Christiano et al., 2017):

\[\small\mathcal{L}_{R}=-\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}}[\log \sigma(r_{\phi}(x,y_{w})-r_{\phi}(x,y_{l}))],\]

where \(\mathcal{D}=\{(x,y_{w},y_{l})\}\) is a dataset of pairwise preferences and \(\sigma\) is the sigmoid function. \(r_{\phi}(x,y)\) is often initialized from \(\pi^{\mathrm{SFT}}(y|x)\) with one additional linear layer that infers the reward value.

Step 3, RL policy optimization:The third step is to use the reward model \(r_{\phi}(x,y)\) to fine-tune the language model. The parameters \(\theta\) of \(\pi\) are trained to maximize

\[\small\mathbb{E}_{x\sim l\epsilon,y\sim\pi_{\theta}(y|x)}[r_{\phi}(x,y)- \beta\mathbb{D}_{\text{KL}}(\pi_{\theta}(y|x)||\pi_{\text{ref}}(y|x))],\]

where \(\mathcal{U}=\{x\}\) is an unlabeled instruction dataset, the language model policy \(\pi_{\theta}(y|x)\) is fine-tuned from the SFT model \(\pi^{\mathrm{SFT}}\) , the reference policy \(\pi_{\text{ref}}\) is also the SFT model \(\pi^{\mathrm{SFT}}\), and \(\beta\) is a regularization coefficient controlling the deviation from \(\pi_{\text{ref}}\). This objective is typically optimized with RL algorithms such as PPO (Schulman et al., 2017).

## 4 Reward Learning on Policy

### Overview

In this study, we propose a novel RLHF framework to fine-tune LLMs with human feedback following five steps: **Step 1-3.** Collect a pairwise human preference dataset \(\mathcal{D}\), then train a reward model \(r_{\phi}\) and fine-tune a language model policy \(\pi_{\theta}\); **Step 4.** Retrain a reward model \(\hat{r}_{\phi}\) using outputs of policy \(\pi_{\theta}\); **Step 5.** Retrain a policy \(\hat{\pi}_{\theta}\) based on the retrained reward model \(\hat{r}_{\phi}\).

Before applying RLP, we assume existing RLHF approaches can be used to train the reward model \(r_{\phi}\) and the policy \(\pi_{\theta}\)(Ouyang et al., 2022). The sample distribution of the policy \(\pi_{\theta}\) can be quite different from the preference data \(\mathcal{D}\) on which the reward model \(r_{\phi}\) is trained (Touvron et al., 2023b). For example, outputs become increasingly longer after applying RLHF methods as shown in the analysis of AlpacaFarm Dubois et al. (2024). The average length of SFT outputs is 278 characters and applying PPO increases it to 637 tokens. These distributional differences make the reward model \(r_{\phi}\) inaccurate off-distribution.

Our goal is to refine the reward model using samples of the policy \(\pi_{\theta}\) and keep it on-distribution. This process is expected to increase the generalization of the retrained reward model \(\hat{r}_{\phi}\) to policy samples. Accordingly, it can maintain an accurate reward during the RL policy optimization phase.

### Reward Retraining

We now describe how to retrain the reward model \(\hat{r}_{\phi}\) in **Step 4** of RLP. We first construct a dataset of policy samples \(\mathcal{P}=\{(x,\mathbf{y})\mid x\in\mathcal{U},\mathbf{y}\sim\pi_{\theta}(y|x)\}\), where \(\mathbf{y}\) is a set of \(n\) outputs from policy \(\pi_{\theta}\) for instruction \(x\). Then, we refine the reward model with policy samples \(\mathcal{P}\) in addition to the human preference dataset \(\mathcal{D}\). Specifically, we propose two different methods for this purpose: unsupervised multi-view learning (UML) and synthetic preference generation (SPG).

Unsupervised Multi-View Learningattempts to learn robust representations of policy samples. For each pair \((x,\mathbf{y})\in\mathcal{P}\), two semantic invariant 

views are constructed: \(v_{i}(x)=(x,y)~{}|~{}y\sim\mathbf{y}\), \((i=1,2)\). These two views preserve the same task-relevant information Zhao et al. (2017). Then, a multi-view information bottleneck (MIB) loss Federici et al. (2020) is optimised for unsupervised representation learning, following the information bottleneck principle Tishby et al. (2000). This optimization process retains task-relevant information in the representations while discarding superficial information.

To facilitate the computation, we parametrize the representation \(z_{i}\) of each view \(v_{i}(x)\) with a factorized Gaussian distribution, i.e., \(p_{\psi}(z|v_{i})=\mathcal{N}[\mu(v_{i}),\Sigma(v_{i})]\). Concretely, we estimate \(v_{i}(x)\) with the final transformer layer of the reward model and use two neural networks \(\mu(v_{i})\) and \(\Sigma(v_{i})\) to produce the mean and deviation respectively. The following MIB loss is optimized:

\[\mathcal{L}_{M}=\mathbb{E}_{(x,\mathbf{y})\sim\mathcal{P}}[-\mathbbm{I}(z_{1};z_{ 2})+\mathbb{D}_{\text{SKL}}(p_{\psi}(z|v_{1})||p_{\phi}(z|v_{2}))],\]

where \(\mathbbm{I}\) calculates mutual information of two random variables, and \(\mathbb{D}_{\text{SKL}}\) represents the symmetrized KL divergence obtained by averaging the expected value of \(\mathbb{D}_{\text{KLI}}(p_{\psi}(z|v_{1})||p_{\psi}(z|v_{2}))\) and \(\mathbb{D}_{\text{KL}}(p_{\psi}(z|v_{2})||p_{\psi}(z|v_{1}))\).

**Synthetic Preference Generation** aims to simulate high-quality preference data with policy samples. For each pair \((x,\mathbf{y})\in\mathcal{P}\), we assume the most frequent item of \(\mathbf{y}\) as the correct prediction and its frequency as the confidence score Si et al. (2023), following the self-consistency assumption Wang et al. (2023). To address semantic equivalence, i.e., different sentences can mean the same thing, we cluster items of \(\mathbf{y}\) into groups \(\mathcal{G}\) with a bi-directional entailment algorithm Kuhn et al. (2023). Sentences from each group \(\mathbf{g}\in\mathcal{G}\) are expected to share the same meaning. We estimate the confidence score of \((x,\mathbf{y})\) as \(\frac{|\tilde{\mathbf{g}}|}{|\mathbf{y}|}\), where \(\tilde{\mathbf{g}}\) is the largest group of \(\mathcal{G}\) and the operator \(|\cdot|\) measures the size of a set. Thus, we can selectively generate a synthetic preference dataset with high confidences \(\hat{\mathcal{D}}=\{(x,y_{w},y_{l})~{}|~{}(x,\mathbf{y})\in\mathcal{P},\frac{| \tilde{\mathbf{g}}|}{|\mathbf{y}|}\geq\gamma,y_{w}\sim\tilde{\mathbf{g}},y_{l}\sim\mathbf{y} \setminus\tilde{\mathbf{g}}\}\), where \(\gamma\) is the threshold for selective generation, the preferred output \(y_{w}\) is sampled from the largest group \(\tilde{\mathbf{g}}\) with the largest reward score and the non-preferred one \(y_{l}\) is randomly sampled from the rest groups.

The overall loss that we optimize for the reward model \(\hat{r}_{\phi}\) is:

\[\begin{split}\mathcal{L}_{R}&=-\mathbb{E}_{(x,y_{ w},y_{l})\sim\mathcal{D}\cup\hat{\mathcal{D}}}[\log\sigma(\hat{r}_{\phi}( x,y_{w})-\hat{r}_{\phi}(x,y_{l}))]\\ &\quad+\lambda\,\mathcal{L}_{M},\end{split} \tag{1}\]

where the coefficient \(\lambda\) controls the weight of the multi-view information bottleneck loss. To simplify computational complexity, we implement two variants: 1. **RLP-UML** removes the synthetic dataset \(\hat{\mathcal{D}}\) in Eq. 1 and learns the representations of policy samples when fitting the reward model. 2. **RLP-SPG** removes the MIB loss by setting \(\lambda=0\) in Eq. 1 and fits the reward model with human and synthetic preference data.

### Policy Retraining

We finally retrain the policy \(\hat{\pi}_{\theta}\) using \(\hat{r}_{\phi}\) in **Step 5** of RLP. Specifically, we optimize \(\hat{\pi}_{\theta}\) to maximize

\[\mathbb{E}_{x\sim\mathcal{U},y\sim t_{\theta}(y|z)}[\hat{r}_{\phi}(x,y)-\beta \mathbb{D}_{\text{KLI}}(\hat{\pi}_{\theta}(y|x)||\pi_{\text{ref}}(y|x))].\]

Our approach RLP is summarized in Algorithm 1.

```
Input: SFT model \(\pi^{\mathrm{SFT}}\), unlabeled data \(\mathcal{U}\). Output: A language model policy \(\hat{\pi}_{\theta}\).
1 Collect a human preference dataset \(\mathcal{D}\). Train a reward model \(r_{\phi}\) using \(\mathcal{D}\). Fine-tune a language model \(\pi_{\theta}\) from \(\pi^{\mathrm{SFT}}\) using \(\mathcal{U}\) and \(r_{\phi}\).
2Retrain a reward model \(\hat{r}_{\phi}\) using \(\mathcal{L}_{\hat{R}}\) (Eq. 1). Fine-tune \(\hat{\pi}_{\theta}\) from \(\pi^{\mathrm{SFT}}\) using \(\mathcal{U}\) and \(\hat{r}_{\phi}\).
```

**Algorithm 1**RLP: RLHF with Reward Learning on Policy

## 5 Experiments

### Datasets

We run experiments on the instruction following task Ouyang et al. (2022); Bai et al. (2022a), which remains a challenging task for the strongest LLMs today Wu et al. (2023); Li et al. (2023).

\begin{table}
\begin{tabular}{c|l|c} \hline \hline  & Dataset & \#Sample \\ \hline \multirow{4}{*}{Training data} & SFT dataset & 10k \\ \cline{2-3}  & Preference dataset \(\mathcal{D}\) & 10k \\ \cline{2-3}  & Unlabeled dataset \(\mathcal{U}\) & 20k \\ \hline \multirow{4}{*}{Evaluation data} & AlpaacFarm & 805 \\ \cline{2-3}  & LLMBar & 100 \\ \cline{1-1} \cline{2-3}  & Vicuna & 80 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Dataset statistics.

 

Training dataof the RLHF procedure come from Alpaca data, which consists of 52k instruction-following demonstrations \((x,y)\)Taiori et al. (2023). Following the data splits of AlpacaFarm Dubois et al. (2024), three splits are used: 1. SFT split are 10k data for fine-tuning the SFT model \(\pi^{\mathrm{SFT}}\); 2. Preference split are 10k instructions on which we collect pairwise preference dataset \(\mathcal{D}\); 3. Unlabeled split are 20k unlabeled instructions \(\mathcal{U}\) used in PPO. Concretely, two variants of preference dataset \(\mathcal{D}\) are curated: simulated \(\mathcal{D}_{\mathrm{sim}}\) are constructed with AlpacaFarm simulated annotators by prompting API LLMs, and human \(\mathcal{D}_{\mathrm{human}}\) are constructed with human annotators.

Evaluation dataof the trained LLMs include three benchmarks: 1. **Alpacafarm** consists of 805 instructions, which includes instructions from Self-instruct evaluation Wang et al. (2022), OASST evaluation, Anthropic helpful evaluation Bai et al. (2022), Vicuna evaluation Chiang and Lee (2023), and Koala evaluation Geng et al. (2023); 2. **LLM-Bar**Zeng et al. (2024) contains 100 instructions with objective preferences. We use the natural set to evaluate performance in real-world distributions; 3. **Vicuna**Chiang et al. (2023) contains 80 instructions to evaluate different skills of LLMs. Table 1 shows the statistics of these datasets.

### Metrics and Experimental Setups

Following Dubois et al. (2024); Touvron et al. (2023b), we use win-rate to evaluate the performance of an LLM \(\pi_{\theta}\), i.e., the percentage of times \(\pi_{\theta}\) is preferred to a reference model \(\pi_{\mathrm{ref}}\) for their instruction-following outputs. Following the experimental setups of AlpacaFarm Dubois et al. (2024), we use simulated win-rate to evaluate methods trained on simulated \(\mathcal{D}_{\mathrm{sim}}\) with simulated annotators by prompting API LLMs. In parallel, we use human win-rate to evaluate methods trained on human \(\mathcal{D}_{\mathrm{human}}\) with human annotators.

### Implementation Details

Our implementations of reward learning, policy optimization, and simulated annotators are based on the AlpacaFarm codebase and its default hyperparameters Dubois et al. (2024). Specifically, we use RL algorithm PPO Schulman et al. (2017) to fine-tune the language model during policy optimization. All reward models and language models are based on LLaMA-7B Touvron et al. (2023a). We sample \(n=10\) outputs for each instance as follows:

Figure 2: The simulated win-rate (%) performance of RLP-SPG compared to PPO on various subsets of AlpacaFarm. Win-rates are computed against reference model text–davinci-003.

 

struction to construct a dataset of policy samples \({\cal P}\). For unsupervised multi-view learning, we implement \(\mu(v_{i})\) and \(\Sigma(v_{i})\) as three-layer MLPs, and use Jensen-Shannon mutual information estimator [17] to estimate mutual information \(\mathbb{I}\) in the MIB loss. For synthetic preference generation, we implement a bidirectional entailment clustering algorithm using Deberta-large model [23] and set the threshold \(\gamma=0.5\) for selective generation. We set \(\lambda=0.5\) in Eq. 1 for RLP-UML. At training and inference time, we use a sampling temperature of 1.0 and 0.7, respectively. All experiments are performed on a single 8\(\times\)A100 machine.

### Baselines

We compare RLP with competitive baselines: **1. LLaMA-7B**[46] directly generates outputs using the base unaligned LLaMA-7B; **2. SFT**[19] is a LLaMA-7B model supervised fine-tuned on 10k Alpaca instruction-following data; **3. Best-of-\(n\)[10]**samples \(n\) i.i.d. responses from the SFT model and returns the response with the highest inferred reward; **4.PPO**[10] is a reinforcement learning algorithm that maximizes surrogate reward, subject to a KL penalty keeping parameters near the SFT model; **5. ChatGPT** uses OpenAI API LLM gpt-3.5-turbo-0301; **6. GPT-4**uses OpenAI API LLM gpt-4-0314.

### Main Results

We compare the win-rate performance of our method RLP and all baselines on three standard benchmarks to assess their instruction-following ability in Table 2. It can be seen that API LLM GPT-4 significantly outperforms all other models due to its obvious advantages. Among the implemented LLMs, RLP-SPG performs the best in both the simulator and human preference data, and achieves SOTA results on all three benchmarks. Compared with the implemented best-performing

\begin{table}
\begin{tabular}{l l} \hline \hline
**Instruction:** & What are different drawers I should have for clothes? \\ \hline
**SFT Output:** & Different drawers for clothes can include a top drawer for socks and underwear, a middle drawer for shorts, pants, and skirts, a lower drawer for long-sleeved shirts, a separate drawer for t-shirts and tank tops, a drawer for sweaters and cardignas, a drawer for jackets, and a drawer for shoes. \\ \hline
**PPO Output:** & Different drawers for clothes can include a master drawer for all items, such as pants, shirts, and jackets; a drawer for socks and underwater; a drawer for delicates, such as lingerie and fine clothing; a drawer for seasonal clothing, such as winter coats and swimwear; a drawer for work clothes, such as dress shirts and skirts; a drawer for shoes; and a drawer for accessories, such as belts, hats, and purses. \\ \hline
* A top drawer for long-sleeved shirts and blouses. \\
- A middle drawer for short-sleeved shirts and blouses. \\
- A bottom drawer for sweaters, sweaters, and jackets. \\
- A separate drawer for parts, shorts, and skirts. \\
- A drawer for socks and underwater. \\
- A drawer for accesses such as belts, scarves, and hats. \\
- A separate drawer for seasonal clothing, such as winter coats and summer dresses. \\ \hline
* A master drawer for folded clothing, such as shirts, pants, skirts, and shorts. \\
- A drawer for reasonal items, like sweaters and jackets. \\
- A drawer for delicate items, like lingerie and fine fabrics. \\
- A drawer for formal wear, like suits and dresses. \\
- A drawer for accessories, like belts, scarves, and jewelry. \\
- A drawer for uniforms, like work clothes and school uniforms. \\
- A drawer for out-of-season items, like swimmer and winter coats. \\
- A drawer for shoes, organized by type and use. \\
- A drawer for cleaning supplies, like a laundry bin or ironing board. \\
- A drawer for miscellaneous items, like first-aid kits and sewing supplies. \\ \hline \end{tabular}
\end{table}
Table 3: Outputs generated by RLP and baselines for an example from AlpaacFarm. RLP-UML and RLP-SPG produce more comprehensive outputs than SFT and PPO in this case.
 

baseline PPO, RLP-SPG brings up from a simulator win-rate of 46.8% to 50.2% in AlpacaFarm, 47.5% to 50.5% in LLMBar, and 57.5% to 62.5% in Vicuna. RLP-SPG also brings up from a human win-rate of 55.1% to 57.4% in AlpacaFarm.

We can also observe that: **1.** Both the two variants of RLP, namely, RLP-UML and RLP-SPG, outperform all implemented baselines that do not train reward models using policy samples. The performance gain demonstrates the advantage of considering policy for reward learning, which can help keep the reward model on-distribution. **2.** RLP-SPG generally outperforms RLP-UML under all circumstances. It demonstrates that synthetic preference generation leads to better performance, which simulates pairwise preference data with policy samples that can be leveraged for optimizing a reward model directly.

To provide a clearer perspective on RLP's superiority over other baselines, we illustrate the simulated win-rate of our best method RLP-SPG compared to the best-performing baseline PPO on various subsets of AlpacaFarm in Figure 2. Instructions from these subsets show diverse coverage over realistic interactions, allowing for an intricate analysis of the proficiency attained through language model fine-tuning Dubois et al. (2024). Notably, RLP-SPG outperforms PPO across all subsets, including Self-instruct evaluation, Anthropic helpful evaluation, OASST evaluation, and Koala evaluation. It further indicates that reward learning on policy leads to a comprehensive enhancement in the capabilities of the LLMs. Meanwhile, RLP also outperforms these baselines on knowledge intensive benchmarks such as MMLU Hendrycks et al. (2021) (See Appendix A.).

The difference between RLP and baselines can be observed qualitatively as well. For example, the case shown in Table 3 makes it sufficiently clear why RLP is so strongly preferred over our baselines from AlpacaFarm. Compared to RLP-UML, RLP-SPG generates even longer and more comprehensive outputs.

### Ablation Studies

This section provides comprehensive ablation studies to understand the efficacy of RLP. For consistency, all ablations are conducted using metric simulated win-rate that is computed against reference model text-davinci-003.

Information Bottleneck LossWe demonstrate the effectiveness of our multi-view information bottleneck loss \(\mathcal{L}_{M}\) by replacing \(\mathcal{L}_{M}\) in Eq. 1 with other alternatives of representation learning: **1. InfoMax**Poole et al. (2019) maximizes the mutual information between an input \(v(x)\) and its representation \(z\), i.e., \(\mathbb{I}(v;z)\); **2. MUI**Bachman et al. (2019) is similar to InfoMax except that it maximizes the mutual information between its two views \(\mathbb{I}(z_{1};z_{2})\); Note that neither InfoMax nor MUI attempts to remove superficial information from representations. **3. CL**Caron et al. (2020) uses a contrastive learning loss. Positive pairs in this variant are obtained using our multi-view construction approach.

Results in Table 4 show that the information bottleneck loss used in RLP-UML performs better than all other variants. We also want to highlight that the approach of explicitly removing superficial information in RLP-UML makes it outperform InfoMax and MUI by \(4.7\%\) and \(1.0\%\) in AlpacaFarm, and \(2.0\%\) and \(1.0\%\) in LLMBar, respectively. This validates our claim that the multi-view information bottleneck loss used in RLP-UML effectively removes the superficial information.

Synthetic Preference GenerationWe compare RLP with two types of synthetic preference generation approaches: **I.** sampling a pair of responses for each instruction and then labeling its preference with LLMs; **II.** sampling a set of outputs for each instruction and then selecting a preferred and a non-preferred one to construct pairwise preference data.

\begin{table}
\begin{tabular}{l|c|c} \hline \hline
**Method** & \begin{tabular}{c} **AlpacaFarm** \\ Win-Rate \\ \end{tabular} & 
\begin{tabular}{c} **LLMBar** \\ Win-Rate \\ \end{tabular} \\ \hline InfoMax & 44.4 & 46.5 \\ MVI & 48.1 & 47.5 \\ CL & 48.2 & 46.5 \\ \hline
**RLP-UML** & **49.1** & **48.5** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Ablation study on the representation learning loss for RLP-UML.

\begin{table}
\begin{tabular}{l|c|c|c} \hline \hline
**Sampling** & **Method** & \begin{tabular}{c} **Preference** \\ Acc. \\ \end{tabular} & \begin{tabular}{c} **AlpacaFarm** \\ Win-Rate \\ \end{tabular} \\ \hline A Pair & \begin{tabular}{c} RLAIF \\ Reward \\ \end{tabular} & \begin{tabular}{c} 52 \\ 56 \\ \end{tabular} & 
\begin{tabular}{c} 46.0 \\ 48.9 \\ \end{tabular} \\ \hline \multirow{3}{*}{A Set} & Select-All & 55 & 48.7 \\ \cline{2-3}  & **RLP-SPG** & **59** & **50.2** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Ablation on different synthetic preference generation approaches.

 

For type **I**, we implement **RLAIF**(Lee et al., 2023) that labels preferences with policy \(\pi_{\theta}\), and **Reward** that rank the two outputs with reward model \(r_{\phi}\) and assume the top ranked output as the preferred one. For type **II**, we study a variant of RLP, **Select-All**, that sets \(\gamma=0\) for selective generation and no longer rejects low confidence data.

Table 5 shows the accuracy of generated preference data and the win-rate of the corresponding LLMs. Golden preferences are labeled with AlpacaFarm simulated annotator. These results indicate that RLP-SPG outperforms all ablation variants in terms of synthetic preference quality and LLM performance. We can also observe that: 1. Sampling a set of outputs rather than a pair for each instruction helps encourage output diversity and leads to high-quality preference generation. 2. The confidence score based on multiple sampling can be used for selective generation and further improve preference quality. 3. LMMs trained with more accurate preference data generally perform better and obtain higher win-rate scores.

### Further Analysis

Here we present further analysis of intermediate results during LLM training. Table 6 shows an example of simulated preference data by RLAIF and RLP-SPG, respectively. The two outputs (\(y_{w}\) and \(y_{l}\)) of RLAIF for this case look quite similar. However, \(y_{w}\) is preferred by RLAIF which would bring in noises in the training process. On the contrary, \(y_{w}\) of RLP-SPG is more comprehensive than \(y_{l}\) of RLP-SPG, resulting in more accurate labels. We also find a major difference in the length distributions of RLP-SPG outputs, with preferred outputs \(y_{w}\) (510 characters on average) significantly longer than non-preferred outputs \(y_{l}\) (449 characters on average).

Table 7 also demonstrates cases of rejected and accepted instructions for selective synthetic preference generation by RLP-SPG (rejecting low confi

\begin{table}
\begin{tabular}{l l} \hline \hline
**Instruction:** & \\ Generate a definition for the term “artificial intelligence”. \\ \hline
**RLAIF Simulated Preference Data:** & \\ \(y_{w}\): Artificial intelligence (AI) is a branch of computer science that enables machines to learn from experience, adjust to new input, and mimic human decision-making processes. AI technology is used to develop systems that can interact with their environment and achieve tasks such as robotic navigation, natural language processing, and computer vision. \\ \(y_{l}\): Artificial intelligence (AI) is a field of study focused on developing computers and machines that are capable of performing tasks that are typically done by humans, such as reasoning, problem solving, and decision-making. AI technology is used in a variety of applications, including robotics, games, speech recognition, natural language processing, and automation. \\ \hline
**RLP Simulated Preference Data:** & \\ \(y_{w}\): Artificial intelligence (AI) is a branch of computer science that involves the use of sophisticated algorithms and technology to enable machines to think, learn, and act like humans. AI systems can be programmed to perform a variety of tasks, from simple to complex, such as recognizing speech, understanding natural language, automating tasks, and more. AI technology is used in a wide range of applications, from self-driving cars to virtual assistants to automated customer service. In short, AI is the science and technology of making computers and machines smart and capable of doing things traditionally done by humans. \\ \(y_{t}\): Artificial intelligence (AI) is a field of study focused on developing computers and machines that can think and act like humans. AI technology allows machines to interact with their environment and to learn from their mistakes, just like humans do. \\ \hline \hline \end{tabular}
\end{table}
Table 6: An example instruction and corresponding simulated preference data from RLAIF and RLP-SPG.

\begin{table}
\begin{tabular}{c|l c} \hline \hline
**Decision** & **Instruction** & **Confidence** \\ \hline  & Describe the life and reign of King Charles II. & 0.1 \\ Reject & What type of fruit would be a good addition to a fruit salad? & 0.2 \\  & Research about a famous scientist and provide a short biography about him/her. & 0.2 \\  & Compose a five word sentence describing your day. & 0.4 \\ \hline  & Write a scientific explanation for why the sky is blue. & 0.6 \\ Accept & Find the synonyms of the following word: ‘Tenacious’. & 0.7 \\  & Find the main idea of the following passage. & 0.8 \\  & Create a tweet summarizing the following news article in 140 characters or less. & 0.9 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Cases of rejected and accepted instructions for selective synthetic preference generation by RLP-SPG.

 

dence generations). It can be observed that open-ended instructions (e.g., more subjective and creative) tend to have low confidences and be rejected.

## 6 Conclusion

In this paper, we propose reward learning on policy (RLP), a novel framework to align LLMs with human preferences. RLP learns robust representations of the policy's data distribution via optimizing a multi-view information bottleneck loss. RLP also simulates preferences with a set of policy outputs, which enables confidence estimation and selective generation. Extensive experiments demonstrate that RLP outperforms SOTA baselines.

### Limitations

While we have carefully studied the effectiveness of RLP compared to several baselines on three benchmark datasets for LLaMA-7B, we have not yet empirically verified our conclusions when aligning larger pretrained LLMs. It would also be interesting to align new SOTA pretrained LLMs such as LLaMA 2 Touvron et al. (2023b) and test other methods for fitting preference data like DPO Rafailov et al. (2024).

Meanwhile, all of our training and evaluation data are in English, and we have not tested in other languages. Performance may degenerate especially in low-resource languages when pretrained LLMs have not been trained with these data.

## Ethics Statement

This work does not raise any direct ethical issues. In the proposed work, we seek to develop a novel RLHF framework to align large language models (LLMs) with human preferences. Concretely, we propose to learn reward models on policy to keep it on-distribution. We believe this work can benefit the field of LLMs, with the potential to benefit other fields requiring NLP models. All experiments are conducted on open datasets.

## References

* Anthropic (2023) Anthropic. 2023. Introducing claude.
* Askell et al. (2021) Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. 2021. A general language assistant as a laboratory for alignment. _arXiv preprint arXiv:2112.00861_.
* Bachman et al. (2019) Philip Bachman, R Devon Hjelm, and William Buchwalter. 2019. Learning representations by maximizing mutual information across views. In _NeurIPS_.
* Bai et al. (2022a) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022a. Training a helpful and harmless assistant with reinforcement learning from human feedback. _arXiv preprint arXiv:2204.05862_.
* Bai et al. (2022b) Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022b. Constitutional ai: Harmlessness from ai feedback. _arXiv preprint arXiv:2212.08073_.
* Buyk et al. (2022) Erdem Buyk, Dylan P Losey, Malayandi Palan, Nicholas C Landolfi, Gleb Shevchuk, and Dorsa Sadigh. 2022. Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences. _The International Journal of Robotics Research_.
* Bommasani et al. (2021) Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosseltut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. _arXiv preprint arXiv:2108.07258_.
* Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Pfraulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In _NeurlPS_.
* Caron et al. (2020) Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. 2020. Unsupervised learning of visual features by contrasting cluster assignments. In _NeurIPS_.
* Casper et al. (2023) Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jeremy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korback, David Lindner, Pedro Freire, et al. 2023. Open problems and fundamental limitations of reinforcement learning from human feedback. _TMLR_.
* Chen et al. (2023) Xinyun Chen, Maxwell Lin, Nathanael Scharli, and Denny Zhou. 2023. Teaching large language models to self-debug. _arXiv preprint arXiv:2304.05128_.
* Chia et al. (2023) Yew Ken Chia, Pengfei Hong, Lidong Bing, and Soujanya Poria. 2023. Intructeval: Towards holistic evaluation of instruction-tuned large language models. _arXiv preprint arXiv:2306.04757_.
* Chiang and Lee (2023) Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evaluations? In _ACL_.
* Chiang et al. (2019) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Tianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al.

 

2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatpgt quality. _See https://vicuna. Imsys. org (accessed 14 April 2023)_.
* Christiano et al. [2017] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In _NeurIPS_.
* Chung et al. [2022] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. _arXiv preprint arXiv:2210.11416_.
* Coste et al. [2024] Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. 2024. Reward model ensembles help mitigate overoptimization. In _ICLR_.
* Dubois et al. [2024] Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2024. Alpacafarm: A simulation framework for methods that learn from human feedback. In _NeurIPS_.
* Federici et al. [2020] Marco Federici, Anjan Dutta, Patrick Forre, Nate Kushman, and Zeynep Akata. 2020. Learning robust representations via multi-view information bottleneck. In _ICLR_.
* Gao et al. [2023] Leo Gao, John Schulman, and Jacob Hilton. 2023. Scaling laws for reward model overoptimization. In _ICML_.
* Geng et al. [2023] Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023. Koala: A dialogue model for academic research. _Blog post, April_, 1.
* Bordeau et al. [2023] Google. 2023. Bard.
* He et al. [2021] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: Decoding-enhanced bert with disentangled attention. In _ICLR_.
* Hendrycks et al. [2021] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In _ICLR_.
* Hjelm et al. [2018] R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. 2018. Learning deep representations by mutual information estimation and maximization. In _ICLR_.
* Kadavath et al. [2022] Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. _arXiv preprint arXiv:2207.05221_.
* Kuhn et al. [2023] Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In _ICLR_.
* Lee et al. [2023] Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. 2023. Blair: Scaling reinforcement learning from human feedback with ai feedback. _arXiv preprint arXiv:2309.00267_.
* Li et al. [2023] Shiyang Li, Jun Yan, Hai Wang, Zheng Tang, Xiang Ren, Vijay Srinivasan, and Hongxia Jin. 2023. Instruction-following evaluation through verbalizer manipulation. _arXiv preprint arXiv:2307.10558_.
* Lin et al. [2022] Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching models to express their uncertainty in words. _arXiv preprint arXiv:2205.14334_.
* Lin et al. [2023] Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2023. Generating with confidence: Uncertainty quantification for black-box large language models. _arXiv preprint arXiv:2305.19187_.
* Longpre et al. [2023] Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. 2023. The flan collection: Designing data and methods for effective instruction tuning. In _ICML_.
* Mishra et al. [2022] Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022. Cross-task generalization via natural language crowdsourcing instructions. In _ACL_.
* OpenAI [2023] OpenAI. 2023. GPT-4 technical report.
* Ouyang et al. [2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhnii Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. In _NeurIPS_.
* Poole et al. [2019] Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. 2019. On variational bounds of mutual information. In _ICML_.
* Rafailov et al. [2024] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. In _NeurIPS_.
* Sanh et al. [2022] Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Ajafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2022. Multitask prompted training enables zero-shot task generalization. In _ICLR_.
* Schulman et al. [2017] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. _arXiv preprint arXiv:1707.06347_.
* Si et al. [2023] Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Boyd-Graber, and Lijuan Wang. 2023. Prompting gpt-3 to be reliable. In _ICLR_.
* Skalse et al. [2022] Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. 2022. Defining and characterizing reward gaming. In _NeurIPS_ 

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. In _NeurIPS_.
* Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Stanford alpaca: An instruction-following llama model.
* Tien et al. (2023) Jeremy Tien, Jerry Zhi-Yang He, Zackory Erickson, Anca D Dragan, and Daniel S Brown. 2023. Causal confusion and reward misidentification in preference-based reward learning. In _ICLR_.
* Tishby et al. (2000) Naftali Tishby, Fernando C Pereira, and William Bialek. 2000. The information bottleneck method. _arXiv preprint physics/0004057_.
* Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. Llama: Open and efficient foundation language models. _arXiv preprint arXiv:2302.13971_.
* Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Wang et al. (2023) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Akanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In _ICLR_.
* Wang et al. (2022) Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharava Naik, David Stap, et al. 2022. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. _arXiv preprint arXiv:2204.07705_.
* Wu et al. (2023) Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Aybarek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. 2023. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. _arXiv preprint arXiv:2307.02477_.
* Yang et al. (2024) Kevin Yang, Dan Klein, Asli Celikyilmaz, Nanyun Peng, and Yuandong Tian. 2024. Rlcd: Reinforcement learning from contrast distillation for language model alignment. In _ICLR_.
* Yu et al. (2023) Tianshu Yu, Ting-En Lin, Yuchuan Wu, Min Yang, Fei Huang, and Yongbin Li. 2023. Constructive large language models alignment with diverse feedback. _arXiv preprint arXiv:2310.06450_.
* Zeng et al. (2024) Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. 2024. Evaluating large language models at evaluating instruction following. In _ICLR_.
* Zhao et al. (2017) Jing Zhao, Xijiong Xie, Xin Xu, and Shiliang Sun. 2017. Multi-view learning overview: Recent progress and new challenges. _Information Fusion_.
* Ziegler et al. (2019) Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. _arXiv preprint arXiv:1909.08593_.

## Appendix A Evaluation on Knowledge Intensive Benchmark

We also evaluate the performance of RLP on knowledge intensive benchmark MMLU Hendrycks et al. (2021), which includes exam questions from 57 tasks such as mathematics, history, law, and medicine. Specifically, we use InstructEval Chia et al. (2023) to perform the evaluation. As shown in Table 8, both RLP-UML and RLP-SPG outperform PPO.

\begin{table}
\begin{tabular}{l|c} \hline \hline
**Method** & **MMLU** \\ \hline GPT-4 & 86.4 \\ ChatGPT & 70.0 \\ PPO & 36.9 \\ LLaMA-7B & 35.2 \\ \hline
**RLP-UML** (ours) & **37.6** \\ RLP-SPG (ours) & 37.3 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Performance of RLP and baselines on the MMLU benchmark. The scores are obtained by running InstructEval.