UNA: Unifying Alignments of RLHF/PPO, DPO and KTO by a Generalized Implicit Reward Function

## Abstract

Abstract An LLM is pretrained on trillions of tokens, but the pretrained LLM may still generate undesired responses. To solve this problem, alignment techniques such as RLHF, DPO and KTO are proposed. However, these alignment techniques have limitations. For example, RLHF requires training the reward model and policy separately, which is complex, time-consuming, memory intensive and unstable during training processes. DPO proposes a mapping between an optimal policy and a reward, greatly simplifying the training process of RLHF. However, it can not take full advantages of a reward model and it is limited to pairwise preference data. In this paper, we propose UN ified A lignment (UNA) which unifies RLHF/PPO, DPO and KTO. Firstly, we mathematically prove that given the classical RLHF objective, the optimal policy is induced by a generalize implicit reward function. With this novel mapping between a reward model and an optimal policy, UNA can 1. unify RLHF/PPO, DPO and KTO into a supervised learning of minimizing the difference between an implicit reward and an explicit reward; 2. outperform RLHF/PPO while simplify, stabilize, speed up and reduce memory burden of RL fine-tuning process; 3. accommodate different feedback types including pairwise, binary and scalar feedback. Downstream experiments show UNA outperforms DPO, KTO and RLHF.

## 1 Introduction

LLMs are trained on extensive and diverse corpora, enabling them to develop robust language capabilities and a deep understanding of various contexts OpenAI et al. (); Anthropic (). However, during inference, LLM can generate undesired responses, which should be avoided. Supervised fine-tuning (SFT) though can improve an LLM on downstream tasks like question answering, it cannot solve these problems. To address these problems, alignment techniques like RLHF Ouyang et al. () and DPO Rafailov et al. () are proposed.

RLHF involves two stages of training from the SFT models as shown in part (b) of Figure [1](#S1.F1). Firstly, it trains a reward model (RM) using a preference dataset consisting of tuples (input, desired response, undesired response). Next, during the RL fine-tuning stage, the policy generates responses to given prompts. These responses are evaluated by the reward model and then used to fine-tune the policy with RL through PPO. However, several problems exist in RLHF. First of all, there exists an overfitting problem in the training stage of the reward model. In addition, RL fine-tuning stage is inherently unstable due to the nature of RL. Lastly, RL increases memory requirements for elements like the policy, reference policy, reward model and value model.

Figure: Figure 1: A figure comparison among (a). UNA, (b) RLHF, (c) DPO and (d) KTO. Each subfigure is composed of four types of data: ”prompt data”, ”preference feedback”, ”binary feedback” and ”score feedback”, LLM policy, response, two reward models: ”generalized implicit reward model” and ”explicit reward model” and a module to minimize the difference between implicit and explicit rewards. The connection between data to other modules are utilizing green dash arrow, while others are connected by black solid arrow. All unused modules are grayed out. In part (b), RLHF firstly utilizes preference feedback to train the explicit reward model, and the use the evaluation provided by the explicit reward model to continuous optimize the policy in a online mode. In comparison, in part (c) and (d), DPO and KTO utilize preference feedback and binary feedback respectively to generate implicit reward to align LLM policy. However, in part (a), UNA can utilize different types of data to get generalized implicit and explicit rewards and minimize their differences to align LLM policy in online and offline modes.

DPO addresses these issues by creating a mapping between the reward model and the optimal policy, combining the RM and RL training stages into a single process as shown in part (c) of Figure [1](#S1.F1). This approach simplifies the two-stage optimization into one stage, eliminating the need to train an explicit reward model, reducing memory costs, and transforming the unstable RL process into a stable binary classification problem. Given a prompt along with desired and undesired responses, the implicit rewards for both responses are calculated. The differences in these rewards are then used to optimize the policy. However, DPO has its own set of challenges. It cannot produce an explicit reward model and will require more preference data to fine-tune the LLM. Moreover, in RL, the pretrained RM can provide accurate guidance for alignment, which is absent in DPO. In summary, DPO’s efficiency in using preference data is lower compared to RLHF/PPO.

KTO extends DPO to handle binary data, such as thumbs up and thumbs down for desired and undesired responses as shown in the part (d) of Figure [1](#S1.F1). However, there have not been work on alignment based on prompt, response and corresponding evaluation scores. In addition, there have not been a work that can unify RLHF/PPO, DPO and KTO to accommodate these different types of data. This work will address these problems.

In this work, we propose UNified Alignment (UNA) which unifies RLHF/PPO, DPO and KTO, and combines the benefits of them. Firstly, inspired by the derivation of DPO, we prove that based on the RLHF objective $\pi^{*}_{\theta}(y|x)=\max_{\pi_{\theta}}\mathbb{E}_{x\sim D}\left\{\mathbb{E}_{y\sim\pi_{\theta}(y|x)}[r_{\theta}(x,y)]-\beta D_{\text{KL}}\left(\pi_{\theta}(y|x)\|\pi_{\text{ref}}(y|x)\right)\right\}$, the optimal policy can be induced by $r(x,y)=\beta\log\left(\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}\right)+f(x)+c$. It can be further simplified to $r(x,y)=\beta\log\left(\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}\right)$ when $f(x)=c=0$. The condition $f(x)=c=0$ indicates that the difference between implicit and explicit rewards is 0.

Based on the new generalized implicit reward function, UNA unifies RLHF/PPO, DPO and KTO into a supervised learning of minimizing the difference between an implicit reward and an explicit reward, where the explicit reward can come from human labelers, reward functions and LLMs as shown in part (a) of Figure [1](#S1.F1). Given a prompt, the trained policy can firstly generate responses, and an implicit reward score can be calculated based on the previous Equation Then, the pair of prompt and response is evaluated by different evaluation tools to derive an explicit reward score. Provided the implicit and explicit reward score, a supervised learning problem like mean square error (MSE) can be constructed to unify RLHF and DPO. Last but not least, for clarity, the unnormalized evaluation is termed as reward and the normalized evaluation is termed as score in this work.

With UNA, RLHF can be simplified through replacing the original RL fine-tuning stage, which is unstable, slow, and memory-intensive with a stable, efficient and memory friendly supervised learning. In addition, UNA can accommodate different types of data including pairwise feedback, binary feedback, score-based feedback. For pairwise data, we mathematically prove that UNA and DPO are equivalent. For binary data, thumb up (positive feedback) and thumb down (negative feedback) can be regarded as explicit rewards with reward scores of 1 and 0 respectively. With these derived implicit and explicit rewards, UNA can accommodate binary feedback. Lastly, for any types of unpaired data composed of a tuple, i.e., (prompt, response, score), UNA can be applied as well. Given the prompt and response, the implicit reward is firstly calculated, and then a supervised learning process is conducted to minimize the difference between the implicit reward and the explicit reward. In conclusion, UNA is a unified alignment framework for RLHF, DPO and KTO. It does not only simplify RLHF but also accommodates different types of data.

We apply UNA on Mistral-7B with Ultrafeedback dataset, and show its superior performances on binary and scalar feedback. For experiments comparing UNA with RLHF/PPO, phi-1.5 is utilized as a policy and RM-Gemma-2B is utilized as a reward model. The prompts in Ultrafeedback are utilized as inputs for comparing UNA with RLHF/PPO. All the tests show that UNA outperforms DPO, KTO and RLHF/PPO.

The contributions of this paper are five-fold:

- 1.
Mathematically prove that based on the RLHF objective function, the optimal policy can be induced by the reward function $r(x,y)=\beta\log\left(\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}\right)+f(x)+c$, which can simplified to $r(x,y)=\beta\log\left(\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}\right)$ when $f(x)=c=0$.
- 2.
Propose UNA which unifies RLHF/PPO with DPO into a supervised learning of minimizing the difference between implicit reward and explicit reward.
- 3.
Propose UNA that outperforms RLHF/PPO while simplifies, stabilizes, speeds up and reduces memory burden of RL fine-tuning process.
- 4.
Propose UNA that can accommodate different types of data: pairwise feedback, binary feedback, score-based feedback on both online and offline mode from different evaluation methodologies including human labeling, reward models and LLMs.
- 5.
Evaluate the performance of UNA on downstream tasks and compare it with DPO, KTO and RLHF/PPO to show its benefits.

## 2 Methodology: UNA

In this section, we will starts with some review of RLHF/PPO and DPO. Then, we will introduce UNA and derive a general loss function and its four applications: 1. Equivalence to DPO for pairwise preference dataset; 2. Replacement KTO for binary feedback; 3. RM / LLM distillation using reward from RM / LLM; 4. Simplification of RLHF in RL fine-tuning stage.

### 2.1 RLHF/PPO

After the SFT phase, the RLHF using PPO consists of two main stages: reward model training and reinforcement learning fine-tuning.

During the reward model training process, an explicit reward model is trained to predict a reward score $r_{\phi}(x,y)$ based on a given prompt $x$ and response $y$. This training utilizes pairwise preference data in the form of tuples, specifically $(x,y_{w},y_{l})$, where $y_{w}$ represents the desired response and $y_{l}$ represents the undesired response. Initially, the probability of $y_{w}$ being preferred over $y_{l}$, denoted as $P_{\phi}(y_{w}>y_{l}|x)$, is calculated based on their respective reward scores $r_{\phi}(x,y_{w})$ and $r_{\phi}(x,y_{l})$ through the Bradley-Terry (BT) model as shown in Equation [1](#S2.E1), which provides a probabilistic framework for comparing the preferences between the two responses.

$$ $P_{\phi}(y_{w}>y_{l}|x)=\frac{e^{r_{\phi}(x,y_{w})}}{e^{r_{\phi}(x,y_{w})}+e^{r_{\phi}(x,y_{l})}}=\sigma(r_{\phi}(x,y_{w})-r_{\phi}(x,y_{l}))$ (1) $$

Given a pre-collected pairwise dataset where humans have selected the desired and undesired responses from two candidates, we have $P(y_{w}>y_{l}|x)=1$ and $P(y_{w}<y_{l}|x)=0$. To train an effective reward model, we minimize the cross-entropy loss between the predicted probabilities and the human-labeled probabilities as shown in Equation [2](#S2.E2). Once the cross-entropy loss is minimized, the training of the reward model is complete.

$$ $\displaystyle L_{\text{RM}}(\pi_{\theta})$ $\displaystyle=-\mathbb{E}_{(x,y_{w},y_{l})\sim D}\left[\log(P_{\phi}(y_{w}>y_{l}\mid x))\right]$ (2) $\displaystyle=-\mathbb{E}_{(x,y_{w},y_{l})\sim D}\left[\log(\sigma(r_{\phi}(x,y_{w})-r_{\phi}(x,y_{l})))\right]$ $$

The second stage of RL fine-tuning has two primary goals. The first goal is to maximize the pretrained explicit reward function $r_{\phi}(x,y)$ to ensure the policy aligns with reward model. To prevent reward hacking, the KL divergence from the initial policy $\pi_{\text{ref}}(y|x)$ is incorporated. The overall objective of RL fine-tuning is detailed in Equation [3](#S2.E3).

$$ $\pi^{*}_{\theta}(y|x)=\max_{\pi_{\theta}}\mathbb{E}_{x\sim D}\left\{\mathbb{E}_{y\sim\pi_{\theta}(y|x)}\left[r_{\phi}(x,y)\right]-\beta D_{\text{KL}}\left(\pi_{\theta}(y|x)\|\pi_{\text{ref}}(y|x)\right)\right\}$ (3) $$

Several limitations exist in RLHF. To begin with, the reward model may suffer from overfitting during training, which can adversely affect the RL fine-tuning process. Then, unlike traditional supervised learning, RL does not have explicit labels for each prompt and response. To address this, the authors employed PPO to optimize the RL objective. However, even with PPO, RL training can still be unstable. Additionally, RLHF with PPO necessitates the use of a policy, reference policy, reward model, and value model, which significantly increases memory requirements, especially for LLMs. These limitations constrain the practical application of RLHF.

### 2.2 DPO

In RLHF, the trained reward model can suffer from overfitting, and RL fine-tuning is notorious for its instability and memory intensity. To address these challenges, the authors of DPO discover that the optimal policy is induced by Equation [4](#S2.E4), based on the objective function in Equation [3](#S2.E3). Here, $Z(x)=\sum_{y}\pi_{\text{ref}}(y|x)e^{\left(\frac{1}{\beta}r_{\theta}(x,y)\right)}$, where $r_{\theta}(x,y)$ represents the implicit reward function.

$$ $r_{\theta}(x,y)=\beta\log\left(\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}\right)+\beta\log Z(x)$ (4) $$

With the derived implicit reward model, it can be plugged into the reward model training process of RLHF in Equation [2](#S2.E2) where $Z(x)$ gets cancelled. Eventually, the loss function for DPO is derived as shown in Equation [5](#S2.E5).

$$ $\displaystyle L_{\text{DPO}}(\pi_{\theta})$ $\displaystyle=-\mathbb{E}_{(x,y_{w},y_{l})\sim D}\left\{\log\left[\sigma(r_{\phi}(x,y_{w})-r_{\phi}(x,y_{l}))\right]\right\}$ (5) $\displaystyle=-\mathbb{E}_{(x,y_{w},y_{l})\sim D}\left\{\log\left[\sigma\left(\beta\log\left(\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\text{ref}}(y_{w}|x)}\right)-\beta\log\left(\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\text{ref}}(y_{l}|x)}\right)\right)\right]\right\}$ $$

By optimizing the loss function in DPO, we can eliminate the need for an explicit reward model and combine the two stages of RLHF into a single, streamlined process, greatly simplifying the RLHF/PPO workflow. However, DPO has several limitations. First, $Z(x)$ cannot be directly estimated, which means only pairwise preference data can be utilized, making single-prompt data unusable during the RL fine-tuning stage. Additionally, while pairwise preference data are typically used only in the reward model stage, DPO requires them throughout, leading to inefficient use of precollected pairwise data. In comparison, after reward model training, it can be applied to prompt data, which are much easier to obtain compared with pairwise data. Lastly, in the RL stage in RLHF, reward model can provide more detailed evaluations of the generated responses. However, DPO cannot offer this level of granularity during training.

### 2.3 UNA

Inspired by the idea of DPO, we aim to establish a new relationship between the reward model and the optimal policy for a unified alignment framework including RLHF/PPO, DPO and KTO on different types of data. By adhering to the same objective outlined in RLHF (Equation [3](#S2.E3)), we can formulate a novel connection between the implicit reward function and the optimal policy, as shown in Equation [6](#S2.E6). The derivation can be found in Section [3](#S3). In the special case where $f(x)=0$ and $c=0$, it is further simplified.

$$ $\displaystyle r_{\theta}(x,y)$ $\displaystyle=\beta\log\left(\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}\right)+f(x)+c$ (6) $\displaystyle=\beta\log\left(\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}\right)\text{when }f(x)=0\text{ and }c=0$ $$

The optimal implicit reward formulation in Equation [6](#S2.E6) implies that we can transform the original unstable, memory-expensive RL training process into a reward function optimization problem, i.e., a stable and memory-efficient supervised learning process. Explicit rewards can be derived from multiple methods including 1. human labeling, 2. pretrained LLMs and 3. reward models. Eventually, the RL fine-tuning process is transformed into a general minimization problem between explicit reward $r_{\phi}(x,y)$ and implicit reward $r_{\theta}(x,y)$ as shown in Equation [7](#S2.E7) where $g(x,y)$ refers to a general function that measure the difference between $x$ and $y$ like MSE.

$$ $\displaystyle L_{\text{UNA-reward}}(\pi_{\theta})=\mathbb{E}_{x\sim D}\mathbb{E}_{y\sim\pi_{\theta}(\cdot|x)}[g(r_{\phi}(x,y),r_{\theta}(x,y))]$ (7) $$

When applying an LLM for evaluation, the scores lie within a specific range, such as [0, 100]. These scores can be easily normalized to the interval [0, 1]. However, the implicit reward function can span from negative to positive infinity. To normalize implicit reward, the implicit score function, denoted as $s_{\theta}(x,y)$, can be derived as shown in Equation [8](#S2.E8). For clarity, the unnormalized evaluation is termed as reward and the normalized evaluation is termed as score.

$$ $s_{\theta}(x,y)=\sigma[r_{\theta}(x,y)]=\sigma\left[\beta\log\left(\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}\right)\right]$ (8) $$

Given the implicit and explicit score functions, an equivalent general loss for UNA can be shown in Equation [9](#S2.E9). The normalized general loss function is more stable and will be used for experiments in this study.

$$ $\displaystyle L_{\text{UNA-score}}(\pi_{\theta})=\mathbb{E}_{x\sim D}\mathbb{E}_{y\sim\pi_{\theta}(\cdot|x)}[g(s_{\phi}(x,y),s_{\theta}(x,y))]$ (9) $$

Based on this general loss function using the new implicit reward function, UNA can be utilized in multiple conditions:
1. Equivalence to DPO for pairwise preference dataset
2. Improvement over KTO for binary feedback
3. RM / LLM distillation using reward from teacher RM / LLM outperforming DPO and KTO
4. Improvement over RLHF in RL fine-tuning stage: simplify PPO with a supervised learning process. Figure [2](#S2.F2) shows how UNA is applied to different types of data and simplifies RLHF.

Figure: Figure 2: The four applications of UNA: (a). equivalent to DPO for pairwise data, (b). improvement over KTO for binary data, (c). RM/LLM distillation for score-based data, (d). simplification of RLHF for online training. The same modules are utilized as in Figure [1](#S1.F1), and unused modules are grayed out. For part (a), the same steps as DPO will be utilized. For part (b), (c), (d), from the different types of data including pairwise, binary and score-based feedback, implicit and explicit rewards are firstly gathered. Then, the difference between implicit and explicit rewards is minimized like MSE loss function to align the LLM policy. More details can be found in Section [2.3](#S2.SS3).

#### 2.3.1 UNA: Equivalent to DPO for pairwise dataset

For pairwise dataset, the implicit rewards of desired and undesired responses can be derived as shown in part (a) of Figure [1](#S1.F1). Then, LLM policy is aligned by maximizing the difference of implicit rewards between desired and undesired responses. The loss function of UNA for pairwise dataset is shown in Equation [10](#S2.E10).

$$ $\displaystyle L_{\text{UNA-pair}}(\pi_{\theta})$ $\displaystyle=-\mathbb{E}_{(x,y_{w},y_{l})\sim D}\left(r_{\theta}(x,y_{w})-r_{\theta}(x,y_{l})\right)$ (10) $\displaystyle=-\mathbb{E}_{(x,y_{w},y_{l})\sim D}\left[\beta\log\left(\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\text{ref}}(y_{w}|x)}\right)-\beta\log\left(\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\text{ref}}(y_{l}|x)}\right)\right]$ $$

It is equivalent to DPO as the loss function is the same as long as $f(x)=\log[\sigma(x)]$ is applied to the difference of implicit rewards of desired and undesired responses: $L^{\prime}_{\text{UNA-pair}}(\pi_{\theta})=L_{\text{DPO}}(\pi_{\theta})=-\mathbb{E}_{(x,y_{w},y_{l})\sim D}\left\{\log[\sigma(r_{\phi}(x,y_{w})-r_{\phi}(x,y_{l}))]\right\}$

#### 2.3.2 UNA: Improvement over KTO for binary feedback

For binary preference, the positive and negative feedback can be transformed to explicit scores. Positive or ’thumb up’ data can be assigned an explicit reward score of 1, i.e., $s_{\phi}(x,y_{w})=1$. In contrast, negative or ’thumb down’ data can be assigned an explicit reward score of 0, i.e., $s_{\phi}(x,y_{l})=0$. After that, similar procedures to DPO will be conducted to estimate implicit reward and minimize the difference between implicit and explicit rewards as shown in part (b) of Figure [2](#S2.F2).

Because the explicit feedback is binary, i.e., score rather than reward, implicit score should be utilized. Based on the implicit and explicit scores, multiple loss functions can be designed using mean square error (MSE) in Equation [11](#S2.E11) and binary cross entropy (BCE) in Equation [12](#S2.E12). As a result, UNA can be utilized to replace KTO for binary feedback data.

$$ $\displaystyle L_{\text{UNA-binary-MSE}}(\pi_{\theta})$ $\displaystyle=-\mathbb{E}_{(x,y)\sim D}[(s_{\theta}(x,y)-s_{\phi}(x,y))^{2}]$ (11) $\displaystyle=-[\mathbb{E}_{(x,y_{w})\sim D}(s_{\theta}(x,y)-1)^{2}+\mathbb{E}_{(x,y_{l})\sim D}(s_{\theta}(x,y)-0)^{2}]$ $$

$$ $\displaystyle L_{\text{UNA-binary-BCE}}(\pi_{\theta})$ $\displaystyle=-\mathbb{E}_{(x,y)\sim D}\left[L_{\text{BCE}}(s_{\theta}(x,y),s_{\phi}(x,y))\right]$ (12) $\displaystyle=-[\mathbb{E}_{(x,y_{w})\sim D}\log(s_{\theta}(x,y))+\mathbb{E}_{(x,y_{l})\sim D}\log(1-s_{\theta}(x,y))]$ $$

#### 2.3.3 UNA: LLM / RM distillation

Researchers have utilized LLM and RM to evaluate responses by outputting scores and rewards according to predefined standards. If the score and reward evaluations are accurate enough, they can be extra information to utilize for alignment. When the tuple type of data (prompt, response, score) is provided, the prompt and response are utilized to calculate implicit reward as shown in Equation [6](#S2.E6), and the score is utilized as the explicit reward as shown in part (c) of Figure [2](#S2.F2). The last step is the minimization of implicit and explicit rewards. However, the explicit reward and score from reward model and LLM are not binary, and as a result, MSE can be used as the loss function, excluding BCE. After normalization, the loss function for UNA using LLM / RM distillation is shown in Equation [13](#S2.E13). When LLMs are utilized for evaluation, it can be regarded as an offline version of RLAIF.

$$ $L_{\text{UNA-LLM-distill}}(\pi_{\theta})=-\mathbb{E}_{(x,y)\sim D}[(s_{\theta}(x,y)-s_{\phi}(x,y))^{2}]$ (13) $$

#### 2.3.4 UNA: Simplification of RLHF

When utilizing reward model for online evaluation, UNA will greatly simplify RL fine-tuning stage of RLHF/PPO with superior performances as shown in part (d) of Figure [2](#S2.F2). Assuming the reward model has already been trained, the focus now shifts exclusively to the RL fine-tuning stage. Prompts are firstly sent to an LLM for online response generation and implicit reward estimation. Then, the prompt and response are sent to the reward model for explicit reward estimation. The last step minimize the differences between implicit and explicit rewards to align the LLM policy. Eventually, the original RL objective in Equation [3](#S2.E3) can be transformed to difference minimization like MSE of implicit reward and explicit reward or scores as shown in Equation [13](#S2.E13)

UNA has several benefits over PPO in RL fine-tuning stage. First of all, it transforms the original unstable RL problem into a stable supervised learning problem by minimizing the difference between implicit and explicit rewards. In addition, UNA removes the necessary of value model in PPO, and partially reduce the burden of memory cost. Then, the computation cost of MSE is much lower compared with the multiple terms in PPO to maintain performance, and as a result, UNA will speed up the training process. Lastly, UNA outperforms RLHF/PPO on downstream tasks.

## 3 Mathematical Proof of UNA

In the section, the mathematical proof of UNA will be provided. For the proof of how to derive the mapping of optimal policy and reward model in DPO can be found in appendix [A](#A1). Inspired by the proof of DPO, we will rigorously prove that $r(x,y)=\beta\log\left(\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}\right)+f(x)+c$ will maximize the objective in Equation [3](#S2.E3) and $r(x,y)=\beta\log\left(\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}\right)$ is the simplest reward with $f(x)=c=0$.

Proposition 1. Let $a_{1},\ldots,a_{n}$ and $b_{1},\ldots,b_{n}$ be non-negative numbers. Denote the sum of all $a_{i}$ by $a$ and the sum of all $b_{i}$ by $b$. The log sum inequality states Equation [14](#S3.E14) with equality if and only if $\frac{a_{i}}{b_{i}}$ are equal for all $i$, in other words $a_{i}=\lambda\times b_{i}$ for all $i$. The proof could be found in [B](#A2)

$$ $\sum_{i=1}^{n}a_{i}\log\frac{a_{i}}{b_{i}}\geq a\log\frac{a}{b}$ (14) $$

Starting from the same objective in Equation [3](#S2.E3), it can be simplified as shown in Equation [15](#S3.E15).

$$ $\pi^{*}_{\theta}(y|x)=\max_{\pi_{\theta}}\mathbb{E}_{x\sim D}\left[\mathbb{E}_{y\sim\pi_{\theta}(y|x)}r_{\theta}(x,y)-\beta D_{\text{KL}}\left(\pi_{\theta}(y|x)\|\pi_{\text{ref}}(y|x)\right)\right]\break=\max_{\pi_{\theta}}\mathbb{E}_{x\sim D}\left\{\mathbb{E}_{y\sim\pi_{\theta}(y|x)}\left[r(x,y)-\beta\log\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}\right]\right\}\break=\beta\max_{\pi_{\theta}}\mathbb{E}_{x\sim D}\left\{\mathbb{E}_{y\sim\pi_{\theta}(y|x)}\left[\frac{1}{\beta}r(x,y)-\log\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}\right]\right\}\break=\beta\max_{\pi_{\theta}}\mathbb{E}_{x\sim D}\left\{\mathbb{E}_{y\sim\pi_{\theta}(y|x)}\left[-\log\left(\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)e^{\frac{1}{\beta}r(x,y)}}\right)\right]\right\}\break=\beta\max_{\pi_{\theta}}\mathbb{E}_{x\sim D}\left\{\mathbb{E}_{y\sim\pi_{\theta}(y|x)}\left[-\log\left(\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)e^{\frac{1}{\beta}\left(r(x,y)-f(x)\right)}}\right)+\frac{1}{\beta}f(x)\right]\right\}\break=\beta\max_{\pi_{\theta}}\mathbb{E}_{x\sim D}\left\{\mathbb{E}_{y\sim\pi_{\theta}(y|x)}\left[-\log\left(\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)e^{\frac{1}{\beta}\left(r(x,y)-f(x)\right)}}\right)\right]+\frac{1}{\beta}f(x)\right\}\break$ (15) $$

Based on the log-sum inequality in Equation [14](#S3.E14), the term can be further simplified as shown in Equation [16](#S3.E16) because both $\pi_{\theta}(y|x)$ and $\pi_{\text{ref}}(y|x)e^{\frac{1}{\beta}\left(r(x,y)-f(x)\right)}$ are non-negative.

$$ $\centering\beta\mathbb{E}_{x\sim D}\left\{\mathbb{E}_{y\sim\pi_{\theta}(y|x)}\left[-\log\left(\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)e^{\frac{1}{\beta}\left(r(x,y)-f(x)\right)}}\right)\right]+\frac{1}{\beta}f(x)\right\}\break=\beta\mathbb{E}_{x\sim D}\left\{-\sum_{y}\left[\pi_{\theta}(y|x)\log\left(\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)e^{\frac{1}{\beta}\left(r(x,y)-f(x)\right)}}\right)\right]+\frac{1}{\beta}f(x)\right\}\break\leq\beta\mathbb{E}_{x\sim D}\left\{\left[-\left(\sum_{y}\pi_{\theta}(y|x)\right)\log\left(\frac{\sum_{y}\pi_{\theta}(y|x)}{\sum_{y}\pi_{\text{ref}}(y|x)e^{\frac{1}{\beta}\left(r(x,y)-f(x)\right)}}\right)\right]+\frac{1}{\beta}f(x)\right\}\break=\beta\mathbb{E}_{x\sim D}\left\{\left[-1\log\left(\frac{1}{\sum_{y}\pi_{\text{ref}}(y|x)e^{\frac{1}{\beta}\left(r(x,y)-f(x)\right)}}\right)\right]+\frac{1}{\beta}f(x)\right\}\break=\beta\mathbb{E}_{x\sim D}\left\{\log\left(\mathbb{E}_{y\sim\pi_{\text{ref}}(y|x)}e^{\frac{1}{\beta}\left(r(x,y)-f(x)\right)}\right)+\frac{1}{\beta}f(x)\right\}\@add@centering$ (16) $$

As a result, the maximum value of the objective function $\max_{\pi_{\theta}}\mathbb{E}_{x\sim D}\left[\mathbb{E}_{y\sim\pi_{\theta}(y|x)}r_{\theta}(x,y)-\beta D_{\text{KL}}\left(\pi_{\theta}(y|x)\|\pi_{\text{ref}}(y|x)\right)\right]$ in eq [15](#S3.E15) is $\beta\mathbb{E}_{x\sim D}\left\{\log\left(\mathbb{E}_{y\sim\pi_{\text{ref}}(y|x)}e^{\frac{1}{\beta}\left(r(x,y)-f(x)\right)}\right)+\frac{1}{\beta}f(x)\right\}$ in Equation [16](#S3.E16), and this inequality reaches the equality condition when Equation [17](#S3.E17) is satisfied where $\lambda$ is a constant.

$$ $\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)e^{\frac{1}{\beta}\left(r(x,y)-f(x)\right)}}=\frac{1}{\lambda}$ (17) $$

By rewriting this term, we can obtain the reward in term of the policy as shown in Equation [18](#S3.E18). In special case, $f(x)=c=0$, it is simplified to $r(x,y)=\beta\log\left(\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}\right)$. The condition $f(x)=c=0$ refers that implicit and explicit reward models are exactly the same.

$$ $\displaystyle r(x,y)$ $\displaystyle=\beta\log\left(\frac{\lambda\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}\right)+f(x)$ (18) $\displaystyle=\beta\log\left(\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}\right)+f(x)+\beta\log(\lambda)$ $\displaystyle=\beta\log\left(\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}\right)+f(x)+c\text{ when }c=\beta\log(\lambda)$ $$

When plugging Equation [17](#S3.E17) in Equation [16](#S3.E16), the upper bound can be simplified into a constant $\beta\log(\lambda)+\mathbb{E}_{x\sim D}(f(x))$ as shown in Equation [19](#S3.E19).

$$ $\beta\mathbb{E}_{x\sim D}\left\{\log\left(\mathbb{E}_{y\sim\pi_{\text{ref}}(y|x)}e^{\frac{1}{\beta}\left(r(x,y)-f(x)\right)}\right)+\frac{1}{\beta}f(x)\right\}\\ =\beta\mathbb{E}_{x\sim D}\left\{\log\left(\mathbb{E}_{y\sim\pi_{\text{ref}}(y|x)}\frac{\lambda\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}\right)+\frac{1}{\beta}f(x)\right\}\\ =\beta\mathbb{E}_{x\sim D}\left\{\log\left(\mathbb{E}_{y\sim\pi_{\theta}(y|x)}\lambda\right)+\frac{1}{\beta}f(x)\right\}\\ =\beta\mathbb{E}_{x\sim D}\left\{\log\left(\lambda\right)+\frac{1}{\beta}f(x)\right\}\\ =\beta\log(\lambda)+\mathbb{E}_{x\sim D}(f(x))\\$ (19) $$

When desired to generalize this into ”infinite dimension”, another constraint needs to be added, i.e., $\sum_{y}\pi_{\text{ref}}(y|x)e^{\frac{1}{\beta}\left(r(x,y)-f(x)\right)}$ should be finite. Then, $f(x)$ is further restricted to $f(x)>\max[r(x,y)]$ with normalization on $r(x,y)$ in advance. Eventually, $\sum_{y}\pi_{\text{ref}}(y|x)e^{\frac{1}{\beta}\left(r(x,y)-f(x)\right)}<\sum_{y}\pi_{\text{ref}}(y|x)=1$, which will be finite.

Here is a brief summary of this section, based on this objective $\pi^{*}_{\theta}(y|x)=\max_{\pi_{\theta}}\mathbb{E}_{x\sim D}\left[\mathbb{E}_{y\sim\pi_{\theta}(y|x)}r_{\theta}(x,y)-\beta D_{\text{KL}}\left(\pi_{\theta}(y|x)\|\pi_{\text{ref}}(y|x)\right)\right]$ in Equation [3](#S2.E3), we can obtain its upper bound $\beta\mathbb{E}_{x\sim D}\left\{\log\left(\mathbb{E}_{y\sim\pi_{\text{ref}}(y|x)}e^{\frac{1}{\beta}\left(r(x,y)-f(x)\right)}\right)+\frac{1}{\beta}f(x)\right\}$ as shown in Equation [16](#S3.E16). The upper bound, i.e., the equality condition is reached when $r(x,y)=\beta\log\left(\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}\right)+f(x)+c$ as shown in Equation [18](#S3.E18). It can be further simplified to $r(x,y)=\beta\log\left(\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}\right)$ if $f(x)=c=0$. In particular, $f(x)=c=0$ refers the implicit reward equals to explicit rewards. Lastly, when the equality condition is reached, the upper bound would be $\beta\log(\lambda)+\mathbb{E}_{x\sim D}(f(x))$ as shown in Equation [19](#S3.E19).

## 4 Experiments

In this section, we will evaluate UNA on three types of experiments: improvement over KTO in binary feedback, RM/LLM distillation for score-based response and simplification to online RLHF. Equivalence of UNA to DPO can be shown mathematically for pairwise preference, and it will not be tested. For the first two test, ”mistralai/Mistral-7B-v0.1” Jiang et al. () is utilized as the policy and ”ultrafeedback-binarized-preferences-cleaned” Cui et al. () is utilized as the alignment dataset, which have a prompt, chosen and rejected responses with corresponding scores. The evaluation scores are from GPT-4, and it can regarded as a distillation of GPT-4. Low rank adaptation (LoRA) Hu et al. () is employed during the fine-tuning process with $r=32$, where $r$ denotes the ranks used in LoRA.

For the simplification of RLHF experiment, due to the computation availability and LoRA is not supported in PPOv2, "microsoft/phi-1_5" Li et al. () is utilized as the policy model and ”weqweasdas/RM-Gemma-2B” Dong et al. () is utilized as reward model. The prompts of the same Ultrafeedback dataset are utilized excluding the prompts longer than 1000 tokens. These experiments shows that UNA outperforms RLHF.

After alignment, two HuggingFace Open LLM Leaderboards are utilized to measure the performance. The new HuggingFace Leaderboard contains 6 tasks: bbh Suzgun et al. (), gpqa Rein et al. (), mmlu-pro Wang et al. (), musr Sprague et al. (), ifeval Zhou et al. () and math-hard Hendrycks et al. (). For all tasks, the average ”Value”s of all tasks are reported. The old Huggingface Leaderboard also contains 6 tasks: gsm8k Cobbe et al. (), truthful-qa Lin et al. (), winograde Sakaguchi et al. (), arc-challenge Clark et al. (), hellaswag Zellers et al. () and mmlu Hendrycks et al. (). In this work, the average match rate in gsm8k, mc2 in truthful-qa, acc in winograde, acc-norm in arc-challenge, acc-norm in hellaswag and acc in mmlu will be reported. Lots of hyperparameters are tested and can be found from Appendix [C](#A3) to Appendix [H](#A8). Only the best-performing hyperparameters for each method are reported in this section. When different hyperparameters are optimized for the two Leaderboards, the best-performing hyperparameters on the new Leaderboard are reported, as models on the old Leaderboard are more prone to reach a performance plateau.

### 4.1 UNA: Improvement over KTO in binary feedback & LLM/RM Distillation for score-based feedback

For binary feedback, borrowing the idea of KTO, the chosen responses are regarded as desired response with score ”+1” and rejected response are regarded as undesired response with score ”0”. In this way, the explicit scores are obtained. The generalized implicit rewards are firstly transformed into implicit reward scores, i.e., $s_{\theta}(x,y)=\sigma[r_{\theta}(x,y)]=\sigma\left[\beta\log\left(\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}\right)\right]$. Then, different loss functions including BCE and MSE are utilized to minimize the differences between implicit and explicit reward scores.

For score-based feedback, the scores in Ultrafeedback are generated by GPT-4, and they are utilized to align LLM. The initial score are from 1 to 5, and they are normalized to 0 to 1 in advance. These normalized scores are utilized as explicit scores. The same implicit reward scores as before are utilized. Because the explicit reward score is a continuous variable in the interval [0, 1], MSE is utilized as the loss function.

The results are shown in Table [1](#S4.T1) for the new HuggingFace Open LLM Leaderboard and Table [2](#S4.T2) for the old HuggingFace Open LLM Leaderboard. The highest scores for each metric and average are stressed in bold. For pairwise preference data, because UNA and DPO are mathematically equivalent, the alignment results should be exactly the same. For binary data, UNA has a better performance on the new Leaderboard, while KTO performs better on the old Leaderboard. As a result, UNA improves over KTO for binary data. Lastly, when score-based feedback, i.e., LLM distillation is provided, it greatly improves both Leaderboards as more information is provided. Consequently, when precise score-based information is accessible, it is essential to apply it.

**Table 1: The table shows the comparison of UNA with DPO, KTO considering pairwise, binary and score-based data on new HuggingFace Open LLM Leaderboard**
| Method | $\beta/lr$ | bbh | gpqa | mmlu-pro | musr | ifeval | math-hard | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Mistral | NA | 44.11 | 29.53 | 30.11 | 41.79 | 23.22 | 2.92 | 28.61 |
| DPO | 0.03 / $5e^{-6}$ | 45.15 | 31.56 | 30.28 | 42.42 | 27.29 | 2.45 | 29.86 |
| KTO | 0.01 / $5e^{-6}$ | 44.37 | 29.03 | 30.25 | 41.89 | 29.42 | 2.77 | 29.62 |
| UNA-binary (MSE) | 0.01 / $5e^{-6}$ | 45.47 | 30.55 | 30.34 | 42.82 | 27.95 | 2.96 | 30.02 |
| UNA-binary (BCE) | 0.03 / $5e^{-6}$ | 44.9 | 29.92 | 30.21 | 42.15 | 28.33 | 2.57 | 29.68 |
| UNA-score (MSE) | $0.03/1e^{-4}$ | 47.57 | 30.22 | 29.53 | 43.45 | 40.78 | 2.85 | 32.4 |

**Table 2: The table shows the comparison of UNA with DPO, KTO considering pairwise, binary and score-based data on old HuggingFace Open LLM Leaderboard**
| Method | $\beta/lr$ | gsm8k | truthful-qa | winograde | arc-challenge | hellaswag | mmlu | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Mistral | NA | 38.02 | 42.58 | 77.58 | 61.43 | 83.44 | 62.51 | 60.93 |
| DPO | 0.03 / $5e^{-6}$ | 42.46 | 46.28 | 79.08 | 63.31 | 84.35 | 62.76 | 63.04 |
| KTO | 0.01 / $5e^{-6}$ | 42.08 | 52.26 | 79.4 | 62.71 | 84.23 | 62.58 | 63.88 |
| UNA-binary (MSE) | 0.01 / $5e^{-6}$ | 42.5 | 48.42 | 79.87 | 62.46 | 84.09 | 62.66 | 63.33 |
| UNA-binary (BCE) | 0.03 / $5e^{-6}$ | 43.03 | 46.95 | 79.87 | 62.71 | 84.34 | 62.68 | 63.26 |
| UNA-score (MSE) | $0.03/1e^{-4}$ | 42.91 | 54.54 | 78.85 | 63.48 | 83.08 | 61.04 | 63.98 |

### 4.2 UNA: Simplification on online RLHF

For the comparison between RLHF and UNA, only prompts of Ultrafeedback are utilized. In RLHF, the prompts are sent to the policy for response generation, to the reward model for reward estimation and to the policy for update through PPO. In comparison, in UNA, the prompts are sent to the policy for response generation and implicit reward estimation, to the reward model for explicit reward estimation and to the policy for update through difference minimization like MSE between implicit and explicit rewards.

The comparison between RLHF and UNA is shown in Table [3](#S4.T3) and Table [4](#S4.T4). UNA outperforms RLHF in 10 out of 12 tasks, with some tasks improved over 25%. Overall, UNA beats RLHF in these two HuggingFace Open LLM Leaderboards (Average: 21.71% vs 16.78% and 50% vs 38.12%).

On the new Huggingface Open LLM Leaderboard, both RLHF and UNA underperform the Phi1.5 in some tasks because of alignment tax caused by the small 2B reward model Bai et al. (). In addition, UNA for RM / LLM distillation, which can been seen as an offline version of RLHF using GPT-4 as reward model has witnessed the overall improvements in the new HuggingFace Open LLM Leaderboard.

**Table 3: The table shows the comparison of UNA with RLHF using Ultrafeedback prompts on new Huggingface Open LLM Leaderboard**
| Method | $\beta/lr$ | bbh | gpqa | mmlu-pro | musr | ifeval | math-hard | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Phi1.5 | NA | 30.52 | 25.55 | 11.88 | 39.36 | 22.76 | 1.04 | 21.85 |
| RLHF | 0.05 / $3e^{-6}$ | 29.93 | 23.77 | 11.88 | 34.34 | 0.73 | 0 | 16.78 |
| UNA | 0.3 / $3e^{-6}$ | 33.71 | 24.98 | 16.26 | 34.32 | 20.4 | 0.56 | 21.71 |

**Table 4: The table shows the comparison of UNA with RLHF using Ultrafeedback prompts on old HuggingFace Open LLM Leaderboard**
| Method | $\beta/lr$ | gsm8k | truthful-qa | winograde | arc-challenge | hellaswag | mmlu | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Phi1.5 | NA | 1.63 | 38.95 | 53.28 | 28.33 | 49.56 | 24.08 | 32.64 |
| RLHF | 0.05 / $3e^{-6}$ | 0 | 44.43 | 67.01 | 39.85 | 50.48 | 26.93 | 38.12 |
| UNA | 0.3 / $3e^{-6}$ | 28.05 | 43.2 | 72.45 | 51.62 | 63.43 | 41.23 | 50 |

Last but not least, because UNA has transformed RLHF from a RL task into a supervised learning problem and it gets rid of the value model, the training memory and time cost are greatly reduced. The training time for 100,000 steps with 8 80G A100 is around 20 hours for RLHF and 5.5 hours for UNA with same batch size. The speed advantages of UNA over RLHF is obvious, and these advantages can be amplified with larger batch sizes for UNA, which is impractical for RLHF due to higher memory costs. In conclusion, with improved performances, more stable loss function, less memory cost and faster training speed, UNA outperforms RLHF from multiple perspectives.

## 5 Related Work

The field of LLM has been greatly revolutionized with billions of parameters, trillions of tokens in parallel during the pretraining stage OpenAI et al. (); Anthropic (); Team et al. (). After pretraining, SFT will be applied to enhance its capability on downstream tasks. However, both pretraining and SFT can not solve the bias and ethic problem of LLM as they exist in the pretriaing data OpenAI et al. (). To solve this problem, RLHF with PPO Ouyang et al. (); Bai et al. () have been proposed, and it is the mostly accepted method to align LLM including GPT and Claude. However, lots of problems exist for RLHF/PPO including large memory burden, unstability of RL and multiple stages of training, i.e. RM training and RL fine-tuning Rafailov et al. (). To decrease the cost of human labelling, AI feedback can be utilized to replace human feedback, which will be termed as RLAIF Bai et al. (); Lee et al. (). RLOO considers PPO an overkill for LLM alignment as LLM has been pretrained, and RLOO should be good enough Ahmadian et al. ().

To simplify RLHF, DPO is proposed to map the optimal policy and reward model, and the two stages can be merged into one step Rafailov et al. (). This process transforms the initial unstable RL into a binary cross entropy problem. DPOP Pal et al. () mathematically prove that during DPO, the reward of desired responses will go down and proposed a maximum term to prevent the rewards of desired responses going down. IPO discovered that under nearly deterministic condition between desired and undesired responses, the effectiveness of the KL divergence constraint imposed by $\beta$ diminished, potentially leading to overfitting, and they proposed a new loss term to prevent this problem Azar et al. (). sDPO proposed to divide given dataset into splits and use these splits to sequentially align the model will achieve performance than using all of them at once Kim et al. (). Iterative DPO argued that LLM can be both response generator and evaluator so that it can iterate and improve it continuously Yuan et al. (); Xu et al. (). TDPO provided an idea to provided reward to each token generation Rafailov et al. (); Zeng et al. ().

There have also been some works on merging SFT with alignment. ORPO proposed a new loss function to increase the ratio of desired responses over undesired responses to realize the goal of both SFT and alignment Hong et al. (). PAFT proposed to conduct SFT and alignment in parallel and merge them together afterward Pentyala et al. (). Some works, i.e., R-DPO Park et al. () and SimPO Meng et al. () have also discovered the verbose problem of LLM generation, and included some length control methods to reduce the length of generated responses while minimizing the impact of LLM performances.

The previous work focused on pairwise dataset, which was more tough to gather. In comparison, binary feedback like ”thumb up” and ”thumb down” will be easier to gather. KTO borrowed the idea of human aversion to desired over undesired data and it can handle binary feedback successfully Ethayarajh et al. (). DRO focused on binary data by estimating the policy and value functions and optimize each sequentially while maintaining the other fixed Richemond et al. (). However, there have not been a work that can unify both pairwise and binary feedback. Nash learning model the LLM improvement as a minmax problem and propose a iterative method to gradually approach the optimal solution Munos et al. (). It can solve the intranstivity problem of human preference. SPPO utilized one model as two sides of the competition Wu et al. (). Though Nash learning provides some hints, it will increase the time of alignment as it will increase the number of iteration before convergence.

LiPO Liu et al. (), RRHF Yuan et al. () and PRO Song et al. () utilized the ranking of a list of responses, and the relative score between these methods were utilized. RPO proposd to utilize KL divergence to mimize the difference between predicted reward and labelled reward by human or AI, which is closer to our idea in this work Nvidia et al. ().

Several problems exist on the current stage of alignment of LLM. To begin with, there has not been a work to unify RLHF and DPO. In addition, RLHF and RLAIF have been known complex, stable and memory intensive, but there have not been a work to revolutize the RL process. Last, though there have been some works on utilizing different types of data like pairwise, binary and list feedback, there have not been a universe method to unify them together. These problems are going to be solved in UNA.

## 6 Conclusion

Despite the trillions of tokens used to pretrain LLMs with billions of parameters, undesired responses persist. RLHF, DPO and KTO can improve the alignment quality. However, RLHF is tough to train due to high memory demands and instability. Though DPO greatly simplify RLHF, DPO cannot fully utilize preference data information and cannot build an explicit reward model for downstream tasks. KTO generalizes DPO to binary feedback. However, RLHF, DPO and KTO each have their own strengths and drawbacks, but they cannot be unified into a single approach. In this work, we propose UNA to integrate the benefits of RLHF, DPO, and KTO into a unified framework. Based on the RLHF objective, the optimal policy is induced by $r(x,y)=\beta\log\left(\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}\right)+f(x)+c$. When $f(x)=c=0$, the reward can be simplified to $r(x,y)=\beta\log\left(\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}\right)$. With this derived implicit reward function, it can be utilized to build UNA, which unifies RLHF, DPO and KTO as a task of minimization between implicit and explicit reward functions. As a result, UNA simplifies, stabilizes and reduces memory cost of RLHF. Downstream tasks demonstrate that UNA significantly outperforms RLHF, despite the alignment tax for smaller models. Then, UNA can deal with pairwise, binary and score-based feedback. For pairwise feedback, UNA is mathematically equivalent to DPO. For binary feedback, UNA can improve over KTO. For score-based feedback, UNA outperforms non-score-based methods including DPO and KTO, and it can be regarded as a distillation of RM and LLM or an offline RLAIF. In conclusion, UNA has proposed a new approach to LLM alignment.

## 7 Discussion

Even though UNA has made breakthroughs in 1. propose a general implicit reward function; 2. unify RLHF and DPO through minimization between implicit and explicit reward functions; 3. simplify, stabilize, reduce memory cost and improve performances of RLHF; 4. accommodate different types of data types: pairwise, binary and score-based feedbacks and 5. conduct multiple experiments on downstream tasks. However, several limitations of this works exist and they remain to be explored:

1. Due to the computation limitation, the current UNA experiments to replace RLHF were conducted on 1B-2B parameters policy and reward models, which will face ”alignment tax” problem. Further tests on larger-scale policy and reward models are needed to compare UNA and RLHF when the ”alignment tax” is reduced in larger models Bai et al. ().

2. The current UNA in replacing RLHF requires two stages. An elegant one-stage training of reward model and alignment should be pursued to further simplify this problem. In addition, the current works require separate reward model, which will cause great memory burden. The memory burden will be further reduced if the reference policy can be utilized to generate reward.

3. The general reward model is $r(x,y)=\beta\log\left(\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}\right)+f(x)+c)$. The appropriate construction of $f(x)$ may be able to further improve the performance of UNA. The costant $c$ can be estimated in advance or gradually updated through exponential moving average (EMA). These parameters estimation may further improve the performance of UNA.

4. Currently, only MSE and BCE are utilized to measure the difference between implicit and explicit reward functions. More measure metrics can be utilized to further improve its quality.

## 8 Default Notation

prompt to LLM

desired response

undesired response

the probability of desired response over undesired response

the explicit reward

the implicit reward

the explicit score: normalized explicit reward

the implicit score: normalized implicit reward

KL divergence

LLM policy to be aligned

reference policy for LLM alignment

any function that measures the difference between implicit and explicit reward functions

## Appendix A DPO: Relationship between optimal policy and reward function

The objective of RLHF / DPO is shown in Equation [3](#S2.E3). From the objective, the relationship between optimal reward and optimal policy can be derived in Equation [4](#S2.E4) where $Z(x)=\sum_{y}\pi_{\text{ref}}(y|x)e^{\left(\frac{1}{\beta}r_{\theta}(x,y)\right)}$. The illustration for deriving DPO is shown in Equation [20](#A1.E20).

$$ $\centering\pi^{*}_{\theta}(y|x)=\max_{\pi_{\theta}}\mathbb{E}_{x\sim D}\left[\mathbb{E}_{y\sim\pi_{\theta}(y|x)}r_{\theta}(x,y)-\beta D_{\text{KL}}\left(\pi_{\theta}(y|x)\|\pi_{\text{ref}}(y|x)\right)\right]\break=\max_{\pi_{\theta}}\mathbb{E}_{x\sim D}\left\{\mathbb{E}_{y\sim\pi_{\theta}(y|x)}\left[r(x,y)-\beta\log\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}\right]\right\}\break=\min_{\pi_{\theta}}\mathbb{E}_{x\sim D}\left\{\mathbb{E}_{y\sim\pi_{\theta}(y|x)}\left[\log\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}-\frac{1}{\beta}r(x,y)\right]\right\}\break=\min_{\pi_{\theta}}\mathbb{E}_{x\sim D}\left\{\mathbb{E}_{y\sim\pi_{\theta}(y|x)}\left[\log\left(\frac{\pi_{\theta}(y|x)}{\frac{1}{Z(x)}\pi_{\text{ref}}(y|x)e^{\frac{1}{\beta}r(x,y)}}\right)-\log\left(Z(x)\right)\right]\right\}\break=\min_{\pi_{\theta}}\mathbb{E}_{x\sim D}\left\{\mathbb{E}_{y\sim\pi_{\theta}(y|x)}\left[\log\left(\frac{\pi_{\theta}(y|x)}{\frac{1}{Z(x)}\pi_{\text{ref}}(y|x)e^{\frac{1}{\beta}r(x,y)}}\right)\right]-\log\left(Z(x)\right)\right\}\break=\min_{\pi_{\theta}}\mathbb{E}_{x\sim D}\left\{D_{KL}\left(\pi_{\theta}(y|x)\|\frac{1}{Z(x)}\pi_{\text{ref}}(y|x)e^{\frac{1}{\beta}r(x,y)}\right)-\log\left(Z(x)\right)\right\}\break\@add@centering$ (20) $$

The objective function is minimized when $D_{KL}\left(\pi_{\theta}(y|x)||\frac{1}{Z(x)}\pi_{\text{ref}}(y|x)e^{\frac{1}{\beta}r(x,y)}\right)=0$, and this is equivalent to $\pi_{\theta}(y|x)=\frac{1}{Z(x)}\pi_{\text{ref}}(y|x)e^{\frac{1}{\beta}r(x,y)}$. By rewriting, the reward model can be expressed in term of the current policy as shown in Equation [4](#S2.E4).

However, the term $Z(x)$ cannot be computed as it needed to be computed by summing all candidate responses $y$. DPO avoids this problem by subtracting the rewards of desired and undesired responses $r(x,y_{w})-r(x,y_{l})=\beta\left[\log\left(\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\text{ref}}(y_{w}|x)}\right)-\log\left(\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\text{ref}}(y_{l}|x)}\right)\right]$. In addition, the authors argue ”We say that two reward functions $r(x,y)$ and $r^{\prime}(x,y)$ are equivalent iff $r(x,y)-r^{\prime}(x,y)=f(x)$ for some function $f$”. However, rigorous proof cannot be provided and it is only provided that $r(x,y)$ and $r^{\prime}(x,y)$ induce the same optimal policy. For Lipo, $r(x,y)=\beta\log\left(\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}\right)$ is directly utilized as rewards for listwise responses and KTO estimates $Z(x)$ by averaging over multiple samples.

## Appendix B Derivation of log-sum inequality

Jesen inequality. For a real convex function $\varphi$, numbers $x_{1},x_{2},\ldots,x_{n}$ in its domain, and positive weights $a_{i}$, Jensen’s inequality can be stated as in Equation [21](#A2.E21):

$$ $\frac{\sum_{i=1}^{n}a_{i}\varphi(x_{i})}{\sum_{i=1}^{n}a_{i}}\geq\varphi\left(\frac{\sum_{i=1}^{n}a_{i}x_{i}}{\sum_{i=1}^{n}a_{i}}\right)$ (21) $$

Proof of log-sum inequality.
Firstly, define $f(x)=x\log(x)$. Then, $f^{\prime}(x)=1+\log(x)$ and $f^{\prime\prime}(x)=\frac{1}{x}$. For the domain $x>0$, $f^{\prime\prime}(x)>0$. As a result, $f(x)=x\log(x)$ is a concvex function and satisfy Jesen’s inequality. Then, the log-sum inequality could be derived in Equation [22](#A2.E22).

$$ $\displaystyle\sum_{i=1}^{n}a_{i}\log\left(\frac{a_{i}}{b_{i}}\right)$ $\displaystyle=\sum_{i=1}^{n}b_{i}f\left(\frac{a_{i}}{b_{i}}\right)$ (22) $\displaystyle=b\sum_{i=1}^{n}\frac{b_{i}}{b}f\left(\frac{a_{i}}{b_{i}}\right)$ $\displaystyle=b\frac{\sum_{i=1}^{n}b_{i}f\left(\frac{a_{i}}{b_{i}}\right)}{\sum_{i=1}^{n}b_{i}}$ $\displaystyle\geq bf\left[\frac{\sum_{i=1}^{n}b_{i}\frac{a_{i}}{b_{i}}}{\sum_{i=1}^{n}b_{i}}\right]$ $\displaystyle=bf\left(\frac{a}{b}\right)$ $$

## Appendix C DPO results on different β 𝛽 \beta

This section shows the results of DPO for pairwise datasets on different $\beta$ using learning rate of $5e^{-6}$ on two HuggingFace Open LLM Leaderboards.

**Table 5: The table shows the results of DPO for pairwise datasets on different $\beta$ that constrains the difference of the current policy and reference policy with learning of $5e^{-6}$ on new HuggingFace Open LLM Leaderboard**
| $\beta$ | bbh | gpqa | mmlu-pro | musr | ifeval | math-hard | average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 0.001 | 44.85 | 30.76 | 30.33 | 38.29 | 25.85 | 3.58 | 28.94 |
| 0.003 | 44.85 | 31.18 | 30.16 | 38.3 | 28.57 | 3.23 | 29.38 |
| 0.01 | 45.06 | 30.93 | 30.45 | 41.36 | 26.63 | 3.19 | 29.6 |
| 0.03 | 45.15 | 31.56 | 30.28 | 42.42 | 27.29 | 2.45 | 29.86 |
| 0.1 | 44.94 | 31.11 | 30.05 | 41.63 | 27.42 | 2.4 | 29.67 |
| 0.3 | 44.71 | 31.09 | 30.67 | 41.12 | 26.05 | 2.65 | 29.38 |
| 1 | 44.57 | 30.9 | 30.24 | 41.52 | 25.37 | 2.39 | 29.17 |
| 3 | 44.65 | 31.28 | 30.24 | 42.05 | 24.49 | 2.59 | 29.22 |

**Table 6: The table shows the results of DPO for pairwise datasets on different $\beta$ that constrains the difference of the current policy and reference policy with learning of $5e^{-6}$ on old HuggingFace Open LLM Leaderboard**
| $\beta$ | gsm8k | truthful-qa | winograde | arc-challenge | hellaswag | mmlu | average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 0.001 | 42.69 | 50.46 | 79.72 | 63.74 | 83.98 | 62.64 | 63.87 |
| 0.003 | 42.69 | 50 | 79.4 | 62.88 | 84.26 | 62.47 | 63.62 |
| 0.01 | 40.98 | 46.52 | 79.64 | 63.4 | 84.25 | 62.62 | 62.9 |
| 0.03 | 42.46 | 46.28 | 79.08 | 63.31 | 84.35 | 62.76 | 63.04 |
| 0.1 | 42.8 | 45.18 | 79.32 | 63.05 | 84.65 | 62.86 | 62.98 |
| 0.3 | 40.83 | 44 | 78.61 | 62.88 | 84.26 | 62.82 | 62.23 |
| 1 | 40.6 | 43.24 | 78.3 | 62.46 | 83.95 | 62.73 | 61.88 |
| 3 | 38.44 | 43.06 | 78.14 | 61.95 | 83.7 | 62.63 | 61.32 |

## Appendix D KTO results on different β 𝛽 \beta

This section shows the results of KTO for binary datasets on different $\beta$ using learning rate of $5e^{-6}$ on two HuggingFace Open LLM Leaderboards.

**Table 7: The table shows the results of KTO for binary datasets on different $\beta$ that constrains the difference of the current policy and reference policy with learning of $5e^{-6}$ on new HuggingFace Open LLM Leaderboard**
| $\beta$ | bbh | gpqa | mmlu-pro | musr | ifeval | math-hard | average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 0.001 | 43.91 | 29.26 | 30.26 | 39.61 | 27.11 | 2.98 | 28.86 |
| 0.003 | 45.09 | 29.4 | 30.29 | 40.81 | 28.77 | 2.91 | 29.55 |
| 0.01 | 44.37 | 29.03 | 30.25 | 41.89 | 29.42 | 2.77 | 29.62 |
| 0.03 | 44.83 | 30.72 | 30.39 | 40.44 | 27.57 | 3.27 | 29.54 |
| 0.1 | 44.65 | 29.52 | 30.44 | 41.63 | 25.55 | 2.31 | 29.02 |
| 0.3 | 45.14 | 29.17 | 30.58 | 41.37 | 27.25 | 1.73 | 29.21 |
| 1 | 44.87 | 30.06 | 30.35 | 41.65 | 24.39 | 2.58 | 28.98 |
| 3 | 44.6 | 29.58 | 30.4 | 41.79 | 23.83 | 2.19 | 28.73 |

**Table 8: The table shows the results of KTO for binary datasets on different $\beta$ that constrains the difference of the current policy and reference policy with learning of $5e^{-6}$ on old HuggingFace open LLM Leaderboard**
| $\beta$ | gsm8k | truthful-qa | winograde | arc-challenge | hellaswag | mmlu | average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 0.001 | 42.84 | 50.93 | 79.4 | 61.77 | 83.83 | 62.59 | 63.56 |
| 0.003 | 42.72 | 53.27 | 79.64 | 62.46 | 83.97 | 62.5 | 64.09 |
| 0.01 | 42.08 | 52.26 | 79.4 | 62.71 | 84.23 | 62.58 | 63.88 |
| 0.03 | 43.22 | 49.38 | 79.87 | 63.14 | 84.41 | 62.69 | 63.79 |
| 0.1 | 42.91 | 47.27 | 78.85 | 63.23 | 84.41 | 62.7 | 63.23 |
| 0.3 | 43.14 | 46.31 | 79.4 | 63.48 | 84.77 | 62.75 | 63.31 |
| 1 | 41.02 | 44.11 | 78.3 | 62.54 | 84.06 | 62.7 | 62.12 |
| 3 | 39.81 | 43.33 | 77.98 | 61.6 | 83.82 | 62.78 | 61.55 |

## Appendix E UNA binary feedback: MSE

This section shows the results of UNA with MSE loss for binary datasets function on different $\beta$ using learning rate of $5e^{-6}$ on two HuggingFace Open LLM Leaderboards.

**Table 9: The table shows the results of UNA-MSE for binary feedback on different $\beta$ that constrains the difference of the current policy and reference policy with learning of $5e^{-6}$ on new HuggingFace Open LLM Leaderboard**
| $\beta$ | bbh | gpqa | mmlu-pro | musr | ifeval | math-hard | average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| $\beta$ | bbh | gpqa | mmlu-pro | musr | ifeval | math-hard | average |
| 0.001 | 45.17 | 30.18 | 29.97 | 41.35 | 27.05 | 3.18 | 29.48 |
| 0.003 | 45.3 | 30.04 | 30.01 | 41.35 | 28.31 | 3.11 | 29.69 |
| 0.01 | 45.47 | 30.55 | 30.34 | 42.82 | 27.95 | 2.96 | 30.02 |
| 0.03 | 45.17 | 29.85 | 30.4 | 41.89 | 26.32 | 1.85 | 29.25 |
| 0.1 | 44.82 | 29.15 | 30.65 | 40.71 | 28.94 | 2.21 | 29.41 |
| 0.3 | 44.94 | 29.68 | 30.47 | 41.25 | 25.39 | 2.46 | 29.03 |
| 1 | 44.17 | 29.75 | 30.19 | 41.39 | 24.31 | 2.68 | 28.75 |
| 3 | 44.37 | 30.02 | 30.33 | 42.59 | 23.81 | 3.02 | 29.02 |

**Table 10: The table shows the results of UNA-MSE for binary feedback on different $\beta$ that constrains the difference of the current policy and reference policy with learning of $5e^{-6}$ on old HuggingFace Open LLM Leaderboard**
| $\beta$ | gsm8k | truthful-qa | winograde | arc-challenge | hellaswag | mmlu | average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 0.001 | 41.21 | 50.37 | 79.4 | 63.05 | 83.62 | 62.33 | 63.33 |
| 0.003 | 41.97 | 48.76 | 79.72 | 62.54 | 83.69 | 62.48 | 63.19 |
| 0.01 | 42.5 | 48.42 | 79.87 | 62.46 | 84.09 | 62.66 | 63.33 |
| 0.03 | 42.23 | 46.64 | 79.48 | 63.05 | 84.52 | 62.53 | 63.08 |
| 0.1 | 42.92 | 45.6 | 79.56 | 63.23 | 84.8 | 62.72 | 63.14 |
| 0.3 | 40.37 | 43.93 | 78.93 | 62.46 | 84.09 | 62.66 | 62.07 |
| 1 | 39.42 | 43.23 | 78.22 | 61.86 | 83.73 | 62.78 | 61.54 |
| 3 | 38.82 | 43.2 | 77.98 | 61.52 | 83.65 | 62.68 | 61.31 |

## Appendix F UNA binary feedback: BCE

This section shows the results of UNA with BCE loss for binary datasets function on different $\beta$ using learning rate of $5e^{-6}$ on two HuggingFace Open LLM Leaderboards.

**Table 11: The table shows the results of UNA-BCE for binary feedback on different $\beta$ that constrains the difference of the current policy and reference policy with learning of $5e^{-6}$ on new HuggingFace Open LLM Leaderboard**
| $\beta$ | bbh | gpqa | mmlu-pro | musr | ifeval | math-hard | average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 0.001 | 45.44 | 30.17 | 29.79 | 41.35 | 27.39 | 2.69 | 29.47 |
| 0.003 | 45.61 | 29.27 | 30.05 | 41.22 | 28.52 | 3.1 | 29.63 |
| 0.01 | 45.12 | 29.66 | 30.39 | 42.42 | 27.34 | 2.94 | 29.65 |
| 0.03 | 44.9 | 29.92 | 30.21 | 42.15 | 28.33 | 2.57 | 29.68 |
| 0.1 | 45.13 | 29.44 | 31.02 | 40.17 | 27.66 | 2.49 | 29.32 |
| 0.3 | 44.82 | 29.73 | 30.37 | 41.38 | 25.37 | 2.4 | 29.01 |

**Table 12: The table shows the results of UNA-BCE for binary feedback on different $\beta$ that constrains the difference of the current policy and reference policy with learning of $5e^{-6}$ on old HuggingFace Open LLM Leaderboard**
| $\beta$ | gsm8k | truthful-qa | winograde | arc-challenge | hellaswag | mmlu | average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 0.001 | 41.47 | 48.01 | 79.4 | 62.63 | 83.67 | 62.54 | 62.95 |
| 0.003 | 42.5 | 48.14 | 79.79 | 62.54 | 83.76 | 62.38 | 63.19 |
| 0.01 | 41.93 | 47.11 | 79.32 | 62.54 | 84 | 62.61 | 62.92 |
| 0.03 | 43.03 | 46.95 | 79.87 | 62.71 | 84.34 | 62.68 | 63.26 |
| 0.1 | 42.15 | 45.68 | 79.32 | 63.4 | 84.78 | 62.7 | 63.01 |
| 0.3 | 40.56 | 43.98 | 78.22 | 62.63 | 84.19 | 62.88 | 62.08 |

## Appendix G UNA: RM/LLM distillation

This section shows the results of UNA with MSE loss for score-based datasets function on different $\beta$ using learning rate of $5e^{-6}$ on two Huggingface Open LLM Leaderboards.

**Table 13: The table shows the results of UNA-RM/LLM distillation for score-based datasets on different $\beta$ that constrains the difference of the current policy and reference policy with learning of $5e^{-6}$ on new Huggingface Open LLM Leaderboard**
| $\beta$ | bbh | gpqa | mmlu-pro | musr | ifeval | math-hard | average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 0.001 | 44.67 | 30.44 | 29.9 | 42.82 | 26.36 | 3.05 | 29.54 |
| 0.003 | 45.27 | 31.03 | 29.85 | 43.08 | 26.25 | 2.53 | 29.67 |
| 0.01 | 45.46 | 30.38 | 30.08 | 42.55 | 26.95 | 2.88 | 29.72 |
| 0.03 | 45.18 | 30.65 | 30.28 | 43.48 | 27.68 | 3.08 | 30.06 |
| 0.1 | 45 | 30.17 | 30.74 | 41.89 | 28.88 | 2.45 | 29.86 |
| 0.3 | 44.75 | 31.06 | 30.44 | 41.11 | 27.43 | 1.97 | 29.46 |
| 1 | 44.94 | 31.23 | 30.28 | 41.12 | 23.92 | 2.8 | 29.05 |
| 3 | 44.75 | 31.38 | 30.22 | 41.26 | 22.94 | 2.73 | 28.88 |

**Table 14: The table shows the results of UNA-RM/LLM distillation for score-based datasets on different $\beta$ that constrains the difference of the current policy and reference policy with learning of $5e^{-6}$ on old HuggingFace Open LLM Leaderboard**
| $\beta$ | gsm8k | truthful-qa | winograde | arc-challenge | hellaswag | mmlu | average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 0.001 | 39.47 | 44.42 | 78.61 | 62.03 | 82.65 | 62.26 | 61.57 |
| 0.003 | 39.46 | 44.54 | 78.53 | 62.12 | 82.68 | 62.46 | 61.63 |
| 0.01 | 41.21 | 45.2 | 78.77 | 62.2 | 82.98 | 62.28 | 62.11 |
| 0.03 | 44.05 | 45.45 | 79.01 | 62.2 | 83.63 | 62.57 | 62.82 |
| 0.1 | 44.43 | 45.52 | 78.93 | 63.48 | 84.57 | 62.91 | 63.31 |
| 0.3 | 40.87 | 44.06 | 78.69 | 62.54 | 84.02 | 62.75 | 62.16 |
| 1 | 38.89 | 43.41 | 77.82 | 61.95 | 83.57 | 62.67 | 61.39 |
| 3 | 39.08 | 43.2 | 78.45 | 61.69 | 83.61 | 62.61 | 61.44 |

The original loss for UNA in binary feedback aims to increase the implicit rewards of desired responses to 1 and decrease the implicit rewards to 0. However, in UNA for score-based datasets, the targets to decrease the difference of implicit and explicit reward, and the explicit reward lies in [0, 1] rather 0, 1. As a result, the loss value will decrease. To further improve performances of alignment, larger learning rates are supposed to be utilized.

**Table 15: The table shows the results of UNA-RM/LLM distillation for score-based datasets on different $\beta$ that constrains the difference of the current policy and reference policy with learning of $5e^{-6}$ on new HuggingFace Open LLM Leaderboard**
| $\beta/lr$ | bbh | gpqa | mmlu-pro | musr | ifeval | math-hard | average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| $0.01/3e^{-6}$ | 44.99 | 30.7 | 30.23 | 42.04 | 25.61 | 2.54 | 29.35 |
| $0.01/1e^{-5}$ | 45.14 | 30.33 | 29.8 | 43.6 | 26.98 | 2.85 | 29.88 |
| $0.01/3e^{-5}$ | 47.05 | 30.98 | 29.55 | 41.33 | 34.38 | 3.21 | 31.08 |
| $0.01/1e^{-4}$ | 46.46 | 30.01 | 29.37 | 41.45 | 37.83 | 3.12 | 31.37 |
| $0.01/3e^{-4}$ | 42.46 | 26.94 | 24.24 | 43.43 | 40.18 | 1.46 | 29.79 |
| $0.03/3e^{-6}$ | 44.81 | 30.87 | 29.94 | 42.03 | 25.74 | 2.98 | 29.4 |
| $0.03/1e^{-5}$ | 46.12 | 29.99 | 29.9 | 41.19 | 30.13 | 2.71 | 30.01 |
| $0.03/3e^{-5}$ | 46.59 | 30.29 | 29.9 | 40.64 | 36.42 | 4.03 | 31.31 |
| $0.03/1e^{-4}$ | 47.57 | 30.22 | 29.53 | 43.45 | 40.78 | 2.85 | 32.4 |
| $0.03/3e^{-4}$ | 42.9 | 13.31 | 23.85 | 44.28 | 41.29 | 1.13 | 27.79 |
| $0.1/3e^{-6}$ | 45.13 | 30.35 | 30.24 | 41.64 | 27.05 | 2.6 | 29.5 |
| $0.1/1e^{-5}$ | 46.78 | 30.13 | 30.19 | 43.46 | 29.73 | 2.69 | 30.5 |
| $0.1/3e^{-5}$ | 46.29 | 29.92 | 29.83 | 43.73 | 31.8 | 3.37 | 30.82 |
| $0.1/1e^{-4}$ | 47.41 | 30.28 | 28.76 | 42.5 | 34.53 | 1.8 | 30.88 |
| $0.1/3e^{-4}$ | 41.57 | 30.15 | 24.09 | 37.76 | 33.43 | 1.71 | 28.12 |

**Table 16: The table shows the results of UNA-RM/LLM distillation for score-based datasets on different $\beta$ that constrains the difference of the current policy and reference policy with learning of $5e^{-6}$ on old HuggingFace Open LLM Leaderboard**
| $\beta/lr$ | gsm8k | truthful-qa | winograde | arc-challenge | hellaswag | mmlu | average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| $0.01/3e^{-6}$ | 39.77 | 43.91 | 78.69 | 61.77 | 83 | 62.41 | 61.59 |
| $0.01/1e^{-5}$ | 44.89 | 48.02 | 79.4 | 62.54 | 83.15 | 62.38 | 63.4 |
| $0.01/3e^{-5}$ | 46.36 | 52.02 | 79.16 | 62.8 | 82.67 | 62 | 64.17 |
| $0.01/1e^{-4}$ | 44.7 | 55.18 | 78.3 | 63.4 | 82.86 | 60.88 | 64.22 |
| $0.01/3e^{-4}$ | 29.01 | 55.01 | 74.11 | 56.57 | 79.3 | 53.66 | 57.94 |
| $0.03/3e^{-6}$ | 40.64 | 44.03 | 78.69 | 62.03 | 83.5 | 62.61 | 61.92 |
| $0.03/1e^{-5}$ | 45.26 | 49.51 | 79.4 | 63.4 | 83.8 | 62.5 | 63.98 |
| $0.03/3e^{-5}$ | 45.76 | 53 | 79.64 | 63.65 | 83.94 | 62.57 | 64.76 |
| $0.03/1e^{-4}$ | 42.91 | 54.54 | 78.85 | 63.48 | 83.08 | 61.04 | 63.98 |
| $0.03/3e^{-4}$ | 26.73 | 53.08 | 74.9 | 57.59 | 77.65 | 53.15 | 57.18 |
| $0.1/3e^{-6}$ | 42.84 | 44.31 | 78.69 | 62.71 | 84.17 | 62.81 | 62.58 |
| $0.1/1e^{-5}$ | 43.94 | 49.65 | 80.74 | 65.1 | 84.87 | 62.89 | 64.53 |
| $0.1/3e^{-5}$ | 44.92 | 53.33 | 80.11 | 63.91 | 84.47 | 62.28 | 64.84 |
| $0.1/1e^{-4}$ | 39.09 | 52.68 | 78.06 | 63.14 | 81.77 | 60.11 | 62.48 |
| $0.1/3e^{-4}$ | 26.23 | 54.27 | 75.69 | 56.57 | 77.53 | 52.59 | 57.18 |

## Appendix H UNA: Simplify RLHF

This section shows the comparison between UNA and RLHF on different $\beta$ using learning rate of $3e^{-6}$ on two HuggingFace Open LLM Leaderboards.

**Table 17: The table shows the comparison UNA with RLHF using ultrafeedback prompts on different $\beta$ that constrains the difference of the current policy and reference policy with learning of $3e^{-6}$ on new HuggingFace Open LLM Leaderboard**
| $\beta$ | bbh | gpqa | mmlu-pro | musr | ifeval | math-hard | average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| $1e^{-4}$ | 32.92 | 23.06 | 16.43 | 34.3 | 7.49 | 0.98 | 19.2 |
| $3e^{-4}$ | 32.14 | 24.13 | 15.25 | 35.36 | 4.65 | 0.44 | 18.66 |
| 0.001 | 33.51 | 24.28 | 15.45 | 37.6 | 6.11 | 1.18 | 19.69 |
| 0.003 | 30.7 | 25.42 | 13.99 | 32.57 | 9.78 | 0.25 | 18.79 |
| 0.01 | 29.48 | 24.83 | 11.21 | 38.73 | 20.6 | 0 | 20.81 |
| 0.03 | 31.38 | 28.78 | 14.58 | 36.05 | 16.81 | 0.17 | 21.3 |
| 0.1 | 32.66 | 26.97 | 15.58 | 32.19 | 18.85 | 0.63 | 21.15 |
| 0.3 | 33.71 | 24.98 | 16.26 | 34.32 | 20.4 | 0.56 | 21.71 |
| 1 | 33.45 | 24.44 | 15.18 | 33.11 | 21.6 | 0.24 | 21.34 |
| 3 | 32.84 | 26.48 | 15.83 | 33.53 | 21.01 | 0 | 21.62 |

**Table 18: The table shows the comparison UNA with RLHF using ultrafeedback prompts on different $\beta$ that constrains the difference of the current policy and reference policy with learning of $3e^{-6}$ on old HuggingFace Open LLM Leaderboard**
| $\beta$ | gsm8k | truthful-qa | winograde | arc-challenge | hellaswag | mmlu | average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| $1e^{-4}$ | 7.96 | 40.3 | 71.9 | 52.73 | 61.7 | 42.76 | 46.23 |
| $3e^{-4}$ | 1.14 | 39.19 | 72.61 | 51.54 | 61.68 | 41.07 | 44.54 |
| 0.001 | 22.56 | 40.68 | 73.16 | 50.85 | 61.4 | 38.9 | 47.93 |
| 0.003 | 8.8 | 42.02 | 69.85 | 50.43 | 60.45 | 38.75 | 45.05 |
| 0.01 | 0 | 46.18 | 65.82 | 28.33 | 28.13 | 24.65 | 32.19 |
| 0.03 | 24.22 | 36.01 | 70.09 | 50.85 | 65.31 | 40.18 | 47.78 |
| 0.1 | 28.85 | 42 | 71.59 | 51.02 | 64.36 | 39.97 | 49.63 |
| 0.3 | 28.05 | 43.2 | 72.45 | 51.62 | 63.43 | 41.23 | 50 |
| 1 | 24.87 | 43.25 | 72.38 | 50.77 | 61.97 | 39.77 | 48.84 |
| 3 | 27.94 | 41.68 | 71.03 | 49.83 | 62.72 | 41.16 | 49.06 |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 3
presentation: 2
questions: 1. The paper states that $f(x)=0$ when implicit and explicit reward models are exactly the same. However, when we optimize a model using UNA with the simplified implicit reward $f(x)=c=0$, will it guarantee that it will lead to the convergence?
2. DPO also use log pi / pi_ref as an indicator of the reward. However, this implicit reward cannot well reflect the training status. Can the UNA framework better stabilize the training process in the DPO manner?
3. In Table 3, maybe you can try other reference models rather than GPT in Alpaca-eval? The win rates against GPT-4 is to low and difficult to compare among different models.
4. Can you include the results of the original DPO?
soundness: 2
strengths: 1. By combining RLHF, DPO, and KTO, UNA supports a range of feedback types (pairwise, binary, and score-based), which enhances its versatility.
2. UNA replaces RLHF’s unstable, memory-heavy RL process with supervised learning, making the alignment process more straightforward and efficient. 
3. The paper provides a mathematical proof linking the RLHF objective to a generalized implicit reward function.
summary: This paper introduces the UNA framework, which integrates three prominent alignment techniques for LLMs: RLHF/PPO, DPO, and KTO. The UNA framework seeks to unify these methods by using a generalized implicit reward function to align language model policies through supervised learning. UNA also accommodates diverse feedback types (pairwise, binary, and score-based) and aims to outperform each technique on downstream tasks, simplifying the RLHF fine-tuning process.
weaknesses: 1. Evaluation of the experiments is relatively small-scale. The authors only evaluate UNA over DPO & KTO when fine-tuning with LoRA and evaluate online RLHF using a 1.5B model. It may be difficult to prove whether UNA can take advantage when using large-scale models. I suggest the authors to try larger models like ~8B models since they claim that they use 8 x 80G A100 GPUs. 
2. The improvement of UNA over three baselines are not significant, which is considerate since they are somewhat equivalent. However, I think the authors should show the advantages of UNA, e.g., the time and computational cost, compared to baselines. 
3. The paper seems written in a rush with several clarity issues. For example, some references (Line 215 and 352) are missing.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 1
questions: 1. What is the intuitive explanation for why a unified approach provides better results? DPO provides better results than RLHF as it does direct optimization of preferences, rather than first creating a reward function and then optimizing. In a similar vein, is there an intuitive explanation. 
2. There are errors in the main proof in Appendix B. While I seem to follow the broad outline, I am not entirely sure and would request the authors to provide more explanation and a corrected version. "y" is present on the left hand side, there is a "y" used in expectation. I am not sure how the expectation over "y" makes sense?
3. Are the experimental improvements significant? The improvements are there, but are they significant? How can it be quantified?
soundness: 3
strengths: 1. The unified framework does seem to have significant advantages.
2. The paper's ideas are strong.
3. Experimental results are detailed and strong.
summary: This paper provides a unifying framework for value alignment that has:
(i) explicit and implicit reward functions
(ii) can handle different types of preferences (pairwise, binary, score etc.)
and seems to provide better performance than state of the art approaches, RLHF, DPO, KTO.
weaknesses: 1. The biggest concern I have is that presentation is extremely poor, with many typos in notation, proofs, text and references. 
2. There are notational inconsistencies in proofs as well.
3. There is no intuitive explanation for why such a unified approach would provide better performance than any of the individual approaches. 

I find that the paper is written in a rushed manner and even though the ideas are interesting, it does not seem ready yet with typos all over the paper. Happy to revise my scores based on the responses of the authors.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 1
questions: It seems to me UNA also needs a reward model to learn the policy, which is not too different from RLHF. But in your abstract, "RLHF requires training the reward model and policy separately, which is complex, time-consuming, memory intensive and unstable during training processes." Can you clarify the advantage of UNA compared to standard RLHF? For example, it could be that UNA saves some computation, or converges faster, etc.
soundness: 1
strengths: 1. There are math derivations for the proposed method.
2. The reward distillation loss is  general for different modes of feedback.
summary: This paper presents an algorithm which they call UNified Alignment (UNA), by utilizing a simplified DPO implicit reward. UNA's loss minimizes the difference between the implicit reward and the explicit reward (provided by an existing reward model), hence can be generalized to different modes of feedback: pairwise, binary or scalar. Empirical results are presented to compare UNA with several baselines.
weaknesses: W1. The paper's presentation does not meet the standard of ICLR. 
* Some references are missing (215,353); there are grammar and spelling errors (026,106,120,141). 
* Figures 1 and 2 are almost identical except for the different parts highlighted, which makes me feel they are redundant. You could refine the two figures and make them respectively focused on the topic you want to highlight: algorithm comparison & applications to different modes of feedback.
* The overall tune is not formal, and the writing is not easy to follow (too many pieces). If your goal is to develop a unified framework for LLM alignment, in your methodology you would start from defining the overall process of alignment, then the objectives of it, followed by your approach. Maybe at the end you can make connections with existing methods like RLHF with PPO, DPO, KTO and say they are some special cases. This suggestion is only one possible way for improvement. There are certainly more that you could consider to improve the current version.

W2. The technical novelty is limited. The implicit reward (eq6) of DPO is standard and well-known. One of the contributions is "simplifying" the implicit reward by setting f(x)=c=0. Could you provide insights behind it? Besides, the other contribution is a supervised loss of minimizing the difference between implicit and explicit rewards (eq7) is also not novel. For example, [1] proposed such a distillation-style loss for robust preference optimization.


[1] Fisch A, Eisenstein J, Zayats V, Agarwal A, Beirami A, Nagpal C, Shaw P, Berant J. Robust preference optimization through reward model distillation. arXiv preprint arXiv:2405.19316. 2024 May 29.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: Please refer to weakness part.
soundness: 2
strengths: It is of interest that the authors try to narrow the gap between the RM-based and RM-free alignment methods.
summary: This paper introduces a method called Unified Alignment (UNA), which aims to unify existing alignment techniques like RLHF, DPO, and KTO. The authors argue that RLHF, though effective, suffers from high complexity and instability, while DPO simplifies the process but is limited in certain aspects, such as not fully leveraging the reward model and being constrained to pairwise preference data. The proposed UNA framework claims to generalize and improve upon these methods by creating a novel mapping between reward models and optimal policies, allowing it to handle different feedback types and improve performance across downstream tasks. The paper suggests that UNA can simplify, stabilize, and speed up fine-tuning processes while reducing memory requirements, and it reports outperforming RLHF, DPO, and KTO in experiments.
weaknesses: The paper is not solid enough for me and the contribution is limited. The reasons are concluded as follow:

First, It is not clear to me why the proposed method differs from DPO, thus the first contribution claimed is not novel at all. In my understanding, the DPO has already proved the equivalent between the optimal policy and the reward model's objective (under the BT model setting). Besides, it is also very confusing why the authors introduce f(x) and c.

Second, for the claimed contribution 2, the author used a objective introduced by DPO model to conduct a supervised learning, which is also confusing. What is the difference between using the proposed method to train an additional model and directly using a well-trained DPO model? And what if the DPO model is not well-trained, then the label would be very noisy? It is a very confusing step and I am open to more discussion.

Third, the proposed method is compared to some baselines and is claimed to be better, however I believe the comparisons are not fair and the experimental results are suspicious.  In Table 1, the authors use 4 UNA variants and only use KTO and Mistral as the baselines, and the best scores in each benchmark are distributed to all 4 variants, so which one is the best? The authors also claim UNA perform better than RLHF while they didn't provide the experimental detail about RLHF. The proposed method is not even a RL algorithm since it doesn't shown any potential in exploration but more like a distillation process, it is meaningless to announce that it perform better than PPO regarding speeds or hardware consuming.

The presentation of the paper needs to be largely improved. There are many compile Errors such as line 215 in the 4th page. Given all these reasons, I will not recommend the paper to be accepted.