A density estimation perspective on learning from pairwise human preferences
Authors: Vincent Dumoulin
Daniel D. Johnson
Pablo Samuel Castro
Hugo Larochelle
Yann Dauphin
Venue: Accepted by TMLR ()

Abstract: Learning from human feedback (LHF)—and in particular learning from pairwise preferences—has recently become a crucial ingredient in training large language models (LLMs), and has been the subject of much research. Most recent works frame it as a reinforcement learning problem, where a reward function is learned from pairwise preference data and the LLM is treated as a policy which is adapted to maximize the rewards, often under additional regularization constraints. We propose an alternative interpretation which centers on the generative process for pairwise preferences and treats LHF as a density estimation problem. We provide theoretical and empirical results showing that for a family of generative processes defined via preference behavior distribution equations, training a reward function on pairwise preferences effectively models an annotator's implicit preference distribution. Finally, we discuss and present findings on "annotator misspecification"—failure cases where wrong modeling assumptions are made about annotator behavior, resulting in poorly-adapted models—suggesting that approaches that learn from pairwise human preferences could have trouble learning from a population of annotators with diverse viewpoints.

Published in Transactions on Machine Learning Research (02/2024)
A density estimation perspective on learning from pairwise human preferences
Vincent Dumoulin, Daniel D. Johnson'! Pablo Samuel Castro ! Hugo Larochelle 1M
Yann Dauphin
(udumoulón,ddjjae
Google Deep
University of Toronto
Mila, Université de Montréal
CIFAR Fellow
Reviewed on OpenReview: https: http::///penrevicee net/ formmo id= YF30ERVY,F
Abstract
Learning from human feedback (LHF))and in particular learning from pairwise preferences has recently become a crucial ingredient in training large language models (LLMs), and has been the subject of much research. Most recent works frame it as a re- inforcement learning problem, where a reward function learned from pairwise preference data and the LLM is treated as policy which adapted maximize the rewards, often under additional regularization constraints. We propose alternative interpretation which centers on the generative process for pairwise preferences and treats LHF density esti- mation problem. We provide theoretical and empirical results showing that for family of generative processes defined via preference behavior distribution equations, training reward function on pairwise preferences effectively modols an annotator's implicit preference dis- tribution. Finally, we discuss and present findings on "annotator misspecification" failure cases where wrong modeling assumptions are made about annotator behavior, resulting in poorly-adapted models suggesting that approaches that learn from pairwise human prefer- ences could have trouble learning from population of annotators with diverse viewpoints
1 Introduction
With the recent surge of interest in large language models (LLMs), learning from human feedback (LHF) has received a great deal of attention from the machine learning community. Pretraining of large Transformer architectures (Vaswani et al.) 2017)) web-scale data has been essential to their success, but by itself the pretraining objective produces unconditional generalive models that are dilficult to control. A critical component in training LLMs is therefore to finetune them to produce "good" responses, which is typically formalized as aligning their outputs with human preferences.
The current dominant approach to aligning LLMs with human preferences relies on pairwise comparisons of model outputs, and frames the problem a reinforcement learning (RL) problem where the LLM is treated
as policy. Preference- based learning of reward functions (Sadigh et al. 2017 Biyik & Sadigh
2018 and
more specifically preference-basee
RL RL from human preferences (abbreviated
RLHF; Wirth et al.
2017 Christiano et al. 2017 Tbarz et al. 2018 Brown et al. 2019 2020
Tee et al. 2021 Shin et al. 2023)
is used to finetune LLMs from model output pairs which have been ranked by annotators to reflect how
preferable those outputs are (Stiennon et al. 2020 Ouyang et al. 2022 Bai et al. 2022).
However, interpreting preference-aligmeent as an RL problem requires a change of perspective from the original pretraining task. For LLM pretraining, the loss derived from a probabilistic modeling formulation
A notebook reproducing all experiments in this paper can accessed

---

Published in Transactions on Machine Learning Research (02/2024)
of the problem: The language model is treated an autoregressite probability distribution, and next-token prediction is used to maximize the likelihood of human language samples. For human preference alignment, the language model is treated as a policy and the loss aims to maximize a (learned) reward which is reflective of human preferences. Given the probabilistic origins of training LLMs, it is worth considering whether cerlain probabilistic modeling tools and techniques are being overlooked when taking the RL perspective.
We argue that treating human preference alignment as probabilistic modeling task forces us to be explicit about our assumptions on the generative process for annotator preferences and provides important insights into the properties and failure modes of algorithms that learn from them. Our main contributions are:
We show that the standard procedure for training a reward function on pairwise human prefer- ences can be reinterpreted as performing donsity cstimation on an annotator's 's implicit preference distribution, assuming annotators behave according to the Luce choice rule (Luce et al. 1963
We generalize the Luce choice rule to family of generative processes for pairwise preferences defined via preference behavior distribution equations (PBDEs) and provide global optimality results in the case where the annotator and model behave according to the same PBDE;
Finally, we show that an unintentional mismatch in generative processes for pairwise proferences between the annotator and model (which we term annotator misspecification) can lead to badly- tuned policies, highlighting the importance of explicitly stating assumptions on annotator behavior.
2 Background
Here we provide some technical background and notation which will be used throughout the manuscript.
2.1
Reinforcement Learning
Reinforcement lcarning tackles sequential decision- decision-making problems and is typically formulated as Markov
decision process M = A, P, R, 7} using the following definitions: X' is the set of states; A is a set. of actions (or "one-step decisions")) "); P X x A + A(X) defines the transition dynamics, where P(x, (x'') is the probability of transitioning to state x' after performing action from state x; R X A - IR is the one-step reward, where R(x,, a) is the reward received after performing action a from state x; 7 [0, 1) is a discount factor that places more emphasis on rewards received in the short-term.
An agent's behaviour is formalized via a policy π X + A(A) which gives distribution over possible actions conditioned on a state (e.g., (a))(a) is the probability given by policy л of picking action a when in state æ). Given a fixed policy π, we can express rewards as a function of state: (=(x) == Ba-x(s) R(x,, R((, a); this notation will prove useful when considering reward functions over LLM outputs. Every policy स induces a value function given by the expected sum of discounted rewards when following policy π.
(T(x)) Σ (R((z,,))) (r) xo x, T(Ee), P( at)
It is uscful to also consider state-action value functions, which quantify the value of taking an action from state x, and following the policy π afterwards: Q" a) = R(x, a) + +7Bxww((e.) V (x²). Finally, we denote by Mx E (X) the stationary state distribution induced by a policy n.
The goal of RL agents is then to find the optimal policy n", where optimality defined uniformly across the state space (e.g. V* > V* for every T). In most modern RL applications, policies are expressed by neural networks with parameters 0, and we denote the respective policy To. There number of approaches for tackling this problem (Sutton & Barto 1998), but for the purposes of this work we focus policy gradient methods. Specifically, the policy gradient Theorem asserts the following relationship:
VVTU Σ Mao (x) *E
Σ (1)
acA
2

---

Published in Transactions on Machine Learning Research (02/2024)
This result establishes that can use gradient-based optimization methods to maximize returns by only requiring gradients of the parameterized policy тT. Therefore we can consider the output of any neural network (such as an LLM) as policy, and we can then update this policy so as to maximize some reward fanction of interest. This observation what underlies RLHF, which we discuss in the next section.
2.2
Reinforcement Learning from Human Feedback (RLHF)
RLHF works in two stages: (1) a reward function is learned from the annotator preference data; and (2) the LLM is treated as an RL policy and finctuned to maximize the learned reward function. The two stages are usually alternated in multiple iteralions of closed loop where annotator feedback is gathered bet each iteration.
More formally, the annotator ranks pairs of model outputs Za and æB, and the outcome can be represented with a binary variable that takes the value otherwise. In the context language modeling, the reward function is usually defined as a parameterized mapping T0( v) from prompt 26 and continuation æ to a scalar reward. To simplify the notation and treatment, we will omit zy and consider the unconditional generation case, with the understanding that the discussion extends trivially to the conditional generation case. The reward function ra( is trained dataset D of annotator preferences:
if æ is preferred to nn (noted Φ > xn) and = 0
① min ERA
- - (.-ogpo((::x..aa) - (1 - y) loog(1 - (
(1)
where
(ro(ma)
Po (A, = o (To((A)) - -r((0))) e'olax)) + cra((nn
(2)
is the probability that æA > BB under the Bradlcy-Terry model (Bradley & Terry 1952 for pairwise comparisons. In plain language, the reward function is trained using binary cross-entropy loss the comparison outcome variable y, which incentivizes increasing the reward margin between the preferred and non-preferred outputs. Then, the LLM (noted TT(x)) is tuned so that
<- max Emo(()) ro(()))
(3)
Using the well-known identity for policy gradient in RL, this results in a gradient of the form
VelE
ro(x)) (e))
Volg
(4)
In order to prevent "catastrophic forgetting" in the model, is common to combine the learned reward with KL-divergence term between the finetuned LLM and the pretrained LLM (noted Tppe) practice known as KL-comtrol-yieldnn a gradient of the form
Ex((a)) (ro(ae) (ro( -Blog no(x;) )pre(x))
Vo log To(2))
Such a constraint is usually handled using policy gradient mothods such as proximal policy optimiza- tion (PPO; Schulman al. 2017).
2The choice of notation from fact that treated as policy for purpose RLHF.
3

---

Published in Transactions on Machine Learning Research (02/2024)
3 Related work
Our work is related to numerous previous works investigating alternatives to RL to learn from pairwise human preferences.
Liu et al. (2023a] move away from RL and propose a technique called Chain of Hindsight which turns annotator-rant rraked pairs of model outputs and their associated contexts into training sequences of the form {Context] {Positive prefix]: (Desirable output} {Negative prefix]: {Undesirable output} The model is finctuned into these chain-of-hindsight sequences and then conditioned inference time by appending " "[Positive prefix)" to the annotator prompt.
Korbak et al. (0022) examine the relationship between reward maximization approaches like RLHF (with or without KI-control) and distribution matching approaches like distributional policy gradients (DPG). They show that RLHF with KL-control can also be thought of as a distribution matching approach, as it is equivalent to minimizing the reverse KL-divergence between Tg(a) and an energy-based model of the form (2(x) x prre (2)e Our work complements their analysis by characterizing what the optimal reward function captures about the annotator (their implicit preference distribution) under various hypothesized generalive processes for pairwise preferences.
Yuan al. (0023) aligu the model with pairwise human preferences through combination of ranking loss on the log- likelihood margin between preferred and non-preferred model outputs and a negative log- ligllikeihhod loss on the preferred model output. Zhao et al. 20233 optimize for pairwise preference aligmment. using a hinge loss on the log-likelihood margin between preferred and non-preferred model outputs and a regularization term towards a model finetuned with supervision. Liu et al. (2023b) extend the loss introduced by Zhaoo et al. 2023) by normalizing the lieeliilldood margin using Trre (z). However, their primary contribution is a statistical rejection sampling algorithm called RSO that draws samples from the optimal policy using samples from (T) and learned reward function.
Rafailov et al. (023) show an equivalence between RLHF with KL-control and binary classification of pairwise comparison outcomes when the reward function used to make a prediction (Equation 2) is defined as r(x)- = (log (=(x)- log (x)) (called DPO, for direct preference optimization). Azar et al. (2023)
frame DPO as a special case of a broader family of approaches called vPO where the maximized quantity is a function Ve of the preference probability defined by the Bradley-Terry model. They introduce another special case called identity preference optimization (IPO) which is optimized via quadratic loss instcad of sigmoid binary cross-entropy loss and is meant to make KL-regularization more elfective in the face of deterministic annotator preferences. Whereas Azar et al. 2023) consider various transformations of the preference probability for the optimal reward under the Bradley-Tery model, we systematically use the logit function on the preference probability and instead examine various transformations of log (< to obtain the preference probability itself.
4 A probabilistic interpretation of learning from pairwise human preferences
We now consider probabilistic interpretation of LHF in the case where the training signal originates from pairwise preferences. First, we need to consider the generative process for the preferences and make explicit assumptions about it.
4.1 Reward learning as density estimation under the Luce choice rule
Let (y; AA EB) be an annotator's preference probability whose functional form we will make more explicit shortly and encapsulates our assumptions on the generative process for pairwise human preferences. Tet. A BB) be proposal distribution over pairs æA and &B to be ranked by the annotator. One common choice of proposal is to draw independent sample pairs from (x). Using the two distributions we can decompose the true distribution D of preference data used in the expectation in Equation into
D(E FB a xn) 50 (y; x #B))
(6)
4

---

Published in Transactions on Machine Learning Research (02/2024)
and we can rewrite Equation
+ min
E =()
(y:a
(og (1 y) -logoog1 P Po (1
(7)
which is equivalen
min Be DKL (p
(8)
and is globally minimized when the KI-divergence wero for all possible æA and æn pairs.
The correct assumption to make about the generative process for pairwise human preferences essentially empirical question, but reasonable hypothesis is model of decision processes known as the Luce choice rule (or ratio rule; Shepard 1957 Luce et. al. 1963). As Sanborn & Grilliths (2007) and Peterson et al. (2018) explain, a common assumption about the behavior of rational Bayesian subjects is that they make choices according to some implicit preference distribution p p²(x:) over outcomes: for any two outcomes æA and BB, they are assumed to prefer DA over Øg with probability
p² (xAA
Prob(æA FB) (*(xA) p" (x
This model also known as the Bradley-Terie model BBaddey & Terry 1952 In fact, Sanborn & Griffiths (2007) and Peterson et al. (2018) use this very assumption to place human subjects the core of a Monte Carlo Markov Chain (MCMC) by having them perform the accept/ /rejectt/ttttttee step and ultimately drawing samples from their implicit preference distribution p" (a:) over outcomes.
Equation 9 and Equation share the same functional form for reason: if we assume the generative process for pairwise human preferences follows the Luce choice rule, then predicting pairwise comparison outcomes using the same generative process results in a well-specified model of annotator behavior.
Let p" (xx be a probability distribulion with support S, and lel q( be a joinl probability distribution with support S S. Assume q( FB) for all A, aB S x S. Let po x xB) and p" (y: AA FB) be defined according to Equations and respectively. The loss function
Theorem 1.
E EE #Ba
DKL (p" (u: po RA (1,
is globally minimized when
cola) xp(()), Ve E S.
A proof provided in Appendix A.I We can therefore reinterpret the training of a reward function on pairwise human preferences as the following probabilistic procedure:
We assume the generative process for pairwise human preferences follows the Luce choice rule (Equa-
tion 9 and consider annotator preferences as samples from this process;
2. We predict comparison outcomes with the learnable reward function using the same generative process [Equation 2);
3. We use maximum-likellhood criterion to optimize the reward-derived preference probability (quaa tion 8) and backpropagate the binary cross-entropy loss through Equation 2 and into the reward function itself; and
Recall that the entropy of p* (-) does depend

---

Published in Transactions on Machine Learning Research (02/2024)
4. Provided the Luce choice rule assumption made in step holds, the resulting optimal reward function is then equal to the annotator's implicit log-preference distribution (up to some additive constant).
To demonstrate this, consider univariate implicit preference distribution
p" (z) -Viruncatedd -2.5, 0.25) (Mruncated 2.5, 1.0)
(10)
defined over the [-10, 10] interval Figuur 1 dashed blue). We simulate learning from pairwisc prefer- ences by drawing 2 observation pairs ø and n uniformly at random in [-10, 10] x [-10, 10] and drawing corresponding comparison outcomes y us- ing Equation 9 We then train reward function in the form of four-layer MLP (refer to Table for details). Finally,
Table Univariate toy experiment hyperparameters.
Hyperparameter
Value
Architecture Hidden layers
MLP
4 layers of width 64 tanh
approximate the normalizing constant for the distribution (x(x) = ("(())/Z()) and visualize (ar) (Figure solid green). We can clearly see that in the large-data regime (in terms of number of ranked pairs), the reward function learns model the implicit preference distribution (see also Figure in the Appendix).
Activation
Optimizer
Adam (Kingma Ba 2015
Optimization steps
8T92
Learning rate Learning rate schedule
5 10-4
Cosine decay to 0.0 over 8192 steps
In other words, if "your language model is secretly reward model" (Rafailov et al. 2023), it is perhaps
equally fair to say that your reward model secretly learns the annotator's implicit preference distribution [ Ultimately, the reward function means to the end of aligning a generative model (for example, an I.T.M "policy") with human preferences, but the above discussion shows that under the Luce choice rule assumption the optimal reward function actually learns the implicit preference distribution p" (x).
4.2 Specifying policies as normalized preference distributions
How, then, should we use our model of human preferences to con- struct a generative model that aligus with those preferences? We could an RL algorithm to search for policy that optimizes pref- erences, but Theorem I suggests that this may be necessary: the "reward model" can already be interpreted a probabilistic model of preferred outpuls, since it estimates the implicit preference dis- tribution (xx) that is assumed to govern the annotator's 's behavior. This means that we could in principle simply use our approxima- tion of the annotator's implicit preference distribution as a policy. The only remaining difficulty is that Theorem only specifies the preference distribution up a constant.
0.6
PDF
Annotator
Zid) er00
0.4
0.2
0.0
-10
X
To address this, suppose we simply define the "reward" re(x) the log-density of a parameterized generative model, i.c. ro(x) log=e(æ) Since Tœ(œ)) is a normalized probability distribution, at- taining the global minimum the loss function directly results in (x) = (xx). This removes the need for a separate reward model, and the original "reward" is now just implicitly encoded our learned approximation of the implicit preference distribution.
Figure 1: Training reward model comparison outcomes stemming from a synthetic implicit preference distri- bution (Equation 10 dashed blue) re- covers the implicit distribution (solid green).
Provided the annotator's preferences follow the Luce choice rule. We purposefully switch from (x) to ra(2)) highlight the fact that the reward function now depends the LLM's parameters 0.

---

Published in Transactions on Machine Learning Research (02/2024)
This idea of bypassing the reward function and directly optimiz-
ing the LLM on pairwise human feedback is explored in recent works like ILHF (uu et al. 2023) and DPO (Rafailov et al.] 2023) (which we will discuss in more detail shortly) as well as IPO
(Azar et al,
2023). RRHF (Yuan et al. 2023). SLiC-HF (Zhao et al.) 2023). and RSO (Liu et al. 2023b) (discussed in Appendix A.2 We also note the probabilistic procedure described above can thought as an inductive bias in and of itself: if one believes that the implicit preference distribution the ideal generative model, then expressing the reward the LLM's log-likelihood is the correct thing to do. In practice, however, stronger inductive biases are used to regularize the tuned LLM towards its pretrained parameterization.
4.3 Expanding to a broader family of generative processes for pairwise preferences
Our results show that assuming the generative process for the annotator's pairwise preferences follows the Luce choice rulc-fitting a model those preferences using the paramcterization in Enquatio 2 recovers the annotator's implicit preference distribution p" (æ:). This recasts the standard reward modeling procedure density estimation procedure. However, it is not necessarily obvious that annotators exactly follow the Luce choice rule, nor that the implicit preference distribution p" (x) we recover is itself good "policy".
Luckily, our density estimation results can be extended to hold for generative processes beyond the Luce choice rule. We focus on the following family of equations for describing generative processes which we name preference behavior distribution equations (PBDEs), noted (p())
- PL) FB) -(0((e)))0(((( (2p(E)), Op(2x) = f(x) log + g(x), f(x)) > 0 Vo.
(11)
Theorem 2. Let the generative processes for pairwise preferences for the annotator and the model be gov- erned by the same PBDE, i.e.,
(x) f(x) llogp" (a) )(x) and arp(10 = f(x) log To()) + g(x).
Then the global minimizer of
DKL (p" (U: xB) Po (o
E (
is
To(x) = p"(x), Va.
A proof is provided in Appendix A.1 in subscction 4.2 This corresponds to setting f(x) = and g(x) 0, and is adopted in practice by the
a simple choice of PBDE the Luce choice rule itself, as we discuss
Inclusive Learning From Human Feedback (ILHF: Xu et al. 2023 approach.
We now give concrete example of an alternative PBDE to the Luce choice rule, which stems from the latter's behavior in the context of variable-length inputs like text sequences. In that case, since the likelihood assigned by the implicit preference distribution to a token sequence x is decomposed into a product of per- token likelihoods in [0. 1] øl, longer sequences naturally have lower likelihood magnitudes. This means that an implicit preference distribution represented by an autoregressive model trained only on long sequences could in some cases assign higher likelihood to short sequences despite having near-zero probability of sampling them. In other words, if we were sample from such an implicit preference distribution we would get long sequences, yet using the same distribution to sample preferences with the Luce choice rulc would tend to prefer shorter sequences. Refer to subsection 5.2 for a practical illustration of this phenomenon. We argue that this is unrealistic: an annotator should not prefer shorter sequences simply because the total number of possible short sequences smaller than the total number of possible long soquences. (In fact, real-world human feedback datasets tend to show a slight preference towards longer responses (Singhal al. 2023

---

Published in Transactions on Machine Learning Research (02/2024)
A more realistic alternative to the Luce choice rule in this context would be the length-normalized variant
p (
Prob( > an)
(12)
p (x mal + p2( Fs
The above has the advantage that its outcome is longer biased by the inherent differences in likelihood magnitudes between sequences of varying lengths. This is likely why been used by Yuan et al. (2023) in RRHF, for instance. Since Equation 12 is PBDE with f(x) = |=|| and g(x) = 0, we know through Theorem 2 that it is possible to recover a globally optimal Te = p" (x) when both the annotator and the model preferences are sampled according to that generative process.
log-likeli liliieiihoods have
Theorem provides a global optimality result for a broad family of generative processes for pairwise pref- erences when the annotator and model share the same process, but it can sometimes be useful define a different generative process for the model than the one assumed for the annotator- instance to influence the properties of the globally optimal (x)) We now show how Direct Preference Optimization (DPO; instance of this general density estimation procedure. DPO bypasses
Rafailov et al. 2023) can be recast the policy optimization step in RLHF and instead directly expresses the reward modeling step in terms of (x:). The authors point out that the KL-controlled policy optimization stage
4- max Exo((e) ro((z) - 3DKI((TO22) (pro(x))
(13)
has a closed-form solution
(rr((a))
(x) x upro (z)
exp exp
(14)
Solving for r=(x)) and performing the substitution in Equation which expresses the probability that a > BB as
they obtain loss function for the reward
To (2
To (x(x)))
pa (, = a 3 log
3 log
(15)
Trre (
Trro (x)
This equation effectively defines generative process for pairwise preferences for TT( that is guided by PBDE with f(x) = B and g(x) = -8 log xpre (x),, i.c.,
2r (x) = B log (x( - 3 log ppre(x)) (x).
(16)
If we assume the preference generative process for the annota- for is governed by the same PBDF--wite could be the case if annotator preferences express only relative improvements that correct for the base fluency of the outputs then Theorem holds and the model recovers TT(x)= = pp (x) Figure 2).
B= 4.0
0.6
PDF Annotator
0.4
ng(x) (DPO-optimized)
Note that diffcrent assumptions about the preference genera- live processes can lead to different relationship between p* and Te, even if those assumptions produce the same observed preference distribution p(y: AA, ÆB). For instance, if we instoad assume the annotator behaves according to the Luce choice rule, but still minimize the DPO objective, the globally opti- mal my (æ) would become
0.2
0.0
-10
0
10
X
Figure 2:
Theorom 2 also holds for DPO the annolator and model share the same generative process (Equation 15).
(1:) Tpro( pp +
(17)

---

Published in Transactions on Machine Learning Research (02/2024)
1.0
B= 4.0
B= 16.0
PDF
0.6
Annotator
ppr((X)
p(x)
0.4
na(x) (DPO-optimized)
0.2
Tre (xlp (xx)
0.0
-10
10
-10
0
10
-10
0
10
X
X
Figure 3: Under the Luce choice rule assumption for the annotator's generative process on pairwise prefer- ences, using DPO lune generative model p (douted orange) on preferences derived from the implicit preference distribution (dashed blue) results in mixture of experts model between the initial model Trre and the temperaturr-smootler implicit preference distribution, demonstrated by the agreement between the empirical (solid green) and theoretical (dash-dotted red) curves.
Under this assumption, DPO converges to a product of experts between a pretrained "prior" policy (a))(a)) and the annotator's tenperature-smoothed implicit preference distribution p" (æ)) This means that whether or not Th (a) converges to p" (æ) depends as much on our assumptions about the preference generating process it does on the algorithm we use.
A proof is provided in Appendix A.1
We illustrate this by finetuning pretrained model (p)e( with DPO pairwise preferences acquired from synthetic annotator whose generative process for pairwise preferences obeys the Luce choice rule with an implicit preference distribution defined in Equation 10 (Figure 3 dashed blue). The model (x) FFigure 3, dotted orange) is an energy-base model with an energy function expressed as a four-layer MLP (we reuse hyperparameters detailed in Table 1):
pre (x)
MLPPpe(a))
(18)
Zpre
We initialize it by regressing the log- loglliillhood of Nuruncated = 0.0, =0 5.00 defined over [-10, 10), and sweep over 3 E (1,4, 16) to adapt it with DPO. The adapted model (Figure 3, solid green) behaves just as predicted by Equation 17 Figure 3, dash-dotted orange).
We point out that Equation 17 is almost a weighted geometric mean but is missing a 1 1- (x). If would result in a globally optimal no (x) (again, assuming the annotator behaves according to the Luce choice rule) of the form
3 exponent on
instead to define f(x) -1 and g(x:) (1 a- log (xr) the PBDE for the model
<α( ( p" (x0)
(19)
Refer to Appendix A.1 for a proof and Figure in the Appondix for an empirical learning behavior visual- ization that case.
5 Annotator misspecification
In Section 4.3 we touched on cases where the mismatch generative processes for pairwise preferences between the annotator and the model was used to our advantage induce specific properties the globally optimal model. In this section, we show that an unwanted mismatch generative processes (which we call annotator misspecifscntion) can lead to badly tuned policies.
9

---

Published in Transactions on Machine Learning Research (02/2024)
Single-annotator
Multi-annotator
Model (misspecified)
Model (well-specified)
1.0
ply: XAx Xg
0.6
Xa
0.4
Xn
XA
XA
Figure 4: a > ΦB) for single-annotator Equation 20 and multi-annotator (Fquation 21 behavior (left two plots) well as models adapted with misspecified and well-specified annotator behavior model (right two plots). The large regions of ncar-0.5 probability in the multi-annotator case (sccond plot) are caused by strong but opposing preferences for the two annotators, which cannot be captured by single- annotator reward model (third plot).
5.1 Annotator misspecification in a toy setting
Consider scenario where two annotators Annotator 1 and Annotator 2-are providing feedback coffees served al various temperatures. Annotator 1 happens to really like iced colfee, while Annolator 2 prefers hot coffee. In the experiment, an annotator is selected at random and is served pair of coffees at temperatures sampled independently at random. The annotator expresses a preference which is added to the feedback dataset alongside the two coffee temperatures, but without noting the annotator's identity.
We simulate this by modifying the way in which foedback is acquired from Equation 10 instead of having single annotator following the Luce choice rule with an implicit preference distribution which covers two modes, assume the distribution of preferences is itself mixture of two preference distributions, each of which is obtained from a distinct annotator following the Luce choice rule with an implicit preference distribution covering one of the two modes. Mathematically speaking, instead of drawing preferences from
p" Ax
Wy (T
Prob( xB)
(20)
D" (x + pr (xn)
Σ wi P.
Σi=
P5 (m))
(where wi 2/5, pi Niruncated -25, 0.25), W2 3/5, and p Neraucatod (p 2.5, 1.0)), sample preferences from mixture of two separate preference distributions
(x))
Prob(æ > xB)
Prob; > xB) =
wi
(21)
P (x + (xB)
A visual way to reason about the differences between Equations 20 and 21 is to plot Prob(æ > BB) in both cases (Figure 4 left half). A cursory examination shows that the two result in very different comparison out- come statistics.
1.5
PDF
Annotator 1
1.0
Annotator 2
p(x)
Tpre(X)
If we adapt (x) on comparison outcomes sampled from Equation 21 using Luce choice rule generative process for the model, we are in annotator misspeci- fication sctting and the adapted model Figure 5] solid green) fails to capture regions of high density under either the two annotators' PDFs Figure 5) dashed blue and dot-dashed pink). In other words, we learn serve lukewarm coffee both annotators despite nei- ther of them enjoying it. This is also reflected in the
0.5
Tx(x)
0.0
-10
0
10
Figure 5: preferences are aggregated across two annotators (dashed blue, dot-dashed pink) but the model is adapted under single-annotator assump- tion, it fails to capture either annotator's PDF (solid green).
10

---

Published in Transactions on Machine Learning Research (02/2024)
0.10
Preference dataset
whole short long
0.05
Annetator mixture
0.08
Density mixture
Probability
Probability
0.04
0.06
0.03
0.04
0.01
0.00
0.00
40
Sequence length (number of tokens)
Sequence length (number of tokens)
Figure 6: Sentence length distributions. (Left) Histogram of sequence lengths for sequences sampled from the whole, short, and long model checkpoints. (Right) Histogram of sequence lengths after adapting the pretrained model (p)) (the whole model checkpoint) on two distinct preference datasets: the "annotator mixture" datasct [Equation 22 and the "density mixture" dataset (Equation 23).
comparison outcome statistics captured by the model Figure third plot). Addressing annotalor misspec- ification open problem, especially in the absence of annotator IDs. In Appendix A.3 we present results
where we succeed in disentangling the two annotators' preferences without annotator IDs (also refer to Fig- ure 4 rightmost, plot), but applying the same approach proved unsuccesful in the larger-scale setting which we will discuss shortly.
This failure mode of learning from pairwise preferences is particularly relevant in a context where LLMs are aligned with human preferences by crowdsourcing the acquisition of pairwise preferences across multiple human annotators. It could appear at first glance that the resulting preferences
inconsistent with one another when in fact annolator identily is a confounding facior that needs to be laken into account. For instance, consider a setting where a group of human annotators are asked to rank pairs of model outputs in terms of fluency. Given equally fluent outputs, some annotators may be biased towards short sentences while others may biased lowards longer sentences.
5.2 Annotator misspecification in language modeling setting
We now move to the One Billion Words Benchmark (LMIB; Chelba el al. 2013) to illustrate the consequences of annotator misspecification a language modeling setting. Starting from Flax's LM1B example code we train three separate instances of a Transformer-based architecture on distinct subsets of LM1B according to sequence length: less than 50 tokens (whole), between 1 and 20 tokens inclusively (short), and between 30 and 50 tokens inclusively (long). We treat. the whole model as the pretrained TMM to adapt via pairwise preference feedback and use the short and long models to form synthetic annotators. Figure 6 (left) shows sequence length distributions for each model.
Recall that for variable-length inputs the Luce choice rule results in "length bias" (in that applying it policy Ieads "length bias" in the resulting preferences), since longer sequences naturally have lower likelihood magnitudes (subsection 4.3). We illustrate this by comparing sequence length distributions FFig- ure 6) left) to the probability given by the Luce choice rule (Equation 9 that synthetic annotator with a long implicit preference distribution will prefer short sequences over long ones (Table 2): that probability is around 97%. When comparing the outcome probabilities for short, medium, and long sequences (Table 2
Equation 12 better aligns with the intuitions built. from examining Figure 6 (left).
We now revisit annotator misspocification when pairwise preferences are acquired from multiple annotators, assuming annotators and the model behave according to the cength-normalize Luce choice rule. We sample 215 pairs from the whole model and compare the outcome of learning on two preference datasets:
11

---

Published in Transactions on Machine Learning Research (02/2024)
Probability of a synthetic annotator preferring some token sequence over another token sequence as function of their respective lengths. Token sequences are sampled from the pretrained LM (whhle) and are binned into small (S, 20 tokens or less), medium (M, between 21 and 29 tokens), and large (L, 30 tokens more) sets of sequences. P(X Y) represents average of P( FB) over all xA X, 2B Y.
Table 2:
P(S > M)
P(S > L)
P(M > L)
Synthetic Annotator
Equ 9
Equ 12
Equ
Equ 12
Equ 9
Equ
whole short long
0.9094
0.4042
0.9928
0.3621
0.9175
0.4554
0.9906
0.9540
1.0000
1.0000
0.9945
0.9783
0.9190
0.2489
0.9729
0.0533
0.7873
0.1575
The "annotator mixture" dataset built as uniform mixture of preferences sampled independently using the short and long model checkpoints. For each (æA, BB) pair, first sample uniformly random which checkpoint will be used as the implicit preference distribution, and we then sample a preference using the length-noue normalized Luce choice rule:
log Te (XA)
log Pire (x
Ply
Σ
{
(22)
2
x B
Opon (Ponhort:Pongg
The "density mixture" dataset is built following the length-normalized Luce choice rule with an implicit preference distribution that a uniform mixture of the distributions computed by the short and long model checkpoints:
( log Tmix (xA) log V x
(log
4.0ntor (2r) + Tohong (x)
P(y:AA, o
Tmixx (x)
(23)
2
In other words, the "annotator mixture" dataset contains a mixture preforences from a synthotic annotator who likes short sequences and a synthetic annotator who likes long sequences, whereas the "density mixture" dataset contains preferences from single synthetic annotator who likes both short and long sequences, but not medium-length sequences. Figure 6 (right) shows sequence length distributions after adapting on either of the two preference datasets. We notice that the model adapted on the "annotator mixture" preference dataset generates medium-length sentences much more frequently than either the short or long model checkpoints, exactly like in the "cold and hot coffee" toy example. This is a direct result of annotator misspecification: pairwise preferences are obtained by querying two synthetic annotators, whereas the learning rule used for adaptation assumes a generative process for preferences that involves a single annotator. On the other hand, when learning from preferences acquired from a synthotic annotator whose implicit preference distribution is a mixture of the short and long model checkpoints ("density mixture" dataset), the generative processes for pairwise preferences the same across annotator and model, and the model succeeds in learning to output bimodal distribution of sequence lengths.
6 Discussion and Limitations
In this work we use a simplified, synthetic experimental framework to empirically verify the theoretical properties of approaches which learn from pairwise human feedback. Specifically, we
1. define probability distribution over random variable with finite support and use it the implicit preference distribution for synthetic annotator following a fixed preference generating process;
provide the model with access to the implicit preference distribution only through fixed number of queries to the synthetic annotator on observation pairs which are ranked stochastically using PBDE; and
3. adapt the model the resulting Aa B. y) triplets.
12

---

Published in Transactions on Machine Learning Research (02/2024)
In this synthetic setting, we have shown that it is indeed possible to approximately recover the annotator's implicit preference distribution. We have also demonstrated how this procedure fails in the presence of annotator misspecification.
Solving annotator misspecification in the presence of multiple annotators requires unsupervised discovery of annotator clusters and difficult and open problem. The naive approach we present in Appendix AA.3 requires knowing the correct number of clusters in advance and represents an important drawback. The literature probabilistic modeling suggests that we could perhaps replace the categorical random variable random for the cluster ID with continuous random variable, but this has yet to be tried in practice.
In some cases practitioners could conceivably use additional metadata to their advantage, for instance by as- signing each annotator with a unique ID and solving the binary preference classification problem conditioned on the annotator ID. Alternatively, if practitioners know in advance that preferences are polarized and if the way each annotator Icans can be predicted from certain covariates (age, location, ctc.). those covariates could be used additional context in the binary preference classification problem.
Determining whether behavioral assumptions are correct is also difficult problem. Practitioners' decisions should be informed by results in disciplincs like mathematical psychology or by their empirical knowledge. Additionally, should a model fail to learn from pairwise preferences, the possibility that a practitioner's behavioral assumptions are wrong should at least be considered.
Beyond aunotator misspecification, there are a number of other ways our hentc density estimation setting differs from practical applications of learning from pairwise human feedback.
Finite data We do not account for the labor necessary to acquire large amounts of human preference data. The theoretical results we present hold in the infinite data limit and the existence of a global minimizer for the loss function does not guarantee convergence through gradient descent. The univariate toy experiment (Figure 1 is carried at the larger end of the data regime spectrum given the dimensionality of the problem and the number of queries made to the synthetic annotator, and in that setting we achieve convergence.
Practical applications are high-dimensional (images, language tokens, etc.), and the number of synthetic an- notator queries make in our LMIB experiment is comparatively small: the size of the vocabulary (30000) and the maximum sequence length (50) result in a set of possible sequences that is considerably larger than the number of synthetic annolator queries (215). It is interesting to note, then, that our LM1B experiment (and more broadly contemporary works on adapting LLMs to pairwise human feedback) are reasonably effective at steering models from pairwise feedback alone. How this is possible despite the combinatorial nature of language is an interesting question for futurr work. Plausible hypotheses include:
Natural language conforms to the manifold hypothesis (Cayton
2005 Narayanan & Mitter 2010);
and the pretrained model (x) is a particularly good proposal distribution to explore the "man- ifold" of natural language. The result is that characterizing a distribution over natural language outputs requires far fewer pairwise comparisons than the curse of dimensionality would suggest.
Density estimation using pairwise human feedback is really hard in the data regimes in which it is deployed, but it is not necessary to fully capture the implicit preference distribution in order to steer the generative model in the right direction. Provided careful regularization is applied (which, to our knowledge, is a requirement in practice), preference datasets are informative enough.
Stationarity Our experiments assume all observation pairs are sampled at once before adapting the model. In practice some approaches (like RLHF) use a noustationary proposal distribution and alternate between querying for human preferences using sample pairs drawn from the model and adapting the model on human preferences. This raises interesting questions on whether stationarity is preferable, and whether active learning could play role in reducing the amount of annotator queries necessary for good performance.
13

---

Published in Transactions on Machine Learning Research (02/2024)
7 Conclusion
In this work we present a probabilistic modeling as an alternative to reinforcement learning for learning from pairwise human preferences. For a broad family of generative processes guided by what we call preference behavior distribution equations (PBDEs) we show through theoretical and empirical results that, 8o long as the human annotator and the model share the same PBDE for generating pairwise preferences, the global optimum of the resulting optimization problem the annotator's implicit preference distribution over observations. We argue that being explicit about the assumed generative process for an annotator's preferences is important both to manipulate the properties of the globally optimal model that learns those preferences and to avoid annotator misspecification failure modes. Finally, illustrate the practical implications of annotator misspecification in language modeling problem where the usual single-annotalor assumption fails to adequately capture two annotators' conflicting preferences.
Broader Impact Statement
Our work investigates learning from pairwise human preferences from a theoretical standpoint, but it has mul- tiple practical implications. Density estimation from pairwise feedback is very difficult much more than density estimation from target distribution samples and most practical applications operate in the low-data regime. This has the polential to exacerbate biases in the pairwise preference dala and makes it all the more important to ensure that annotators are representative of diverse population of end-users. The insights we provide on annotator misspecification also highlight the importance reasoning explicitly about the generative process for pairwise preferences, as a misspecified process has the potential to steer the model towards unwanted behavior or erase underrepresented perspectives altogether. We believe future research on both these aspects of learning from pairwise human preferences important in order to achieve inclusive alignment with human judgement.
arrgubly
Author Contributions
Vincent Dumoulin wrote a first draft of the mamuscript which co-authors then addod to and helped edit. He implemented and carried out the experiments (starting from Yann's code and model checkpoints the case of LM1B). He derived the proofs for [Theorem (with input from Laurent Dinh) and Theorem 2 (with input from Daniel) and developed the length-normalized Luce choice rule solution to the "length bias" problem jointly with Yann.
Daniel D. Johnson identified the problem of annotator misspecification with two annotators and proposed the "family of generative processes" framing to unify the Luce choice rule, DPO, and the length-normalized Luce choice rule. He contributed to writing and editing, and helped with some of the proofs. He explored alternative variants of the LM1B experiment which ended up not making it into the paper.
Pablo Samuel Castro provided regular input in the research discussions and helped write and edit the manuscript.
Hugo Larochelle proposed the rescarch direction that eventually led to the paper's narrative of framing learning from pairwise human preferences as a probabilistic modeling problem. He advised on the project and helped revise the mamuscript.
Yann Dauphin helped with experiments and trainod the LM1B model chockpoints used in the submission. He proposed latent variables as a solution to user misspecification with two annotators (Section A.3). He identified the "length bias" issue with the Luce choice rule for autoregressive models and he developed the Length-normalize Luce choice rule solution jointly with Vincent. He helped develop and frame the theoretical results in the paper.
Acknowledgements
We would like thank Laurent Dinh for his help in deriving the proof for Thcorem 1 Jesse Engel for his input in the early stages of the project; Christian Walder for his input on inducing specific properties in globally optimal models by manipulating the model's generative process for preferences; Ben Poole for his
14

---

Published in Transactions on Machine Learning Research (02/2024)
thoughtful feedback in revising the manuscript; and Rishabh Joshi for clarifying the implementation details of STiC-HF-direct.
References
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm understand learning from human preferences. ar preprint arXiv:2310. 12066, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human foedback. arXiv preprint Xxx::2204005666 2022.
Erdem Biyik and Dorsa Sadigh. Batch active preferenco-basce learning of reward functions. In Proceedings of the Conference Robol Learning, 2018.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 393344)324--35,, 1952.
Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Nickum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. Proceedings of the International Conference on Machine Learning. 2019.
Daniel Brown, Wonjoon Goo, and Scott Niekum. Better--han-demonssrttorr imitation learning via autoomatically-rankked demonstrations. In Proceedings of the Conference Robot Learning, 2020.
Lawrence Cayton. Algorithms for manifold learning. Technical Report C82008-0923, University of California, 2005.
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robin- son. One billion word benchmark for measuring progress in statistical language modeling. arXio preprint arXiv:1312.300 2013.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforce- ment learning from human preferences. Advances in Neural Information Processing Systems, 2017.
Borja lbarz, Jan Leike, Tobias Pohlen, GeofTrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in Neural Information Processing Systems, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Leorning Representations, 2015.
Tomasz Korbak, Hady Elsahar, Germán Kruszewski, and Mare Dymetman. On reinforcement learning and distribution matching for fine-tuning language models with no catastrophic forgetting. Tn Advances in Newral Processing Systems, 2022.
Kimin Lee, Laura Smith, and Picter Abbeel. Pebble: Feedback-officien interactive reinforcement lcarning via relabeling experience and unsupervised pre-training. In Proceedings of the International Conference Machine Learning, 2021.
Hao Liu, Carmelo Sferrazza, and Picter Abbeel. Chain of hindsight aligns language models with feodback. arXiv preprint NNiv:2302.02670 2023a.
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. arXio preprint or orxi:223090060 2023b.
R Luce, Robert R Bush and Eugene Ed Galanter. Handbook of mathematical psychology- John Wiley, 1963.
15

---

Published in Transactions on Machine Learning Research (02/2024)
Hariharan Narayanan and Sanjoy Mitter. Sample complexity of testing the manifold hypothesis. Advances in neural information processing systems, 23, 2010.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhiní Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddic Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, 2022.
Joshua C Peterson, Jordan W Suchow, Krisha Aghi, Alexander Y Ku, and Thomas L Griffiths. Capturing human category representations by sampling in deep feature spaces. In CogSci, 2018.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is sccretly a reward model. arXiv preprint arXiv:2305.18200 2023.
Dorsa Sadigh, Anca D Dragan, Shankar Sastry, and Sanjit A Seshia. Active preference-based learning of reward functions. Technical report, UC Berkeley, 2017.
Adain Sanborn and Thomas Griffiths. Markov chain monte carlo with people. In Advances in Newral Information Processing Systems; 2007.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms. arXiv preprint aiv:: (tt:17770000 707.06947, 2017.
Roger N Shepard. Stimulus and response generalization: A stochastic model relating generalization to distance in psychological space. Psychometrika, 22(4):325 345, 1957.
Daniel Shin, Anca. D Dragan, and Daniel S Brown. Benchmarks and algorithms for offline preference-based reward learning. Transactions on Machine Learning Research, 2023.
Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in RLHF. arXiv preprint arXiv:2310.0311 037166, 2023.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Aduances in Neural Information Processing Systems, 2020.
Richard S. Sutton and Andrew G. Barto. Introduction to Reinforcement Learning. MIT Press, Cambridge, MA, USA, 1st edition, 1998. ISBN 0262193981.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
Christian Wirth, Riad Akrour, Gerhard Neumann, Johannes Fürnkranz, et al. A survey of preferenco-based reinforcement learning methods. Journal of Machine Learning Research, 18(136):1 -46, 2017.
Wanqiao Xu, Shi Dong, Dilip Arumugam, and Benjamin Van Roy. Shattering the agent-environment interface for fine-tuning inclusive language models, arXiv preprint arXi::2305. 11455, 2023.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songlang Huang, and Fei Huang. RRHF: Rank responses to align language models with human feedback without tears. In Advances in Neural Information Processing Systems, 2023.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Salch, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. Xiv preprint 23w:23355.0423 2023.
16

---

Published in Transactions on Machine Learning Research (02/2024)
A Appendix
A.1 Theorems and proofs
Lemma 1. Let A, vn) ) be joint probability distriluution with support S x 8 and for which A, > 0 for all AA, aB E S x S. Let x(x): S - R and w(x); S + R be two injective functions mapping æ to scolar "scores". ". Let
ex(eaa
(x ex(aa) ex(xx) o (x(xx)) x((x1) and p0( of (wW(xA) wwarn))
The loss function
Ex ooggee (1s (1) w (I
is globally minimized when
X(x) = w()) C
Proof. We know from the properties of the KL-divergence that minimized when the left- and right-hand side represent the same distribution, which means that the expectation itself has global minimum zero when
ex((x))
(a((a)
V E E
ex(mx) + ex(aa) (w(mn) + (=())))
=exlextootaal (xlea))((mm) paleaktotea ((len))
vo 8
+x(xx)) = w( (x(#n) w(#)))
V 8
For any given AA, the above equation needs to hold for all øB 8, which is only possible if the term x(xB)= w(xB) is constant for all BB S. We therefore conclude that
x(x) wmx)+ +C, CE
Proof for Theorem
Proof. The proof follows directly from applying Lemma with x(x) = lloop'' (x) and w(x) = (x(x): at the global optimum we have that
( C
which mcans that
(a(m) x (x) Vo E. S.
17

---

Published in Transactions on Machine Learning Research (02/2024)
Proof for Theorem 2
Proof. Using Lemma with x(x) = (2-((2) and w(x) = Qmo(a)) at the global optimum we have that
Pp (z) - Aup (x) [ +,
C R Va ES
= f(x) log (a) ((x))
g(z) + C.
C V
-p (x) (
C V
Since both p² (a:) and (x)) are probability distributions, summing integrating both sides results in
E (m) e = 1.
There are three possibilities for C: either C C<< 0, C > 0, or C = 0. Since know f(x) > 0 for all æ, C < 0
would result in an expectation smaller than and C > 0 would result in expectation larger than 1. We
conclude that C' = 0 and therefore
To(x)) = 10 (ar), Va..
Proof of [qquation 17ss global optimality
Proof. We observe that
Po (V = ( log
(T(xaa)
To B log
Trre (xa)
Tpre (xg)
Using Lemma ! with x(x) logp"(x)) and w(x:) = 3(log (xx log Tpro (x)), the global optimum we have
that
log (x:) = Blog Te(x)) log ppre +C, C CE CE
from which we conclude that
no TT(x)) x (x) pp' (2:).
Proof of Equution 19's global optimality
Proof. Using Lemma with x(x) log pxx and w(x) 1 (log Tx(x)) loog Tpre pp)e + log xpre (x), at the
global optimum we have that.
log (x) (log (xx)) log ppre(x)) + log (x) C E
a
from which conclude that
TT( (2)) .pp.ppp'e (2=)
18

---

Published in Transactions on Machine Learning Research (02/2024)
A.2 Alternatives to the binary cross-entropy loss
Other recent works adapt the model directly on pairwise preferences but move beyond binary cross-entropy loss formulation. As such, Theorem 2 is not applicable, and more work is needed to characterize their learning properties when viewed through the density estimation lens.
Rank Responses to align Human Feedback (RRHF; Yuan el al. 2023 uses a ranking loss instead of the binary cross-entropy loss and considers annotator rankings obscrvations æ1 through ay:
Trank (0) D(E xb)
Σ 2n
log no (
log (
(24)
[z)
where |a]] is the length of the token sequence a. log-likelihood on the highest-ranked 2; is also used.
An additional loss component in the form of negative
Sequence Likelihood Calibration from Human Feedback (SLiC-HE; Zhao et al. 2023 considers (among other alternatives) STiC-HF-direet variant which optimizes To( directly on pairwise human preferences. The loss function is defined
(s-(vv log (a( + (1 y) -log To (
To (T(B))))) To To xA))
)
((0)- =
(6-(u)
A EE (x) flg To (x:)
(25)
The first expectation represents hinge loss the difference in log-likclihoods between æ and æg it encourages the difference to be positively large in favor of the higher-ranked observation, up to margin 6. The second expectation regularizes Tx( towards supervised finetuned (SFT) model trained on a dataset DSFT of target domain examples, either using the supervised finetuning data directly, drawing samples from the supervised finetuned model, or maximizing the likelihood preferred observations.
Ignoring the regularization term, the STaC-HF-direet loss function resembles Equation but uses hinge loss on the difference in log- logllikeisoodd between A and æB rather than the sigmoid binary cross-entropy. We examine the properties of SLiC-HF Hirecrect using synthetic density estimation problem by finetuning the pretrained model pre (x) with SLiC-HF-diress
pairwise preferences acquired from the implicit preference distribution defined in Equation 10 Since we consider a purely preference-based setting, we cannot assume access to a finetuning dataset, however we can still regularize the model towards the pretrained model by sampling from (x) in Equation 25 second expectation. We sweep over E {0.0, 0.25, 0.5, 1.0, 2.0} and and
A AE (0.0, 1.0 to adapt the pretrained model with SLiC-HF-direct. Figure 9 shows the effect The 5 hyperparameter has a non-uniform, smoothing effect on TT(2), as bounds the maximum difference in log- pog-prbbabilitie allowed between any two observations. The À hyperparameter has the expected effect of regularizing the adapted model towards Tpre (x).
2023b) samples from the optimal policy (023bb) also invest igate an extension to SLiC-FH which normalizes the model's log-likelshoods using the pretrained model (x))
Statistical Rejection Sampling Optimization (RSO; Liu et al.
using samples from the pretrained model (x)), but Liu et al.
(6-
(0
To Te (Tpr + (1 - y) log To 1 a pre (x)
( prre (x )
[(0) = ED(
log no To( Tpro (
(26)
VPO with identity mapping (IPO; Azar al. 2023 derives learning rule for KT-controlled preference probability maximization problem
0 6- max
E ExANTA(
p" (us as (v:
- DRR.. (x) || Tppe(()) (x))
(27)
2 =))0)
19

---

Published in Transactions on Machine Learning Research (02/2024)
which has the form
( log no pre + (1 y) log Te ( T (x
To (T( x prre (xaA) (2 Tpre (x
x-1 27
[(0) m)())
(28)
Practically speaking, the loss similar to the one investigated in SLiC-FH (and RSO) but constructed using& a two-sided quadratic expression rather than one-sided hinge expression. We examine the properties of IPO using synthetic density estimation problem by finetuning the pretrained model Tpr (æ) with IPO
pairwise preferences acquired from the implicit preference distribution defined in Equation 10. We sweep over E {0.1, 0.5, 1.0) to adapt the pretrained model with IPO. Figure 10 shows the regularizing effect of т towards the pretrained model's density.
A.3 Addressing annotator misspecification in toy setting
To accounl for annotator identity in the experiment described in Section 5
we initialize two copies of p (adding Gaussian noise with standard deviation 1 x 10 to break symmetries) to create a mixture model
2
W%
To(x))
=
(29)
W1 + W2
and adjust our annotator behavior model account for Annotator and Annotator 2 that
(x( W1 W2 (x() TT(
Σ
(30)
We then train as usual with binary cross-entropy loss on the comparison outcome variable Figuue 11 Doing so allows to recover the correct marginal implicit preference distribution, even in the absence of annotator IDs in the preference data. This is particularly interesting in contexts where human annotators "cluster" their preferences. In the coffee example presented in section the two annotators may represent entire subpopulations of annotators.
10a
10-5
p(x)
10-13
10-21
PDF
Annotator
10-29
(x)
-10
0
10
X
Figure A reward model trained on comparison outcomes sampled according to an implicit preference distribution |Equatin 10: dashed blue) converges to the annotator's unnormalized preference distribution (solid green), as indicated by the constant offset between the two curves.
20

---

Published in Transactions on Machine Learning Research (02/2024)
1.0
a= 0.25
a= 0.0625
0.6
PDF
Annotator
Mrre (x)
p(x)
0.4
molx)
0.2
(x) -a. (x)
0.0
-10
0
10
-10
0
10
-10
0
10
Figure 8: Under the Luce choice rule for the annotator, using a PBDE for the model with f()) = a and g(x) (1 (1-a-1) log pre to tune (x) (dotted orange) on pairwise preferences derived from the implicit preference distribution (dashed blue) results in weighted geometric average between the initial model Tpre and the implicit preference distribution, demonstrated by the agreement between the empirical (solid green) and theoretical (dash-dotted red) curves.
6=0.0.
6= A=
6 -=
5=1.0, A=
6= 2.0, A
0.6
PDF
Annotator
pbo)
ny/x)
0.2
0.0
0.0.0, 1
6= 0.25, A.10
6= 0.5, 1.0
5=0.0 A= 1.0
6= 2.0, 1.0
0.6
pbx
0.4
0.0
-10
10
-10
10
-10
10
-10
-10
0
10
X
X
Figure 9: Using SLiC-HF-direct to tune a generative model (x) (dotted orange) on pairwise preferences derived from the implicit preference distribution (dashed blue) results in different adapted models (solid green) depending on the choice of hyperparameters 5 and X.
0=0.1
== 0.5
==1.0
0.6
PDF
Annotator
p(x)
0.4
Hpra (x)
lx(xx)
0.2
0.0
-10
0
10
-10
0
10 -10
0
10
X
X
x
Figure 10: Using IPO to tune generative model (x) (dotted orange) pairwise preferences derived from the implicit preference distribution (dashed blue) results in different adapted models (solid green) depending on the choice of the 7 hyperparameter.
21

---

Published in Transactions on Machine Learning Research (02/2024)
Marginalized
Components
PDF
0.6
PDF
1.5
Annotator
Annotators
Annotator 2
pix)
0.4
(x)
1.0
(x,(
Tx(x)
0.2
0.5
(x)
0.0
0.0
-10
10
-10
0
10
X
Figure 11:
Choosing the correct annotator behavior model Equation 30) allows to recover the correct marginal implicit preference distribution (left) and individual implicit preference distributions (right), even without amnotator IDs in the preference data.
22


Reviews:
[{"review_id": "uC7Y85D0rC", "reviewer": "Reviewer_mn8Y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The authors formulate a new perspective of reinforcement learning from human feedback and show that under some assumptions, optimizing the loss function of RLHF is equivalent to learning the implicit preference distribution. From this perspective, it is possible to avoid fitting the reward function directly and learn from the preference pair directly. \n\n2. They gave some simple examples in the annotation misspecification, which is easy to understand and emphasizes the importance of explicitly stating the assumptions on the annotation process. They also did sufficient experiments on learning the implicit preference distribution, and learning preference under the annotation misspecification.\n\n3. The theory part seems solid and the proof is good o my knowledge.\n\nWeaknesses:\n\n1. The paper structure can be improved from my perspective. Here is my suggestions: you should give some motivations on why the Luce rule is unrealistic before section 4.3. The length-normalized variants (equation 26) will be a good example. In case that the context lengths vary in the dataset, there will be an implicit preference towards longer sentences. So a length-normalized variant of Luce rule is preferred and it belongs to the class of PBDE. Additionally, I am not sure what role the section 4.4 plays in your paper. Is it just a review for alternative loss functions in learning from human preference? If so and if this section does not connect very closely with your other parts, then probably this section can be moved to the appendix. \n\n2. In section 5, you propose two generative models. You can successfully learn the 'density mixture' using a same model as the annotator, but if we want to learn the first type (the annotator mixture), how can we do it? In the appendix A.4, the authors proposed a way to deal with it, but to implement it, we need to know 'how many clusters are there in the annotation preference distribution.' In practice, how do we know it? More generally, you claimed that it is essential to explicitly state the assumption for the annotator behavior, but how can we know the annotator behavior if the annotators are some groups of people? (I know this question is kind of beyond the scope of this paper, but for the completeness of paper, I suggest at least the authors should discuss how people can get to know which assumption is true for the annotator behavior in existing literatures).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates into the probabilistic perspective of reinforcement learning from human feedback. They showed that, assuming the generative process of human annotator and the model we trained share the same preference behavior distribution equation(PBDE) (or the so-called Luce choice rule as a special example), learning the global optimum of the loss function can be equivalent to learning the implicit preference distribution function (which is called by the authors a 'density estimation problem'). They also reviewed some alternative loss function for addressing the problem of learning from human preference. They also considered the problem of annotator misspecification and showed empirically that a mismatch between generative process and models can lead to a severely bad policy. They hence emphasize the importance of explicitly stating the assumptions of annotation generative process.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above.", "broader_impact_concerns": "/", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JGmg9MUEjD", "reviewer": "Reviewer_yDZ9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths:\n* the paper is clear and well-written\n* the framework is sound\n* the question of annotator misspecification is relevant.\n\nweaknesses:\n* the originality and contributions are low\n* the experiments are toy\n* most importantly, the paper doesn't have much content: theorem 1 and 2 are trivial as far as I can see, the \"general formula\" is just a general formula, but there is no particular insight into how to choose the specific values, there are two toy experiments with unclear practical value.\n\nCompared to papers coming out lately on similar topics (e.g., papers for DPO, IPO and the likes) this particular paper falls short in technical novelty and experimental insights.\n\nother comments:\n--- section 4.1: I don't understand the contribution in that section. This seems to be paraphrasing the fact that the pairwise logistic assumption to learn the reward model is correct under the assumption that human preferences follow the bradley-terry model, this seems to be known and is clearly explained in the DPO paper.\n\n-- Th 1 and 2: I don't understand what is non-trivial in the theorems, it is well-known that  the KL is 0 if and only if the probability distributions are the same\n\n-- 5.1: annotator misspecification: maybe the authors could clarify what result they would expect in cases where there is no agreement between annotators. I also easily believe that in the case of \"LLM alignment\" where annotators are supposed to follow fairly strict guidelines such extreme disagreement where the middle ground is undesirable doesn't really happen.\n\n-- section 5.2 \"the long model assigns higher likelihoods to short token sequences despite\nhaving a near-zero probability of sampling them\": this sentence seems inconsistent (high likelihood for sequences of low probability), maybe it's because the authors don't compute an actual likelihood (e.g., without taking into account the probability of \"end of sequence\" token)?\n\n-- section 5.3: the length bias is fairly obvious and the difference between sum log prob(token|history) vs mean log prob (token|history) is folklore. mean log prob also has its issues (e.g., favoring long sequences with low token-level entropy).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper reviews the modeling assumptions underlying the practice of learning from human feedback in LLM alignment. The authors make the link between assuming that human feedback follows a Luce model and the reward moeling step, and propose a general formula for directly learning the policy from human feedback, which includes as special cases a large number of recent works regarding different loss functions or regularizers.\n\nThe authors discuss the effect of \"annotator misspecification\", meaning that the model of human feedback is incorrect, and in particular show undesirable effects that come from using the usual model in cases where annotators have conflicting preferences. The authors provide an example in which the model turns out to be biased towards shorter outputs.\n\nThe paper is moslty theoretical, the experiments are mostly of qualitative nature in artificial settings", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "the paper should go through another round of review anyway\n\nrequested changes:\n* less space dedicated to the beginning and theorem 1 and 2 since most of it is already known, focus on what's different from DPO/IPO etc.\n* the discussion on annotator misspecification is interesting, but without real-life experiments it is unclear what the take away of the paper is", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "V1WGPKsWks", "reviewer": "Reviewer_GBeD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\nAuthors exactly knows their expansion direction as stated in Section 4.3, and it is a classic probabilistic modeling to approach the preference modeling as distributions. They are kind enough to interpret others' methods in their probabilistic framework, i.e. the author's interpretation of Direct Preference Optimization in the density estimation procedure.\n\nOnce the preference learning becomes the density estimation problem, all sorts of density estimation tricks can be applied as authors demonstrate in Section 4.4. Sequential likelihood modeling, rejection sampling... All these are classical methods of probabilistic modeling.\n\nI am pretty sure that authors would include a researcher who knows the value of old-fashioned PGM in 2000s and 2010s, which still means a lot in these days, as well.\n\nGood work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work presents a probabilistic modeling perspective upon the preference behavior distribution equation, which transforms the human feedback into a reward function in the RL framework. I can see that the authors are versed in modeling probabilistic models in proposal distribution and its relevant inference method, as demostrated in Eq 6,7,8. This work will significantly solidify the theoretic basis of the presvious works in Section 2.2.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I don't need any change at the moment", "broader_impact_concerns": "no ethical problem", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new perspective on RLHF. Specifically, it is shown that a generative process for preferences (the Luce choice rule) leads to a probabilistic interpretation of RLHF, namely as minimizing KL between the underlying human preferences and the distribution induced by the reward model.

Due to a split in the judgment of the reviewers, and owing to the uniqueness of the criteria for TMLR, I read through this paper somewhat carefully, and I am not sold on its value myself. Unlike other papers like DPO, the reformulation in this paper does not appear to buy us a whole lot, at least from the evidence presented here.

The paper emphasizes:
> your reward model secretly learns the annotator’s implicit preference distribution

I go back and forth on the value of this. It feels like a minor reformulation of the Bradley-Terry model, basically just making a generative assumption which doesn't dramatically change our interpretation of the existing procedure.

Moreover, as the paper itself notes,
> if one believes that the implicit preference distribution is the ideal generative model, then expressing the reward as the LLM’s log-likelihood is the correct thing to do

but this feels to me like somewhat circular reasoning: the Luce choice rule explains the algorithm already in use, and this paper doesn't really offer any techniques to evaluate whether or not the Luce choice rule is the right assumption to keep making.

The discussion of broader PBDEs and the reinterpretation of DPO is interesting. I'm again not quite sure how valuable this reformulation is. In particular, the theory in the DPO paper itself, showing the optimality of the DPO policy under the Bradley-Terry model, goes significantly beyond what is here in terms of its value.

The idea of dealing with annotator misspecification is interesting, but other work like rewarded soups ( https://arxiv.org/pdf/2306.04488.pdf ) and follow-ups push this further. The experiments presented here would unquestionably be valuable if they were the first forays in this area, but they are not. I am not sure about the experiments in section 5.2; while they deal with language modeling, the setting here is very toy.

My complaints here reflect the skepticisms of two reviewers who ended up being in favor of rejection. The objections were on the basis of technical novelty and the triviality of the theorems. I do feel that TMLR has some bar for this, represented by the "audience" bar. However, I am not confident to say that there are *no* individuals in the audience who would find this paper interesting. I think it is a thought-provoking piece of work that at least contributes to the conversation around these methods.  I also believe the edits help significantly in raising the technical sophistication of the discussion and in tying it to recently proposed methods.

In defense of the paper, I will say that it is technically correct (as far as I could tell) and quite well written. I found it enjoyable to read. I could see myself recommending it to students who want to gain some understanding of the formulations of these different methods and understand them from a new perspective. I also think this *could* be a platform for future work in this area; perhaps others in the audience besides myself and the critical reviewers will find significant value in what it presents!
Decision: Accept (poster)