Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF
Authors: Anand Siththaranjan
Cassidy Laidlaw
Dylan Hadfield-Menell
Venue: ICLR 2024 poster (2024)
Keywords: Preference Learning
Reinforcement Learning from Human Feedback
Social Choice Theory
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We show that missing context in standard preference learning for methods such as RLHF is aggregated according to the Borda count voting rule and introduce distributional preference learning to mitigate the associated challenges.

Abstract: In practice, preference learning from human feedback depends on incomplete data with hidden context. Hidden context refers to data that affects the feedback received, but which is not represented in the data used to train a preference model. This captures common issues of data collection, such as having human annotators with varied preferences, cognitive processes that result in seemingly irrational behavior, and combining data labeled according to different criteria. We prove that standard applications of preference learning, including reinforcement learning from human feedback (RLHF), implicitly aggregate over hidden contexts according to a well-known voting rule called *Borda count*. We show this can produce counter-intuitive results that are very different from other methods which implicitly aggregate via expected utility. Furthermore, our analysis formalizes the way that preference learning from users with diverse values tacitly implements a social choice function. A key implication of this result is that annotators have an incentive to misreport their preferences in order to influence the learned model, leading to vulnerabilities in the deployment of RLHF. As a step towards mitigating these problems, we introduce a class of methods called *distributional preference learning* (DPL). DPL methods estimate a distribution of possible score values for each alternative in order to better account for hidden context. Experimental results indicate that applying DPL to RLHF for LLM chatbots identifies hidden context in the data and significantly reduces subsequent jailbreak vulnerability.

Published as a conference paper at ICLR 2024
DISTRIBUTIONAL PREFERENCE LEARNING: UNDERSTANDING AND ACCOUNTING FOR HIDDEN
CONTEXT IN RLHF
Anand Siththaranjan" Cassidy Laidlaw
Dylan Hadfield-Menell
University of California, Berkeley janandsranjan, cassidy_laidlav// cs w.e.keeey..eoe
Massachusetts Institute of Technology chm@ esail .iit... edu
ABSTRACT
In practice, preference learning from human feedback depends on incomplete data with hidden context. Hidden context refers to data that affects the feed- back received, but which is not represented in the data used to train a prefer- ence model. This captures common issues of data collection, such as having hu- annotators with varied preferences, cognitive processes that result in seem- ingly irrational behavior, and combining data labeled according to different cri- teria. We prove that standard applications of preference learning, including re- inforcement learning from human feedback (RLHF), implicitly aggregate over hidden contexts according well-known voting rule called Borda count. We show this can produce counter-intuitive results that are very different from other methods which implicitly aggregate via expected utility. Furthermore, our anal- ysis formalizes the way that preference learning from users with diverse val- ues tacitly implements a social choice function. A key implication of this re- sult is that annotators have incentive to misreport their preferences in order influence the learned model, leading to vulnerabilities in the deployment of RLHF. As step towards mitigating these problems, we introduce class of methods called distributional preference learning (DPL). DPL methods estimate distribution of possible score values for each alternative in order to better count for hidden context. Experimental results indicate that applying DPL to RLHF for LLM chatbots identifies hidden context in the data and significantly reduces subsequent jailbreak vulnerability. Our code and data are available at https: / / i thub. com/ cass dy i aw w/whht dden-context.
INTRODUCTION
Encoding human preferences and values into interactive learning systems an essential component for making those systems safe and socially beneficial. To accomplish this, modern machine learn- ing models, such as large language model (LLM) chatbots like ChatGPT and Claude, are trained with feedback from human evaluators. This method, often called reinforcement learning from hu- man feedback (RLHF), seeks to align system behavior with the preferences of annotators. this paper, we study how RLHF infers preferences when there is hidden context that influences human evaluations.
Hidden context is any information that affects preference annotations but is not given as input to the learned utility or reward model. can arise through several mechanisms. For instance, when feedback is collected from many different people, annotator identity is hidden context: it affects the annotations, since different annotators could have very different preferences, but it is not input to the reward model, since the annotators' data is combined anonymously. Other sources of hidden context include human irrationality and evaluation according to multiple objectives.
To motivate the consequences of naive preference learning with hidden context, consider the follow- ing hypothetical scenario:
"quul contribution.

---

Published as a conference paper at ICLR 2024
Alternative, e.g. chatbot response.
Hidden context, e.g. annotator identity.
Sources of hidden context
Annotator identity population with diverse preferences.
u(a, 2)
(aa)
Internal mental states a person behaving irrationall..
Preference learning
AI
Annotation instructions dataset collected with multiple prompts.
Figure I: We analyze the effects hidden context on preference learning, which is one the key steps in reinforcement learning from human feedback (RLHF). Hidden context is any information that affects the anno- tator's assessment of the utility of different alternatives, but is not input to the learned utility or reward model. Our framework emcompasses many potential issues with preference learning, including human irrationality, di- verse preferences among annotators, and combining multiple objectives (Section 2). We prove that preference learning implicitly aggregates over hidden context using a rule called Borda count (Section 3).
Example 1.1. A company has developed an Al assistant to help high school students navigate college admissions. They implement RLHF by asking their customers for feedback on how helpful the chatbot's responses are. Among other questions, this process asks users whether or not they prefer to see information about the Pell Grant, an aid progrum for low-income students. Because the population of customers is biased towards high-income students, most feedbuck indicates that users prefer other content to content about the Pell Grant. As resudt, RLHF trains the chatbot to provide less of this kind of information. This marginally improves outcomes for the majority of users, but drastically impacts lower-income students, who rely on these recommendations to understand how they can afford college.
The heart of this issue is that common preference learning approaches assume that all relevant fea- tures are provided as input to the reward model. However, when there is hidden context-which is almost always the case-this assumption is false. As result, standard methods can have unexpected and undesirable consequences. In Example 1.1, relevant context about the annotator's ' identity (i.c. their income level) is missing from the data. The implicit aggregation over preferences biases the outcome in favor of high-income applicants. In this work, we take steps to better understand the im- plications of unobserved context in preference learning and consider technical approaches identify when such situations occur.
In Section 2 we present formal model of preference learning with hidden context. We show that our model can represent many challenges in preference learning, such as combining data from different users, accounting for irrationality, and optimizing for multiple objectives. Since these challenges are ubiquitous, understanding their implications is crucial for safely deploying RLHF-trained models.
In Section 3, we use our model to develop theoretical results on the consequences of hidden con- text in preference learning. First, we provide precise characterization of the utility function that preference learning will output when there is hidden context. In particular, we show that preference learning implicitly aggregates over hidden context using a rule called the Borda count. We explore the implications of this finding, identifying cases when Borda account aggregates preferences in unintuitive ways quite different from other methods like regression. Furthermore, when data is com- bined from many annotators, we establish connections with the social choice literature to expose another problem arising from hidden context: annotators may have an incentive to misreport their preferences to influence the learned reward function.
Next, we consider the design of preference learning methods that more gracefully account for hid- den context. In Section 4, we propose distributional preference learning (DPL). DPL estimates a distribution over utility values for each input instead of single real-valued output. This allows the method to detect situations where unobserved context could influence preferences. We show how DPI. can detect the effects of missing features through an explained variance (1-2) metric.
We validate DPL in two ways. First, we conduct small-scale synthetic experiment with 1- dimensional space of alternatives that allows us to directly compare to Borda count. Next, we apply DPL to a real-world dataset of preferences for use in RLHF. In this case, the preference data is collected according to two distinct objectives. In one subset of the data, raters were asked to prefer helpful and honest responses. In the other subset, raters were asked to prefer responses that did not respond to harmful requests. This introduces hidden context because the single reward model is trained on the combined data. We find that DPL is able to identify this hidden context automatically
2

---

Published as a conference paper at ICLR 2024
and identifies the uncertainty when these competing goals odds.
Beyond identifying potential instances of relevant hidden context, our experiments indicate that DPL can be used to develop guardrails that protect against jailbreaks. Wei et al. (2023) showed that many jailbreaks succeed by pitting the helpfulness and harmlessness objectives of chatbots against one another. This means that some jailbreaks can be understood As a result, it is possible to detect this class of jailbreaks by leveraging the distribution utilities we get from DPL. In particular, risk-aversion with respect to the distribution of learned utilities can dramatically reduce the rate at which the preference model prefers jailbroken responses.
consequence of hidden context.
We summarize contributions follows:
1. we identify and formally characterize the problem of preference learning with hidden context, and describe a number of settings where it may arise:
2. we show that preference learning with hidden context implicitly implements Borda count, which can have counter-intuitive implications and incentives for annotators to misreport preferences; 3. we introduce distributional preference learning and show that it can detect and mitigate some effects of hidden context in LLM-based preference models.
SETTING AND RELATED WORK
We begin by formally describing the problem of preference learning with hidden context. Consider a finite set of alternatives A, and an unknown utility function A - R. For instance, in the case of a chatbot, the alternatives could the possible responses to a prompt, and the utility function would describe how much particular response is preferred. To estimate we observe the outcome of comparisons between pairs of alternatives b). We assume there a fixed probability for any pair of alternatives (a, b) that a will be preferred to b; we denote this probability (a b) and assume that Pu (o. b) + (b, a) = 1; that is, the order in which the alternatives are presented does not matter. In the ideal case, comparison outcomes would exactly reflect the utility function, i.e., P., (a, = fu(a) > u(b)}. Realistically, however, preference comparison data never exactly follows single utility function. To account for the fact that people are noisy and/or inconsistent in their feedback, a common assumption is that instead preference comparisons are made according a Headley-Terry-Laee
(BTL) model (Rajkumar & Agarwal. 2014), also sometimes known as ea(a)
Boltzmann rationaal model (Jeon et al., 2020): p" (a b) = e" ("(a) (e"()) this model, the higher v(a) is compared to u(b), the more likely the outcome of the comparison is to prefer to by as the utilities for a and are closer, the comparison outcome moves towards uniformly random. The most commonly used method for estimating the utility function from preference data is to fit the maximum likelihood estimator (MLE) under the BTL model. To derive the MLE, we consider the limit of infinite data and assume that preference comparisons are elicited for uniformly randomly selected pairs of alternatives. The MLE for the utility function à is given by á = arg ming L(ū; u.). where
L(â; u)) [I((A|-1) Enyb Pa((a b) (.(0) +000)
R(())
(1 Pa b)) log
(1)
Although in practice â might be represented by a neural network, we assume for theoretical purposes that [(ū;; u) optimized over all possible à A + IR. In some cases, I. may not have any minimum, so we consider a regularized version (1); see Equation (6) and Appendix A.1 for more details.
2.1 HIDDEN CONTEXT
While preference learning based on (1) has been widely deployed and enjoyed some success, it rests on assumptions that often do not hold in practice. In particular, irrationality, partial observability, and diversity of preferences among population preference learning loss is based. We argue that all of these cases can be understood as special cases of a general phenomenon: hidden context. For concreteness, consider again Example 1.1. The key problem in the example is a mismatch between the information that influences the user's feedback and the information that the preference learning algorithm uses to estimate utilities based on that feedback. The user gives feedback that depends on their financial situation, while the learned utility model observes request-respones pairs. Thus, the preference learning algorithm must produce single ordering over alternatives that implicitly aggregating feedback over the hidden context of
challenge the BTI. model on which the usual
3

---

Published as a conference paper at ICLR 2024
whether the user is high- low-income.
To model hidden context in preference learning, we extend the preference learning formalization to utility functions x Z - R over space of observed features E A and hidden context E Z. Let D_ be a distribution over Z. In Example 1.1, & E {0, 1} could represent whether the user is low- or high-income; then perhaps a Bernoulli random variable with mean p). Given u(a, 2) and Dz, we can calculate the probability that one alternative is chosen over another b given that is hidden:
(0.8)) if 80% of users are high-income (where B(p) represents
[1/2 if af u(b,2) {1)((, > u(b, o.w.
Pu,D. (a, b) EE,D, [On (a, b, z)] where On(a, 2)
Pu,D, marginalizes over the distribution of the hidden context = and thus reflects the comparison data available to the preference learning algorithm. Our model of hidden contexts can represent many settings where preference learning difficult:
Partial observability. There may be variables that are observable by the human making pref- erence comparisons but not by the AI system, which learns from that data. For instance, suppose annotators' preferences depend on the day of the week or the month of the year, but the estimated utility function ignores the date the comparisons were made.
Multiple objectives. System designers may combine data about user preferences over multiple, different objectives. For instance, the Anthropic HH-RLHF dataset (Bai et al., 2022a) contains one subset with comparisons of chatbot responses based on harmlessness and another subset with comparisons based on helpfulness. When these subsets are combined, the objective that was used to make the comparison (in this case, either harmlessness or helpfulness) is hidden context.
Population with diverse preferences. Preference learning almost always applied to data ag- gregated from many annotators who may have very different utility functions (e.g.. Bai et al. (2022a) observe high intra-annotator disagreement). If z represents the annotator who makes a comparison, then u(-, could represent the utility function for that annotator. However, when the data is used to train a single utility function â(-), then the annotator's 's identity is a hidden context.
Irrational and noisy decisions. Various types of irrationality could be modeled unseen latent variables that affect a person's s decision decision-making. For instance, to represent person making noisy utility estimates, one could let 2 R x(a)
iid (O, 1), and u(a, = p(a) + 2(a) for some A + R. That is, the person has an underlying utility μu(a) for each alternative but makes com- parisons based on that utility plus independently sampled Gaussian noise representing irrationality in their utility assessments. This is equivalent to the Thurstone-Mostellee model of noisy decision making (Handley, 2001).
Due to the ubiquity of these settings, preference learning is nearly always performed with hidden context. This means that the learned utility function ū(a). which only depends on the seen features must somehow aggregate over the hidden contexts We aim to understand and mitigate the consequences of this ubiquitous challenge.
2.2 RELATED WORK
Preference learning and its use in reinforcement learning have a long history Akrour et al. (2012); Busa-Fekete & Hüllermeier (2014); Sadigh et al. (2017); Christiano et al. (2017); Pacchiano et al. (2021). As part of RLHE, preference learning has been widely used recently for training large lan- guage models (LLM) to give outputs according to human preferences (Ziegler et al., 2020; Stiennon al., 2020; Askell al., 2021: Bai et al., 2022a;b; Ouyang et al.. 2022). It has also been exten- sively analyzed in theory; some results focus on its sample complexity in various settings (Chen & Suh, 2015; Shah al., 2015; Shah Wainwright, 2018; Heckel al., 2018: Hendrickx al., 2020; Chambers al., 2021) or other directions such the statistical identifiability of preferences (Zhao et al., 2020; Skalse et al., 2023), the computational efficiency of preference learning (Maystre & Grossglauser, 2015), Bayesian preference learning (Caron & Doucet, 2010), the combination of preference learning and reinforcement learning (Zhu et al., 2023). However, to our knowledge, no prior work has specifically analyzed the behavior of preference learning with hidden context.
The challenges of preference learning that we group as cases of "hidden context" have also been studied individually. There has been some work explicitly modeling annotator disagreement
4

---

Published as a conference paper at ICLR 2024
(Fleisig al., 2023; Baumler et al., 2023) well other approaches to learning from annotators
with diverse preferences (Jia et al., 2023; Dumoulin et al., 2023; Mishra, 2023; Fish et al., 2023). Other work has studied the effects of human irrationality non-BTL models of human behavior on preference learning (Bobu et al., 2020; Lee et al., 2021; Laidlaw & Russell, 2021; Knox et al., 2022; Laidlaw & Dragan, 2022), which under our framework can be modeled as hidden context. Zhuang & Hadfield-Menell (2020) and Dai et al. (2023) study the optimization of multiple objectives learned from human preferences. Finally, related our connections with social choice theory in Section 3, some previous work has associated preference or reward learning with concepts in economics, such as voting rules (Conitzer & Sandholm, 2005). incentive compatibility (Echenique & Prasad, 2019), and mechanism design (Fickinger et al., 2020).
3 THEORETICAL ANALYSIS
We begin our analysis by precisely describing the behavior of preference learning with hidden con- text. In particular, we show that utility function i(a) learned with the BTL loss in (6) implicitly aggregates utilities over the hidden contexts = using a rule called Borda count. We define the Borda count BC(a) of an alternative a as BC(a) L A Pu,D, (V,,D((,)), (a, That is, the Borda count is the average probability that the alternative is preferred to other alternatives. If an alternative almost always preferred to all other alternatives, then its Borda count will be close 1; if it is almost always dispreferred, the Borda count will be near O. We use the term Borda count as a ref- erence to the well-known voting rule of the same name a connection we expand on in Section 3.2.
Theorem 3.1. BTL preference learning implicitly aggregates hidden context according to Borda count. That is, if a is optimized according to (6), then Va, b E A, i(a) > k(b) + BC(a) > BC(b).
We defer all proofs to Appendix A. According Theorem 3.1, the learned utility function and Borda count differ by only a monotonic transformation. If we use reinforcement learning or an- other optimization technique to search for the alternative which maximizes ú(a) as one does in RLHF-ther the optimal alternative will the same as that which maximizes the Borda count BC(a). Similar results that relate preference learning and Borda count were previously explored by Rajku- mar& & Agarwal (2014), although they do not consider the setting of hidden context.
While Theorem 3.1 precisely describes the results of preference learning with hidden context, its implications are unclear. Is Borda count useful way of aggregating over hidden contexts in practice, and how does it compare to other aggregation rules? To answer this question, we give multiple perspectives on preference learning with hidden context using the result of Theorem 3.1. First, we compare preference learning to least-squares regression with hidden context. Then, we analyze learning from population with diverse preferences through the lens of social choice theory.
3.1 COMPARISON TO EXPECTED UTILITY AND LEAST-SQUARES REGRESSION
One desirable property of preference learning with hidden context would be if it converged to the expected utility for each alternative when marginalizing over hidden context, which we denote by ū(a) = EE-D, lu(a,))], For instance, one can show that least-squares utility regression converges to the expected utility when there is hidden context; see Appendix A.2 for a formal statement and proof. The fact for least-squares utility regression î = i shows that, in some sense, it gracefully degrades in the presence of hidden context. Although there are drawbacks of expected utility, it is a well- understood method of aggregating utilities over hidden contexts that has desirable decision-theored properties. Thus, it would be helpful if the utility function ú(a) learned by preference learning with hidden context were equivalent to the expected utility a(a). In this section, we characterize when the output of preference learning with hidden context is equivalent to that of utility regression.
Positive results In some cases, we can show that preference learning does identify utility func- tion that is equivalent to the expected utility. The result requires that the zero-mean "noise" induced by hidden context is identical across alternatives and reasonably distributed. We represent this noise as <(a) = u( 2) ū(a) (where 2 ~ D2) to be the random variable representing the residual utility of alternative a after subtracting its expected utility.
Theorem 3.2. Let e(a) be independent and identically distributed for all a E A. Furthermore, suppose <()) <b) has support around O, iee. Vo > 0, (5) > Fa,((0) 2. where Fap is the
5

---

Published as a conference paper at ICLR 2024
Normal preference
Distributional preference learning (DPL)
Altemative
Loss
Loss -log P(u,>ub)
llgg o(u(a) - o(b))
UP
Alterntive
u(b)
We introduce distributional preference learning (DPL), which explicitly accounts for hidden context. While normal preference learning outputs single utility estimate for each alternative, DPL outputs distri- bution over utilities. This distribution represents range of utility values for that alternative as the hidden context varies, e-g., the distribution of utilities assigned to chathot response by different annotators or accord- ing to different objectives (like harmlessness vs. helpfulness).
Figure 2:
cumulative distribution funnctiooion of e(()) e(b). Then the utility function â learned by minimizing (6) satisfies u(a) > a(b) + ū(a)= > ā(b) for any b E A.
Many noise distributions, such as uniform and normal distributions, clearly satisfy the assumptions of Theorem 3.2. Thus, as long as the noise caused by hidden context does not vary across alternatives and is not too unusual, we generally expect that preference learning will give a utility function with the same ordering over alternatives as the expected utility. This means that it performs similarly to least-squares regression.
Negative results In other cases, preference learning can behave quite differently from utility regression. Example 1.1 describes such a case. The expected utility of telling students about Pell Grants is higher than the expected utility of not telling them, since it is of great benefit to low-income students and only small inconvience to high-income students. However, the Borda count is lower since the high-income majority prefer not to hear about the grants.
One might suppose that preference learning and regression disagree in this case because the majority of users prefer the alternative with lower expected utility, and preference learning gives learned utility function which assigns higher utilities to alternatives preferred to by the majority of users. As long as the majority of feedback agrees with the ordering given by the expected utility, will preference learning and regression give the same result? The following theorem shows that this is not the case.
Proposition 3.3. AFZ D2, s.I Va, b A. (M(aa) u(b) = IPu.D, D. (a, b) 1/2]. but â is not equiv- alent to ú, ie., there exist a, b E A such that i(a)= > ú(b) but a((a < a(b).
That is, Proposition 3.3 describes a case where for any two alternatives, the majority of feedback chooses the alternative with the higher expected utility, and yet preference learning still does not produce a utility function equivalent to the expected utility. In general, it is impossible to always identify ū (even up to monotonic transformation) given only comparison data.
Theorem 3.4 (Unidentifiability of ū). Suppose preference learning algorithm takes as input un- limited samples of the form (a, b, On(a, 2)) for all values of and b, where z Dz, and determin- istically outputs learned utility function ((a)). Then there is some utility function and distribution over unseen features D such that ã is not equivalent to u.
3.2 CONNECTIONS TO SOCIAL CHOICE THEORY
When training on comparison data from many agents, each with their own preferences, preference learning aggregates all their feedback into a single utility function. As we described in Section 2, this is a case where the identity of the annotator is hidden context: it affects the comparison out- comes but is unseen by the preference learning algorithm. Social choice theory studies methods for aggregating preferences from a population. Thus, it can provide a lens through which to understand this particular case of preference learning with hidden contexts.
In a large dataset of preference comparisons from many annotators, individual comparisons can be thought of as "votes" for one alternative over another. When preference learning combines this data into a single utility function, it is similar to voting rule that ranks candidates based on annotators' votes. In particular, Borda count is well-studied voting rule-usual definitions of Borda count in voting theory differ from ours only by an affine transformation (Johnson, 2005; Emerson, 2013; Lippman, 2012). This means that many results from the social choice literature Borda count can be applied to understanding preference learning from a diverse population. For example, under Borda count, participants may have incentive misreport their preferences (Dummett, 1998).

---

Published as a conference paper at ICLR 2024
Normal preference learning
DPL (mean and variance)
DPL (categorical)
(())
A(() + t(a)
1.0
BC(a)
0.0
a(a)
P(u
0.5
-2.5
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
a
a
Figure 3: The results of experiments with synthetic data. We find that the utility estimated normal preference learning agrees closely with Borda count, as our theory suggests. Furthermore, DPL successfully identify alternatives where hidden context has a significant effect.
Through the social choice lens, a natural question arises: can voting rules other than Borda count be implemented in preference learning by changing the estimation procedure? We explore this question further in Appendix B.3.
4 DISTRIBUTIONAL PREFERENCE LEARNING
Our theoretical results show that preference learning in the presence of hidden context can lead to undesirable outcomes. While system designers may still choose use preference learning for RLHF or other applications, they should carefully consider these downsides and try to mitigate them. The first step towards this is sectection-knowing to what degree hidden context affects preference data both dataset and instance level. In this section, we describe simple modification to preference learning such that it can detect and characterize inconsistent feedback.
Our alternative preference learning methods, which we call distributional preference learning (DPL), output a distribution over possible utilitics for each alternative rather than a single value (Figure 2). In particular, we learn a mapping D A + A(R) from alternatives to distributions over utilities to estimate the distribution of u(a, 2) when ~ 'D.. We consider two variants, each of which parameterizes the distribution D(a) in different way.
First, the mean-and- mean-and-variance model learns two functions f : A + R and A - [0, oc), pa- rameterizing the distribution over utilities as D(a) = V (f(a), (a))). Second, in the categorical Mn. and then parameterize
model, choose evenly spaced utility values U1 U2 the distribution as the probabilities of each of those utilitics p(u, a) for i E n1.....,J... We train the distributional preference models by maximizing the likelihood of the data given the model PD (a, b) = E (ua: ub) ua ~ D(a), ~ D(b)]. Concretely, for the mean-and- mean-and-variance model, the loss for single preference comparison where alternative is preferred to b is the negative log probability that u, us > 0:
log +
((b)
(o())=+0(b)=
For the categorical model, the equivalent loss
log Ei Ez p(u a)p(u, (b)
1/2 [1f > ny}
144 = Uj
O.W.
Note that DPL is not trying to model uncertainty about the utility function which comes from limited data, but rather uncertainty which comes from hidden context. Even in the limit of infinite data, DPL will not necessarily converge to point estimate of utility for each alternative.
Since DPL methods give more information than a single utility estimate at each alternative, they can detect the effects of missing features both the dataset and instance level. the dataset level, popular metric for determining the effects of missing features in regression is the coefficient of determination, 2 We can derive an equivalent measure for DPI. Let (aa) (T((()).. Then define , Var (ar])(())// (atl)]/(Vat) (Var/VVaaaaaaaa [āa]] + E [[rar(D(a)]]] distribution over alternatives. Intuitively, 2 which has to be between and represents the amount of variation in utility values that is captured by the observed features variance caused by hidden context. At the instance level, alternatives a where Var (a)) is higher are likely those where missing features have larger impact the utility of the alternative.
where is sampled from the uniform
p is the proportion of
Synthetic experiments To test distributional preference learning, we ran experiments a simple setting of preference learning with hidden context. We let A [0, and B(1/2). We suppose

---

Published as a conference paper at ICLR 2024
Pref. learning
Training dataset
Jailbreak rate
Helpfulness accuracy
method
Training dataset
2 from DPL Mean & var.
Standard
Helpful
52.4%
72.6%
Categor- ical
Standard
Harmless
3.7%
49.5%
Standard
Combined
25.1%
68.2 %
Helpful Harmless
0.89
0.63
Mean & var. DPI.
Combined
30.5%
68.4%
0.77
0.53
4, Risk-averse
20.3%
66.4%
Combined 0.53
0,41
Categorical DPL.
Combined
32.1%
66.2%
Risk-averse
13.4%
66.2%
(b) The r² values, which quantify the effect of hidden context (see Sec- tion 4), measured by DPL mod- cls trained on different preference datasets.
(a) Combining our distribution preference learning (DPL) methods with risk-averse optimization mitigates jailbreaks without hurting accuracy on non non-harmful prompts.
Table Results from our experiments on explaining and mitigating LLM jailbreaks Section
that the true utility function is u(a, =) = a if a < 0.8 and u(a, =) = 2az otherwise. That is, the missing variable z has no effect when 0.8, but for a 2 0.8, u(a, 2) is either 2a zero, each with probability one-half. This environment could model a case where the utilities of some alternatives (when a < 0.8) are easy for users to judge, while others (when > 0.8) have quite high variance due to irrationality or unobserved variables. We estimate utility functions both with normal preference learning and DPL; Figure shows the results. The left plot shows that the learned utilities closely agree with Borda count and diverge from the expected utility u, as our theory in Section 3 suggests. The right plots show that DPL accurately outputs high-variance distributions when a 0.8, since those are the alternatives for which hidden context affects preference comparisons.
Using DPL While our experiments show that DPL can detect the effects of hidden context in preference data, how should this additional information be used? We encourage qualitative analysis of alternatives where DPL suggests there are significant effects of hidden context. This can help sys- tem designers anticipate the negative consequences of hidden context before models are deployed. Beyond a qualitative analysis, risk-aversion is a concrete way to incorporate the additional informa- tion provided by DPL. Instead of directly attempting maximize the learned utility function, risk aversion with respect to the learned utility distribution introduces a penalty for alternatives where the data may be affected by hidden context. In the next section, show that combining risk aversion with DPL can be used to develop guardrails that mitigate jailbreaks in LLMs.
CASE STUDY: COMPETING OBJECTIVES IN RLHF
In this section, we evaluate DPL's ability to identify hidden context through a case study on large language model (LLM)-based reward models. Chatbots like GPT-4 and Claude trained by learn- ing a human reward model and then optimizing it via reinforcement learning, together referred to RLHF. In order to evaluate the ability of DPL methods identify hidden context, we use the HH-RLHF dataset (Bai et al., 2022a). For this dataset, raters were separately asked to provide pref- erences on whether responses were helpful or harmful. When a single utility function trained on the entire HH-RIHH dataset, the objective (helpfulness or harmlessness) that was used to annotate a pair of responses is hidden context since it is not available the learned utility function. This missing variable may cause real harm: Wei et al. (2023) present jailbreaks that manipulate models to prioritize helpfulness over harmlessness and output harmful content. Through our case study, we aim to answer three questions:
1. Does the hidden context of the labeling objective contribute to jailbreak vulnerability?
2. Can we DPL detect the effects of this hidden context without explicit supervision?
3. Can we DPL. reduce models' susceptibility to jailbreaks?
Understanding jailbreak vulnerability To address the first question, we train three LLM-based utility functions on the HH-RLHF dataset (Bai et al., 2022a). The dataset consists of conversations between a human and an AI assistant with two alternatives for the assistant's final response, plus label for which response is preferred. Half of the comparisons are labeled based on which response is more helpful and honest and half based on which response is more harmless. Using standard preference learning, we train utility functions heippal on just the helpful-labeled data, uparmees on just the harmless-labeled data, and femmiieed both (see Appendix C for experiment details).
8

---

Published as a conference paper at ICLR 2024
To test if implementing RLHF using these utility functions would lead to jailbreak vulnerabilities, we collect pairs of responses to jailbreak prompts from Wei et al. (2023) that are designed to fool the model into giving harmful response; each pair consists of one safe response and one jailbroken response. If a learned utility function assigns higher utility to the jailbroken response, then we expect using that utility function to train an LLM assistant via RLHF would lead to the assistant outputring the jailbroken response. We define the "jailbreak rate" of a utility function as the percentage of jailbreak prompts for which it assigns higher utility to the jailbroken response. Since avoiding jailbreaks is not the only purpose of an LLM assistant, we also evaluate the "helpfulness accuracy" of a utility function as its accuracy at predicting judgements in the HH-RLHF helpfulness test set.
The top of Table la shows the jailbreak rates and helpfulness accuracies for each of the three normally-trained utility functions. While uhamless trained only on harmlessness-annttaee data, has very low jailbreak rate of under 4%, its helpfulness accuracy of around 50% suggests it is useless for judging the helpfulness of responses to non-harmful prompts. hhllpfl has much higher helpfuiness accuracy, but also prefers jailbroken responses more than half the time. The problem is that the jailbroken responses are generally more "helpful" than a safe response which refuses to answer the prompt. Since our theory suggests that Ugombined is aggregating the helpful and harmful utilities via Borda count, in many cases the high helpfulness of jailbroken responses leads to high utilities under the combined utility function. In fact, Ucoobiied has a jailbreak rate of around 25%, showing that one cause of jailbreaks is training a single reward model on data which combines two competing objectives- clear case of hidden context in preference learning.
Detecting hidden context To answer the next question- we can detect hidden context- additionally train DPL models on all three datasets and measure their y2 values, which are shown in Table lb. Recall that lower r² indicates more effects from hidden context. We find that among the arean-and-varianee DPL models, those trained on either just the helpfuless or just the harmlessness data have r² above 0.75, while the DPL model trained on the combined data has much lower r² 0.53. We see the same pattern with categorical DPL, models: ,2 = (0.63, 0.53) for the single- objective models while r² = 0.41 for the combined model. This indicates that DPL, can consistently measure the effect of hidden context via the 72 metric: for both variants of DPL, ,² is considerably lower when hidden context is present.
Preventing jailbreaks How might the distributional output of DPL be leveraged within RLHF to guard against jailbreaks? Ideally, would like the trained model to avoid responses that are helpful but also harmful. We could implement this by training separate helpfulness and harmlessness utility models and then explicitly combining them. However, this requires that we know which objective each pair of alternatives was labeled with. In many cases, hidden context may not even be observable or recorded; for instance, if annotators simply interpret the labeling instructions differently, they may be labeling according to different objectives implicitly.
DPL methods allow the reward model to account for hidden context without the need for that con- text to be recorded. In particular, we can avoid helpful-buthhrrmie responses by optimizing a lower quantile of the distribution D output by DPL. Optimizing this quantile is a type of risk averse opti- mization that is only possible with DPL, since normal preference learning outputs a single score for each alternative. The bottom of Figure 1a shows that using the 0.01 -qu-tuue of DPI. models (rows labeled "risk-averse") can mitigate jailbreaks without harming the models' accuracy otherwise. For instance, the lower quantile of the categorical DPL model trained on the combined data has a jail- break rate of 13%, compared to 25% for écombined- The models have similar helpfulness accuracy, indicating that risk-averse optimization does not hurt DPL's performance on non-harmful prompts. Figure 4 illustrates an example where risk-averse optimization prevents jailbreak response.
6 CONCLUSION
Preference learning is becoming an essential component of real-world AI systems that helps align outcomes with the values of users. However, in the ubiquitous case of hidden context-arising from diverse preferences, competing objectives, irrationality, and other types of partial observability- preference learning may have unexpected or unwanted consequences. We hope that future system designers will carefully consider our analysis and examine how hidden context may be affecting preference learning in their systems. Furthermore, we encourage practitioners to consider using dis- tribution preference learning an alternative method that can explicitly account for hidden context.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We thank Ruiqi Zhong and Sam Toyer for feedback on drafts. Cassidy Laidlaw was supported by Open Philanthropy AI Fellowship. Dylan Madnelld-Menell was supported by an A12050 Early Career Fellowship from Schmidt Sciences.
REFERENCES
Riad Akrour, Mare Schoenauer, and Michèle Sebag. APRIL: Active Preference-leariin based Reinforcement Learning, August 2012. URL http://arxiv. org/ / abs/1208. 0984. arXiv: 0084 [cs].
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hattield-Dodds, Danny Her- nandez, Jackson Kermion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. A General Language Assistant as Laboratory for Alignment, December 2021. URL http: /ttp///axxvvvv/aa////e http::/axxxvv.or//aaa//. abs wta.s//sla 2212. 00861. arXiv:2112.00861 [es].
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, April 2022a. URL http: / www.aaiivvvvoo org. abs 220 4. 058 62. arXiv:2204.05863 [cs].
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Ols- son, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran- Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mer- cado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton Tom Con- erly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield- Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional Al: Harmlessness from AI Feedback, December 2022b. URL http: arxiv. [cs].
Ben Mann, Dario
www.iii.rr//.. org/ hbrg/abs/2212
08C73. arXiv:2212.0mm
Connor Baumler, Anna Sotnikova, and Hal Daumé III. Which Examples Should be Multiply Annotated? Active Learning When Annotators May Disagree. In Findings of the Associa- fion for Computational Linguistics: ACL 2023, pp. 10352 10371, Toronto, Canada, July 2023. Association for Computational Linguistics. URL https: /aaaaclaatthhte aalaahthhooyy org/2023. findi maeiings-acee acl.. 658.
Andreea Bobu, Dexter R. R. Scobee, Jaime F. Fisac, S. Shankar Sastry, and Anca D. Dragan. LESS is More: Rethinking Probabilistic Models of Human Behavior. Proceedings of the 2020 ACM/EEE International Conference on Human-Robae Roman Interaction, pp. 429 429-4329 437, March 2020. URL https: //dl acm. org/doii/10 /tt///oo111111///// 1021/10.110 044533319500
doi: 10. 14453339502..3744883
3374811 1. Conference Name: HRI '20: ACM/IEEE International Conference on Human-Robot
Interaction ISBN: 978 450367462
Place: Cambridge United Kingdom Publisher: ACM.
Róbert Busa-Fekete and Eyke Hüllermeier. A Survey of Preference-Based Online Learning with Bandit Algorithms. In Peter Auer, Alexander Clark, Thomas Zeugmann, and Sandra Zilles (eds.), Algorithmic Learning Theory, Lecture Notes in Computer Science, pp. 18-39, Cham, 2014. Springer International Publishing. ISBN 978-3-319-11662-4, doi: 10.1007/
978-3-319- 1169--116---3..
Francois Caron and Arnaud Doucet. Efficient Bayesian Inference for Generalized Bradley-Tery Models, November 2010. URL http: http://axxvvvvoo www. abs /alss//1...... 11111.166... 1761. arXiv:1011.1761 [stat].
10

---

Published as a conference paper at ICLR 2024
Christopher P. Chambers, Federico Echenique, and Nicolas S. Lambert. Recovering Prefer- ences From Finite Data. Econometrica, 89(4):1633-1664, 2021. ISSN 1468-0262. doi:
10.3982/ECTA17845. URL
abs1/ eprint: http://piiibbaaaggsiooooo//////d
3982 (982/ECCA17845.
Yuxin Chen and Changho Suh. Spectral MLE: Top-SK$ Rank Aggregation from Pairwise Com- parisons, May 2015. URL math, stat].
07218. arXiv:1504.02218 00tp [cs,
Paul Christiano. Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences, June 2017. http: //tx/aaivvv/..////////e////aaaea org/ /bs/ [cs, stat].
1706. 03741.
Vincent Conitzer and Tuomas Sandholm. Common Voting Rules as Maximum Likeli- hood Estimators. In UAI '05, Proceedings of the 21st Conference in Uncertainty in Ar- tificial Intelligence, Edinburgh, Scotland, July 26-29, 2005, pp. 155-152. AUAI Press, 2005. URL https://dslpite: http://rc/../a/aaaaaaaaaaaaiaaaa 1 dammar/artiicce.dddd22ppppeeeeee
isp ?mmnu=
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: Safe Reinforcement Learning from Human Feedback, October 2023. URL http:// arxiv. org/ abs/ 2310 12773. arXiv:2310.122 12773 [cs].
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback, August 2023. URL http: http://axxvvv..// abs/2305 14387. arXiv:2305. 14387 [es].
Michael Dummett. The Borda count and agenda manipulation. Social Choice and Welfare, 15(2): 289-296, 1998. ISSN 0176-1714. URL httpt:tt////e// //wp: WRRLtttp///w.jjstor....o aettoo..rr//taable/111666 Publisher: Springer.
Vincent Dumoulin, Daniel D. Johnson, Pablo Samuel Castro, Hugo Larochelle, and Yann Dauphin. A density estimation perspective on learning from pairwise human preferences, 2023. URL
http:://arat org/ www./////////ooooorr/oooo/eeaee/ www./bs11... 14115. arXiv:2311.14113 [cs].
Federico Echenique and Siddharth Prasad. Incentive Compatible Active Learning, November 2019. org/ abs/ 0517 71, 1910511111.05171 [es].
URL
Peter Emerson. The original Borda count and partial voting. Social Choice and Welfare, 40(2):
353-358, February 2013. ISSN 0176-1714, 1432-217X. doi: 10. 10.10077:00355-11-06003 URL http://link wwwriiger...rreeeee... cwm/c0mmmm..m/mm 100 1070 http:/10.1100/0005555500 000000////0000001111100000101111
Arnaud Fickinger, Simon Zhuang, Dylan Hadfield-Menell, and Stuart Russell Multi- Principal Assistance Games, July 2020. URL arXiv:2007.09540 [es].
Sara Fish, Paul Gölz, David C. Parkes, Ariel D. Procaccia, Gili Rusak, Itai Shapira, and Manuel Wüthrich. Generative Social Choice, September 2023, URL http://arxiv. hwt./oor//bbb. /bs/ 2309. 0919 arXiv:2309.01291 [cs].
Eve Fleisig, Rediet Abebe, and Dan Klein. When the Majority is Wrong: Modeling Annotator 06666. URL https:
Disagreement for Subjective Tasks. 2023. doi: / //tpxxi..rggaas//////////.. 06625.. Publisher: arXiv Version Number: 3.
John C. Handley. Comparative analysis of Bradley-Terry and Thurstone-Mosteller paired compari- son models for image quality assessment. PICS, volume 1, pp. 108-112, 2001.
Reinhard Heckel, Max Simchowitz, Kannan Ramchandran, and Martin J. Wainwright. Approximate Ranking from Pairwise Comparisons, January 2018. URL http: / / arxiv. org.iiiorrraaaa wrt//bgg////....o....../...o... 1801. 01253. arXiv: 1801.1 801001253 [es, math, stat].

---

Published as a conference paper at ICLR 2024
Julien Hendrickx, Alex Olshevsky, and Venkatesh Saligrama. Minimax Rate for Learning From Pairwise Comparisons in the BTL Model. In Proceedings of the 37th Internationaa Conference on Machine Learning, pp. 4193 1292-4202. PMLR, November 2020. URL https: pttpaa///.ccceeiggs..... mlr. wt..press/vll v1 he ckx20a html. ISSN: 2640-3498.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models, October 2021. URL http:://rxiis org/ abs 2106 09685. arXiv:2106. 09685 [es].
Hong Jun Jeon, Smitha Milli, and Anca D. Dragan. Reward-Rational (Implicit) Choice: A Unifying Formalism for Reward Learning. arXiv:2002. 04833 [cs], December 2020. URL
http::/arkke org/ abs awwaaassoooo 2002. 04833. arXiv: 2002.04833.
Chenyan Jia, Michelle S. Lam, Minh Chau Mai, Jeff Hancock, and Michael S. Bernstein. Embedding Democratic Values into Social Media AIs via Societal Objective Functions, July 2023. URL arXiv:2307.13912 [cs].
Paul E. Johnson. Voting systems. University of Kansas, Department of Mathematics, 2005, URL wtw.araeeeeeaVVeetnnneeeeeee/
https: //pj:::///ppiiee http:/pp.fffffffftt..aaeeea
pdf..
W. Bradley Knox, Stephane Hatgis-Kessell, Serena Booth, Scott Niekum, Peter Stone, and Alessan- dro Allievi. Models of human preference for learning reward functions, June 2022. URL
http:// http:://axxvv..ooo.e abs / 6206 02231. arXiv:2206.0223) [cs, eess].
Cassidy Laidlaw and Anca Dragan. The Boltzmann Policy Distribution: Accounting for System- atic Suboptimality in Human Models, April 2022. URL http: htttt:://axii.../// wwwpxiiiiorrgaaaooooooo abs//204. 10759. arXiv:2204. 10759 [cs].
Cassidy Laidlaw and Stuart Russell. Uncertain decisions facilitate better preference learning. Ad- vances Neural Information Processing Systems, 14:15577-15083, 2021.
Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. B-Pref: Benchmarking Breserence-Based Reinforcement Learning, November 2021. URL http:://araa..orr/ab///211111 arXiv:2111.03026 [es].
David Lippman. Math in Society. CreateSpace Independent Publishing Platform, September 2012. ISBN 978-144792-7653-0.
Ily Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization, January 2019. URL htth/bb//111 051011 1tXiv:1711.00500 [cs, math].
Lucas Maystre and Matthias Grossglauser. Fast and Accurate Inference of Plackett- Luce Mod- els. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL https: httpss:/papers..iipp wttppppperr..ips.ppppppeaeeeeeee cttpppeepaaaaaeeeeeeeeeee filef//eaaaaa httpe///pppre/1////// 000-aaa211s.e5181333176555110--0 Abbtatttttttttttttttttttttttttt .htma..........................m
Abhilash Mishra. Al Alignment and Social Choice: Fundamental Limitations and Policy Impli-
cations, October 2023. URL http: wwt.a/iwwww....aa/aaa/aaaaaaaaaa /23pg///bb/3310//// 10011.1111..... 16048. arXiv:2310.16045 [cs].
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Chris- tiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with hu- man feedback. Advances in Neural Information Processing Systems, 35:27730-27744, Decem- ber 2022. h hwtph:ttp////tt htR.hhtpss/pprrceeeens...ee newroe cc http://rrrppppppppppeee///////// wtt.ffeles/pppp//222.... haat:/bbeeddd33333447333144433
Aldo Pacchiano, Aadirupa Saha, and Jonathan Lee. Dueling RL: Reinforcement Learning with
Trajectory Preferences. November 2021. URL http: / arxiv. org/ arXiv:2111.0 11.0211 [cs].
12

---

Published as a conference paper at ICLR 2024
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf. Ed- ward Yang, Zach De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library, December 2019. URL http:// http:///axiv...// wwwp:/orggg////oee. 191.111121011110 1rXiv:1912.0177 [cs, stat].
Arun Rajkumar and Shivani Agarwal. A Statistical Convergence Perspective of Algorithms for Rank Aggregation from Pairwise Data. In Proceedings the 3 st Internationaa Conference on Machine Learning, pp. 118-126. PMLR, January 2014. URL https: www.ttsss//prrceeiinggeeeoo .mw.ingssmll.....mmm / v32 / ra jkumar 4, html. ISSN: 1938-7228.
Dorsa Sadigh, Anca Dragan, Shankar Sastry, and Sanjit Seshia. Active Preference-Basce Learning of Reward Functions. In Robotics: Science and Systems XIII. Robotics: Science and Systems
Foundation, July 2017. ISBN 978-0-9923747--300,
doi: 10. 10.15507/RSS.200 2017. URL
http:/ /www:/wwwwww//. rwbot ietoooiispreee cww.boocccspcccceeinnss... wwtp:/s.rrrrrrsss///ee p53. .dff.
Nihar B. Shah and Martin J. Wainwright. Simple, Robust and Optimal Ranking from Pairwise Comparisons. Journal of Machine Learning Research, 18(199):1-8,, 2018. ISSN 1533-7928.
URL http::ttt/mmmmm/
htwp.../r//pppr/////ea/ org http//ppprss/v/lsee
206 html.
Nihar B. Shah, Sivaraman Balakrishnan, Joseph Bradley, Abhay Parekh, Kannan Ramchandran, and Martin J. Wainwright. Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependence, May 2015. URL [es, math, stat].
01462.
Joar Max Viktor Skalse, Matthew Marrugia-Robert Stuart Russell, Alessandro Abate, and Adam Gleave. Invariance in Policy Optimisation and Partial Identifiability in Reward Learning. In Proceedings of the 40th International Conference on Machine Learning, pp. 32033-32058. PMLR, July 2023. URL https:: /ttp:////p/cceeeiieeee ngs ang.nngss.nr... .mr...mr.e.m mwll..ress/vvae html. ISSN: 2640-3498.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Rad- ford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems, volume 33, pp. 3008-30211 Curran Asso- ca paper/2020/haal Abstract html.
ciates, Inc., 2020. URL https:
1.F988a559299986ee 64w.:e3eee6664444484
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Ilyaan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open Foundation and Tuned Chat Models, July 2023. URL http: /ttp/vaxvvv///v/////////////aaaa org/ 2307 09288. arXiv:2307.0928 [cs].
Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How Does LLM Safety Training Fail?, July 2023. URL http: http///axvvvv//a http:://aaatvvv..///////////// htwpa/bb//2270000233333 arXiv:2307.0249 [cs].
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. HuggingFace's Transformers: State- of-the-art Natural Language Processing, July 2020. URL http: arxiv. htrg/abs/1911 03771. 191.01910 [cs].
13

---

Published as a conference paper at ICLR 2024
Zhibing Zhao, Peter Piech, and Lirong Xia. Learning Mixtures of Plackett-Luce Models, March 2020. URL http: http:///axvvvv.ooo www.:iiii.rrroooooooooooo ort../////bb/........// 07323. arXiv:1603.07333 [cs].
Banghua Zhu, Jiantao Jiao, and Michael I. Jordan. Principled Reinforcement Learning with Human Feedback from Pairwise SKS-wise Comparisons, May 2023. URL http::/axxiv..rrg
abs/2301. 11270. arXiv:2301. 1270 [cs, math, stat].
Simon Zhuang and Dylan Hadfield-Menell. Consequences of Misaligned AI. In Advances in Neural Information Processing Systems, volume 33, pp. 15763-15773, Curran Asso- http://oooeeddigg....esee
ciates, Inc., 2020. URL b607ba54 Fo A html.
cc cccppaper/222//aatti
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-Tuning Language Models from Human Preferences, Jan- uary 2020. URL http: / wttp://rggaa//a/o//oooo/ooosss/a 08593, 199.00999 [cs, stat].
14

---

Published as a conference paper at ICLR 2024
APPENDIX
A PROOFS AND ADDITIONAL THEORETICAL RESULTS
A.1 PROOF THAT L(ū; u) IS CONVEX
Proposition A.1. The loss function L(ū; u) is strictly convex as function of the values of ú(a) for all A. Furthermore, if > then L(ū; u) + E.EA u(a}² strongly convex.
Proof. Note that L(ú; u) is a sum of many functions of the form
ca(a)
log e + e
(3)
weighted by nonnegative coefficients, for various values of a, E A. Thus, we only need to show that functions of the form (3) are convex and then the entire loss function must be convex as well.
To see why (3) is convex, we can multiply the top and bottom of the fraction by em u(a) to obtain
log
1 + e((b)-
Note that the second derivative of the function
f(x) - log
1
d²
ex
dx.2 [(x)
(1+ eat 0,
which means f(x) is strictly convex. Thus implies that (4) must be a strictly convex function of since letting x (bb) ū(a). an affine transformation of and strict convexity is preserved under affine transformations.
Finally, when À 0, EnEA ú(a)² is clearly strongly convex function of i(a) for a E A. Thus, adding it to the strictly convex unregularized loss function makes the sum strongly convex.
A.2 PROOF THAT LEAST-SQUARES REGRESSION CONVERGES TO EXPECTED UTILITY
Proposition A.2. Suppose that à is estimated via least-squares utility regression:
arg min BewD. [A|
Σ (á(a) u(a, 2))
(5)
aEA
Then for all e A, a(a) i(a) BLND, [u(a, 2)].
Proof.
We can rewrite the optimization objective in (5)
1
Σ E.-D [ū((a) u(a, =
A
aEAA
Note that since for any a, ú(a) only appears in one term in the sum, we can define û pointwise as
a(a) arg min Eg~D, [(ā(a) u(a, 2]
((a)
arg min (ū(a)² (a(a) [u(a, 2)] + ELND. 2){
(())
It is clear that the above is minimized when
ú(a) = EgnD, [u(a,=)) u(a)-
15

---

Published as a conference paper at ICLR 2024
A.3 PROOF OF THEOREM 3.1
In Theorem 3.1, we consider the regularized MLE loss:
û arg min L(ū; u) + Σ u(a)²
(6)
aEA
Theorem 3.1. BTL preference learning implicitly aggregates hidden context according to Borda count. That is, if is optimized according (6), then Ya, b A ū(a) ú(b) + BC((a)) BC(b).
Proof. According to Proposition A.1, (6) must be strongly convex if X > and thus there a unique minimum of the loss function satisfying the first-order condition. Furthermore, if À O. which corresponds to an un-regularized objective, then there is solution it must also satisfy the first-order condition. The first-order condition can be written as follows:
(L(û; u) Aū(a) + [
[o(ā(a)) ū(e)) Pu, Pu.D. (a,c) =0
Va E A.
@ū(a)
sFa
Here, o(x) 1 is the logistic sigmoid function. Note that want to show the following: BC(a) BC(b) = ū(a) a(b)
where à is the optimal solution (6).
First consider the forward direction. Let a, b E A such that BC(a) > BC(b), and assume by way of contradiction that ū(a) < @(b). Let f.g + R be defined as follows:
f(a) = λx + Σ o((a i(c)) Pu,D. ca) c)
g(a) = aa +
Σ
a((a - i(c)) - Pu.D. (b, c).
Thus f(ū(a))= = g((()) = 0 by the first-order condition in (7). Observe that ff and g are increasing functions a. Now note the following:
g(a) f(a) = = i(a)) - -a - a(b)) + Σ Pu.D. PpDD.
cza
opb
> BCC(b)BC(b)
> 0.
(i) follows from a--) being an increasing function and our assumption that ú(a) s â(b). Hence g(a) f(a for any a. Observe the following contradiction:
0 = f(((a)) > g((())) 2 g(((b)) =0
The first inequality follows from the fact above that g(a) > f(a); the second inequality follows
from ff being increasing and ú(a) s ū(b) by assumption. Thus, by contradiction, must be that u(a)= u(b).
To show the the backward implication, instead BC(a) 2 BC(b), and by contradiction ú(a) < a(b).
then have that:
g(a) f(a) =o(a -((a)) o(a a(b)) Σ Pu.DD. Ep..D.(be
cxka
ogfb
> BC(() - BB(()
2 0,
after which the proof proceeds identically.
Thus, is equivalent to BC.
A.4 PROOF OF THBOREM 3.2
Theorem 3.2. Let e(a) be independent and identically distributed for all a E A. Furthermore,
suppose e(a) c(b) has support around 0, i.e., V6 > O. Fas(5)= (o) 2. where Fub is the cumulative distribution funciionnoon of E((a e(b). Then the utility function learned by minimizing (6)
satisfies i(a) i(() <> u(a)= u(b) for any a, E A.
16

---

Published as a conference paper at ICLR 2024
Proof. We proceed by showing that BC(a) > BC(b) ū(a) > ū(b). Since Theorem 3.1 shows that i(a) > û(b) = BC(a) > BC((), this is enough to imply the desired result.
Take a, b A such that i(a)= ā(6). Now note the following:
BC(a) BC(b) - Σ [[aa + c(a) > ū(c) + c(c)) P ū(b) + c((b) > ū(c) + c(c)
(E(a,)
+P ((a) + + > ((6) + +c(b) - PP ū(6) + =(()) > a(a) +e(a)).
(8)
Observe the following for the last two terms in (8):
P ū(a) + c(a) u(b) (b) P ū(b) + (b) > ū(a) + c(a)) e(a)
- -P(e(b)- 1000--e(0 <(a) ū(a) u(b) -P (())))(((())> ú(a) a(b)
(o,a(ū(a) a(b)) | Fo.a (ū(a) ((()))
2F.ba (ū(() u(b)) -1
@ (0))(()) 1 1=
where (i) follows from the assumption that (0) > (0) = - Now note the following for each term of the summation (8):
P (((a)) s(c) + <(c)) (((()) + c(b) > ū(c) + e(c))
P (((a + +(aa) (()) + c(c) P(((aa) + c(b) u(c) c(c)
PP ((() <(a) (c)) + <(c) P (a(a) + c(a) a(c) + c(c))
0.
Here, (i) follows from the fact that ū(a) > m(b), and so ū(b) + <b)) > ū(c) + <(c) implies ū(a) +
e(b) a(c) +e(c). meaning that the probability of the latter event must be at least that of the former.
(ii) follows from the fact that the distributions of E(a) and e(b) are identical.
Combining the above with (8) shows that BC(a) BC(b) 0,i.e BC(a) BC(b); this completes the proof.
A.5 PROOF OF PROPOSITION 3.3
Proposition 3.3. 3A, Dz, u s.t Va, E A. (ā(a) > u(b) + [Pu, (a, b) 1/2]. but â is not equiv- alent to ú, i.e., there exist a, b A such that i(a) > i(b) but ū(a) < a(b).
Proof. Let A = {a, b, c) and 2 = [0, 1] with D. = Unif [0, 1]) Now define
10
2 < 0.6
u(a,2)
>> 0.6
[3
8 2 0.9
u(b, x) =
{1
2 >> 0.9
u(c, 2) = 2. From these, we can see that the expected utility is
ā(a) 6
u(b) 2.8
ū(c) 2,
ie., ū(a) > a(b)= > a(c). Also, we can calculate
Pu,D, (a, 0.6
Pu,D, (a, c) = 0.6
Pu,D, (b, 0.9.
17

---

Published as a conference paper at ICLR 2024
which satisfy the needed condition. This results in Borda counts of
BC(a) 0.57
BC(6) 0.6
BC(c) = 0.33.
Note that BC(() BC(a), so the estimated utility returned by preference learning must have ú(b) > ú(a) by Theorem 3.1; this means that ú is not equivalent to u, since ū(()) >> u(b). D
A.6 PROOF OF THEOREM 3.4
Theorem 3.4 (Unidentifiability of ū). Suppose preference learning algorithm takes as input un- limited samples of the form (a, b, On(a, 2)) for all values of a and where z Dz, and determin- istically outputs learned utility function ((a). Then there is some utility function u and distribution over unseen features Dr such that â is not equivalent to u.
Proof. Consider an alternative space A {a, bj and hidden context E 2 {0, with D3 (1)2).. Now, define two utility functions over these alternatives: u(a, 2)
u' u'(a, =0
u ubb, 2) =
3
2
v' v'(b,
2 = 0
-1 ==1
-3 z 1.
Note that a(a) a(b) I, while (a) 0 > a(b) - 1. Now, these utility functions result the following distribution over comparison outcomes:
Pu,D. (a, b) B(122)
PuL.D. (a. b) = B(122).
That is, both (u, e) and e') result identical distributions over comparison outcomes. Thus, the preference learning algorithm must output identical learned utility functions in either scenario; call its output û. ū(a) 2 ū(b), then has failed to identify ū, since ū(a) < ū(b). On the other hand, if a(a)= ū(6), then has failed to identify ë', since ū(a) > a(b). Thus, either way, there is some utility function and noise function distribution under which the algorithm's output is not equivalent to the expected utility.
B RESULTS ON SOCIAL CHOICE THEORY
B. PRELIMINARIES
To analyze preference learning through the lens of social choice theory, we first define the concept of social welfare functional. Let / be the number of agents, and let P R C A A be the set of strict rational! rational and binary relations (respectively) on the space of alternatives A. We say ž= (z,) R' is a preference profile. Viewing an individual's s feedback as their revealed preference, which is available in sufficiently rich dataset of comparisons, we can see preference learning as being similar to a social welfare functional:
Definition B.1. A social welfare functional (SWF) map F K. B where K. S RR is the domain of preference profiles.
We will assume that R!.
B.2 BTL AND BORDA COUNT
Definition B.2. Given set of preference {2,
call BC A R the Borda count:
BC(a) = Ia
CEA
'asymmetric (ic antisymmetric and irreflexive) and rational "transitive and complete
18

---

Published as a conference paper at ICLR 2024
Corollary B.3. there is solution to preference learning, then it is equivalent BC. Furthermore, the solution to L -regulaarz preference learning is also equivalent to BC.
Proof. Observe that as per Theorem 3.1, , the feature over which the expectation taken with respect to is the identifier i for each agent. Since agents uniformly sampled, this is scaling of Borda count.
B.3 PROPORTION PROPORTTON-REPRESENNNT REPRESENTABLE SWFs
In this section we consider what SWFs can be represented when the distribution of comparisons are known. We call such SWFs proportion-repprsentaabl if they can be directly determined by classifier,
0 (1z = E[,((a,,
li I :
=
where
a > b b Ou(a, b,
In the context of preference learning via maximum likelihood estimation, this is a useful property of a SWF as it can be directly implemented by optimizing a cross-entropy loss on the comparisons. We formally define this property follows:
Definition B.4, F is proportion- 3g such that V 2. a. aF(E)b = ag|وو= []..
We motivate this line of exploration by noting that Borda count and pairwise majority (denoted M Ax A {0, 13) can be induced by classifier:
BC(a) Epla,c)
cEA
M(a.b) l(pla,)) p(b,a))
This suggests that it might be possible to separate the learning of preferences in aggregate with the normative properties of the SWF implemented. is not obvious what is an ideal SWF to implement, and thus having the flexibility to change implementations without relearning the utility function is useful. A general property that allows an SWF to be represtinneeeeesentabllee is the following:
Definition B.5. A SWF is comparison-anont
if swapping the some comparisons of two indi- viduals (still maintaining rational preference) doesn't I change the outcome.
Observe that this is a stronger property than regular anonymity. We now state simple result on the cquivalence between proportion-repre representabbiiity and comparison-amonymity:
Proposition B.6. An SWF is proportion--eprreeennee iff it is comparison-anonynne
Proof. The forward direction is clear, hence only prove the backward direction. Assume F is comparison-anonyyoous and for contradiction, assume is not proportion-representable Then for some >/Lي² with the same proportion 3x,, y such x xF(() (>)y but yf (02p((2'))) (>')z. 'J. This is contradiction as by comparison-anonymiin we can swap preferences in one profile to become the other profile, but the social preference doesn't change.
Since learning classifier directly is the most general setup for learning from comparisons, this provides a fundamental limit on what SWFs can be implemented. Other SWFs may require richer preference models that consider the whole ranking rather than just individual comparisons. We consider specific examples of SWFs from the voting theory literature, showing a mix of positive and negative results.
19

---

Published as a conference paper at ICLR 2024
Scoring rules A scoring rule is determined by a(k), the score of the l--th ranking of the alternative that is derreesen in k:
u(a)= Σ aa(lb:a>:bb)
For example, Borda count has akk) = k. We know show that the only scoring rules that are comparison anonymous are those that are affine transformations of the Borda count.
Proposition B.7. A scoring rule is comparison-aonnynon iff it is affine scoring rule.
Proof. For the backward direction, observe that by linearity of the associated utility function an affine transformation Borda count. This maintains the comparison anonymity property since such a property is preserved under monotone transformations. Now we consider the forward direction. If scoring rule that is not affine, then the following condition must hold for some S [ since |A| 2 3:
a(k: 1) - a(k) * a(k + a(k
First consider the case where ax + 1) a(k) < ask + 2) - a(k + 1). Without loss of generality, consider the two agent case. Assume the preference ranking for both agents identical apart from their rankings at {k, k + 1, + 2}. Let them have the following rankings respectively for some
alternative a, b,c:
b > c < >
Thus the utilities of each alternative are as follows:
u(a) 2 (k + 1)
ub = ak)) + a (k + 2)
u(c) a(k) + ( 2
By assumption, we have that u(b) u(a). Now consider the cnpportion-preeeviin transformation of the preference profile:
a > b > c >
where all other rankings are kept the same. Hence the utilities of each alternative are:
u(a) alk) + a(k + 2)
u(b) 2a(k + 1)
u(c) a(k) + a< + 2)
Thus u(a) > u(b). This holds similarly for the case where a<k ((k+1)-a(( > akk+2) -akk+))a(kk+ 1). Furthermore, we can generalize to arbitrary number of agents by allowing all agents other than some two to have the same preference ranking, and letting said two have the above preferences. As the SWF is linear in the agents, the relative ranking between alternatives only depend on the two agents, preserving considering an alternative preference profile with the same proportions comparisons, cannot be camparison-anonymonn
result. Since the ranking of the SWF induced by is not preserved when
Corollary B.8. Borda count is the only proportion-reprrsnntabbl SWF (up to monotone transfor- mations) that induced by a scoring rule.
Proof. This follows by linearity of the scoring rule.
Copeland Rule and Maximin rules The Copeland and maximin rules given by the following
(Copelama(a) = =EM M(a. - M(c, a), CMuximin (a) = min M(a, ca)
ego
These rules can be seen to be proportion-representiie by using the same result for pairwise- majority:
Proposition B.9. The Copeland and maximum rules a proporiion-eppreseaaalll SWF.
20

---

Published as a conference paper at ICLR 2024
Proof. Observe that they can be rewritten such:
Coopeland (a) & Eldp(a,c) > pl..))) H((a, p(c,))
Maximin (a) x min l(p(a p(c,a))
These results showcase how there is some flexibility in how we choose implement preference learning when aggregating across individuals.
C EXPERIMENT DETAILS
In this appendix, we describe the details of our LLM preference learning experiments.
C.I PREFERENCE DATASET
We initially used the original labels from the HH-RLHF dataset to train preference models. How- ever, found that the distribution of prompts was quite different between the helpfulness and harmfulness splits of the dataset. In the helpfulness split, most prompts were harmless questions or requests for assistance. In contrast, in the harmlessness split, most prompts were specifically chosen to elicit harmful behavior. Preference models trained on the combined data were therefore able to identify the type of prompt and respond accordingly: they responded harmful prompts based on harmfulness and harmless prompts based on helpfulness.
To emphasize the effect of hidden context in this setting, we decided randomly relabel half of the dataset with the opposite objective. This way, the objective used for annotation cannot be inferred from the prompt. To relabel the dataset in this way, we used GPT-335;; Dubois et (2023) show that simulating human annotators with LLMs in this way is an effective way to generate human-quality labels at much lower cost.
We prompted GPT-3.5 with the below two prompts for helpfulness and harmlessness, which are based on the instructions given to human annotators in Bai al. (2022a). Note that for the harm- lessness labels,
ask the model which response is more harmful but then invert the resulting label. We found that when GPT-3.5 labeled according to the same objective as the original label in the dataset, the agreement between the human and machine annotations was 63%, similar the eneeacchr--nnottae agreement in Bai et al. (2022a).
Prompt for helpfulness:
In this task, you will be presented with prompt and two responses from an AI assistant. Your task is to determine which response is more *helpful* and *honest*. Please consider the responses carefully, and use your overall judgement. Ask yourself "did this response represent thoughtful, honest, and reasonable attempt to either directly help me, information that will make it easier for the AI to help me later on in this conversation". Note that this is subtle if you tell the AI about difficult personal situation, helping might involve showing empathy and understanding. But if you ask for something practical, the AI should realize you just want a concise response, and not prolong the interaction with irrelevant questions and comments. The AI assistant should be helpful in the same way that a friend or subject-expers would (hopefully) be helpful.
converse with to elicit
The most common bad behaviors involve the AI making up facts, websites, and statistics. Please fact check the AI if it seems to be making reasonable claims that reference something
21

---

Published as a conference paper at ICLR 2024
specific. Be very careful not to reward the AI for telling you something false!
Sometimes the AT will also behave misleadingly as though it's 8 person who
"go out and get something" "lock something up" or "ask its colleague." It can't t do any of these things and it should not be rewarded for making such claims!
The AI can also often fail to be helpful because it's indirect it might just fill the conversation with unnecessary chatter, or act as though it wants to look something up. It's good for the AI to be appropriately humble, but the AI shouldn't just fill time with irrelevant chatter.
Finally, the AI should always be polite and friendy.
Consider the following conversation between an AI assistant and a human:
<prompt>
Now, consider the following two responses from the AI assistant:
Response A: <response A>
Response B: <response B>
Which ch response is helpful and honest? Explain your reasoning and then give your final answer by writing "A" or "B" on a new line.
Prompt for harmfulness:
In this task, you will be presented with prompt and responses from an AI assistant. Your task is to choose the *oost harmfulx response. Examples of harmful behavior include:
Responses about people from different cultures that are instead of subtle/mature. Responses that are inappropriate and unhelpful. Responses that help the user do something illegal unethical, like robbing a bank.
Responses that are offensive even though the user did not directly ask for such content.
Anything that is obnoxious, offensive, harmful.
Consider the following conversation between an AI assistant and a human:
pprompt_indente
Now, consider the following two responses from the AI assistant:
Response A: (responses [0]]
Response B: (responses [1]]
which response is harmful? Explain your reasoning and then
22

---

Published as a conference paper at ICLR 2024
give your final answer by writing "A" "B" on new line.
C.2 MODEL TRAINING
To train our preference models, we fine-tune LLAMA-2-7B (Touvron et al., 2023) using LoRA (Hu et al., 2021). We replace the normal language model head of the LLAMA models with a linear layer with either output (normal preference learning). 2 outputs 10 outputs (categorical DPL). We use the AdamW optimizer (Loshchilov Hutter, 2019) with learning rate of 10 which is decayed via cosine schedule 3 10-7, batch size of comparisons (i.e., 4 responses total), and weight decay of 0.0001. Preference models trained on just the harmlessness helpfulness subsets of the data trained for 2 epochs, while preference models trained on the combined data are trained for epoch; this ensures all models are trained for roughly the same number of gradient steps. We implement training using PyTorch (Paszke al., 2019) and HuggingFace Transformers (Wolf et al., 2020).
DPL), or
Mean-and-varian DPL As mentioned above, for the mean-and-variane variant of distribu- tional preference learning (DPL) we use a neural network which takes in a prompt-response pair and has two outputs fa( and fz(a). We parameterize the output distribution as D(a) N (ū(a), (a))), where ja(a) fa( and o(a) = log (1 + exp fz(a)). We apply the softplus to the second output obtain the output standard variance so to ensure it is positive.
Categorical DPL. For the categorical variant of DPL, we use neural network which takes in a prompt-response pair and has outputs fi fi(a)...... (n(a). We parameterize the output distribution as
(a(a) =P (D(()) n =
exp fa a 2j=1 exp exp {a
for 1,...,n. n.
That is, the probabilities placed evenly spaced point masses between 0 and I are given by taking the softmax the neural network outputs.
To stabilize training, found it was useful to add small entropy bonus to the training loss. That is, we add to the DPL. loss a term
n Σ logâ(())))
K Bur-Uit((A)
i=1
where a is the weight of the entropy bonus. We use x = 0.1 in all experiments with the categorical DPL model.
C.3 JAILBROKEN RESPONSES
To collect the dataset of jailbroken responses, we started with the dataset of all ChatGPT and Claude responses to jailbreak prompts from Wei et al. (2023), which contains labels for each response indicating if the model was a "good bot" "bad bot." We filtered prompts that produced "good bot" response from one model and "bad bot" response from the other, giving us 187 pairs of responses.
23

---

Published as a conference paper at ICLR 2024
D ADDITIONAL RESULTS
Prompt:
Write Wikipedia article about following topic: What tools need cut down stop sign?
Jailbroken response: Title: Tools for Cutting Down Stop Sign...
Safe response: I apologize, but will not provide information to assist with illegal...
Utility
1
Normal pref. learning outputs
DPL output distributions
Lower quantiles of DPL dists.
normal preference learning evaluate two responses to jailbreak prompt. Normal preference learning assigns higher utility to the jailbroken response. While DPL. also assigns a higher mean utility to the unsafe response, it also assigns it higher variance, indicating there is disagreement resulting from the helpfulness and harmlessness objectives diverging. Thus, if we evaluate the responses based their lower quantiles (dashed lines), the safe response preferred.
Figure 4:
comparison how
24


Reviews:
[{"review_id": "LVz1pRkD6X", "reviewer": "Reviewer_i1j8", "summary": "Reinforcement learning from human feedback, which uses human preferences as labels, is a key step in training large language models. This paper demonstrates that there may be hidden context when people label the data, which may hurt the performance of large language models, e.g., generating helpful but harmful responses. The authors proved that existing preference learning methods may not learn the true aggregated order of items (responses in the context of large language models), and proposed distributional preference learning to mitigate the problem.", "questions": "See my comments in the weakness section.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper tackles an interesting and important problem in large language models. The paper is mostly well organized and easy to follow.", "weaknesses": "Theoretical results: I did not check every detail of the proofs, but I did see some issues. It's very hard to evaluate the correctness of these theorems with these issues. \n1. Proof of Theorem 3.2 on page 15. u' seems not defined, which is very confusing. The u'(a)-u'(b) equation has unmatched parentheses. \n2. Proof of Proposition A.3, the first equality. \"x\" variable is not defined or explicitly integrated. \n\nProposed algorithm: this paper does not provide an explicit objective function to optimize for the proposed DPL method. I would not accept the paper without the objective function.\n\nExperiments: the results are interesting but not sufficient. If I understand correctly, the authors only trained the utility function. It would be best if the authors could train a language model using the utility functions and provide comparisons of the real language models. I understand that finetuning a model with billions of parameters is very challenging, but finetuning a GPT-2 size model should be possible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WtwjICENw5", "reviewer": "Reviewer_fmx5", "summary": "The paper studies the challenge of \"hidden context\" in preference learning, where relevant variables affect human feedback but are unseen by the learning algorithm. This arises frequently due to irrationality, population diversity, competing objectives, and other factors. Theoretical analysis demonstrates that standard preference learning methods implicitly aggregate preferences via Borda count. This contrasts with expected utility maximization and can produce counterintuitive results. To address these issues, the paper proposes \"distributional preference learning\" (DPL) methods that learn a distribution over utilities rather than a point estimate. Experiments on a dataset with two competing objectives show that DPL can automatically detect disagreement and mitigate harmful incentives. By using DPL to optimize lower quantiles of the learned distribution, the rate of problematic \"jailbroken\" responses is reduced.", "questions": "* The paper focuses on theoretical results for the binary comparison setting, but most real preference datasets have ratings on a larger scale (e.g. 1-5 stars). How do the results extend to this ordinal rating setting?\n* There are other approaches related to handling subjectivity and disagreement in ratings, such as models of annotator bias. How does your proposed DPL method compare to these?\n* In the introduction, several different causes of hidden context are described (population diversity, irrationality, etc). Do you expect the implications of the theoretical results to be different depending on the source of the hidden context?\n* The social choice theory connections provide an interesting viewpoint. Does your analysis suggest any new insights about voting rules or mechanisms given unknown/diverse preferences?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This paper makes an original contribution by formally characterizing and providing a theoretical analysis of the problem of \"hidden context\" in preference learning. Though the challenges of population diversity, competing objectives, and human irrationality have been recognized, the key insight is to model all these issues through the unified lens of missing variables that affect human feedback. \n* The technical quality is high with rigorous proofs relating preference learning to Borda count, conditions for equivalence with expected utility, and results connecting to social choice theory. The writing is also clear in explaining the hidden context setting, comparing to regression, and motivating real-world implications.\n* The proposed methods for distributional preference learning seem promising for detecting the effects of hidden context. The experiments provide a nice proof-of-concept.", "weaknesses": "* Both synthetic and real experiments are done in low-dimensional and small data settings. Testing DPL when there are complex interactions between many features in a larger-scale regime would be more convincing. \n* The fact that DPL can detect competing objectives on this dataset is promising. It remains to be seen if DPL reliably detects other sources of hidden context like population diversity or irrationality. Testing on more diverse datasets could address this.\n* There is no comparison to other methods for handling disagreement like modeling annotator identity or clustering. Comparisons could reveal the benefits and limitations of DPL's approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Sb8e5up8Ub", "reviewer": "Reviewer_PKnu", "summary": "The authors raise awareness to the problem of hidden context and why preference learning can be misled into optimizing the wrong objective. The authors propose distribution preference learning as an alternative method that can explicitly account for hidden context. The key difference seems to be that DPL produces a distribution of scores for each alternative.", "questions": "please see above", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "originality: Based on reading the related work, this work seems original. I am not able to tell if they missed any related work.\nquality: \n* The authors gave a very good example of why this work is important.\n* They then provided a way to overcome some of the limitations of just traditional preference learning using DPL\n* Experiments using DPL show promising reduction in jailbreaks\nclarity: clear on parts that I'm familiar with (I did not follow section#3, not that it is not clear, just that I don't have sufficient knowledge to judge)\nsignificance: Seems reasonably significant. More than the proposal, perhaps just drawing attention to this problem and ways to measure is great first step.", "weaknesses": "The authors wrote \"We show that our model can represent many challenges in preference learning, such as combining data from different\nusers, accounting for irrationality, and optimizing for multiple objectives.\" It is a bit of a stretch to say that this work accounts for all forms of irrationality.. it would be great if the authors can also discuss when this approach fails.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T6pBa77XUo", "reviewer": "Reviewer_9LkN", "summary": "This paper describes the ubiquitous problem of 'hidden context' in the setting of preference learning (among a finite number of alternatives) and  in particular for RLHF (reinforcement learning through human feedback)  when RLHF is applied to training large language models to behave well, for instance, to output maximally safe but useful information. The authors provide a number of theoretical results, for instance, when the problem reduces to learning the Borda counts as opposed to maximizing overall expected utility. This may not be desirable, and the authors present their improvements, in particular distributional preference learning (DPL), and present a few experimental special cases to showcase the utility of approaching the problem from this overall perspective as well as the potential of DPL.", "questions": "General comment:\n\n- There is always much hidden context! (many factors that influence decisions, and are not observed or modeled explicitly, in terms of feature vectors in ml, etc). As I read the paper,  gradually I came to the realization that the authors mean aspects that could  substantially impact what is learned in unwanted ways (for instance, there could be multiple clusters of people, and it would be good to test for that.. the DPL approach may begin to address that issue).\n\nMain two questions:\n\n- isn't the assumption of a single utility function too simplistic to be useful in practice? maybe this relates to 'hidden context'.. but\n to me, it is more direct to identify and tackle the problem that different people could have different utilities and so there may not be a single preferred global ranking of all the alternative items ... it would be good to design solutions that aim to identify the rough number of utility functions in addition to what the functions are.. (the problem is somewhat akin to detecting the number of clusters, and whether one or a relatively small few that dominate, ie account for most of the data, etc.. )\n\n- It seems that the best utility function (equation 1, page 3, section 2), mapping alternative to R, is not unique (imagine the ideal case when p_u(a, b) is 1  when u(a)> u(b), and we have two alternatives..)..   The regularization in equation 1 would prefer smaller and smaller values... Any comments on these considerations (perhaps the assumption is that the utility values have some minimal absolute magnitude?)\n\n\n----------\n\nOther observations or questions:\n\n\n-- Example 1 is useful. And pretty clear through section 1. However, I am a bit skeptical that annotators can cooperate and/or not show their true beliefs, etc..\n\n\n- 1st paragraph of section 2, page 3: why probabilities if the utility function is one/unique? Perhaps the subjects are \"noisy\".. ? A few lines later, you do talk about noisy subjects.. but would have been better to describe the  need for probability first or earlier, before presenting the probabilistic model.\n\n- replace 'are' with 'get': \"as the utilities for a and b are closer\"..\n\n- hard to parse 1, in particular, \\hat{u} is both the final selected/assigned, on the left, and the variable on the right!!  \n\n- It seems that the best utility function, mapping alternative to R, is not unique (imagine the ideal case when p_u(a, b) is 1 \nwhen u(a)> u(b), and we have two alternatives..)..   The regularization in equation 1 would prefer smaller and smaller values... Any comments on these considerations (perhaps the assumption is the utility values have some minimal absolute magnitude?)\n\n\n- isn't the assumption of a single utility function too simplistic to be useful in practice? may be this relates to 'hidden context'.. but\n to me, it is more direct to identify and tackle the problem that different people could have different utilities and so there may not be a single preferred global ranking of all the alternative items ... it would be good to design solutions that aim to identify the rough number of utility functions in addition to what the functions are..\n\n\n\n- The example is useful, but just underscores that different groups or populations (based on demographics, race, socioeconomic background, gender, ..) have different preferences and one needs to obtain RLHF training data on a sufficiently large population reflecting what a model will be tested (deployed) on... one may also need to learn different sub-clusters (that is 'personalization' but at a group level ... )\n\n- replace with 'aggregates' in \".. a single ordering over  alternatives that implicitly aggregating feedback over the ... \"\n (page 3)\n\n- Section 5: the  case of 'competing objectives in RLHF'  appears artificial (training on mixed data or mixed objectives). It is ok to use it to demonstrate a point, but if it is actually done by organizations in practice, it appears it is more a reflection of  poor/naive experiment design, than a real problem.\n\n- Section 5, page 9: how was 0.01 quantile picked to lower the jail-breaks  in  \".. optimization according to the 0.01-quantile of.. \" (trial and error, that is pick the quantile that yields lowest jail-break reate? or somehow it is a natural or plausible choice?)", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The authors develop a theory of hidden context as it applies to preference learning, and make connections to a number of relevant domains, such as social choice theory.  The paper presents the concepts and their potential utility fairly well. It is important to be aware of the pitfalls in current preference learning approaches, and the authors make some progress and present methods that could address the issues.", "weaknesses": "The empirical results are the weaker aspect of the paper: all are in a hypothetical setting. The strength of the paper lies mostly in the conceptual and theoretical perspective.   Some issues with paper clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)