Models of human preference for learning reward functions
Authors: W. Bradley Knox
Stephane Hatgis-Kessell
Serena Booth
Scott Niekum
Peter Stone
Alessandro G Allievi
Venue: Accepted by TMLR ()

Abstract: The utility of reinforcement learning is limited by the alignment of reward functions with the interests of human stakeholders. One promising method for alignment is to learn the reward function from human-generated preferences between pairs of trajectory segments, a type of reinforcement learning from human feedback (RLHF). These human preferences are typically assumed to be informed solely by partial return, the sum of rewards along each segment. We find this assumption to be flawed and propose modeling human preferences instead as informed by each segment’s regret, a measure of a segment’s deviation from optimal decision-making. Given infinitely many preferences generated according to regret, we prove that we can identify a reward function equivalent to the reward function that generated those preferences, and we prove that the previous partial return model lacks this identifiability property in multiple contexts. We empirically show that our proposed regret preference model outperforms the partial return preference model with finite training data in otherwise the same setting. Additionally, we find that our proposed regret preference model better predicts real human preferences and also learns reward functions from these preferences that lead to policies that are better human-aligned. Overall, this work establishes that the choice of preference model is impactful, and our proposed regret preference model provides an improvement upon a core assumption of recent research. We have open sourced our experimental code, the human preferences dataset we gathered, and our training and preference elicitation interfaces for gathering such a dataset.

Published in Transactions on Machine Learning Research (02/2024)
Models of human preference for learning reward functions
W. Bradley Knox*1
bradknon&cs.. wtw.aa..ess..m... edui
Bosch
The University of Teras at Austin Google Research
Stephane Hatgis-Kessed
stephane@cs. utwzas. edu
The University of Tezos Austin
Serena Booth
sbooth.@mit.c edu
Bosch
MIT CSAIL
Scott Niekum
snickum@cs. utezas. edu
The University of Texas Austin University of Massachusells Amherst
Peter Stone
pstone@es.ulee www.eessssssstsss...o.... edu
The Unéversity of Teras at Austin Sony Al
Alessandro Allievi
alessandro
wwwievi@ws.........ooo..oooo alessandro.alleewi@uer
bosch. com
Bosch
Reviewed on OpenReview: https: optpr/vvee.rereeee.eme/e net/ forum? hd=hoKJ
Different end state
Different start state
Suboptimal segment
Optimal segment
Suboptimal segment
Optimal segment
GOAL
GOAL
Equal partial return Higher regret
Equal partial return Lower regret
Equal partial return Higher regret
Equal partial return Lower regret
Figure 1: Illustratoon of segment pairs for which the common partial return preference model poorly explains intuitive human preference. The task has -1 reward each time step, penalizing time taken to reach the goal. In both pairs, both segments have the partial return (-2), but the the right segment in each pair consists only of optimal actions, whereas the left segment includes at least one suboptimal action. Regret, which proposed preference model is based upon, designed optimal decision making. The right segment in each pair is therefore more likely to be preferred by regret preference model. In the left pair, the preferred segment has higher end state value. In the right pair, the preferred segment has lower start state value, indicating lower opportunity cost (i.e., it did not waste more valuable start state).
the right is nonetheless the intuitive choice. Additionally,
segment's deviation from
"The first two authors contributed equally. Work partially done while Bosch and Google Research

---

Published in Transactions on Machine Learning Research (02/2024)
Abstract
The utility of reinforcement learning is limited by the alignment of reward functions with the interests of human stakeholders. One promising method for alignment is to learn the reward function from human-generaled preferences between pairs of (rajectory segments, a type of reinforcement learning from human feedback (RLHF). These human preferences are typically assumed to be informed solely by partial return, the sum of rewards along each segment. We find this assumption to be flawed and propose modeling human preferences instead as informed by each segment's regret, a measure of a segment's 's deviation from optimal decision-making Given infinitely many preferences generated according to regret, we prove that we can identify a reward function equivalent to the reward function that generated those preferences, and we prove that the previous partial return model lacks this identifiability property in multiple contexts. We empirically show that our proposed regret preference model outperforms the partial return preference model with finite training data in otherwise the same setting. Additionally, we find that our proposed regret preference model better predicts real human preferences and also learns reward functions from these preferences that lead to policies that are better human- aligned. Overall, this work establishes that the choice of preference model impactful, and our proposed regret preference model provides an improvement upon a core assumption of recent research. We have open sourced our experimental code, the human preferences dataset we gathered, and our training and preference elicitation interfaces for gathering a such dataset.
1 Introduction
Improvements in reinforcement learning (RL) have led to notable recent achievements (Silver et al. 2016 Senior et al. 2020 Vinyals et al. 2019 Bellemare et al. 2020 Berner et al., 2019 Degrave et al. 2022 Wurman et al. 2022). increasing its applicability to real-world problems. Yet, like all optimization algorithms, even perfect RL optimization is limited by the objective it optimizes. For RI., this objective is created in large part by the reward function. Poor aligument between reward functions and the interests human stakeholders limits the utility of RL and may even pose risks of financial cost and human injury or death (Amodei al. 2016 Knox et al., 2021
Influential recent research has focused on learning reward functions from preferences over pairs of trajectory segments, common form of reinforcement learning from human foedback (RLHF). Nearly all of this recent work assumess that human preferences arise probabilist ically from only the sum of rewards over a segment, i.e., the
segment's 's partial return CChristiano et al. 20177] Sadigh et al. [017 Ibarz et al. 2018 Biyık et al.
2021 Lee et al.
2021a b Ziegler et al., 2019 Wang et al. 2022 Ouyang et al.,
2022 Bai et al. 2022 Glaese et al.
2022 OpenAl
2022). That is, these works assume that people tend to prefer trajectory segmnntts that yield greater accunulated
rewards during the segment. However, this preference model ignores seemingly important information about the
segment's desirability, including the state values of the segment's start and end states. Separately, this partial return preference model can prefer suboptimal actions with lucky outcomes, like buying a lottery ticket.
This paper proposes an alternative preference model based on the regret of each segment, which is a measure of how much each segment deviates from optimal decision-making, More precisely, regret is the negated sum of an optimal policy's advantage of each transition in the segment (Section 2.2 Figures [ and 2ssow intuitive examples of when these two models disagree. Some examples of domains where the preference models will differ are those with constant reward until the end, including competitive games like chess, go, and soccer as well as tasks for which the objective is minimize time until reaching a goal.
For these two preference models, we first focus theoretically a normative analysis (Section 3 i.e., what preference model would we want humans to use if we could choose one based on how informative its generated preferences are proving that reward learning on infinite, exhaustive preferences with our proposed regret preference model identifies a reward function with the same set of optimal policies as the reward function with which the preferences are generated. We also prove that the partial return preference model is not guarantoed to identify such reward function in three different contexts: without preference noise, when trajectories of different lengths are possible from a state, and when segments consist of only one transition. We follow up with descriptive
2

---

Published in Transactions on Machine Learning Research (02/2024)
analysis of how well each of these proposed models align with actual human preferences by collecting human-
labeled dataset of preferences in rich grid world domain (Section 4 and showing that the regret preference
model better predicts these human preferences (Section 5p. Finally, we find that the policies ultimately created
through the regret preference model tend to outperform those from the partial return model learning both
when assessed with collected human preferences or when assessed with synthetic preferences (Section Our
code for learning and for re-running our main experiments can be found here alongside our interface for training
subjects and for preference elicitation. The human preferences dataset is available here Knox al. (2023).
In summary, primary contributions are five-fold:
1. We propose new model for human preferences that is based regret instead of partial return.
We theoretically validate that this regret-based model has the desirable characteristic of reward identifiability, and that the partial return model does not.
3. We empirically validate that when each preference model learns from a preferences dataset created, this regret-basod model leads to better-aligned policics.
4. We empirically validate that, with a collected dataset of human preferences, this regret-based model both better describes the human preferences and leads to better-aligned policies.
5. Overall, we show that the choice of preference model impacts the alignment of learned reward functions.
2 Preference models for learning reward functions
We assunee that the task environment is Markov decision process (MDP) specified by the tuple (8, A, T, Do, r). S and A are the sets of possible states and actions, respectively. T is a transition function, T:Sx A= pp-ss,,)); y is the discount factor; and Do is the distribution of start states. Unless otherwise stated, we assume all tasks are undiscounted (i.e., 7= 1) and have terminal states, after which only reward can be received. Discounting is considered in depth Appendix B.2 r is reward function, S:Sx Ax R, where the reward Tt time is a function of st, ae, and sc+1. An MDP\r is an MDP without a reward function.
Throughout this paper, refers to the ground-truth reward function for some MDP; f refers to learned approximation of r; and F refers to any reward function (including or f). A policy (T:Sx x A [0,1]) specifies the probability of an action given state. Q7 and V," refer respectively to the state-action value function and state value function for policy, x, under f, and are defined as follows.
V (s) def
Fl 01 Ex
(7,a,) Em ((, +VV7(())))
Apottimal policy n² is any policy where V QF and V Q and VV,, respectively. The optimal advantage function is defined AŞ e this measures how much an action reduces expected return relative to following an optimal policy.
at every stale for every policy æ. We writesh for
Throughout this paper, the ground-truth reward function r is used to algorithmically generate preferences when they are not human-generated, is hidden during reward learning, and is used to evaluate the performance of optimal policies under a learnod
2.1 Reward learning from pairwise preferences
A reward function can be learned by minimizing the cross-entropy loss
i.e., maximizing the likelihood of
observed human preferences, common approach in recent literature
(Christiano al.) 2017 Ibarz et al.)
2018
Wang et al. 2022 Biyık et al. 2021 Sadigh al. 2017 Lee et al.
2021a Ziegler al. 2019 Ouyang
et al.
2022
Bai al.] 2022 Glaese al. 2022 OpenAl
2022
3

---

Published in Transactions on Machine Learning Research (02/2024)
Segments Let denote segment starting at state sg. Its length |o| is the number of transitions within the segment. A segment includes |a|+1 states and |o| actions: (s) (g i "0, ......).)))) In this problem setting, segments lack any reward information. As shorthand, we define o; a (sf of +1+)). A segment o is optimal with respect to if, for every iE [ (sf (sf). A segment that is not optimal is suboptimal. Given some i and a segment o, r. - f66. aj (811), and the undiscounted partial return of a segment a is El ft, denoted in shorthand as F.
Preference datasets Each preference over a pair of segments creates a sample (01,072,14 in preference dataset D,. Vector μ= (M1)22) represents the preference; specifically, if o, is preferred over d2, denoted 01 02, p= (1,0). μ is (0.1) ai <20 and is (0.5,0.5) for m-02 (no preference). For a sample (01,02,11,, we assume that the two segments have equal lengths (i.e., |oi = [(2]).
Loss function To learn a reward function from a preference dataset, Dy, a common assumption is that these preferences were generated by preference model P that arises from an unobservable ground-brule bruuh reward function r. We approximate by minimizing cross-entrope loss to learn f:
loss(f,Dz)=
-E (oL.D..N)ED.
(1)
For a single sample where 01 T2, the sample's likelihood is P(o 02lf) and its loss is therefore -logP(o1> > ozlf). If o1 <00, its likelihood is 1 P oz))). This loss is under-speciffed until P( o2l is delined, which is the focus of this paper. We show that the common partial return model of preference probabilities is flawed and introduce an improved regret-based preference model.
Preference models A preference model determines the probability of one trajectory segment being P(a1 <<alr)= and P -a))) for the preference models considered herein. Preference models could be applied to model preferences provided by humans other systems. Preference models can also directly generate preferences, and in such cases we refer to them as preference generators.
preferred over another, P(a > o2))). P(o1 +z))+
2.2 Choice of preference model: partial return and regret
Partial return All aforementioned recent work assumes human preferences are generated by a Boltzmann distribution over the two segments' partial returns, expressed here as a logistic function
(0)0) (anatticco,,---- E -oor
(2)
Regret We introduce an alternative preference model based on the regret of each transition in a segment. We first focus on segments with deterministic transitions. For transition (5,,)11141) in deterministic segment, regreta (a, F) A Vi (sf) Fr VV² (s]+1)). The subscript d in regreta signifies the assumption of deterministic transitions. For a full deterministic segment,
|al-1 Σ
regreta(oli)
)),
(3)
with the right-hand expression arising from cancelling out intermediate state values. Therefore, deterministic regret measures how much the segment reduces expected return from V/2 (s)). An optimal segment, a*, always has 0 regret, and a suboptimal segment, ,-*, will always have positive regret, an intuitively appealing property that also plays a role in the identifiability proof of Theorem
derivstion of this logistic expression from a Boltzmann distribution with temperature of 1. Unless otherwise stated, ignore the temperature because scaling reward has the effect when preference probabilities not deterministic. The temperature is allowed vary for our theory Section Another context when the temperature parameter would be useful is when learning single reward function with loss function that includes the formula Equation in such case, scaling reward might undesirably affect the other loss term(s), whereas the varying the Roltzmann temperature changes the preference entropy without affecting the other loss term(s).
See Appendix B.1
loss terms in addition to
4

---

Published in Transactions on Machine Learning Research (02/2024)
Stochastic state transitions, however, can result in regretaff""))>>rrrrrefffff (egreta(o=*))) losing the property above. For instance, an optimal action can lead to worse return than suboptimal action, based on stochasticity in state transitions. To retain this property that optimal segments have a regret of 0 and suboptimal segments have positive regret, we first note that the effect on expected return of transition stochasticity from a transition (81, is [Fc+ ( (s))0)) and add this expression once per transition to gel regrel(a): removing the subscript that refers to determinism. The regret for a single transition becomes regret(o, [ (sf)) [it V2 (6((2444)) (+1+))l)+ [[+++ (+++1) - [[](s[aa]]]-1
.a]) - -7 .a)). Regret for a full segment is
-|-1
|-|-1 (egret(av(F)===
|0|-1
regret(ali)= =
Σ (6),000
Σ (a)))
(4)
1=
The regret preference model is the Boltzmann distribution over negated regret:
Pregret(a1 > o2lrt) logisticc (eegret(ol))
(5)
Lastly, we note that if two segments have deterministic transitions, end in terminal states, and have the same starting state, this special case the regret model reduces to the partial return model: (-IF) (-1).
In this article, our normative results examine both tasks with deterministic transitions and Lasks with stochastic transitions. These normative results include the theoretical analysis in Section 3 and the empirical results with synthetic data. in Section 6.2 and Appendix F.2. with stochastic tasks specifically examined empirically in Appendix F F.2.4 We gather human preferences for a deterministic task, which allows us to investigate the results with the more intuitive expression of regrela that includes partial ueturn as a component.
Algorithms in this paper All algorithms in the body of this paper can be summarized as "minimize Equation D They differ only in how the preference probabilities are cal- culated. All reward function learning via rettrn return uses
Suboptimal segment
Optimal segment
end state (safe)
state (crnsh imminent)
Equation2 2 replicating the dominant algorithm in recent
sum of reward for bumping the wall
literal ture ure
Christiano et al.
2017 Tbarz el al. 2018 Wang 2021
sum of reward
ct al. 2022 Biyik et al.
Sadigh et al. 2017 Lee et al..
2021a by Ouyang et al.
2022). We use two algorithis for
state (safe)
start state (crush likely)
reward function learning via regret. The theory in Section 3 assumes exact measurement of regret, using Equation
Higher partial return Higher regret
Lower partial return Lower regret
Section G introduces Equation D to approximate regret replacing Equation 5 to create another algorithm and uses the resulting algorithm for our experimental results later in that section. Appendix B introduces other algorithms that use Equation] [ as well as one in Appendix B.4 that generalizes Equation
Two segments of car moving at high speed near brick wall. On the left, a car moves toward brick wall; a bad crash is imminent, but has not yet occurred. On the right, car escapes an imminent crash against a brick wall with only scrape. Assume the right segment is optimal and the left segment is suboptimal (as defined in See. 2.Tp. The left segment. has higher sum of reward, so is preferred under the partial return preference model. The right segment is preferred under the regret preference model since optimal segments have munnimal regret. If we also assume deterministic transitions, then the regrel model includes the difference in values between the start state and the end state (Equation g and the right segment would tend to be preferred because it greatly improves its state values from start to end, whereas the left segment's 's state values greatly worsen. We suspect our human readers will also tend to prefer the right. segment.
Figure 2:
Regret as model for human preference P Pregret makes at least three assumptions worth noting. First, it keeps the assumption that human preferences follow Boltz- mann distribution over some statistic, which is a common model of choice behavior in economics and psychology, where it is called the Luce-Shepard choice rule [Luce] 1959 Shepard 1957). Second, Pregret implicitly assumes humans can identily optimal and suboptimal segments when they see them, which will be less true in domains where the human has less expertise. This assumption is similar to a common assumption of many algorithms for imitation learning, that humans can provide demonstrations that are optimal or noisily optimal (e.g.. Abbeel & Ng 2004)).
Lastly, Pregret assumes that in stochastic settings where the best outcome may only result from suboptimal decisions (e.g., buying lottery ticket), humans instead
5

---

Published in Transactions on Machine Learning Research (02/2024)
prefer optimal decisions. We suspect humans are capable of expressing either type of preference based on decision quality or desirability of outcomes- -ndees---dddd interface. Curiously, for stochastic tasks in which preferences are based upon segments" observed outcomes, a preference model that uses deterministic regreta in Equation I appearrs fitting, since it does not subtract out the effects of fortunate and unfortunate transitions but does include segments' start and end state values.
and can be influenced by training the preference elicitation
In practice we determine that the regret model produces improvements over the partial return model (Section and its assumptions represent opportunity for follow-up research.
3 Theoretical comparison
In this section, we consider how different ways of generating preferences affect reward inference, setting aside whether humans can be influenced to give preferences in accordance with a specific preference method. In economics, this analysis and all of our later analyses with synthetic preferences could be considered a normative analysis. In artificial intelligence, this analysis might be cast as step towards defining criteria for a rational preference model.
The theorems and proofs below focus on identifiability, a property which determines whether the parameters of model can be recovered from infinite, exhaustive samples generated by the model. model is unidentifiable if any two parameterizations of the model result in the same model behavior. In our setting, the model of concern is preference model and the parameters constitute the ground-truth reward function, A preference model is idintifiable ble if an infinite, exhaustive preferences dataset created by the preference model contains sufficient. information to infer behaviorally equivalent reward function to r. Note that identifiability focuses on the preference model alone as a preference generator, not on the learning algorithm that uses such a preference model.
This section uses preference models that include discounting (sce Appendix B.2) We allow for discounting make the theory more general and also because discounting integral Section 3.2.3 Here the notation for (s,a) and V (s) is expanded to (7 (s,a) and V(A,a) (s) respectively include the discount factor. To make the other content in this section specific to undiscounted tasks, simply assume all instances of 11, including the ground-truth ry and the 9 used during reward function inference and policy improvement.
Definition 3.1 (An identifiable preference model). For preference model P, assume an infinite dataset D > 02)r), using P
of pairs of segments is constructed by repentedly choosing (01,02) and sampling a label μ~ (0ai as a preference generator. Further assume that, in this dataset, all possible segment pairs appear infinitely many times. For some M that is an MDP (r.") -an MDP with neither reward function nor a discount factor-let let Mir,4) be M with the reward function and the discount factor 7. Lel (,4) M(ra)- Let problem-equivaeere
5) be the set of optimal policies for class A be the set of all pairs of a reward function and discount factor such (. Preference model P is identifiable if and only if, for any choice of segment length n and ground-truth M(rg), there erists an operation on D2 that always outputs a (f,5) that is in the same problem equivalence class (r.7). Le., II; mi =I[
that if (12,1) ER then 11 I1;
3.1 The regret preference model identifiable.
We first prove that our proposed regret preference model is identifiable.
Theorem 3.1 (Pregret is identifiable). Let Pregret be any function such that (f <regret(o21 (o > o2) > 0.5, and if regret(oi)ñ,f)) regrel(oz)r.5). Pregret(a) o2l7,7) = 0.5. Pregret is identifiable.
This class of regret preference models includes but is not limited to the Boltzmann distribution of Equation 5 Additionally, it includes a version of the regret, preference model that noiselessly always prefers the segment. with lower regret, as Theorem 3.2 considers for the partial return preference model
be extended include such noiscless preference models by including the temperature paramcter of the Boltzmann distributions (after converting from their logistic formulations, reversing the derivation in Appendix B. where assume that setting temperature results in hard maximum. In other words, when the temperature is the preference is given deterministically the segment with the higher partial return in Equation Zor regret in Equation
Equatioos 2 and
6

---

Published in Transactions on Machine Learning Research (02/2024)
Consider reviewing the definitions of optimal segments and suboptimal segments in Section 2.1 before proceeding.
For the proof below, we will apply the following sufficiency test for identifiability. Preference model P is identifiable if, for any ground-truth M(r,)), any (f.1) - aagmangggmnnnn.... (l
for the cross-entropy loss which is Eqn. generalized to include discounting), with as the preference model is in the same problem equivalence class as (1,7). Le., (T..)) = [T (P,4)*
(Equ. S
Proof Make all assumptions in Definition 3.1 Let (f,4) = argmin (o,ss(T,, where loss is the
cross-entropy loss from Equ. g with Pregret as the preference model.
Since (f.6) minimizes cntsopy loss and chosen from the complete space of reward functions and discount
factors, Pregree(-11:7)) ((trr)>>rrerrr((/ (-1.:5) for all possible segment pairs. A lsso, by Equation 12 (which generalizes Equation 4 to include discounting) regret(a)f, (R)) = 0 if and only if is optimal with respect to f. And regret(a (.)) > 0 if and only if a is suboptimal with respect to (F,7).
With respect to some (f, 7), let or be any optimal segment and a be any suboptimal segment. regret(a"1;; (F.9) < regret(a"" (1,7), so Pregret(a" > (,,5) > 0.5. (-1F, induces a total ordering over segments,, defined by regret(ai)iññ) < regrel(o2)), (,,7) - Pregret > o2lf, > 0.5 + 01 >> 02 and (
< P 005 = 02= Because regret has a minimum (0), there must be set of segments which are ranked highest under this ordering, denoted ((,9)) These segments in Ex (F,4) are exactly those that achieve the minimum regret (0) and so are optimal with respect to (f,7).
Since the dataset (D₂) contains all segments by assumption, (F.))) contains all optimal segments with respect to (F,5). If a state-action pair (s,a) is in optimal segment, then by the definition of an optimal segment. (o (8,a) = (ir,5) (s). The set of optimal policies II; for f is all such that, for all (s,a), if TTs,a) > 0, then (F,,) (s.a) ((,,)) (s). In short, ((.5g) determines the set of each state-action pair (s,a) such that (i,,) (s.a) = (1,5) This set determines II (7,8)) Therefore (F,9) determines I1 (F,)) and we will refer to this determination as the function g.
We now focus on the reward function and discount factor used to generate preferences, (r:7), and on the inferred reward ionction and discount factor, (F:5). Since Pregret(-(r:y)) P (-rggrrt--11 (-),,),, (1:7) and (f,4) induce the same total ordering over segments, and so ((r,)) = (Faa Therefore g g()))))))) = (( Since g (>x())))) II (oo) and
g ((())))) - "'r,>) = =I((FA)
The proof above establishes the identifiability of Pregret regardless of whether preferences are generated noiselessly or stochastically.
3.2 The partial return preference model is not generally identifiable.
In this subsection, we critique the previous standard preference model, the partial return model Per. by proving that this model can be unidentifialle in three different contexts.
Given noiseless preference labeling by Psr in some MDPs, preferences never provide sufficient information to recover the set of optimal policies.
In variable-horizon tasks when the lengths of both segments in pair are always equivalent. Variable-horizon tasks include common tasks that terminate upon reaching success or failure states, reward functions that differ by a constant can have different sets of optimal policies. Yet for two such reward functions, the preference probabilities according to partial return will be identical.
With segment lengths of 1 (la] = 1), the discount factor y does not affect the partial return preference model and therefore will not be recoverable from the preferences it generates. Since different. values of y can determine differont sets of optimal policies, an inability to recover y is a third type of inidenti
We now prove in each of these three contexts that the partial return preference model is not identifiable.
7

---

Published in Transactions on Machine Learning Research (02/2024)
For each, will apply the following sufficiency test for non-identifiability Preference model P is not identifiable if there exist two ground-truth MDPs, ((...)) and M (r).7)) such that ( ) and the infinite preference datasets created as described in Definition 3.1 by P for M(rin) and ((r2.79) are identical. Note that such identical preference datasets lack the information to diffcrentiate which MDP they came from.
3.2.1 Partial return is not identifiable when preferences are noiseless.
Theorem 3.2 (Noiscless Par is not identifiable). Let Per be any function such that &o, Eo,F, Per 02 and f= (1r(01>0221)) 0.5. There erists an MDP in which Psr is not identifiable.
Below we present two proofs of Theorem 3.2 Each are proofs by counterexample. The first is a general proof and the second proof assumess a common characteristic, that all segments in the preference dataset are the same length. Though only one proof is needed, we present two because each counterexample demonstrates a qualitatively different category of how the partial return preference model can fail to identify the set of optimal policies.
Proof based on stochastic transitions: Assume the follow- ing class of MDPs, illustrated in Figure 3 The agent always begins at start state sy- From 80, action asafe always transitions to Szafe: getting a reward of O. From so; action Orisk transitions to Swin with probability 0.5, getting reward of Twins and tran- sitions to Slose with with probability 0.5, gelting a reward of -10. In all MDPs in this class, Twin > 0. All 3 possible resulting states (Ssafe: Swin: and Stose) are absorbing states, from which all further reward is 0.
100%, 0
S safe
a safe
S 0
S lose
risk
50%, -10
If Twin 2 10, Grisk is optimal in sg. If Twin I 10, asafe is optimal in Sp- Three anglo-transition segments exist: saynaaafesssfell (rskk))s): and (sp.arisk By noiseless Per, (rokkrssk,Suin) > (S0.0safe:Ssaff) less of the value of Twin- In other words, Per is insensitive to what the optimal action is from so in this class of MDPs.
50%, ade
S win
(80, (8sk regard-
Figure A class of MDPs in which, if Twin > 0, the partial return preference model fails the test for identifiability.
Now assume MDP M, where Twin = 11. In linear form, the weight vector for the reward function TM can be expressed as WrM, 1,cc-10,0111 >. Let far have WFm =< - 10,0,9 Both r'M and A' have the same preferences as above, meaning that fM minimizes loss on an infinite preferences dataset D2 created by Per, yet it has different optimal policy. Therefore, noiscless Psr is not identifiable.
]
In contrast, note that by noiseless Pregret, the preferences are different than those above for Per If Tanin 10, then (so.arisk,$win) (so,arisk ($0,0safe,$saf) If Twin < 10, then (so,asafe > (50,Grish swin) (sskkrrss.Slose)) Intuitively, this differonce comes from Pregret always giving higher preference probability to optimal actions, even if they result bad outcomes. Another perspective can be found from the utility theory of Von Neumann & Morgenstern (1944). Specifically, Per gives preferences over outcomes, which the terms of utility theory can only be used to infer an ordinal utility function. Ordinal utility functions are merely consistent with the preference ordering over outcomes and do not generally capture preferences over actions when their outcomes are stochastically determined. The deterministic regret preference model, Pregreta also has this weakness in tasks with stochastic transitions. On the other hand, Pregret forms preferences over so-called lotteries the distribution over possible outcomes and can therefore learn a cardinal utility function, which can explain preferences over risky action. See Russell & Norvig 2020 Ch. 16) for more detail on these concepts from expected utility theory.
Since the proof above focuses upon stochastic transitions, we show the lack of identifiability for noiseless Psr can be found for quite different reasons in deterministic MDP when the preferences dataset has common characteristic: that all segments are the same length.
8

---

Published in Transactions on Machine Learning Research (02/2024)
in Figure 1 and and assume preferences are given over segments with length (i.e., containing one transition). The optimal policy for M1 is to move rightward from So, whereas optimal behavior for Mi is to move downward from so- In both Mj and Mi, preferences by PIr are as follows, omitting the action for brevity: (8) ~ ((ppsttrm)m((osa)))))) > (S0..)))(( (Su.S)erm)). As
Proof based on segments of fixed length: Consider the MDP M1
in the previous proof, Per is insensitive to certain changes in the reward function that alter the set of optimal policies. Whenever this characteristic is found, =T; IT not guaranteed, failing the test for identifiability. Here specifically, the reward function for M1 would achieve the minimum possible cross-entropy loss on an exhaustive preference dataset created in M1 with the noiseless preferences from the partial return preference model, despite the optimal policy in Mj conflicting with the ground-truth optimal policy.
The logic of this proof can be applied for tra- jectories of length in the MDP M2 shown in Figure 5 Together, M1 and M2 suggest a rule for constructing an MDP where II; II; is not guaranteed for Pxr, failing the iden- tifiability test for any fixed segment length, lol: set the number of states to the right of so to |or| (not counting Sterm), set the reward fast for (80,Sterm) such that ffail < 0, and set the reward for each other transition to c+r/aa//(o [[oo| + 1); where e > Given an MDP constructed this way; an alternative reward function that results in the same pref- erences under Par yet has a different optimal action from so can then be constructed by changing all reward other than rasi to faaa//((| +1), where e now is constrained to c<0 and ex |a| (ffll- Note that the set of preferences for each of these MDPs is the same even when including segments that reach terminal state before |ol transitions (which can still be considered to be of length |o| if the terminal state is an absorbing state from which reward is 0).
S 0
S a
S 0
-2
M,
S term
M
S
term
Figure 4: An MDP (M1 where IT; = ITç not guaranteed for the partial return preference model, failing the test for identifiability with segments of length The ground-truth reward function shown to the left, and an MDP Mj with an alternative reward function is shown the right. Under partial return, both create the same set of preferences despite having different optimal actions from
Discussion of preference noise and identifiability Of the two proofs by example for Theorem 3.2 the first proof's example reveals issucs when lcarning reward functions with stochastic transitions with either Psr or deterministic Pregreta. These issues directly corre- spond to the need for preferences over distributions over outcomes (i.e., lotteries) to construct a cardinal utility function (see RRssell & Norvig (2020 Ch. 16)). Corre- spondingly, when (Skalse et al. (0222 consider reward
S
S
0
a
b
-4
M,
S
term
identifiability with the partial return preference model, they change the learning problem such that a training sample consists of preferences between distributions over trajectories. Intuitivvlyy Theorem 3.2 says that Ps, is not identifiable without the distribution over preferences providing information about the proportions of rewards with respect to each other. In contrast, to be identifi- able, the regret preference model does not, require this preference error (though it can presumably benefit from it in cortain contexts).
-1.5
-1.5
S 0
S
S
-1.5
a
-1.5
b
-1.5
M 2
S
term
Figure 5: An MDP (M2) where IT; = II; not guaranteed for the partial return preference model, failing the test for identifiability with segments of length The ground-trri reward function is shown in the top diagram, and an MDP Mi with an alternative reward function is shown the bottom diagram. Under partial return, both create the same set of preferences despite having different optimal actions from 8g-
3.2.2 Partial
return is not identifiable in horiable-horizon tasks.
Many common tasks have the characteristic of having at least one state from which trajectories of different
9

---

Published in Transactions on Machine Learning Research (02/2024)
lengths are possible, which we refer to as being variable-horizon task. Tasks that terminate upon completing goal typically have this characteristic. We also assumee that for any pair of segments in the preference dataset, the lengths of those two segments are equivalent. This assumption describes typical practice (e.g., Christiano
ct al. (2017); Sadigh et al. 2017): : Iharz et al. 2018); Bryak et al. (2021 ;: Lee et al. (021a)) Wang et al. (2022))
In this context, we show another way that the partial return preference model is not identifiable, a limitation
that has arisen dramatically in our own experiments and is not limited to noiseless proferences: adding a
constant to a reward function will often change the set of optimal policies, but it will not change the probability of preference for any two segments. Therefore, those preferences will not contain information to recover the set of optimal policies.
We now explain why such a constant shift will not change the probability of preference based upon partial return. Consider a constant value c and two reward functions, r1 and r2, where T1 (5,, (r)())),,,,,, =c for all transitions ($,,0,;8++1)- The partial return of any segment of length |o| will be ex |o| higher for r1 than for r2 (assuming an undiscounted task, 7=1). In the partial return preference model (Equation [ this addition of ex |o| to each segment's 's partial return cancels out, having no effect on the different in the segments" partial returns and therefore also having no effect on the preference probabilities. Consequently, adding c to reward function's 's output will also not affect the distribution over preference datascts that the partial return preference model would create via sampling from its preference probabilities.
1f, for each state in an MDP, all possible trajectories from that state have the same length, then adding a constant e to the output of the reward function does not affect the set of optimal policies. Specifically, the set of optimal policies is preserved because the return for any trajectory from a state is changed by ex |r], where |r] is the unchanging trajectory length from that state, so the ranking of trajectories by their returns is unchanged and also the expected return of policy from that state is unchanged. Continuing tasks and fixed-horizon tasks have this property.
However, if trajectories from a state can terminate after different numbers of transitions, then two reward functions that differ by a constant can have different sets of optimal policies. Episodic tasks are often vulnerable to this issue. To illustrate, consider the task in Figure [ a simple grid world task that penalizes the agent with -1 reward for each step it takes to reach the goal. If this reward per step is shifted to +1 (or any positive value), then any optimal policy will avoid the goal, flipping the objective of the task from that of reaching the goal. So, for variable-ly ble-hhrizohnnn lasks, Psr is not generally identifiable.
Though identifiability focuses on what information is encoded in preferences, this issue has practical consequences during learning from preferences over segments of length 1: for a preferences datasct, all reward functions that difler by a constant assign the same likelihood to the dataset, making the choice bet ween such reward functions arbitrary and the learning problem underspecified. Some past authors have acknowledged this insensitivity to a shift (Christiano et al., 2017 Lee al. 2021a Ouyang et al., 2022 Hejna & Sadigh 2023), and the common practice of forcing all tasks to have fixed horizon (Christiano et al. 2017 Gleave et al 2022 may be partially attributable to Pxy 's lack of identifiability in variable-horizon tasks, leading to its low performance in such tasks. In Appendix F 2.2 we proposc a stopgap solution to this problem and also observe that in episodic grid worlds that the partial return preference model performs catastrophiclly poorly without this solution, both with synthetic preferences and human preferences.
The regret preference model is appropriately affected by constant reward shifts. Here we give intuition for why adding a constant e to the output of a reward function does not threaten the billt of the regret preference model, as established in Theorem 3.1 As stated above, adding to reward can change the set of optimal policies. Any such change in what actions are optimal would likewise change the ordering of segments by regret, so the likelihood of a preferences dataset according to the regret preference model would be affected by such a constant shift in the learned reward function (as it should be).
3.2.3 Partial return is not identifiable for segment lengths of 1.
Arguably the most impactful application to date of learning reward functions from human preferences is to fine-tune large language models. For the most notable of these applications, the segment length |ø| == [Zigllrr
ct al. 2019 OpenAl 2022
Glaese et al. 2022 Ouyang et al. 2022: Bai ct al. 2022)
10

---

Published in Transactions on Machine Learning Research (02/2024)
Changing Y often changes the set of optimal policies, yet when |a|= changing the discount factor does not change preference probabilities based upon partial return preference model. We elaborate below.
Here we make an exception to this article's default assumption that all tasks are undiscounted. As we describe in Appendix B.2 the discounted partial return of segment is - We follow the standard convention that = When |oj 11, the partial return simplifies to the immediate reward, fo regardless of the value of 7 Consequently the partial return preference model is unaffected by the discount factor when |a|=1. 1. We leave to the reader the task of a precise proof by counterexample that partial return is not identifiable when |o| = 1; any two MDPs that differ only by their discount factor and have different sets optimal policies will suffice provide proof, since the distribution of preferences according partial return will be identical in each of these MDPs, establishing the lack of identifiability.
To remove this source of unidentifiability, the preferences dataset would need to be presented to the learning algorithm with a corresponding discount factor. Past work on identifiability in this setting (Skalse et al. 2022 has assumed that the discount. factor is given and does not discuss the topic further.
As with the other identifiability issues demonstratod in this subsoction, this issuc has practical consequences during learning from preferences. When |o| 1, the choice of is arbitrary, making the learning problem underspecifid..
The regret preference model is identifiable even when the discount factor is unknown. Note that
Theorem 3.1 already includes this case. To add some intuition, the discounted regret of geggeett-pesennee in Appendix B.2 does include the discount factor in its formulation, regardless of segment length. Therefore, the discount factor used during preference generation does impact what reward function is learned.
4 Creating a human-labeled preference dataset
To empirically investigate the consequences of each preference model when learning reward from haman preferences, we collected a preference dataset labeled by human subjects via Amazon Mechanical Turk. This data collection was app-appo Appendix Daadds detail to the content below.
4.1
The general delivery domain
The delivery domain consists of a grid of cells, each of a specific road surface lype. The delivery agent's 's state is its location. The agent's action space moving one cell one of the four cardinal directions. The episode can terminate either at the destination for +50 reward or in failure at sheep for -50 reward. The reward for a non-terminal transition is the sum of any reward components. Cells with a white road surface have a -1 reward component, and cells with brick surface have a -2 component. Additionally each cell may contain a coin (+1) or a roadblock (-1). Coins do not disappear and at best cancel out the road surface cost. Actions that would move the agent into a house or beyond the grid's perimeter result in no motion and receive reward that includes the current cell's surface reward component but not any coin roadblock components. In this work, the start state distribution, Du, is al always uniformly random over non-terminal states. This domain was designed to permit subjects to easily identify bad behavior yet also to be difficult for them to determine optimal bchavior from most states, which is representative of many common tasks. Note that this intended difficulty forces some violation of the regret preference model's assumption that humans always prefer optimal segments over suboptimal ones, therefore testing its performance in non-ideal conditions.
Figure 6: The delivery task used to gather human preferences. The yellow van the agent and the red inverted teardrop is the destination.
11

---

Published in Transactions on Machine Learning Research (02/2024)
4.1.1 The delivery task
We chose one instantiation of the delivery domain for gathering our dataset human preferences. This specific MDP has a x0x 10 grid. From every state, the highest return possible involves reaching the goal, rather than hitting a sheep or perpetually avoiding termination. Figure shows this task.
4.2 The subject interface and survey
This subsection describes the three main stages of each data collection session. A video showing the full experimental protocol can be seen at youtu.be/ Vet.n///X//aAAA
Teaching subjects about the task Subjects first view instructions describing the general domain. To avoid the jargon of "return" and "reward," these terms are mapped to equivalent values in US dollars, and the instructions describe the goal of the task as maximizing the delivery vehicle's financial outcome, where the reward components are specific financial impacts. This information is shared amongst interspersed interactive episodes, in which the subject controls the agent in domain maps that are each designed to teach one or two concepts. Our intention during this stage to inform the later preferences of the subject by teaching them about the domain's dynamics and its reward function, as well as develop the subject's sense of how desirable various behaviors are. At the end of this stage, the subject controls the agent for two episodes in the specific delivery task shown in Figure G
Preference elicitation After each subject is trained to understand the task, they indicate their preferences between 40-50 randomly-ordered pairs of segments, using the interface shown in Figure 7 The subjects select a preference, no preference ("same")) or "can't tell". In this work, we exclude responses labeled "can't tell", though one might alternatively try to extract information from these responses.
Subjects' task comprchension Subjects then answered questions testing their understanding of the task, and we removed their data if they scored poorly. We also removed subject's data if they preferred colliding the vehicle into sheep over not doing so, which we interpreted as poor task understanding or inattentiveness. This filtered dataset contains 1812 preferences from 50 subjects.
WHICH SHOWS BETTER BEHAVIOR?
2/48
SAME
LEFT
CAN'T TELL
RIGHT
Figure 7: Interface shown to subjects during preference elicitation. The left trajectory shows the yellow van doubling back on itself before hitting a sheep. The right trajectory shows the van hitting a road block.
4.3 Selection of segment pairs for labeling
We collected human preferences in two stages, each with different methods for selecting which segment pairs to present for labeling. The sole purpose of collecting this second-stape data was to improve the reward-learning performance of the partial return model, Psr. Without second-stage data, Psr compared even worse to Pregret than in the results described Section 6 performing worse than a uniformly random policy (sce Appendix 3.333)
12

---

Published in Transactions on Machine Learning Research (02/2024)
Proportion that prefer o
100%
02
83%
66%
50%
33%
16%
0%
A6VV V - A
Figure 8: Proportions at which subjects preferred each segment in a pair, plotted by the difference in the segments' changes in state values (x-axis) and partial returns (y-axis). The diagonal line shows points of preference indifference for Pregret- Points of indifference for Ps lie on the x-axis. The shaded gray area indicates where the partial return and regret models disagree, each giving different segment a preference probability greater than 0.5. Each circle's 's area is proportional the number of samples it describes. A visual test which preference model better fits the data as follows: if the human subjects followed the partial return preference model, the color gradient would be orthogonal to the x-axis. If they followed the regret preference model, the color gradient would be orthogonal to the diagonal line, since regret this plot is z z+y. Visual inspection of the color gradient reveals the latter color gradient, suggesting they more closely followed the regret preference model.
Both stages' data are combined and used as a single dataset. These methods and their justification are described in Appendix D.3.
5
Descriptive results
This section considers how well different preference models explain our datasct of human preferences.
5.1 Correlations between preferences and segment statistics
Recall that with deterministic transitions, the regret of segment has components: =egreta(a1))=V) (E, Vi )), one of which is partial return, E. We hypothesize that the other two termss the values of segments' start and end states, which are included in Pregret but not in Ps Ps-affect human preferences, independent of partial return. If this hypothesis is true, then we have more confidence that preference models that inchude start and end state values will be more effective during inference of the reward functions.
The dataset of preferences is visualized in Figure To simplify analysis, we combine the two parts of regreta(olr) that are additional to F and introduce the following shorthand: V. A V/2 --
Note that with algebraic manipulation (sce Appendix E. regreta (1))) (Ao, V An, (E,, Therefore, on the diagonal line in Figure regrela(a2)1) regrela (o) making the Pregreta preference model indifferent. This plot shows how Ag V, has influence independent of partial return by focusing only on points at a chosen y-axis value; if the colors along the corresponding horizontal line reddens as the x-axis value increases, then A, V, appears to have independent influence.
To statistically test for independent influence of A, V. on preferences, we consider subsets of data where Eo Eo,ris is constant. For Σ,, r ---r-- and Σ,, - the only values with more than 30 samples that also include informative samples with both negative and positive values of regret(az -regretttc))))) the Spearman's 'ss rank correlations between A, V, and the preferences are significant (r>=0.3, p p This result indicates that A V, influences human preferences independent of partial return, validating our hypothesis that humans form preferences based on information about segments' start states and end states, not only partial returns.
13

---

Published in Transactions on Machine Learning Research (02/2024)
5.2 Likelihood of human preferences under different preference models
To examine how well each preference model predicts human preferences, we calculate the cross-entropy loss for each model (Equation[) [ i.e., the negative log likelihood- of the preferences in our dataset. Scaling reward by a constant factor does not affect the set optimal policies. Therefore, throughout this work we ensure that our analyses of preference models are insensitive reward scaling. To do for this specific analysis, we conduct 10-fold cross validation to learn reward scaling factor for each of Pregret and Pr. Table [ that the loss of P Pregret is lower than that of Pgr, indicating that regret is more reflective of how people actually express preferences.
Preference model
Loss
P(.)=0.5 (uninformed)
0.69
Per (partial return) Pregret
0.62
0.57
Table 1: Mean cross-entropy test loss over 10-fold cross validation (n=1812) from predicting human preferences. Lower is better.
6 Results from learning reward functions
Analysis of a preference model's predictions of hu- Ground man preferences is informative, but such predictions are a means to the ends learning human-aligned reward functions and policies. We now examine each preference model's performance in these terms. In all cases, learn reward function f according to Equation and apply value iteration (Sutton & Barto 2018) to find the approximately optimal Q; function. For this Q7, we then evaluate the mean return of the maximum-entrop optimal policy- which chooses uniformly randomly among all opli-
Learned
Preferences dataset
evarlvated by
The general design pattern used for learning a reward function from preferences and evaluating that reward function. The generic gridworld shown is for illustrative purposes only.
Figure 9:
mal actions with respect to the ground-truth reward function r, over Do. This methodology is illustrated in Figure
To compare performance across different MDPs, the mean return of policy म, V is normalized to (V / where V,* is the optimal expected return and V is the expected return of the uniformly random policy (both given Dp). Normalized mean return above 0 is better than V Optimal policies have a normalizod mean return of 1, and we consider above 0.9 to be near optimal.
6.1 An algorithm to learn reward functions with regret(ar())
A Algorithm Dis is a general algorithm for learning a linear reward function according to Pregret- This regret-specific algorithm only changes the regret-based algorithm from Section 2.2 by replacing Equation 5 with a tractable approximation of regret, avoiding expensive repeated evaluation V (-) and Q?(y.) to compute Pregret (-If) during reward learning. Specifically, successor features for set of policies are used to approximate the optimal state values and state-action values for any reward function. This algorithm straightforwardly applies general policy iteration (GPI) with successor features to approximate optimal state and action values for arbitrary reward functions, as described by Barreto et al. (2016).
Following the notation of Barreto et al., assume the ground-truth reward is linear with respect to a feature vector extracted by $: x Ax S Rd and a weight vector w, E Rd: =
Approximating Pregret with successor features
ur, During learning, wp similarly expresses as fs..,,') "o..a,,)""=(ss........ =ss,,,s")) wp.
Given a policy म, the successor features for (s,a) are the expectation of discounted reward features from that state-action pair when following (s, E D 2 (((( at $111) so 8,40 a]. Therefore, Q7 (s,aa) - 17 (s.a) wp. Additionally, state-based successor features can be calculated from the vo above as - A T (als) (s,a), making (T((s) + (w),.
(T)))=
W...
Given set V of state-action successor feature functions and a set V, of state successor feature func- tions for various policies and given reward function via wi, Q (s) ' mar ev, [ (s, a) Twy]] and
14

---

Published in Transactions on Machine Learning Research (02/2024)
Algorithm 1 Linear reward learning with regret preference model (Pregret), using successor features
1: Input: a set of policies
2: V @
3: for each reward function policy TS in the input set do
4: estimate das and verr (if not estimated already during step 4)
add v8 to V. add vise to V. end for
8: repeat
optimize by loss of Equ. calculating P01 > (2))) via Eqn. using Vo and Vp until stopping criteria are met
10:
11: return We
these two maximizations approxi- mations of (s,a,) and Vi (s), respectively. In practice, to enable gradient-based optimization with cur- rent tools, the maximization in this expression is replaced with the softmax-weighted average, making the loss function linear. Focusing first on the approximation of (s), for each v E V softmax weight calculated for ()(:: softmar (15 (w)) - [[<b, (s) w²) (T))(T//((EW Eo, (s)T)) wT)" where temperature T is a constant hyperparameter. The resulting approximation of V7( there- fore defined as (s)s) & Enyev, softmar,, (ab (s) ( (s) Similarly, (7 a); softmar (wb (s, a)TTun wF) & [( (s, Tw) uF w/: ev wa (s) (up)) and (; a) a Ev, softmar (4bg (s, a) T w (s,a) wy). Consequently, from Equs. and 5 the corresponding approximation Pregres of the regret preference model is:
(s) 2 mar EV, J (s) wel Barreto et al. 2016),
approximate
(01 o2 logiissti 1 - (2 (1666)))000)))
(6)
The algorithm In Algorithm lines 8-11 11 describe the supervised-learning optimization using the approximation Pregret, and the prior lines create VA and M. Specifically, given a set of input policies (line 1), for cach such policy TSF, successor feature functions Te and T are estimated (line 4), which by default would be performed by minor extension of a standard policy evaluation algorithm detailed by Barreto et al. (2016). Note that the reward function that ultimately learned is not restricted to be in the input set of reward functions, which is used only to create an approximation of regret.
One potential source of the input set of policies is set of reward functions, where each input policy is the result of policy improvement. on one reward function. We follow this method in our experiments, randomly generating reward functions and then estimating an optimal policy for each reward function. Specifically, for each reward function, we seek the the maximum entropy optimal policy, which resolves ties among optimal actions in state via uniform distribution over those optimal actions.
Further details of our instantiation of Algorithm | for the delivery domain can be found in Appendix F.1 along with preliminary guidance for choosing an input set of policies (Appendix F.1.1 and for extending it to reward functions that might be non-linear (Appendix F.1.2).
6.2 Results from synthetic preferences
Before considering human preferences, we first ask how each preference model performs when it correctly describes how the preferences in its training set were generated. In other words, we investigate empirically how well the preference model could perform if humans perfectly adhered to it. Recall that the ground-truth reward function, r, used to create these preferences but is inaccessible to the reward-learning algorithms.
For these evaluations, either a stochastic noiseless preference model acts preference generator to create a preference dataset. Then the stochastic version of the same model is used for reward learning, which prevents
15

---

Published in Transactions on Machine Learning Research (02/2024)
100%
Regret (noiseless)
% of MDPs which performance is near-optimal
80%
Partial return (noiseless)
Regret (stochustic)
60%
Partial return (stochastic)
40%
10
100
300
1000
3000
Preferences Per Training Set
performance is near-opping cor--pptmalll
1.6
performance is not near- near-opting optinoppmaal (purtial return/negret)
Ratio MDPs for which
(regret/partial return)
Noiseless
Ratio of MDPs for which
8.0
Noiseless
-- Stochasicc
Stochastic
1.4
6.0
1.2
4.0
2.0
1.0
3
10
30
100
300
1000
3000
10
30
100
300
1000
3000
Preferences Per Training Set
Preferences Per Training Set
Figure 10: Performance comparison over 100 randomly generated deterministic MDPs when each preference model creates its own training dataset and learns from Performance with the regret preference model is consistently better, regardless of training set size whether preferences generated stochastically. The bottom-left. and bottom-right plots are created from the top plot. The bottom-left plot shows the ratio of between each preference model's success rate. The bottom-right plot shows the ratio between each preference model's ss rate of failure reach near-optimal performance. For easier visual comparison, the ratios of each plot are chosen such that higher values indicate beller performance by the regret preference model.
the introduction of a hyperparameter. Note that the stochastic preference model can approach determinism through scaling the reward function, so learning a reward function with the stochastic preference model from deterministically generated preferences does not remove our ability to fit a reward function to those preferences. For the noiseless case, the deterministic preference generator compares a segment pair's 's E.r values for Psr or their regret(a)rr) values for Pregret. Note that through reward scaling the preference generators approach determinism in the limit, this noiscless analysis examines minimal-entropy versions of the two preference- generating models. (The opposite extreme, uniformly random preferences, would remove all information from preferences and therefore is not examined.) In the stochastic case, for each preference model, each segment pair is labeled by sampling from that preference generator's 'ss output distribution (Eqs) or using the unscaled ground-truti reward function.
We created 100 deterministic MDPs that instantiate variants of our delivery domain (sco Section 4.1 To create each MDP, we sampled from sets of possible widths, heights, and reward component values, and the resultant grid cells were randomly populated with destination, objects, and road surface types (see Appendix F.2 for details). Each segment in the preference datasets for each MDP was generated by choosing start state and three actions, all uniformly randomly. For set number of preferences, each method had the same set of segment pairs in its preference dataset. Figure 10 shows the percentage of MDPs in which each preference model results in near-optimal performance. The regret preference model outperforms the partial return model at every dataset size, both with and without noise. By Wilcoxon paired signod-rank test on normalized mean returns, p< 0.05 for 86% of these comparisons and p< 0.01 for 57% of them, as reported in Appendix F.2
Further analyses be found in Appendix F.2 with stochastic transitions, with different segment lengths, without segments that terminate before their final transition, and with additional novel preference models.
16

---

Published in Transactions on Machine Learning Research (02/2024)
6.3 Results from human preferences
We now consider the reward-learning performance of each preference model on preferences generated by humans for our specific delivery task. We randomly as- sign human preferences from our gathered dataset to different numbers of same-sized partitions, resulting in different training set sizes, and test each preference model on each partition. Figure 11 shows the results.
100%
% of partitions in which performance is near optimal
80%
Regret
60%
Partial
40%
return
20%
With smaller training sets (20- 100 partitions), the
regret preference model results in ncar-optimal per-
36
91
182
363
906
1812
formance more often. With larger training sets (1 10 partitions), both preference models always reach near- optimal return, but the mean return from the regret preference model is higher for all of these partitions except for only 3 partitions in the 10-partition test. Applying Wilcoxon paired signed-rank test nor- malized mean return to each group with 5 or more p<
(100)
(50)
(20)
(10)
(5)
(2)
(l)
Preferences per training set
(partitions)
Figure 11: Performance comparison over various amounts
of human preferences. Each partition has the number of preferences shown or one less.
partitions, p<0.00 0.05 for all numbers of partitions ions except 100 and p<0..0 for 20 and 50 partitions. To summarize, we find that both the regret and the partial return preference models achieve near-optimal performance when the dataset is sufficiently large although the performance of the regret preference model nonetheless almost always higher-ande and we also find that regret achieves near-optimal performance more often with smaller datasets.
Using the human preferences dataset, Appendix F3 contains further analyses: learning without segments that terminate before their final transition, learning via additional novel preference models, and testing the learned reward functions on other MDPs with the same ground-trute reward function.
7 Conclusion
Over numerous evaluations with human preferences, our proposed regret preference model (Pregret) shows improvements summarized below over the previous partial return preference model (Psr). When each preference model gencrates the preferences for its own infinite and exhanstive training set, we prove that Pregret identifies the set optimal policies, whereas PEr is not guaranteed to do so in multiple common contexts. With finite training data of synthetic preferences, Pregret also empirically results in learned policies that tend outperform those resulting from Psr. This superior performance of Pregret is also seen with human preferences. In summary, our analyses suggest that regret preference models are more effective both descriptively with respect to human preferences and also normatively, as the model we want humans to follow if we had the choice.
Independent of Pregret, this paper also reveals that segments' changes in state values provide information about human preferences that not fully provided by partial return. More generally, we show that the choice of preference model impacts the performance of learned reward functions.
This study motivates several new directions for research. Future work could address any of the limitations detailed in Appendix A.T Specifically, future work could further test. the general superiority of Pregret or apply it to deep learning settings. Additionally, prescriptive methods could be developed via the subject interface elsewhere to nudge humans to conform more to Pregret or to other normatively appealing preference models. Lastly, this work provided conclusive evidence that the choice of preference model is impactful. Subsequent. efforts could seek preference models that are even more effective with preferences from actual humans.
Acknowledgements
We thank Jordan Schncider, Garrett Warnell, Ishan Durugkar, and Sigurdur Orn Adalgeirsson, who each gave extensive and insightful feedback on earlier drafts. This work has taken place in part in the Safe, Correct, and Aligned Learning and Robotics Lab (SCALAR) at The University of Massachusetts Amherst. SCALAR research is supported in part by the NSF (11S-1749204, IIS IIS- (1S-1925,,2), AFOSR (FA9550-20-1-0077)) and ARO
17

---

Published in Transactions on Machine Learning Research (02/2024)
(78372-CS, W911NF-19-2-0333) This work has also taken place in part in the Learning Agents Research Group (LARG) at UT iustin. LARG research is supported in part by NSF (FATN-2019844), ONR (N00014-18-2243) ARO (W91LNE-19---0333), DARPA, Bosch, and UT Austin's Good Systems grand challenge. Peter Stone serves as the Executive Director of Sony AI America and receives financial compensation for this work. The terms of this arrangement have been reviewed and approved by the University of Texas at Austin accordance with policy on objectivity research.
References
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning, In Proceedings of the twenty-first first international conference on Machine learning, pp. 1, 2004.
Riad Akrour, Marc Schoenauer, and Michele Sebag. Preference-based policy learning. In Joint European Conference on Machine Learning and Knowledgee Discovery in Dalabases, pp. 12 27. Springer, 2011.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. orXiv preprint or Xiv: 16::6606.060 0666006565 2016.
Saurabh Arora and Prashant Doshi. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence, 297:103500, 2021.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXie preprint ar KVv:2224.05866 2022.
André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado Van Hasselt, and David Silver. Successor feal for transfer in reinforcement learning. arXie preprint wxx.v:1666.0555 05311, 2016.
Marc G Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong, Marlos C Machado, Subhodeep Moitra, Sameera S Ponda, and Ziyu Wang. Autonomous navigation of stratospheric balloons using reinforcement 2020.
learning. Nature, 588(786))
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. Xix preprint arXiv:1912.068 2019.
Erdem Bıyık, Dylan Losey, Malayandi Palan, Nicholas C Landolfi, Gleb Shevchuk, and Dorsa Sadigh. Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences. The International Journal of Robotics Research, pp. 0278364921 10416622, 2021.
Daniel Brown, Russell Coleman, Ravi Srinivasan, and Scott Niekum. Safe imitation learning via fast bayesian reward inference from preferences. In International Conference on Machine Learning, pp. 1165 1177. PMLR, 2020.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Newral Information Processing Systems (NIPS), pp. 2299-4307, 2017.
Yuchen Cui, Qiping Zhang, Brad Knox, Alessandro Allievi, Peter Stone, and Scott Niekum. The empathic framework for task learning from implicit human feedback. In Conference on Robot Learning, pp. 604 626.
PMLR, 2021.
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Magnelic control of tokamak plasıas through deep reinforcement learning. Nature, 602((897): 4414-4444411, 419,,,99,400 2022.
William Fedus, Carles Gelada, Yoshua Bengio, Mare Bellemare, and Hugo Larochelle. Hyperbolic discounting and learning over multiple horizons. arXiv preprint ar 1902 90660500 2019.
18

---

Published in Transactions on Machine Learning Research (02/2024)
Shane Frederick, George Loewenstein, and Ted O'donoghue. Time discounting and time preference: A critical review. Journal of economic literature, 40(2):351-40) 2002.
Amelia Glaese, Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint (rXXv:2209.14999 2022.
Adam Gleave, Mohammad Taufeeque, Juan Rocamonde, Erik Jenner, Steven H. Wang, Sam Toyer, Maxi- milian Ernestus, Nora Belrose, Scott Emmons, and Stuart Russell. imitation: Clean imitation learning implementations. arXiv:2211.11972 [es.L.G], 2022. URL httt:as://arii..orr/aba///////// 11972.
Joey Hejna and Dorsa Sadigh. Inverse preference learning: Preference-based rl without a reward function. arXiv preprint arXie:2305.15360 2023.
Borja Tbarz, Jan Leike, Tobias Pohlen, Geolfrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. arXiv preprint Xre: 01.2211,, 2018.
Kuno Kim, Shivam Garg, Kirankumar Shiragur, and Stefano Ermon. Reward identification in inverse reinforcement learning. In International Conference on Machine Learning, pp. 5496-5505. PMLR, 2021.
W Bradley Knox, Alessandro Allievi, Holger Banzhaf, Felix Schmitt, and Peter Stone. Reward (mis design for autonomous driving. arxiv preprint ar urXiv:2104.13906 2021.
W Bradley Knox, Stephane Hatgis-Kessell, Serena Booth, Scott Nickum, Peter Stone, and Alessandro Allievi. Reproduction Data for: Models of Human Preference for Learning Reward Functions, 2023. URL https: 1tbbi..rr/10..178887777 SAWWV
Zeb Kurth-Nelson and A David Redish. Yemporal-different reinforcement learning with distributed represen- tations. PLoS One, 4(10):e7362, 2009.
Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via
relabeling experience and unsupervised pro-training. arXiv preprint arXiv:2106.0500) 2021a.
Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. B-pref: Benchmaarinng reinforcement learning. Xixiv preprint XrXi:2111..3006, 2021b.
R Duncan Luce. Individual choice behavior: A theoretical analysis. John Wiley, 1959.
Andrew Y Ng and Stuart Russell. Algorithms for inverse reinforcement learning. In Seventeenth International Conference on Machine Learning (ICML), 2000.
OpenAI. Chatgpt: Optimizing language models for dialogue. OpenAl Blog https://oppnat..combbugg
chatgpt/ 2022. Accessed: 2022-12-20.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXie preprint arXiw:2203.021 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bui Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019. URL http:://ppp/ nwwrips..... cc/ paper/9015-pytorcch an- imperative-styye http::/0ooooroooooooooooaaaaaeer
styye--ighh periogrnnproornccceee pertorrance-dee-leemneee deep-
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825 2830, 2011.
19

---

Published in Transactions on Machine Learning Research (02/2024)
A David Redish and Zeb Kurth-Nelson. Neural models of delay discounting. In Impulsivity: The behavioral and neurological science of discounling,, pp. 123-158, American Psychological Association, 2010.
Stuart Russell and Peter Norvig. Artificial intelligence: modern approach. Pearson, 2020.
Dorsa Sadigh, Anca D Dragan, Shankar Sastry, and Sanjit A Seshia. Active preference-based learning of reward functions. Robolics: Science and Systems, 2017.
Andrew W Senior, Richard Evans, John Jumper, James Kirkpatrick, Laurent Sifre, Tim Green, Chongli Qin, Augustin Zídek, Alexander WR Nelson, Alex Bridgland, et al. Improved protein structure prediction using potentials from deep learning, Nature, (77())22) 710, 2020.
Roger N Shepard. Stimulus and response generalization: A stochastic model relating generalization distance in psychological space. Psychometrika, 22(4):325-345, 1957.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Toannis Antonoglou, Veda Panneershelvam, Marc Lanclot, el al. Mastering the game of go with deep neural networks and tree search. Nature, :29(7787):484 489, 2016.
Joar Skalse, Matthew Farrugia-Ro Roberts, Stuart Russell, A Alessandro Abate, and Adam Gleave. Invariance in policy optimisation and partial identifiability in reward learning. arXiv preprint arXiw:2203.074 2022.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018
Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in StarCraft 11 using multi-agent reinforcement learning. Nature, (56(7782)33003333 2019.
John Von Neumann and Oskar Morgenstern. Theory of games and economic behavior. Princcton university press, 1944.
Xiaofci Wang, Kimin Lee, Kourosh Hakhamaneshi, Picter Abbeel, and Michael Laskin. Skill preferences: Learning to extract and execute robotic skills from human feedback. In Conference on Robot Learning, pp. 1259-1268. PMLR, 2022.
Peter R. Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas J. Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, Leilani Gilpín, Varun Kompella, Piyush Khandelwal, HaoChih Lin, Patrick MacAlpine, Declan Oller, Craig Sherstan, Takuma Seno, Michael D. Thomure, Houmehr Aghabozorgi, Leon Barrett, Rory Douglas, Dion Whitehead, Peter Duerr, Peter Stone, Michael Spranger, and Hiroaki Kitano. Outracing champion gran turismo drivers with deep reinforcement learning. Nature, 62:223-222 Feb. 2022. doi: 10.1038 01118860--21-------
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and GeofTrey Irving. Fine-tuning language models from human preferences. ar preprint ar Xiv: 1 1909.999900 08593, 2019.
20

---

Published in Transactions on Machine Learning Research (02/2024)
After Appendix A the appendix is organized according to the major sections and subsections of the main content.
A Limitations and ethics
A.1
Limitations
Some limitations of the regret preference model are discussed in the paragraph "Regret as model for human preference" in Section 2.2, including assumptions that a person giving preferences can distinguish between optimal and suboptimal segments, that they follow a Boltzmann distribution (i.e., a Luce Shepard choice rule), and that they base their preferences on decision quality even when transition stochasticity results in segment pairs for which the worse decision has better outcome.
Our proposed algorithm (Section 6.1 has a few additional limitations. Generating candidate successor features for the approximations Q7 and Vi may be difficult in complex domains. Specifically, challenges include choosing the set of policies or reward functions for which to compute successor features (line 3 of Algorithm [ discussed in Appendix F. and creating a reward feature vector & for non-linear reward functions (discussed in Appendix F.1.2p. Additionally, although learning with Pregret is more sample efficient in our experiments, it is computationally slower than learning with Per because of the additional need to compute successor features and the use of the softmax function to approximate Q7 and Vi* Nonetheless, we may accept the tradeoff of an increase in computational time that reduces the number of human samples needed or that improves the reward function's ss aligument with human stakeholders interests. Lastly, the loss during optimization with Pregret was unstable, which we addressed by taking the minimum loss over all epochs during training, Therefore, for more complex reward feature vectors (Ø) than our 6-element vector for the delivery task, extra care might be needed to avoid overfitting f, for example by withholding some preference data to serve as a test set.
We also generally assume that the RL algorithm and reward learning a algorithm use the same discount factor as in the MDP specification. One weakness of contemporary deep RL is that RL algorithms require artificially lower discount factors than the truc discount factor of the task. The interaction of this discounting with Our expectation though is that this weakness of deep RL algorithms is likely temporary one, and so we focused our analysis on simple tasks in which we do not need to artificially lower the RL algorithm's 's discount factor. However, further investigation of the interaction between preference models and discount factors would aid near-term application of Pregret to deep RL domains.
preference models is considered in Appendix F 22
This work also does not consider which segment pairs should be presented for labeling with preferences used for reward learning. However, other research has addressed this problem through active learning (Lee al. 2001aa]
Christiano et al.) 2017 Akrour et al. 2011), and future work could consider the compatibility of these active learning methods with our Algorithm combining the improved sample efficiency of P Pregret with that of these active lcarning methods.
Additionally, all preference models considered in this paper are evaluated with the assumption that for any pair of segments in the preference dataset, both segments have the same length. However, for segment pairs of different lengths, the regret preference model may act in ways that violate our intuition for how humans would give preferences. In particular, a short, highly suboptimal segment might be more preferable under the regret. preference model than much longer segment that is near-optimal yet nonetheless has higher regret, since the regret of segment is the sum of the regret of its individual transitions. The partial return model which already breaks our intuition with same-length segment deviates from intuition as the segment. lengths diverge. Allowing such different-length segment pairs appears uncommon in practice but may be difficult to avoid in some task domains.
from a similar limitation, in that it further
Regarding the human side of the problem of reward learning from preferences, further research could provide several improvements. First, we are confident that humans can be influenced by their training and by the preference elicitation interface, which is particularly rich direction for follow-up study. We also do not consider how to handle learning reward functions from multiple human stakeholders who have different preferences, a topic we revisit in Appendix A.2 A.2 Lastly, we expect humans to deviate from any simple model, including P Pregret.
21

---

Published in Transactions on Machine Learning Research (02/2024)
and fine-grained characterization of how humans generate preferences could produce preference models that further improve the alignment of the reward functions that are ultimately learned from human preferences.
A.2 Ethical statement
This work is meant to address ethical issues that arise when autonomous systems are deployed without properly aligning their objectives with those of human stakeholders. It is merely step in that direction, and overly trusting in our methods-even though they improve on previous methods for alignment-could result in harm caused by poorly aligned autonomous systems.
When considering the objectives for such systems, a critical ethical question is which human stakeholders interests the objectives should be aligned with and how multiple stakcholders' interests should be combined into single objective for an autonomous system. We do not address these important questions, instead making the convenient-but butttt--btttllwwddd assumption that many different humans' preferences simply be combined. In particular, care should be laken that vulnerable and marginalized communities are adequately represented in any technique or deployment to learn a reward function from human preferences in figh-impact settings. The stakes are high: for example, a reward function that is only aligned with a corporation's 's financial interests could lead to exploitation such communities or more broadly to exploitation of or harm to users.
In this specific work, our filter for which Mechanical Turk Workers could join our study is described in Appendix(] We did not gather demographic information and therefore cannot assess how representative our subjects are of any specific population.
A.3 On the challenge of using regret preference models in practice
We have provided evidence theoretically and with experimentation -tht the regret, preference model is more effective when precisely measured effectively approximated. The challenge of efficiently creating such approximations presents one clear path for future research. We believe this challenge does not justify staying within the local maximum of the partial return preference model.
Like the regret preference model, inverse reinforcement learning (IRL) was founded on an algorithm that requires solving an MDP in an inner loop of learning a reward function. For example, see the seminal work on IRL by Ng & Russell (2000). IRL has been an impactful problem despite this challenge, and handling this inner-loop computational demand is the focus of much IRL research.
Future work on the application of the regret preference model can face the challenge of scaling to more complex problems. Given that IRL has made tremendous progress in this direction and Brown et al. (2000) have scaled an algorithm with similar needs to those of A Algorithm [ we are optimistic that the methods to scale can be developed, likely with light adaptation from existing methods (c.g., in Brown et al. or in Appendix F. 1. and F.1.2..
Preference models for learning reward functions
Here we extend the content of Section 2 focusing on preference models and learning algorithms that use them. This corresponding section of the appendix provides a simple derivation of the logistic form of these preference models, discusses extensions of the regret preference model, sketches an alternative way to learn a policy with it, and discusses the relationship of inverse reinforcement learning to learning reward functions with regret preference model.
B.1 Derivation of the logistic expression of the Boltzmann distribution
For the reader's convenience, below we derive the logistic expression of function that is based two subtracted values from the Boltzmann distribution (i.e., softmax) representation that is more common in past work. These values are specifically the same function f applied to each segment, which is a general expression of both of the
22

---

Published in Transactions on Machine Learning Research (02/2024)
preference models considered here.
exp [f(r)) exp [ [f(
P(01+02)=
[s(az]]
1+ exp [[()]]]
(7)
1
1 [[(02)- = (
B.2 Learning reward functions from preferences, with discounting
For the equations from the paper's body that assume that there no temporal discounting (i.c., y= we share in this section versions that do not make this assumption. 7f then the equations below simplify to those in the body of the paper. To allow for fully myopic discounting with y=0, we define =0=1.
Recall that i indicates an arbitrary reward function, which may not be the ground-truth trrth reward function, r, and refers to a learned reward function. Similarly, 5 refers to an arbitrary exponential discount. factor, which may not be the ground-truth discount factor, 7, and f refers to the discount factor during learning, which could be inferred or hand-coded. Also, the notation of V;* and QF are expanded this subsection to denote the discounting in their expected return: ((F.5) and (1,5)) respectively.
In most of this article, the discount factor used during reward function inference is hard-coded as 2= However, in the theory of Section 3 assume is not known to reach more general conclusions. In this subsection, for generality we likewise assume that ry is not known, using Ý generally and using & in notation we consider specific to reward function inference.
The discounted versions of the preference models below rely on a cross entropy loss function that identical to Equation I except for the inclusion of discounting notation:
loss(f,f,DD.) - E 1
(8)
(01.0214)ED
Partial return With discounting, the partial return of a segment a is E "FF 1 This notation differs from that in Section 2.1 in that the subscript of the reward symbol fa,e is now expanded to include which segment it comes from.
The preference model based on partial return with exponential discounling is expressed below, generalizing Equation 2
1011-1
loal-1
02 logistic
5/
Fon
Σ
Regret With discounting, for transition (51,4))1441) in segment containing only deterministic transitions, regreta A (5%) [r+ Pirt-vVV. ((
Vi
For full deterministic segment, regrela (1175)) with exponential discounting is defined as follows, generalizing Equation 3
1
regreta(o)i.9)) 4
5f regreta(a.lF.f)
(10)
la1-1 For + zl glolve
(())))(a))
Σ
Vin (5% (s))
Like Equation 3 this discounted form of deterministic regret also measures how much the segment reduces expected return from the start state value, V (6))
23

---

Published in Transactions on Machine Learning Research (02/2024)
To create the general expression of discounted regret that accounts for potential stochastic transitions, note that, with discounting, the effect on expected return of transition stochasticity from transition (8,,01,8++1) is and add this expression once per transition to get regret(alř, removing the subscript that refers to determinism. The discounting does not change the simplified expressions in Equation the regret for a single transition:
+ (F.9) (5t+1)] (s, (s),
regret(ov.)ã ( ( (1 (h
= (ins)))))))))))))))))))))))))))) (57
(11)
With both discounting and accounting for potential stochastic transitions, regret for a full segment
regrel(ali,f)
x regree((o)17.)
-
(12)
-5t (6)
The expression of regret above is the most general in this paper and can be used in Equation 5 iddntically as the undiscounted version in Equation 4
Equation 6 the approximation Pregret of the regret preference model derived in Section 6.1 is expressed with discounting below.
(1g)))) Eleal- 4r
( (57",08))
(6,))))))))))))0)0) - -l ( (s"")) (5 .o7
(13)
Note that the successor features used in Section 6.1 to determine these approximations, Vi and (7 already include discounting.
As with the undiscounted versions the above equations, if two segments have deterministic transitions, end in terminal states, and have the same starting state, this regret model reduces the partial return model: Progret ( (-1,5)= (1155))
If hard-coding 9, when to set & <1 during reward function inference In reinforcement learning, both and r together determine the set of optimal policies. Changing either y or r while holding the other constant will often change the set of optimal policies.
For both preference models, we suspect that learning would benefit from using the same discounting during reward inference as the human used while evaluating sogments to provide preferences (i.c., setting f= And this same would be used for learning policy from the learned reward function. On the other hand, when & is hand-coded and *Y, the reward inference algorithm will regardless attempt find an f that explains those preferences; however, set of optimal policies is determined by a reward function with the discount, and the set of optimal polices created by the human's 's reward function and discounting may not be determinable under a different discountinng.
Not only is a specific human rater's y unobservable, but psychology and economics researchers have firmly established that humans do not typically follow exponential discounting (Frederick et al., 2202)). which should evoke skepticism for hard-coding << during reward function inference. One exception humans who have been trained to apply exponential discounting, such in certain financial settings. The best model for how humans discount future rewards and punishments is not settled, but one popular model is hyperbolic discounting. Some exploration of RL with hyperbolic discounting exists, including approximating hyperbolically discounted value
24

---

Published in Transactions on Machine Learning Research (02/2024)
function using mixture of exponentially discounted value functions Kurth-NNlson & Redish Kurth-Nelson 2010). However, it has not. found clear usage beyond as auxiliary task to aid representation learning (Fedus et al. 2019). The interpretation of human preferences over segments appears to us to be a strong candidate for using these methods to approximate hyperbolic discounting.
2009 Redish &
This research topic currently lacks a rigorous treatment of discounting when learning reward functions from human preferences and such an investigation beyond our scope, and we leave our guidance above speculative.
B.3 Logistic-linear preference model
In Appendices E.2, F.2.5 and F.3.2 we also consider preference models that arise by making the noiseless preference model a linear function over the components of Pregreta- Building upon Equation aabove, set. f()) (V; 2., (=))))) This preference model, Plog-tine can be expressed after algebraic manipulation
88
Prog-tin (01 > 02) llogisicc i
Fo F- Do Vi (8 ) V2 (o
(14)
This logistic-linear preference model is a generalization of Psy and also of Pr Pregreta the regret preference model for deterministic transitions. Specifically, if w= (0,1,)), then (- Piog-tin = Pregreta (-IF). More gencrally, for some constant c, w = (0,c,0) and xt = (-c.c,e) recreate Psr and Pregret, respectively but with different reward function scaling, which is the same as allowing a different temperature in the Boltzmann distribution that determines preference probabilities. In Appendix F.2 we fit of to maximize the likelihood of the human preference dataset under Piog (-rr,, using the ground-truth T, and compare the learned weights those of Psr and Pregreta
And if = then
B.4 Adding constant probability of uniformly distributed preference
Appendix E.2 also considers adaptations of Per: Pregretas and Piog-kin that add a constant probability of uniformly distributed preference, as was done by Christiano al. (017)) The body of the paper does not. consider these adaptations.
We create this adaptation, which we will call P' here, from another preference model P by P' 02) [11 (1-iogistic(c))) P(a) > x2]] [logistic(c)/2), where e is constant that in practice we fit to data and logistic(e) is the constant probability of uniformly random preference. The logistic(c) allows any constant e to result in a the constant probability of uniformly distributed preference to be in (0.1). The term logistic(c)/2 gives half of the constant probability to Ti and half to 02. The term [1-logistic(C) scales the P(o, > 02) probability which could be Per: Pregreta, or Plog-tin to a proportion of the remaining probability. The only difference this adaptation and Christiano al.'s 0.1 probability of uniformly distributed preference that we learn the value of e from training data (in a k-fold cross-valle validation setting), as we see in Appendix E whereas Christiano et al. do not share how 0.1 was chosen.
B.5 Expected return preference model
In Appendix F.3 test reward learning on third preference model. This expected return preference model is VV; derived by making ((o)==(EEf++V)) (6% in Equation 7 This segment statistic (aa can be considered be in between deterministic regret (Equation and partial return, differing from each by one term.
We include this preference model because judging by expected return is inluitively appealing in that it considers the partial return along the segment and the end state value of the segment, and we found it plausible that human preference providers might tend to ignore start state value, as this preference model does. However, reward learning with the regret model outperforms or matches that by this expected return preference model, as we show in Appendix
25

---

Published in Transactions on Machine Learning Research (02/2024)
B.6 Relationship to inverse reinforcement learning
Like learning reward functions from pairwise preferences, inverse reinforcement learning (IRL) also involves learning reward function. However, the inputs to IRL and learning reward functions from pairwise preferences are difforent: IRL requires demonstrations, not preferences over segment pairs. However, because a a regret-basod preference model always prefers optimal segments over suboptimal segments, at least one further connection can be made. If one assumes that demonstrated trajectory segment noiselessly optimal as in the foundational IRL paper on apprenticeship learning (Abbeel & Ng. (2044) then such a demonstration is equivalent to expressing preference indifference for the demonstrated segment over all other segments. In other words, no other segment is preferred over the demonstrrted segment. However, IRL. has its own identifiability issues in noiseless scttings (e.g., see Kim et al. (021]) that, viewed from the lens of preferences, come in part from the "indifference" part of the above statement: since there can be multiple optimal actions from a single state, it is not generally correct to assume that a demonstration of one such action shows a preference over all others, and therefore remains unclear in IRL what other actions are optimal. Note that since partial-return-bass preferences can prefer suboptimal segments over optimal segments, the common assumption in IRL that demonst.ration are optimal does not map as cleanly to partial-return-basel preferences.
The regret preference model also relates to IRL in that the most basic version of IRL requires solving an MDP
in the inner loop (sec Algorithm 1 in the survey of IRL by Arora & Doshi (021]) ), as appears necessary for a perfect measure of regret while learning a reward function. The progress that IRL has made addressing this challenge gives us optimism that it is similarly addressable for complex tasks in for our proposed algorithm. We
discuss potential solutions in Appendix F.1II 1.1 and 1.2
C Theoretical comparisons
The relevance of noiseless preference generators Because we model preferences as stochastic in Section 2 one might reasonably wonder how the above theoretical analysis of noiseless preference generators are relevant. We offer four arguments below.
First, having structured noise provides information that can help both preference models, but these proofs show that there are cases where the signal behind the noise-either regret or partial return-is not sufficient in the partial return case to identify an equivalent reward function. So, in a rough sense, regret more effectively uses both the signal and the noise, which might explain its superior sample efficiency in our experiments across both human labels and synthetic labels. Relatedly, the noiseless setting can help us understand each preference model's sample efficiency in a low-noise setting.
Second, noiseless preferences are also feasible, even if they rare. Therefore, understanding what can be learned from them is worthwhile. Theorem shows that there are MDPs in which there is no class of preference models stochassiccc or deterministic- that can identify an equivalent reward function from nartial-rtuun-bbscol preferences if the preference generalor noiselessly prefers according to partial return. Specifically, we show that the mapping from two reward functions with different sets of optimal policies to partial-return based preferences and therefore the information simply does not exist to invert that mapping and identify a reward function with the same set of optimal policies. In contrast, Theorem 3. shows that preferences generated noiselessly (and in certain stochastic settings) by regret do not have this issue.
is a
Third, noise is often motivated as modeling human error. Having algorithm rely noise structured in a very specific, Boltzmann rationaal way-is is an undesirable crutch. Skalse et al. (2022) justify including noiseless preferences in their examinations of identifiability with a similar argument: "these invariances rely heavily on the precise structure the decision noise revealing cardinal information in the infinite-data limit".
Beyond the work of Skalse et al. (022), there is broader precedent for considering noiseless human input for theory or derivations. For instance, the foundational IRL research on apprenticeship learning (Abbeel &
Ng. 2004) treats demonstrations noiselessly optimal. Recent work by Kim et al. (2021) focuses reward identifiability with noiseless, optimal demonstrations.
26

---

Published in Transactions on Machine Learning Research (02/2024)
D Additional information for creating human-labeled preference dataset
D.1 The preference elicitation interface and study overview
Here we share miscellancous details about the preference clicitation interface from which we collected human subjects' preferences. This description builds on Section 4.2
In selecting preferences, subjects had four options. They could prefer either trajectory (left or right), or they could express their preference to be the same or indistinguishable. To provide these preferences, subjects could either click on each of the buttons labeled "L.EET,, 'RIGHT' 'SAME", "CAN'T TELL" (shown in Figure or by using the arrow keys to select amongst these choices.
For the interface, all icons used to visualize the task were obtained from icons8.com com under their Paid Universal Multimedia Licensing Agreement.
We paid all subjects $5 per experiment (i.e., for each a Mechanical Turk HIT), which was chosen using the median time subjects took during a pilot study and then calculating the payment to result in $15 USD per hour. This hourly rate of $15 was chosen because it is commonly recommended as an improved US federal minimum wage. The human subject experiments cost $2,145 USD in total.
An experimental error resulted in the IRB-approved consent form not being presented to human subjects after Mechanical Turk Workers accepted our study. We reported this error to our IRB and received their approval to use the data.
D.2 Filtering subject data
To join our study via Amazon Mechancial Turk, potential subjects had to meet the following criteria. They had to be located in the United States, have an approval rating of at least 99%, and have completed at least 100 other MTuuk HITss. We selected these criteria to improve the probability collecting data from subjects who would attentively engage with our study and who would understand our training protocol.
We assessed each subject's understanding of the delivery domain and filtered out those who did not comprehend the task, as described below. Specifically, subjects completed task-comprehension survey, through which we assigned them a task-comprehension score. The questions and answer choices are shown in Table 2 Each fully correct answer was worth 1 point and each partially correct answer was worth 0.5 points. Task--compeeenssion scores were bounded between and 7. We removed the data from subjects who scored below a threshold of 4.5. The threshold 4.5 was chosen based on visual analysis of a histogram of scores, attempting to balance high standards for comprehension with retaining sufficient data for analysis.
In addition to filtering based off the task comprehension survey, we also removed subject's data if they ever preferred colliding the vehicle into a sheep over not doing so. Since such collisions are highly undesirable in this task, we interpreted this preference as evidence of either poor task understanding or inattentiveness.
In total, we collected data from 143 subjects. Data from 58 of these subjects were removed based on subjects' responses to the survey. From what remained, data from another 35 subjects were removed for making the aforementioned sheep-collision preference errors. After this filtering, data from 50 subjects remained. This filtered data consists of 1812 preferences over 1245 unique segment pairs and is used in this article's experiments.
Regarding potential risks to subjects, this data collection had limited no risk. No offensive content was shown to subjects while they completed the HIT. Mechanical Turk collected Worker IDs, which were used only to link preference data with the results from the cask-comppeeennsoo survey for filtering data (see Appendix D.2) and then were deleted from our data. No other potentially personally identifiable information was collected.
D.3 The two stages of data collection
We collected the human preference dataset in two stages, as mentioned in Section 4.2 Here we provide more detail on each stage. These stages differed largely by their goals for data collection and, following those goals, how we chose which segment pairs were presented to subjects for their preference.
27

---

Published in Transactions on Machine Learning Research (02/2024)
Table The task comprehension survey, designed to test participant's comprehension the domain for the purpose of filtering data, Each full credit earned point; each partial credit answer earned 0.5 points. We discarded the data of participants who scored less than 4.5 points overall.
Question
Full credit
Partial credit
Other choices
What is goal of this world? (Check all that apply.)
maximize profit.
get specific loca- tion.
To drive as far possible explore the world.
maximize profit
To collect as many coins as possible,
Partial credit was given if both answers were selected.
To collect as many sheep possible.
To drive sheep to specific location.
What happeus when you into house? (Check all that apply.)
You pay gas penalty.
You pay gas penalty.
The episode ends.
You can't 't run into house;
can't rum into a house; the world doesn't you move
You get stuck.
the world doesn't 't let you
To collect as many sheep as possible.
move into
Full credit was given if both answers were selected.
Partial credit was given if ofnly one answer was selected.
What happeus when you run into a sheep? (Check all that apply.)
The episode ends.
The episode ends.
You are rewarded for collecting a sheep.
You are penalized for run- ning into sheep.
You penalized for run- ning into sheep.
Full credit given if both
Partial credit given if oolly selected.
selected.
What happens when you roadblock? (Check all that apply.)
You pay penalty.
The episode ends.
You get stuck.
You can't run into roadblock; the world doesn't let you move into it.
Is running into a roadblock ever good choice in any town?
Yes, in certain circum- stances.
No.
What happens
when you into the brick area? (Check all that apply.)
You pay extra for gas.
The episode ends.
You stuck in the brick area.
You can't go brick area; the world doesn't let you into
entering the brick area ever good choice?
Yes, in certain circum-
No
stances
First stage Figure 12 illustrates the coordinates that segment pairs were sampled from in the first stage of data collection, varying by state value differences and by differences in partial returns over the segments. We sought a range of points that would allow a characterization of human preferences that is well distributed across different parts of the plot. To better differentiate the consequences each preference model, we intentionally chose a large number of points in the gray area of Figure where the regret and partial return preference models would disagree (i.e., each giving a different segment preference probability greater than 0.5).
28

---

Published in Transactions on Machine Learning Research (02/2024)
5
Figure 12:
Coordinates from which segment pains were sampled from during the first stage of data collection. The r-axis is state value differ- ences between the two segments and the y-axis is partial return differences between the two segments. The areas of the circles are proportional to the number of samples at that point, and the proportionality is consistent. across this plot and the 3 subplots of Figure 13
A.V - A,V
We now describe our segment-pair sampling process more specifically. We first we constructed all unique segments of length 3 and then exhaustively paired them, resulting in nearly 30 million segment pairs, Each segment pair's partial returns, start-state values, end-state values place the segment pair on a coordinate in Figure S and segment pairs that are not on any of the dots in Figure were discarded. For the segment pairs at each coordinate, we further divided them into bins: non-terminal segments with the same start state and different end states, non-terminal segments with different start states and different end states, terminal segments with the same start state and same end state, terminal segments with different start states and the same end state, and bin of segment pairs that fit in none of the other bins. Segment pairs in the 5th bin were discarded. From each of the 4 bins corresponding to each point in Figure S pairs. If the bin did not have at least 20 segment pairs, all segment pairs in the bin were "sampled". All sampled segment pairs from all bins for all points in Figure g made up the pool of segment pairs used with Mechanical Turk. For each subject, 50 segment pairs were randomly sampled from this pool. We gathered data until we had roughly 20 labeled segment pairs per bin. After filtering subject data, this first stage contributed 1437 segment pairs out of the 1812 pairs used in our reward learning experiments in Section 6.3 and Appendix(F..
we randomly sampled 20 segment
Second stage When we conducted the reward- reward-learning evaluation in Cootionwwith. only the data from the first stage, Par performed very poorly, always performing worse than uniformly random. This performance difference is shown in Appendix F.3. In contrast Pregret performed well, always achieving near-optimal performance. To better assess Per, we investigated its results with synthetic preferences in detail and spoculated that two types of additional segment pairs would aid its performance. The first of these two lypes include one segment that is terminal and one that is non-terminal, which expected to help differentiate the reward for reaching terminal states from that of reaching non-terminal ones.
5
52
46
30
50
42
44 A A.V
-56- V.V A
A.V A.V
Figure 13: Coordinates from which segment pairs were sampled from during the second stage of data collection. The points are in 3 distant clusters, they are presented in 3 separate subplots for readability. The areas of the circles are proportional to the number of samples at that point, and the proportionality is consistent across these 3 subplots and Figure 12
The second of these two types are two segments that each terminate at different t values. For example, one segment terminates on its end state, s% and another terminates after its first (ransition, at 8f- These early-terminating segments can be viewed either as shorter segments or as segments of the same length as the other sogments 33), where they reach absorbing state from which no future reward can be received. We speculated that this second type of segment pairs would help learn the negative reward component for each move (i.e., the gas cost). Specifically, in the first stage's data, both segments in pair always have the
29

---

Published in Transactions on Machine Learning Research (02/2024)
same number of non-terminal transitions, seemingly preventing preferences from providing information about whether an extra transition (from non-absorbing state) generally resulted in positive or negative reward. These segment pairs were included in all results unless otherwise stated. Note that this second type addresses the identifiability issuc of the partial return model related to a constant shift in the reward function and discussed in Section 3.2.2 and Appendix F.2.2. The speculation described above was a conceptual predecessor of our understanding of this identifiability issuc. In particular, any change to this gas cost cost-which given at every constant shift in the reward function.
time step is equivalent
We now describe our segment- pair sampling process for the second stage more specifically. For the first additional type of segment pair, where one segment is terminal and one is not, we randomly pair terminal and non-terminal segments from the first-stage pool of segment pairs drawn from to present to subjects. In this pairing, each segment is only used once, and pairing stops when one of all terminal segments or all mon-terminall segments have been paired. The corresponding coordinates for these pairs are shown in the two right most plots of Figure 13] For the second additional type of segment pair, we utilize all terminal segments from the pool of segment pairs shown to subjects in the first. stage. For each of these terminal segments, we construct two additional segments: one that shifts the segment earlier, removing the first state and action and adds dummy transition within absorbing state at the end, and another that shifts the segment two timesteps carlier and adds two such dummy transitions at the end. These two newly constructed segments are then each paired with the original segment, producing two new pairs for each terminal segment in the data set. The corresponding coordinates for these segment pairs are shown in the left most plot of Figure 13
All of both types of additional segments pairs are then characterized by the coordinates shown in Figure B Then, with the first stage, we randomly sampled 20 segment pairs from each coordinate to make the experimental pool for the second round of Mechanical Turk data collection. If 20 segment pairs were not available at coordinate, we used all segment pairs for that coordinate. As in the first stage, 50 segment pairs were randomly sampled from this pool to be presented to each subject during preference elicitation. After filtering subject data, this first stage contributed 311 segment pairs out of the 1812 pairs used in our reward learning experiments in Section 6.3 and Appendix F3
D.4 The study design pattern
This work follows an experimental design pattern that is often used for studying methods that take human input for evaluating the desirability behaviors or outcomes. This pattern illustrated for the specific case of learning reward functions from preferences in Figure In this pattern, human subjects are taught understand specific task metric and/or are incentivized to align their desires with this metric. The human subjects then provide input to some algorithm that has no knowledge of the performance metric, and this algorithm or learned model is evaluated on how well its output performs with respect to the hidden metric. For another example, see Cui et al. (2021).
E
Descriptive results
E.1 Derivation of
V
A,, Vz)+ F-E Em
The derivation below supports our assertion in the first paragraph of Section 5.1
regreta(02 (regreta
(11188)-88,4147111)))11177771)11 --; (5
E0ogF - -Eon
(15)
(MNT + -- 1
(Ao, (A V1) -2),
E.2
Losses of an expanded set of preference models on the human preferences dataset
30

---

Published in Transactions on Machine Learning Research (02/2024)
Table 3: Expanding on Table mean cross-entropy
test loss over 10-fold cross validation (n=1812) from predicting human preferences. Lower is better.
Table 3 shows an expansion of Table I including models intuodee in Appendix B The logistic linear preference model, Plog-lin provides a lower bound, given that it can express either Pregret or Per and that we do not not observe any overfitting of its parameters.
Loss
Preference model
(n= 1,812)
P()) 0.5 (uninformed)
0.693
Per (partial return)
0.620
Including the constant probability of uniformly random
P Pregvot (regret)
0.573
response, as in Christiano et al. (2017) also increases the
Plogdin (logistic linear)
0.548
expressivity of the model. The final three results in Table 3
Per with prob of uniform response
0,620
show the best test loss achieved across different training
Pregret with prob of uniform response
0,573
Piog-lin with prob of uniform response
0.548
runs that differ by initializing logistic(e) of this model to
be 0.01, 0.1, or 0.5. Surprisingly, no benefit observed from including a constant probability of a uniformly random response. Because this augmentation of our models appears to have no effect on the likelihood, we do not include it in further analysis.
where each weight Across the 10 folds, the mean weights learned for Prog-tin( (-17) were w = -- applies respectively to segments" start state values, partial returns, and end state values. Scaled to have a maximum weight of 1 for easy comparison with Per and P Prerrets Usculed = that these weights are closer to those that make Prog-tin lin Pegret,, (i.e., == than to those that make Plog-tin= = Pxr (i.e., w= (0,1,0)). Also, the notable deviation from the weights of Pregreta is the weight for the start state value, which has half as much impact as the regret preference model gives it. In other words, this lower weight suggests that our subjects did tend to weigh the maximum possible expected return from each segment's 's start state, but they did so less than they weighed the reward accrued along each segment and the maximum expected return from each segment's end state.
First, we note
F
Results from learning reward functions
This section provides additional implementation details for Section discussion of potential improvements, and additional analyses that thematically fit. in Section 0
F.1 An algorithm to learn reward functions with (a)r)
We describe below additional details of our instantiation of Algorithm
Doubling the training set by reversing preference samples Because the ordering of preference pairs is arbitrary-i.e. (0) <0))= σ1) for all preference datasets we double the amount of data by duplicating each preference sample with the opposite ordering and the reversed preference. This provides more training data and avoids learning any segment ordering effects.
Collecting the policies from which successor feature functions are calculated For this article's instantiation of Algorithm ! we collect successor feature functions by randomly sampling a large number of reward functions and then calculating the successor feature functions for their optimal polices. This procedure is more precisely described below.
1. Create reward function by sampling with replacement each element of its weight vector, Wa, from 1-50,-10,- 1,0,,,,,,,11
2. For this reward function, use value iteration to approximate its maximum entropy optimal policy and that policy's 's successor feature function.
3. If this successor feature function policy differs from all previously calculated successor feature functions, save it and go to step 1.
4. Otherwise it is a redundant policy. Tf less than 300 consecutive redundant policies have been found, go to step 1.
31

---

Published in Transactions on Machine Learning Research (02/2024)
The policy collection process above is terminated after 300 consecutive redundant policies are found. Finally, we calculate the maximum entropy optimal policy for the optimal policy for the ground-truth reward function, T, and remove the successor feature function for any policy that matches the optimal policy for r. In other words, we remove any policies for other reward functions that were also optimal for r, making the regret-basod learning problem more difffcull.. We ensured that the ground-truth reward function was not represented to better approximate real-world reward learning applications, in which one would be unlikely to have the optimal policy for learning a successor features function. On the specific delivery task on which we gathered human preferences, the process above resulted in 70 reward functions.
Early stopping without a validation set During training, the loss for the Pregret model tended to show cyclical fiuctuations, reaching low loss and then spiking. To handle this volatility, we used the f that achieved the lowest loss over all epochs of training, not the final f. For Per and Pregret. we found no evidence of overfitting with our linear representation of the reward function, but with a more complex representation, such early stopping likely should be based upon the loss of the model on a validation set. beller understanding of the cyclical loss fluctuations we observe during training could further improve learning with Pregret.
Discounting during value iteration Despite the delivery domain being an episodic task, low- wow-performing policy can endlessly avoid terminal states, resulting in negative-infinity values for both its return and successor features based on the policy. To prevent such negative-infinity values, we apply a discount factor of 7 0.999 during value iteration which is also where successor feal ure functions are learned and when assessing the mean returns of policies with respect to the ground- reward function, r. We chose this high discount factor to have negligible effect on the returns of hig--performig policies (since relatively quick termination is requirod for high performance) while still allowing value iteration to converge within a reasonable time.
Hyperparameters for learning Below we describe the other specific hyperparameters used for learning reward function with both preference models. These hyperparameters were used across all experiments. For all models, the learning rate, softmax temperature, and number of training iterations were tuned on the noiseless synthetic preference data sets such that each model achieved an accuracy of 100% on our specific delivery task and then were tuned further on stochastic syuthetic preferences on our specific delivery task.
Reward learning with the partial return preference model learning rate: 2; number of training epochs: 30,000; and optimizer: Adam (with B1 0.9 and 32 = 0.9999, and eps= 18-
Reward learning with the regret preference model learning rate: 0.5; number of training epochs: 5,000; optimizer: Adam (with 31 = =0.9, 82 = 0.999, and 18); and softmax temperature: 0.001.
Logistic regression with both preference models, for the likelihood analysis in Section 5.2 and Appendia E.
learning rate: 0.5; number of training itcrations: 3,000; optimizer: stochastic gradicnt descent; and evaluation: 10-fold cross validation.
Computer specifications and software librarics used The computer used to run experiments shown in and
Figures 15
16
17
18
19
21
22
23 had the following specification. Processor: 2x AMD EPYC 7763 (64
cores, 2.45 GHz); Memory:
284
GB.
The computer used to run all other experiments had the following specification. Processor: 1x TM 19-9980XE (18 cores, 3.00 GHz) & 1x WS X299 SAGF/10G ASUS MOBO; GPUs: 4x RTX 2080 Ti; Memory: 128 GB.
Pytorch 1.7.1 Paszke et al. 2019 was used to implement all reward learning models, and statistical analyses were performed using Scikit- learn 0.23.2 (Pedregosa et al. 2011
F.1.1 For Algorithm 1 choosing an input set of policies for learning successor feature functions
A set of policics input to Algorithm Tand used to create successor feature functions, which are turn used for generalized policy improvement (GPI) to efficiently estimate optimal value functions for f during learning. Which policies to insert is important open question for senteessor-feauee---ee methods in general, but our
32

---

Published in Transactions on Machine Learning Research (02/2024)
intuition is that the performance of GPI under processor-featurr-baee methods is improved with greater diversity of successor feature functions (via a diverse set. of policies) and by having some policies that perform decently (but not necessarily perfectly) on the reward functions for which V* and Q* outputs values are being estimated via GPI.
Recalling that an input policy can come from policy improvement with an arbitrary reward function, we offer the following ideas for how to source an input of policies.
Choose reward function parameters according to some random distribution (as we do).
Create set of reward functions that differ in a structured way, such each reward function being point in a grid formed in parameter space.
Learn policies from a separate demonstration datasct, using an imitation learning algorithm.
Bootstrap from f. Specifically, during learning augment the current set of successor feature functions (Werr and we ) by learning one new successor feature function via policy improvement on the current. A then contimue reward learning with the augmented set of successor feature functions and repeat this process as desired.
The input set of policies could come from multiple different sources, including the ideas above.
F.1.2 Instantiating Algorithm I for reward functions that may be non-linear
(earithhm[operateess under the assumption that the reward function can be represented as a linear combination of reward features. These reward features obtained through a input to the algorithm. Here we address situations when the lincarity assumption docs not hold.
function Φ, which is given
If the state and action space are discrete, one could linearly model all possible (deterministic) reward functions by creating a feature for each (s,a,s') that is for (s,a,s") and 0 otherwise. downside this approach is that the learned reward function cannot benefit from genoralization, which has two negative consequences. First, complex lasks, generalization would lypically have decreased the training set size required to learn reward function f with optimal policies that perform well the ground-truth reward function, r. Second, the reward function will not generalize to different tasks that share the same symbbolc reward function, such as always having the same reward from interacting with a particular object type.
If the reward features are unknown or the reward is known to be non-linear, another method is to create a reward features function that permits a linear approzimation of the reward function. Several methods to derive some or all of these reward features appear promising:
Reward features can be learned by minimizing several auxiliary losses in a self-supervised fashion, as by Brown et al. (020).. After optimizing for these various objectives using a single neural network, the activations of the penultimate layer of this network can be used as reward features. Such auxiliary tasks may include minimizing the mean squared error of the reconstruction loss for the current state from a lower-dimensional embedding and the original state, predicting how much time has passed between states by minimizing the mean squared error loss (i.e., learning temporal difference model), predicting the action taken between two states by minimizing the cross entropy loss (i.e., learning an inverse dynamics model), predicting the next state given the current state and action by minimizing the mcan squared error loss(i.c., learning a forward dynamics model), and predicting which of two segments is preferred given a provided ranking by minimizing the t-rex loss.
Reward features could also be learned by first learning a reward function represented as a neural network using a partial return preference model, and then using the activations of the penultimate layer of this neural net work to provide reward features.
33

---

Published in Transactions on Machine Learning Research (02/2024)
F.2 Results from synthetic preferences
F.2.1
Learning reward functions from 100 randomly generated MDPs
Here we describe how each MDP the set of 100 MDPs discussed in section 6.2 was generated. We also exlend the analysis to illustrate how often each preference model performs better than uniformly random and give further details on our statistical tests.
Design choices The 100 MDPs are all instances of the delivery domain, but they have different reward functions. The height for each MDP is sampled from the set (5,6,10), and the width is sampled from [3,6,10155) The proportion of cells that are terminal failure states is sampled from the set {0,0.1,0.3). There is always exactly one terminal success state. The proportion of "mildly bad" cells were selected from the set (0.0.1.0.5.0.8), and the proportion of "mildly good" cells were selected from {0,0.1,0.2). Mildly good cells and mildly bad cells respectively correspond to cells with coins and roadblocks in our specific delivery task, but the semantic meaning of coins and roadblocks irrelevant here. Each sampled proportion is translated (rounding down to integer when needed) and then cells are randomly chosen to fill the grid with each of the above types of states until the proportions are satisfied.
number of cells
Then, the ground-truth reward component for each of the above cell types were sampled from the following sets:
Terminal failure states: 10,1,5,10,50)
Terminal success states: {-5, 11-,--00-550)
Mildly bad cells: (-2.-5.-10}
Mildly good cells always have reward component of 1, and the component for white road surface cells is always -1. There are no cells with higher road surface penalty (analogous to the bricks in the delivery domain).
Better than random performance Figure 14 complements the results in Figure 10 showing the percentage of MDPs in which cach preference model outperforms policy that chooses actions according to a uniformly random probability distribution. We can see that at this performance threshold, lower than that in Figure 10 the regret preference model outperforms the partial return preference model in most conditions. Even when their performance in this plot- blst-based outperforming uniformly random actions is nearly identical, Figure 10 shows that the regret preference model achieves near optimal performance at a higher proportion.
Details for statistical tests We per- formed a Wilcoxon paired signed-rank test on the normalized average returns achieved by each model over the set of 100 randomly generated MDPs. All normalizod average re-
100%
% of MDPs in which
performance is above
uniformly random
Regret (noiseless)
80%
Partial return (noiseless)
turns below 1 were replaced with -1, so that all such returns were in the range [-1,1]. This
60%
Regret (stochastic)
clipping was done because any normalized av- erage return below 0 is worse than uniformly random, so the difference between a normal- ized return 1 and - 10000 is relatively unim- portant compared the difference between and Results are shown in Table
40%
Partial return (stochastic)
3
10
30 100
300 1000 3000
Preferences per training set
Figure 14: Comparison of performance over 100 randomly generated deterministic MDPs, showing the percentage of MDPs in which each model performed better than an agent taking actions by a uniformly random policy. This plot complements Figure 10] which shows the percentage of MDPs in which the models perform mear-optimally.
Additionally, we investigale whether Pregret and Per learn near-optimal policies on the same MDPs within this set of 100 randomly generated MDPs. Results for this analysis are shown below.
34

---

Published in Transactions on Machine Learning Research (02/2024)
Table 4: Rosults of the Wilcoxon paired signod-rank test on normalized average returns for cach proference model.
Preference
D =3
1
D
D 100
D
= 1000
D 3000
generator
type
Noiseless
w= w= 1003,
w=917,
739,
W=487,
w=284,
w=301,
w=289,
(Pregret Par)
vs.
p=0.115
p=0.007
p=0.012
p=0.007
p<00.001
p=0.002
p=0.001
Stochastic
w-979,
w- 1189.5,
w= w= 891,
w=710, p=0.018
w=285, p<c0.001
W6460, p=0.002
w= 199,
(Pregret 0-5441
0=0.018
0.027
p<c0,001
Por)
Table A table showing the count of the number MDPs where both, either, or neither of the models achieved near optimal performance.
Model(s)
D2
33
DD, =
10
DD
30
D
= 100
D,
300
D,
1000
D,
3000
Both models
40
72
87
Only Pregret
20
26
17
18
14
8
Ouly Pir
10
12
7
3
3
Neither
22
F.2.2
The effect of including transitions from absorbing state
In Section 3.2.2 we describe one context in which partial return is not identifiable because reward functions that differ by a constant can have different sets of optimal policies yet will have identical preference probabilities according to the partial return preference model (via Eqs. and 9 This lack of identifiability arises specifically in tasks that have the characteristic of having at least one state from which trajectories of different lengths are possible. Tasks that terminate upon completing goal reaching failure state typically have this characteristie. Below we describe imperfect method to remove this lack of identifiability. Then we show that the partial return preference model does much worse in our main experiments with synthetic preferences and human preferences when this method not appliod.
An imperfect method to prevent this source unidentifiability for partial return Put simply, the approach to prevent all constant shifts of reward from resulting in the same preference probabilities to force f(s,a,s))=0 in at least tuple of state, action, and next state, (s,a,s'), and include in the training dataset more segments with that tuple.
Technically, this solution addresses the source of this identifiabilily issue, that any
Regret
Includes early terminating segments
(noiseless)
constant shift in the output of the reward function will not change the likelihood of preferences dataset (but can change the set of optimal policies). With this solution, constant shift cannot be applied to all oulpuls, since at least one reward output is hard-coded to 0. And applying a constant shift to all other outputs would change the likelihood of the infinite, exhaustive preferences dataset that assumed in iden- tifiability theory.
100%
Includes early temnirtting segments (stoxhastix)
MDPs which performance is near-optimal
No early terminsting (noiseless)
80%
No carly torminating segments (stochisiie)
60%
Partial return
Includes early terminating segments (noiseless)
40%
Includes terminating segments (stochustic)
100
300
1000
3000
No eurly terminating (naiseless)
Preferences training
No early fomminating segments (suxchaiicc
Figure 15: Performance comparison over 100 randomly generated de- terministic MDPs. The results in this plot expand upon the experiments Figure I0 adding results for datasets that do not have any segments that terminate early.
More intuitively, setting one f(s,0,5)) =0 for one (s,0,5") tuple anchors the reward function. To explain, reward function in- ference effectively learns an ordering over (s,a,s') tuples. Let us arbitrarily assume this ordering is ascending order of preference. Then setting the
35

---

Published in Transactions on Machine Learning Research (02/2024)
reward for one such tuple to 0 forces all (s,a.s") tuples that are later in the ordering to have positive reward and all (s,a,s") tuples that are earlier in the ordering to have negative reward. (s,a,s") tuples that are equal in the ordering are assigned a reward of O.
However, assuming that reward is O in any state has at least two undesirable properties. First, it requires adding human task knowledge that beyond what the preferences dataset contains, technically changing the learning problem we are solving. Second, while it resolves some ambiguity regarding what reward function has the highest likelihood, this resolution may not actually align with the ground-truth reward function. In an attempt to reduce the impact of these undesirable properties, we only set the reward for absorbing state to O. A Absorbing state is the thcoretical state that an agent enters upon terminating the task. All actions from the absorbing state merely transition to the absorbing state.
In practice to include segments with transitions from absorbing state, we add early terminating segments, meaning that termination in an n-length segment occurs before the final transition.
The method above is not perfect fix, however. Assuming that transitions from absorbing state have 0 reward also consequently assumes that humans consider time after termination to have 0 return. We are uncertain that humans will always provide preferences that are aligned with this assumption. For instance, if termination frees the agent to pursue other tasks with positive reward, then we might be mistaken to assume that humans are giving preferences as if there is only 0 reward after termination.
As mentioned in Section 3.2.2, past authors have ackuowledged this issue with learning reward functions under the partial return preference model, apparently unaware of this solution of including transitions from absorbing states. Instead, they have used normalization (Christiano et al. 2017 Ouyang et al. 2022), tanh activations (Lee et al. 2021a), and L2 regularization (Hejna. & Sadigh 2023) that resolve their algorithms' insensitivity to constant shifts in reward. However, these papers do not discuss what assumptions regarding alignment are implicitly made by these ambiguity- choices. Curiously, artificially forcing episodic tasks to be fixed horizon is common (c.g., as done by Christiano et al. (2017) p. 14) and Gleave ct al. (2022)) but has not, to our knowledge, been justified as a way to address this identifiability issue. Our solution above involving early termination could be cast as a specific form of forcing a fixed horizon (infinite in this case). In one recent analysis of reward identifiability, [Skalsc et al. (2022) make assumptions that are identical to our solution above but do not discuss their assumptions' relationship to the partial return preference model otherwise being insensitive to constant shifts in reward when the lengths of segments in pairs are the same. We advise researchers, when forcing a fixed horizon via other methods, to explain what bias they are introducing regarding the set of optimal policics, either for generating synthetic preferences or during reward function inference.
Empirical results Figure 15 expands upon the synthetic- syuthetic-prefernnces preferences results of Figure 10 in Section 6.2 adding results for datasets that lack segments that terminate carly. For each combination of preference model and whether early terminating segments are included (i.e., for each different color in Figure 15). we used learning rate chosen from testing on set of 30 MDPs. Specifically, we chose the learning rate from the set (0.005,0.05,0.3 1,2 that had the highest mean percentage of near-optimal performance these 30 MDPs across training sets of 30, 300, and 3000 preferences. These 30 MDPs were taken from the 100 randomly generated MDPs used in Section 6.2 So, for testing with the chosen learning rates, we replaced them with 30 new MDPs generated by the same sampling procedure, determining the 100 MDPs used for these results. Also, a different random seed is used here than in Section 6.2 In these results shown Figure [5] performance by the partial return model is worse without early terminating segments, whereas the regret model does not appear to be sensitive to their inclusion.
The effect of removing such early-terminat segments from the human preferences dataset is tested in Appendix F.3.4] There we similar results: only performance with the partial return model is harmed by removing segment pairs with carly-terminating segments and the decrease in performance is severe.
F.2.3 Varying segment length
Here we consider empirically the effect of different fixed lengths of the segments in the preference data set. All other experiments in this article assume the segment length |ø| =3, and this analysis is a limited investigation into whether changing |o| affects results. In general, we suspect that increasing segment length has a trade off:
36

---

Published in Transactions on Machine Learning Research (02/2024)
performacce is near optimal
Regret
performance is optimal
Regre
% of MDPs in which
100%
trandiion
carntions
% of MDPs in which
100%
crandion
tonssions
3 trunsiions
75%
Io transtions
75%
10 trandinss
50%
transtion
50%
cranetion
sansions
tranctions
10 transtions
10 tranations
100 1000 3000
30
100
300 1000 3000
20 trastions
of preferences
of preferences
Using noiselessly generated preferences, comparison of performance over 100 randomly gener- ated deterministic MDPs, showing the percentage MDPs in which each model performed near-optimally after learning from preference datasets of different seg- ment lengths.
Figure 16:
Figure 17: Using stochastically generated preferences, comparison performance over 100 randomly gener- ated deterministic MDPs, showing the percentage of MDPs which each model performed near-optimally after learning from preference datasets of different seg- ment lengths.
% of MDPs which
performance near optinal
100%
75%
30%
Pantal roum
arssions
trnations
constion
% of MDPs which
performance is near optimal
100%
50%
75%
10 transtions
retum
tnsoins
tnnoiinss
30
1000
3000
20 carotions
100
300 1000 3000
20 transiinss
Reyael
100%
75%
Begret
inniiooss
tunstin
100%
75%
2 tmmstions
ttaasstion:
Innntion
50%
50%
30
100
300
1000 3000
20 tranatinss
imndiuua
30
100
300
1000
3000
20 trunstors
preferences
of preferences
Figure 19: The results from Figure 17 (stochastically generated preferences) divided by preference model, allowing further perspective. Segment lengths and are added here.
Figure 18: The results from Figure 16 (noiselessly generated preferences) divided by preference model, allowing further perspective.
it makes credit assignment within the segment more difficult yet also gives preference information about more unique and complex combinations of events, which could reduce the probability of the preference providing new information that is not already contained in the remainder of the training dataset. Put more simply, preferences over larger segments appear to give information that is harder to use but covers more transitions.
To conduct this analysis, the following process is followed for each of 30 MDPs that were sampled as described in Appendix F.2 For each n E (1,2,3,4,6, 4,,,,0,20000 we syuthetically create preference datasets with segment length la|=n. Each segment was generated by choosing a non-terminal start state and n actions, all uniformly randomly. As in Appendix F.2. each preference model acts as preference generator to label these segment pairs, resulting in datasets that differ only in their labels, and then each preference model is used for reward learning the same dataset it labeled.
We observe the following:
With noiselessly generated preferences (Figure 16), the performance with each preference model is similar for segments with 20 transitions, though it is sometimes slightly better with the partial return preference model. At segment length of 10, performance with the regret preference model is better or similar, depending on the number of preferences. For all other segment lengths, performance with the regret preference model is better.
37

---

Published in Transactions on Machine Learning Research (02/2024
With stochastically generated preferences (Figure 17]. the performance with each preference model is generally better with the regret preference model, although sometimes the partial return preference model has marginally better performance. Additionally, the variance of performances with the partial return preference model is higher.
Additionally, we conduct Wilcoxon paired signed-rank tests for positive effect of segment length performance. Four tests are conducted, one per combination of preference model and whether preferences are generated noiselessly or stochastically. The steps to conduct this test are below. For each combination of preference model and each preference dataset of size [D~|E 100,00000000000,0000,,,,,0000 measures between the following two orderings:
we calculate Kendall's T correlation
segment lengths in ascending order, (1,2,3,10,20), and
segment lengths in the ascending order of the corresponding percentage of MDPs in which near-optimal performance was achieved (e.g., if performance is near-optimal in 90% of MDPs for |a| 20 and is
near-optimal in 60% of MDPs for |o| = 10, then |o| = 20 would be later in the ordering than |or = 10).
For each of the resultant values one for cach | we create a pair for Wilcoxon paired signed-ranked lest: (7,0). The in the pair is the expected Kendall's value for uniformly random orderings of segment lengths. Therefore, the pair represents a comparison between the correlation between training dataset's 's segment length and its performance and no correlation. Each Wilcoxon paired signed-rank test is conducted on these 7 pairs.
With noiselessly generated preferences and the partial return preference model (Figure P= p= and the mean 7 across the 7 training dataset sizes is 0.80, indicating a significant and large correlation between segment length and ordering by performance. Our visual inspection of Figure [8 shows that the effect size is large, so we conclude that in this experiment increasing segment length meaningfully improves performance.
With stochastically generated preferences and the partial return preference model and (Figure p=0.016 and the mean 7 across the training dataset sizes is 0.51, indicating a significant and moderate correlation between segment length and ordering by performance. Our visual inspection of Figure [9] shows that the effect size is smaller but still observable, so we conclude that in this experiment increasing segment length somewhat improves performance.
With noiselessly generated preferences and the regret preference model (Figure 18), p= 0.219 and the mean across the 7 training dataset sizes is 0.25, which is not significant between segment length and ordering by performance. Likewise, our visual inspection of Figure 19 does not reveal any effect, so we conclude that in this experiment increasing segment length does not improve performance.
With stochastically generated preferences and the regret preference model (Figure 19p. p= and the mean across the 7 training dataset sizes is 0.47, indicating a significant, and moderate correlation between segment length and ordering by performance. Our visual inspection of Figure 19 does not reveal this effect, indicating that the effect size is small, so we conclude that in this experiment increasing segment length improves performance with only minor effect.
F.2.4 Reward learning in stochastic MDPs Although thcoretically consider MDPs with stochastic transitions in Section 3 we have not yet empérically compared Ps. and Pregret in tasks with stochastic transitions, which we do below.
We randomly generated 20 MDPs, each with a 5 x grid. Instead of terminal cells that are associated with succoss or failure, those MDPs have terminal cells that are cither risky or safe. A single terminal safe cell was randomly placed, and the number of terminal risk cells was sampled from the set (1,2,7) and then these terminal risk colls were likewise randomly placed. No other special cells were used in this set MDPs. To add stochastic transitions, the delivery domain was modified such that when an agent moves into terminal risk cell
38

---

Published in Transactions on Machine Learning Research (02/2024)
Table 6: Stochastic MDPs: Proportion of 10 MDPs in which performance was near optimal, with varied reward functions. Entering terminal risk cell results in Twin and Plose; each with 50% probability.
Preference generator
Twis =
Pwsin = 1000
Twin = 100
Twin 100
= -50
Pose = -00
Flose =-1
Flose 1000
Noiseless Pregget
1.0
1.0
1.0
1.0
Stochastic Pregret
1.0
1.0
1.0
1.0
Noiseless Per
1.0
0.0
1.0
0.0
Stochastic Psr
1.0
0.0
1.0
1.0
there is a 50% chance of receiving a lower reward, Plose, and a 50% chance of receiving a higher reward, Twin- All other transitions are deterministic. As in the unmodified delivery domain, moving to any non-terminal state results reward of -1. Moving the terminal safe state yields a reward of +500, like the terminal success state of the unmodified delivery domain. Therefore, depending on the values of Twin and Tloses it may be better to move into terminal risk state than to avoid it. All segments were generated by choosing a start state and three actions, all uniformly randomly. For each MDP; the preference dataset D2 contains 3000 segment, pairs.
The 10 MDPs of each condition differed from those of the other conditions by their ground-truth reward function r, with different Twin and Flose values. As in Section 6.2 regardless of whether the stochastic or noiseless version of preference model generates the preference labels, the stochastic version the same preference model is used for lcarning the reward function.
The results shown below in Table indicating that for both noiseless and stochastic preference da tasets, Pregret is always able to achieve near-optimal performance, whereas Per is not. These results expand upon and support the first proof of Theorem 3.2 in Section
F.2.5 Generating preferences and learning reward functions with different preference models
Using synthetically generated preferences, here we investigate the effects of choosing the incorrect model. Specifically, either Pregret or Par generates preference labels, and then the other preference model is used to learn reward function from these preference labels. Through this mixing of preference models, we add two new conditions to the analysis in the delivery domain in Section 6.2 The results are shown in Figure 20 We observe that, as expected, each preference model performs best when learning on preference labels it generated. Of all four combinations of preference models during generation and reward function inference, the best performing combination is doing reward inference with Pregret on preferences generated by Pregret-
Between the two mixed-model conditions, observe that learning with the partial return preference model on preferences cre- ated via regret outperforms the reverse. These results suggest that learning reward functions with the regret preference model may be more sensitive how much the preference generator deviates from it. How-
100%
Partial return preference labeling. partial retur reward learning Noiseless
performance is near optimal
Stochastic
MDPs in which
80%
Regret preference partian
return reward learning
Noiscless
Stochastic
60%
Regret preference labeling. regret
reward learning
NNiieeees
ever, note that humans do not ap- pear to be giving preferences according to partial return, so these though informative are not worrisome.
-Soochastic
40%
return preference laheling,
regret reward learning
Noiscless
10
30
100 300
1000
3000
Stochastic
preferences
Figure 20: Performance comparison over 100 randomly generated de- terministic MDPs, with synthetically generated preferences. This plot. expands upon Figure 10 by including mismatches between the preference model generating preference data and the preference model used during learning of the reward function.
F.3 Results from human preferences
In this section we expand upon the results described in Section 6.3 which involve learning reward functions from the dataset of human preferences we collected.
39

---

Published in Transactions on Machine Learning Research (02/2024)
performance near optimal
% of partitions in which
100%
Regret
% of partitions which
performance is ahove
100%
80%
uniformly random
Regret
Partiol return
80%
60%
Partial return
Expected return
60%
40%
Expected return
Logistic linear
40%
Logistic lincar
20%
20%
19
36
91
182
363
906
1812 (1)
19
36
182
363
906
1812
(100)
(50)
(20)
(10)
(5)
(2)
(100)
(50)
(20) (10)
(5)
(2)
(1)
Preferences per training (partitions)
Preferences per training
(partitions)
Performance comparison over various amounts of human preferences. Each partition has the number of preferences shown or one less. This plot is identical to Sigure[[[esst except that results for the expected return preference model are included and different random seed was used to partition the human preferences.
Figure 21:
Performance comparison over various amounts of human preferences. Each partition has the number of preferences shown one less. This plot tests the same learned reward functions as in Figure 21 but thresholds outperforming uniformly random policy rather than on near-oplimal performance.
Figure 22:
F.3.1 Wilcoxon paired signed-rank test
For the Wilcoxon paired signed-rank test described in Section 6.3 [-1,1] as in Appendix F.2.1 The result from each test is shown in Table 7 These results are surprisingly significant, given that both models reach 100% near-optimal performance for 5 and 10 partitions in Figure [1] However, this setting, learning reward function with the regret preference model tends result in a higher mean score than learning one with the partial return preference model, even when both above the threshold for near optimal performance.
normalized mean returns were clipped to
Table Results from Wilcoxon paired signed-rank tests.
5 partitions
10 partitions
20 partitions
50 partitions
100 partitions
Pregrat Vs. Per preference models
w=0
w
w=24
w=216
w-939
p=0.043
p-0.028
p=0.007
p=0.003
p=0.076
F.3.2
Expanded plots, with the expected return and logistic linear preference models included
Figure 21 show the same results as Figure 11 but with additional results from the expected return preference model introduced Appendix B.p and the logistic linear preference model (Piog-tin) introduced in Appendix B.3 Figure shows the same results with different performance threshold, that of performing better than uniformly random action selection, which receives matches outperforms both other preference models in all partitionings of the human data, at both thresholds (near optimal and better than random).
our normalized mean return metric. The regret preference model
F.3.3 Performance on only human preferences from the first stage of data collection
As previously mentioned in Section D and Appendix when learning reward functions only from the data from the first slage of human data collection, the partial return model does worse. This first stage of data contains 1437 preferences, which is 79% of the full dataset. The specific performance of the partial return preference model on the full set of first-stage data (i.e., 1 partition) is normalized mean return of 12.77, worse than a uniformly random policy. The regret preference model achieves 0.999, close to optimal performance of 1.0.
F.3.4 Performance without carlytterminatig segments
Segment pairs with aary-terminating segments second stage of data collection but not in this first stage. The identifiability issue of the partial return model related to a constant shift in the reward function function-discnne discussed in Section 3.2.2 and Appendix F22 provides
type of segment pairs (of two types) that are in the
40

---

Published in Transactions on Machine Learning Research (02/2024)
sufficient justification for the low performance in Appendix /. F.3.3 observed without inclusion of carly-terminating segments.
To more directly test the effect of early-terminating segments, we repeat the analysis in Appendix F.3.3 while only removing the segment pairs from the second stage that have early-terminating segments. We get the same normalized mean return, 12.7 for the partial return preference model and 0.999 for the regret preference model. Therefore, removal of the tereinttrmin segments is sufficient to cause the partial return preference model to perform poorly.
F.3.5
Generalization of learned reward functions to other MDPs with the same ground truth reward function
We also test how well these learned reward functions generalize to other instantiations of the domain. To do so, keep the same hidden ground-truth reward function but otherwise randomly sample MDP config-
100%
% of MDPs where performunce
80%-
urations identically as done for the analysis of Sec. 6.2
optimal
Regret
which is detailed in Appendix F.2. Procedurally, this analysis starts identically as that in Section 6.3
Partial return
60%%
learning a reward function from randomly sampled partitions of the human preference data. Each learned reward function is then testod within the 100 MDPs. For example, for two partitions of the data, each algo- rithm learns two reward functions and each such A is tested on 100 MDPs. To test a reward function in an MDP, the same procedure of value iteration used as in Sections 6.2 and 0.3 to find an approximately opti- mal policy, the performance of which is then tested on the MDP (with the ground-truth reward function).
19
36
91
182
363
906
1812
(100)
(50)
(20)
(10)
(5)
(2)
(1)
Preferences per training set (partions)
Figure 23: Performance comparison of learned reward func- tions' generalization to 100 MDPs with the same reward function, when learned from various amounts of human pref- erences, Each partition has the number of preferences shown or one less.
The results are shown in Figure 23 The two pref- erence models perform similarly at 100, 10, 2, and 1 partition(s), and otherwise the regret preference model oulperforms the partial return preference model. The most pronounced differences are at 20 and 5 partitions, where the partial return preference model fails to reach near-optimal performance approximately twice as often as the regret preference model.
For each training set size, we conduct the same Wilcoxon paired signed-rank test in Section 6.3 and Appendix F.3.1 except that for each of the 100 MDPs, calculate the normalized mean return, and the mean of normalized mean returns across all 100 MDPs represents a single sample for the statistical test. Across all 7 training set sizes, no statistical significance is found (p 0.2). Unlike most other analyses in this paper, we cannot here conclude superior performance from assuming the regret, preference model.
41


Reviews:
[{"review_id": "8Fhu383iCZ", "reviewer": "Reviewer_QGea", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n* The paper is clear and well-written, and easy to follow.\n* The paper's claims are all carefully made and backed up by theoretical or empirical results in all places.\n* The theoretical results are interesting and well-explained.\n* The proposed regret model of human preferences is interesting for the community, and points to lots of future work in this space.\n\n# Weaknesses\n\n* While this does not make the paper not worthy of acceptance, the propose model of human preferences seems much more difficult to learn a reward function under than the partial return model. The proposed algorithm relies on a linear setting and successor features, neither of which are generally the case in the standard RLHF setting of fine-tuning language models.\n* Similarly, in this setting (in particular single-turn dialogue or instruction following) the partial return and regret models are the same, so this contribution is unlikely to improve results there. I don't think the authors need to change the paper to address these two points, but I thought it was worth mentioning.\n* The empirical results are somewhat limited to a single environment which is proposed by the authors, which makes it difficult to asertain how performant using this model and algorithm would be in other more realistic settings. I think the paper is still worthy of acceptance, but if the authors performed experiments on other settings, particularly pre-exising environment from the literature, then that would enable them to make stronger claims about the empirical promise of this modelling strategy.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the problem of how to model human preferences for learning reward functions, which can then be used in Reinforcement Learning from Human Feedback. The paper proposes a new model for human preferences, the Regret model, which models the probability of one sequence being preferred to another as a logistic function over the differences in the *regret* of the two sequences. They compare this model to the standard partial return model both theoretically and empirically. On the theoretical side, they show their model is identifiable (in the limit of infinite noiseness data using the model can recover the underlying reward function), wherease the partial return model is shown by counter-example to not be identifiable in 3 scenarios: noiseless preferences, variable-horizon tasks and segments of length 1. On the empirical side, the authors introduces a toy delivery driver gridworld MDP, and gather human preference data on segments in this setting. They show that the regret model better-models the human preference data, implying it is a better descriptive model. They present an algorithm to learn this a reward function under regret model in a linear setting using successor features, and show that this algorithm can learn reward functions from synthetic and human data better than the partial return model in terms of the size of the dataset.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As mentioned in the Strengths and Weaknesses, none of these changes are critical for securing my recommendation, and are rather just changes that would strengthen the claims the authors could make in this work.\n\n* If the authors performed experiments on other settings, particularly pre-exising environment from the literature, then that would enable them to make stronger claims about the empirical promise of this modelling strategy.\n* Related to this, if the authors could present an algorithm for learning under the regret preference model that will work in more general settings (for example RLHF fine-tuning of language models), that would be beneficial.", "broader_impact_concerns": "I don't have any concerns of the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yrsYDEHBzM", "reviewer": "Reviewer_G5ch", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nI really like the paper. As far as I know the way the authors address the problem of learning from preferences hasn't been addressed either in the \"preference learning\" literature nor in the \"RL\" literature (and still not in the most recent \"RLHF\"/LLM literature).\n\nI think the authors make an interesting point between rewards cumulated over small trajectories vs regret (vs rewards comulated over entire trajectories). The claim coes in two parts (that's what humans do + that's the model we should use when learning from human rewards), which are validated independently.\n\nThe theoretical part is nice. I agree with the authors that in the current format, keeping the proofs in the body of the paper (instead of putting them to Appendix, which tends to be more standard nowadays) is useful as the actual insights are in there. \n\nThe algorithm is a nice-to-have more than a game changer, but somehow makes the paper feel more complete\n\nThe experiments are carried out thoroughly. The various steps are well-detailed and my feeling is that they support the claims sufficiently well for the sake of this paper.\n\nWeaknesses:\nTo me the main weakness is that the paper may miss its audience. Considering the huge interests and investments in RLHF for LLMs and foundation models in general, it seems to me that the value of this paper would lie in how the insights translate for these particular applications. I think that the paper, as insightful as it can be, doesn't really address the specific LLM setting in enough depth to be directly applicable.\n\nOn the other hand, I appreciate that the application of regret vs patial return idea to RLHF in LLMs is a whole different endeavour that would likely require an entirely new paper. So I'm happy with the contribution as it stands and I think that \"leaving this for future work\" is reasonable.\n\nThe second main weakness from my perspective is presentation -- but this again may only be a matter of perspective. It seems to me that the presentation is directed towards a theory-inclined audience that is familiar with RL and concepts such as regret, which IMHO makes the paper hard to read for more applied researchers working on RLHF. Also, from the write-up, it is hard to see how much the results are corner cases or actual phenomena that we are likley to occur. I understand that only experience can answer the question, but I would have liked some discussion of what the authors would expect on RLHF-types of tasks, or a list of the main unknowns that should be answered in that context.\n\nother questions/comments:\n- the notion of regret (Eq 3) is based on the optimal policy but most of the value of this definition is when the trajectory doesn't end in a terminal state. At this stage of the paper I wonder what happens if we constrain the (sub)trajectories to end up in terminal states\n- I am not sure to understand why identifiability (Def 3.1) should be limited to compare segments of the same length. Also, I think it should be said more clearly (now it's hidden in the middle of the definition). I missed it in a first read which made me struggle at the start of section 3.2.2\n- the paragraph \"regret as model for human preference\" right before section 3 discussed the assumptions underlying the regret model. This paragraph is definitely nice to have, but I would also have liked some discussion on the differences in assumptions between the reward and the regret model regarding human data collection.\n- Figure 2: it feels strange to me to compare the two segments since they do not  have the same starting state. As a human, it is unclear which one is \"better\" without doing an evaluation of each segment irrespective of the other (segment one is bad because is goes from safe to unsafe apparently because of the driver, while segment two seems good because it goes from unsafe to safe), which kills the point of comparisons in the first place. As far as I can see, RLHF as used in LLMs always focus on the same starting state, so I would encourage the authors to make sure their examples also follow this case. In particular, to me it is not clear if the human assessment should assess the starting state -- in which case I prefer to start from a safe place than an unsafe one -- it seems easier to improve from there. My understanding is that the regret cancels out the effect of the starting state within a trajectory, which seems a pretty big deal in practice if we compare trajectories with different starting states.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies how much existing ways of learning from human preferences based on reward [models] can recover optimal behavior, and under what assumptions. The idea proposed by the authors is that humans are more likely to rank (sub)trajectories based on regret than rewards cumulated in that segment, which is what current reward modelling paradigms assume. \n\nIn the theoretical part, the authors show that regret-based models of human preferences allow to recover optimal behaviors within standard workflows (preferences -> value function -> policy) but in some non-trivial cases models involving rewards cumulated on the subtrajectory cannot. The authors also propose an algorithm based on successor features to learn the optimal value function for practical cases where it is not available.\n\nIn the experiments, the authors study two claims: 1) whether humans do seem to follow more a regret-based model than a cumulated rewards-basesd model, 2) whether learning from human preferences works better with a regret-based model, and 3) how much we lose if the preferences actually follow a cumulative reward model compared to a regret-bsaed model because of non-identifiability. The experiments overall support the claims of the authors, with a little caveat for the last experiment where eventually \"partial return\" catches up with \"regret\"", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "nothing requested per se but I would encourage the authors to address my questions on the presentation", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YRNbV2J2Nw", "reviewer": "Reviewer_4vjA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: The paper makes a first step towards understanding how dense regret helps the better align the policy with human preferences in RL. The proposed method associates the regret of each segment with human preferences, which utilizes the negated sum of an optimal policy’s advantage of each transition in the segment. \n\nWeakeness: \n\n- 1. My biggest concern is about how the regret can be approximated in practice. For RL problems, approximately knowing the optimal Q function up to $\\epsilon$ error is equivalent to finding an $\\epsilon$-optimal policy, assuming that maximizing over $Q^\\star$ can be directly solved. Thus, it is unclear whether we can really utilize the regret notion for RLHF in practice.\n\n- 2. Regarding the identifiability issue, I understand that the regret over segments provide more signal than partial return, making it identifiable under weak assumptions. However, as the authors mentioned, traditional approach focuses on maximum likelihood -- meaning that the preference model follows a certain statistical model (or precisely, the Bradley-Luce-Terry model for pairwise comparisons). If such model is well-specified and with enough samples, the MLE shall be identifiable. How shall we compare this with the regret based scenario in terms of identifiability?\n\n- 3. I don't get the point of Section 3.2.3. If $|\\sigma|=1$, meaning that it is a contextual bandit problem. Then it is expected that the discount factor does not play a role. Why would we say that this is an unidentifiability issue?\n\n- 4. The algorithm focuses mostly on the linear reward case. Can the authors comment on how this can be generalized to neural networks?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the problem of modeling human preferences. A new preference model is proposed based on the regret rather than partial return. The author shows that the partial-return based model is unidentifiable, while the regret based on the negative advantage function value of the segment is identifiable. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please clarify all my comments above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper was a bit cursed, through no fault of the authors, as we struggled to find a full set of reviewers. Once we did, the reviewer consensus was clear, and I am satisfied that the authors have taken onboard all feedback in making changes to the paper during the discussion period. I encourage the authors to look once more over the feedback and incorporate any outstanding recommendations into their camera ready draft.
Decision: Accept (poster)