Models of human preference for learning reward functions
Authors: W. Bradley Knox
Stephane Hatgis-Kessell
Serena Booth
Scott Niekum
Peter Stone
Alessandro G Allievi
Venue: NeurIPS 2022 Submitted (2022)
Keywords: reinforcement learning
learning reward functions
reward design
alignment
learning from preferences
regret
advantage
successor features

Abstract: The utility of reinforcement learning is limited by the alignment of reward functions with the interests of human stakeholders. One promising method for alignment is to learn the reward function from human-generated preferences between pairs of trajectory segments. These human preferences are typically assumed to be informed solely by partial return, the sum of rewards along each segment. We find this assumption to be flawed and propose modeling preferences instead as arising from a different statistic: each segment's regret, a measure of a segment's deviation from optimal decision-making. Given infinitely many preferences generated according to regret, we prove that we can identify a reward function equivalent to the reward function that generated those preferences. We also prove that the previous partial return model lacks this identifiability property without preference noise that reveals rewards' relative proportions, and we empirically show that our proposed regret preference model outperforms it with finite training data in otherwise the same setting. Additionally, our proposed regret preference model better predicts real human preferences and also learns reward functions from these preferences that lead to policies that are better human-aligned. Overall, this work establishes that the choice of preference model is impactful, and our proposed regret preference model provides an improvement upon a core assumption of recent research.

Models of human preference for learning reward functions
Anonymous Author(s)
Affiliation Address
email
Abstract
The utility of reinforcement learning limited by the alignment of reward functions with the interests of human stakeholders. One promising method for alignment is learn the reward function from human-generated preferences between pairs of trajectory segments. These human preferences are typically assumed to be informed solely by partial return, the sum of rewards along each segment. We find this assumption to be flawed and propose modeling preferences instead as arising from a different statistic: each segment's regret, a measure of a segment's deviation from optimal decision-making. Given infinitely many preferences generated according regret, we prove that we can identify reward function equivalent the reward function that generated those preferences. We also prove that the previous partial return model lacks this identifiability property without preference noise that reveals rewards' relative proportions, and we empirically show that our proposed regret preference model outperforms it with finite training data in otherwise the same setting. Additionally, our proposed regret preference model better predicts real human preferences and also learns reward functions from these preferences that lead policies that are better aligned Overall, this work establishes that the choice of preference model is impactful, and our proposed regret preference model provides an improvement upon assumption of recent research.
13
14
15
18
Introduction
20 Improvements in reinforcement learning (RL) have led to notable recent achievements 1-6
21 increasing its applicability to real-world problems. Yet, like all optimization algorithms, even perfect
22 RL optimization is limited by the objective it optimizes. For RL, this objective is created in large part by the reward function. Poor aligmment between reward functions and the interests of human stakeholders limits the utility of RL and may even pose catastrophic risks
24
25 Influential recent research has focused on reward learning from preferences over pairs of fixed-length
26 trajectory segments. Nearly of this recent work assumes that human preferences arise probabilis-
27 tically from only the sum of rewards over segment, i.c., the segment's partial return M [6]] That is. these works assume that people tend to prefer trajectory segments that yield greater rewards during the segment. However, this preference model ignores seemingly important information about the segment's s desirability, including the state values of the segment's 's start and end states. Separately, this partial return preference model can prefer suboptimal actions with lucky outcomes, like buying a lottery ticket.
30
31
32 This paper proposes an alternative preference model based on the regret each segment, which equiv-
33 alent the negated sum of optimal policy's advantage of each transition in the segment (Section
Submitted to 36th Conference Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

63 T: Sx5xx A S. y is the discount factor and Do is the distribution of start states. Unless otherwise 34 Figure shows an intuitive example of when these two models disagree. Other classes of domains that
35 the models will differ on are those with constant reward until the end, including competitive games like chess, go, and soccer as well as tasks for which the objective is to minimize time until reaching a goal.
36
37
For these two preference models, we first focus the- normative analysis (Section i..., what preference model would we want humans
38 oretically
end state (safee
39
cnd state (crash imminent)
40 to use if we could choose proving that reward learning on infinite, exhaustive preferences with
-1 sum of reward for bumpping the wall
41
suma reward
42
our proposed regret preference model identifies a reward function with the same set of optimal poli- cies the reward function with which the prefer- ences generated. We also prove that the par- tial return preference model is not guaranteed to identify such a reward function without preference noise. We follow up with a descriptive analysis of how well each of these proposed models align with actual human preferences by collecting a human- labeled dataset of preferences a rich grid world domain ( and showing that the regret pref- erence model better predicts these human prefer- ences (Section 5 Finally, we find that the policies ultimately created through the regret preference model tend to outperform those from the partial return model learning- both when assessed with collected human preferences or when assessed with
43
xtart state (sufe)
start state (crash likely)
Two segments of car moving at high speed near brick wall. Assume the right segment opti- mal and the left segment is suboptimal (as defined in Sec. 2. The left segment has a higher sum of reward, the partial return preference model tends prefer it. The regret preference model instead tends to prefer the right segment because optimal segments have mini- mal regret. If we also assume deterministic transitions, then the regret model includes the difference in values between the the right segment would tend be preferred because it greatly improves its state values from start to end, whereas the left segment's state values greatly worsen. We suspect human readers will also tend to prefer the right segment.
Figure I:
46
47
48
49
50
51
59
and the end state (Eq. 3 and
54
55
56
57
58
59
synthetic preferences (Section d).
2
Preference models for learning reward functions
81 We assume that the task environment is a Markov decision process (MDP) specified by the tuple (S, A,
62 T. 7, Do. r). and are the sets of possible states and actions, respectively. 7 is a transition function,
64 stated, assume undiscounted tasks (i.e., 7=1) that have terminal states, after which only 0 reward can be received. r is a reward function, : X A S + R. where the reward rt at time is a function of 66 8,, Gt and 81+1- An MDP,r (r is an MDP without reward function.
67
Throughout this paper, r refers to the ground truth reward function for some MDP; f refers to a learned
approximation of r; and i refers to any reward function (including r or f). A policy (T:S A (0,1) specifies the probability of an action given a state. QF and V; refer respectively to the state-action value function and state value function for an optimal policy, π under F. The optimal advantage function is defined as (s.a) é (s,a) Vi* (s). Throughout this paper, the ground-truth reward function is used to algorithmically generate preferences when they are not
68
89
71
is hidden during 73 reward learning, and is used to evaluate the performance of optimal policies under learned f.
72
74
2.1 Reward learning from pairwise preferences
75 A reward function can be learned by minimizing the cross-entrops loss i.c., maximizing the likelihood observed human preferences, a common approach recent literature 1 14 16
76
Segments Let denote segment starting state $o,0- Its length |o| is the of within the segment. A segment includes /a|+1 states and |o| actions: (80,0;0a,0:8a, ((o,,,...995...)) In this
78
79
problem segments lack any reward information. shorthand,, we o, (s, 80 segment a is optimal with respect to F if, for every i E (1,...,/o' -1], Q7 (So,,, = V/ A
81 segment that is not optimal is suboptimal. Given some and segment a, F, A (( and the partial return of segment is E|rl >FF denoted in shorthand as E,

---

83
Preference datasets Each preference over pair of segments creates a sample (2 in preference dataset Dr. Vector μ= (/11/12) represents the preference: specifically, if a1 is preferred over 02, denoted 01 > d2. p= (1,0). f is (0,1) o1 <0² and is (0.5.0.5) for 01 ~022 (no preference).
84
85
86
Loss function To learn reward function from preference dataset, D common assumption that these preferences were generated by preference model P that arises from unobservable ground-truth reward function We approximate r by minimizing cross-entropy loss to learn A:
88
(D= - (a (o.0),,))D
poloo x02lf)
This loss is under-specified until P(ai > o2lr) is defined, which is the focus of this paper. We show that the common model of preference probabilities is flawed and introduce an improved preference model.
90
91
Preference models A preference model determines the probability of one trajectory segment being preferred over another, P oglr). Preference models could be applied to model preferences provided by humans or other systems. Preference models can also directly generate preferences, and in such cases refer to them as preference generators.
92
95
2.2
Choice of preference model: partial return and regret
96 Partial return Recent work assumes human preferences are generated by Boltzmann distribution over the two segments' partial returns HH expressed here as logistic functior
97
Pss,, (a) > 02 = E,, F., Eng
(2)
98 Regret
We introduce an alternative preference model based on the regret of each transition in segment. We first focus on segments with deterministic transitions. For transition (81,01,86411) 1) in deterministic segment, regreta(o, ()) e (5o, - VV For a full deterministic segment,
99
regretu(o regreta (egreta(o./))=v - V
(3)
102
with the right-hand expression arising from cancelling out intermediate state values. Therefore, deterministic regret measures how much the segment reduces expected return from V, (8o,o)) An optimal segment, a², always has 0 regret, and suboptimal segment, '*, will always have positive regret, a intuitively appealing property that also plays role in the identifiability proof of Theorem 3.1
103
105
106
Stochastic transitions, however, can result in regreta(a" > regreta(o "I)). losing the property above. To retain it, we note that the effect on expected return of transition stochasticity from transition (5+) is [F. + V (s+++))] -)] Q) (s),a;) and add this expression once per transition to get regret(a)), removing the subscript d that refers to determinism. The regret for single transition becomes regret(o, [7) = [V; (s. ) [Ft + V (1),+1)]]] + [lrr + (so, (s,,,, (s (((s..)-Q((s.... QU,, - - (5, Regret for a full segment
107
108
|-|-1
|m|-1
al-1
regret(al7)=
Σ regre((o.)))=
Σ
VV Q
Σ 1=0
A (sna,
(4)
t=
t=
The regret preference model is the Boltzmann distribution over negated regret:
Pregret (01 + o2l7) - logistic (
(5)
113
Lastly, we note that if two segments have deterministic transitions, end in terminal states, and have the same starting state, this regret model reduces to the partial return model: Pregret(-1F) Ps, (-IF).
114
Algorithms in this paper All algorithms in the body of this paper are defined simply "minimize Equation D: They differ only in how the preference probabilities are calculated. All reward function learning via partial return uses Equation 2
116
117
We use two algorithms for reward function learning
See Appendix B for derivation of this logistic expression from a Boltzmann distribution with temperature of 1. Unless otherwise stated, ignore the temperature because scaling reward has the same effect.
3

---

118
via regret. The theory in Section assumes exact measurement of regret, using Equation 5
Our
119
experimental results in Section 6 use Equation 6 approximate regret. Appendix
[ introducess other
120
algorithms that use Equation as well as one in Appendix B.2 that generalizes Equation D
121
Regret as a model for human preference Pregret makes at least three assumptions worth noting. First, it keeps the assumption that human preferences follow a Boltzmann distribution over some statistic, which is a common model of choice behavior in economics and psychology, where it is called the Luce-Shepard choice rule [7 1 Second, Pregret implicitly assumes humans can identify optimal and suboptimal segments when they see them, which will less true in domains where the human has less expertise. Lastly, Peegret assumes that in stochastic settings where the best outcome may only result from suboptimal decisions (e.g., buying a lottery ticket), humans instead prefer optimal decisions. on decision quality or desirability of outcomes- and can be influenced by training or the preference elicitation interface. In practice we determine that the regret model produces improvements over the partial-reture model (Section Gp. and its ussumptions represent an opportunity for follow-up research.
122
123
124
125
126
127
128
We suspect humans are capable of expressing either type of preference-
129
130
131
132
Alternative methods for learning reward functions Other methods for learning reward functions
133
include inverse reinforcement learning from demonstrations [19 20 (discussed in Appendix (B.5 and inverse reward design from trial-and-error reward design in multiple instances of a task domain 21
134
3
Theoretical comparisons
135
136
In this section, we consider how different ways of generating preferences affect reward inference, setting aside whether humans can be influenced to give preferences in accordance with a specific preference method. In economic terms, this analysis- and all of our analyses with synthetic preferences could be considered normative analysis. In artificial intelligence, this analysis might be cast as step towards defining criteria for rational preference model.
137
138
139
140
141
Definition 3.1 (An identifiable preference model). For a preference model P, assume an infinite dataset D, of n-length pairs of segments is constructed by repeatedly choosing (a) (02) and sampling label μ~ P(11 02 (r). using P as a preference generator. Further assume that in this dataset, all possible n-length segment pairs appear infinitely many times. For some MDP M, let Mf be M with the reward function F. Let 112 be the set of optimal policies in M,. Let rnvard-equivalence class R be the set of all reward functions such that if 12 J22 E R then II,, = -I; Preference model P is identiifiale if. for any choice of n and Mr, any f = ar argming,DD D Ross [F)ss [f)] the cross-entropy loss (Eqn. 7y with P as the preference model in the same reward equivalence class a. r. Lee, II; =II-
142
144
145
146
147
148
149
Theorem 3.1 (Pregret is identifiable). Let Pregret be any function such that if regret <
r ( Pregret(01 > 02l7) 0.5, and if regret(a11 rregret(ra)) Pregret(01 o2l
150
151
0.5. Pregret is identifiable.
152
This class of regret preference models includes but is not limited to the Boltzmann distribution of Eqn. and the narrower class that Theorem focuses upon.
Theorem 3.2 (Noiseless Pz, is not identifiable). Let Pz. be any function such that if f> Eo, R, Ps. (0, > (2) 1,, and if f Eo Pz, (01 > o2) not identifiable.
155
())2))=000. = 0.5. There exists an MDP in which Ps. is
156
157
Appendix C contains a proof of Theorem 3.1 and two proofs by example for Theorem 3.2 each focusing on a different weakness of Tse first proof by example reveals issues when learning reward functions with stochastic transitions with either Pz, or deterministic Pregreta- These issues directly correspond to the need for preferences over distributions over outcomes (i.c., lotteries) to construct a cardinal utility function (see Russell and Norvig 22 Ch. 16)). Note that the noiseless version of Ps. in Theorem 3.2 is achieved in the limit as reward values are scaled higher; equivalently, one could include a Boltzmann temperature parameter in Equation 2 and scale it towards O. Intuitively, 3.2 says that Ps., is not identifiable without the distribution over preferences providing information about the proportions of rewards with respect to each other. In contrast, to be identifiable, the regret preference model does not require this preference error (though it can presumably benefit from it in certain contexts).
158
159
160
161
162
164
Theorem
185
186
167

---

168
4
Creating a human-labeled preference dataset
169
To empirically investigate the consequences of each preference model when learning reward from human preferences, we created a preference dataset labeled by human subjects via Amazon Mechanical Turk. This data collection was IRB-approved. Appendix adds detail to the content below.
170
171
172
4.1
The general delivery domain
173
The delivery domain consists of a grid of cells, each of a specific road surface type. The delivery agent's state is its location. The agent's 's action space is moving one cell in one of the four cardinal directions. The episode can terminate either at the destination for +50 reward or in failure at sheep for -50 reward. The reward for nerminal transition is the sum of any reward components. Cells with white road surface have -1 1 reward component, and cells with brick surface have -2 component. Additionally, each cell may contain coin (+1) or roadblock (-1). Coins do not disappear and at best cancel out the road surface cost. Actions that would move the agent into a house or beyond the grid's s perimeter result in no motion and receive reward that includes the current cell's surface reward component but not any coin or roadblock components. In this work, the start state distribution, Du. is always uniformly random over non-terming states. This domain was designed to permit subjects to easily identify bad behavior yet also to be difficult for them determine optimal behavior from most states, which representative of many common tasks.
174
175
178
179
180
181
182
183
184
185
4.1.1
The delivery task
186
We chose one instantiation of the delivery domain for gath- ering our dataset of human preferences. This specific MDP has a 10x x 10 grid. From every state, the highest return pos- sible involves reaching the goal, rather than hitting a sheep or perpetually avoiding termination. Figure shows this task.
188
189
180
191
4.2
The user interface and survey
192
This subsection describes the three main stages of the ex- prrimnntll session. A video showing the full experimental
193
194
protocol can be seen at bit.ly/humanprefS/
Figure 2: The delivery task used to gather human preferences. The yellow is the agent and the red inverted teardrop is the destination.
195
Teaching subjects about the task Subjects first view in- structions describing the general domain. To avoid the jargon of "return" and "reward," these terms are mapped to equiv- alent values in US dollars, and the instructions describe the goal of the task as maximizing the delivery vehicle's 's financial outcome, where the reward components specific financial impacts. This information is shared amongst interspersed interactive episodes, in which the subject controls the agent in domain maps that are each designed to teach one two concepts. Our intention during this stage is to inform the later preferences of the subject by teaching
196
197
198
199
200
202
203
them about the domain's 's dynamics and its reward function, as well as to develop the subject's sense of how desirable various behaviors are. At the end of this stage, the subject controls the agent for two episodes in the specific delivery task shown in Figure 2
204
205
Preference elicitation After each subject is trained to understand the task, they indicate their preferences between 40-50 randomly-orderes pairs of segments, using the interface shown in Figure 3 The users select a preference, no preference ("same"), ") or "can' tell". In this work, we exclude responses labeled "can't 't tell", though one might alternatively try to extract information from these responses.
208
209
210
task comprehension Subjects then answered questions testing their understanding of the task. and we removed their data if they scored poorly. We also removed subject's s data they preferred colliding the vehicle into sheep over not doing so, which we interpreted as poor task understanding or inattentiveness. This filtered dataset contains 1812 preferences from 50 subjects
Users'
211
5

---

214
4.3 Selection of segment pairs for labeling
215
We collected human prefer- ences in two stages, each with different methods for selecting which segment pairs to present for label- ing. The second stage's sole purpose was to im- prove the reward-learning performance of Br.. With- out second-stage data, Ps, compared even worse to
219
220
221
222
223
225
226
regret than in the results described in Section (see
LEFT
CAN'T TELL
RIGHT
227
Figure 3: Interface shown subjects during preference elicitation.
228
Appendix ??). Both stages'
229
data are combined and used single dataset. These methods and their justification are described in
230
Appendix D.3
5
Descriptive results
Proportion prefer
100%
232
This section considers how well different prefer- ence models explain our dataset of human pref- erences.
83%
66%
233
50%
234
33%
16%
0%
235
5.1 Correlations
between preferences and segment statistics
A Ao!
237
hypothesize that the values segments' start and end states which are included Pregret but not in Ps. affect human preferences, inde- pendent of partial return. To simplify analysis, we combine the two parts of regrela(o that are additional to Σn F and introduce the follow- ing shorthand: A V/ & V/2 ($a.jar) V;" (8,,0).
Figure 4: Proportions which subjects preferred each segment in pair, plotted by the difference the seg- ments' changes state values (x-axis) and partial returns (y-axis). The diagonal line shows points of preference indifference for P Pregret- Points of indifference for Ps: lie the x-axis. The shaded gray area indicates where the two models disagree, a preference probability greater than 0.5. Each circle's area is proportional to the number of samples it describes.
238
242
243
giving different segment
244
Note that with an algebraic manipulation (see Ap- pendix E.I. regreta F) (o)
245
246
A. V+)+ (Zi, , f Mo, Therefore, the diagonal line in Figure 4 regretu(a2)r)= (a) making the P Pregret, preference model indifferent.
249
The dataset of preferences is visualized in Figure This plot shows how A, V, has influence independent of partial return by focusing only on points at a chosen y-axis value: if the colors along the corresponding horizontal line reddens the rr-xiss value increases, then A, V, appears to have independent influence. To statistically test for independent influence of A, V, on preferences, we consider subsets of data where E,, r -E,rr is constant. For Σg r- Σ,, r - and Σ,, r- Σgg r 22, the only values with
Preference model
Loss
250
P(5)=0.5 0.5 (uninformed)
0.69
Ps. (partial return) Pregret
0.62
251
0.57
Table 1: Mean cross-entropy test loss over 10-fold cross validation (n=1812) from predicting human preferences. Lower better.
255
256
257
more than samples that also include informative samples with both negative and positive values of regret(o; -r) -eegrre(orr))) the Spearman's rank correlations between A, and the preferences are significant (033, 0.0001). This result indicates that A, V, influences human preferences independent of partial return, validating hypothesis that humans form preferences based on information about segments' start states and end states, not only partial returns.
258
259

---

262
5.2
Likelihood of human preferences under different preference models
263
To examine how well each preference model predicts human preferences, we calculate the cross- entropy loss for each model (Eqn. D i..., the negative log likelihood of the preferences in our dataset. Scaling reward by a constant factor does not affect the set of optimal policies. Therefore, throughout this work we ensure that our analyses of preference models are insensitive to reward scaling. To do so for this specific analysis, we conduct 10-fold cross validation to learn reward scaling factor for each of Pregret and Pz.. Table I shows that the loss of Pregret is lower than that of Ps,, indicating
264
266
257
288
269
that it more reflective of how people actually express preferences.
6
Results from learning reward functions
270
Analysis of a preference model's predictions of human preferences is informative, but such predictions 272
are means to the ends of learning human- maian-- reward functions and policies. We now examine each
273
preference model's performance on these ends. In all cases, we learn a reward function / according
274
to Eqn. and apply value iteration [3 to find the approximately optimal Q2 function. For this Qj.
275
we then evaluate the mean return of the -entroo optimal policy- which chooses uniformly
278
randomly among all optimal actions- with respect to the ground-truth reward function r. over Dy-
To compare performance across different MDPs, the mean return of policy æ, V75 is normalized to (VT - /V²", where V is the optimal expected return and V is the expected return of the uniformly random policy (both given Do). Normalized mean return above O is better than V Optimal policies have normalized mean return of 1, and we consider above 0.9 to be near optimal.
279
280
6.1
An algorithm to learn reward functions with regret(a=lf))
282
Al Algorithm is general algorithm for learning linear reward function according Pregret- This regret-specific algorithm only changes the regret-based algorithm from Section 2.2 by replacing Equation
5 with a tractable approximation of regret, avoiding expensive repeated evaluation of Vi (-) and (-) to compute Pregree(-1 (-)) during reward learning. Specifically, successor features for set of policies are used to approximate the optimal state values and state-action values for any reward function.
285
286
287
288
Approximating P Pregret with successor features Following the notation of Barreto et al. 24 assume the ground-truth reward is linear with respect to feature vector extracted by Φ: Ax S >Bd and weight vector w, E Rd: rss,a,s)) = (ss,a,,')" w,, During learning, wi similarly expresses ñ as F(s,a,5') = ((s,a,s)) Wp.
290
281
292
Given a policy æ, the successor features for (s,a) are the expectation of discounted reward features from that state-action pair when following क:
E' -T (oliii-bsss,aa,ss ($t,,,(,,5++ | 5,a; = a). Therefore, (s,a) vir (s,a) "... Additionally, state based successor features can be calculated
293
294
from the vh above as +b (s) EoEA (als) (s,a). making V.7 (s) u (s) Twp.
295
296
Given set V of state-action successor feature functions and set V of state successor feature func- tions for various policies and given reward function via wP Q7 (s.a) marr ev a 1 (s.a) T wy}
297
and V. (s) 2mary eM,, T (s)T w 24
these two maximizations as approximations of Q7 (s,a) and (s), respectively. In practice, to enable gradient- optimization with current tools, the maximization in this expression is replaced with the softmax weighttd average, making the loss function linear. Focusing first on the approximation of V. (s), for each vb E V softmax weight is calculated for & (s): softmax,, (17 (s) wP) [( (s) "/T1////(((
300
301
eV, 2 (s)Tw;) wT)
302
303
where temperature is constant hyperparameter. The resulting approximation Vx is there- fore defined as (s) A Es ev, softmax (1b77 (s) T "w ( (s)T wP]. Similarly, to approxi- mate (s) softmax (v2 (s, wP) 2 [[ (s, a) T (wr //T//// ev va (s,a)Tee T w and (s,aa A EV 2 so coflmary (vbg (s.a) T wo (a/7 (s.a) T wz). Consequently, from Eqns.
304
305
306
7

---

Algorithm 1 Lincar reward learning with regret preference model (Pregret), using successor features
1: Input: set of reward functions and set of policies (where one set can be 2)
2: V -+- 2
3: for each rewardfunctionn TsF policy #sF in the input sets do
4: rsr then TSF t- estimate of optimal maximum -entunpppeenoyyyyyooo policy for TSF
5: estimate a and 55 (if not estimated already during step 4)
6: add waser to V
7: add eh V
8: end for
9: repeat
10: optimize we by loss of Eqn. calculating P Pregret x x020>0221) o2l via Eqn. using Va and V,
11: until stopping criteria are met
12:
return Wv
and the corresponding approximation Pregret of the regret preference model is:
307
Progrea (0) lolf)=
Σ 1
(Sond) (alena))-Q((0000000000 (2(001111002)))))
Elab (So (((onssmmaa))
(6)
The algorithm In Algorithm lines 9-12 describe the pnperviseddlearriing optimization using the approximation P Pregret, and the prior lines create V. and V, Specifically, given set of reward functions, a corresponding set of policies is created (line 4), where each policy is an estimate of the maximum entropy policy for a reward function. Standard policy improvement methods can be used to create each such policy. Alternatively, some or all of the set of policies can be given as input directly, not derived from input reward functions. For each such policy TsF, successor feature functions & and S are estimated (line 5), which by default would be performed by minor extension of standard policy evaluation algorithm detailed by Barreto et al. [4] Note that the reward function that is ultimately learned is not restricted to be in the input set of reward functions, which is used only
310
311
312
313
316
317
to create approximation of regret.
318
The details our instantiation of Algorithm T for the delivery domain can be found in Appendix along with guidance for extending reward functions that might be non-linear.
319
320
6.2
Results from synthetic preferences
321
Before considering human preferences, we first ask how each preference model performs when it is correct. In other words, we investigate empirically how well the preference model could perform humans perfectly adhered Recall that the ground-truft reward function, r, is used create these preferences but inaccessible to the reward-learning algorithms.
322
323
324
For these evaluations, either stochastic or noiseless preference model acts preference generator to create a preference dataset, and then the stochastic version of the same model used for reward learning. For the noiseless case, the deterministic preference generator com- pares segment pair's Egr values for PE, or their regrel(a (r) values for Pregret- Note that through reward scaling the preference generators approach determinism in the limit, so this noise- less analysis examines minimal minimal-entropy versions of the two preference-gen gennrccingg models. (The opposite extreme, uniformly random preferences, examined.) the stochastic case, for each preference model, each segment pair is labeled by sampling from that preference generator's output distribution (Eqs 2 for using the unscaled ground-truth reward function.
100%
328
of in which performance neca--optimal
Regret
80%
(nniseless)
327
Partial retur (noiseless)
328
60%6
Regret
329
(sochastic)
330
Partial return (stochasiic)
10
30
1000
300
1000
Preferences per training set
333
Figure 5: Performance comparison over 100 randomly generated deterministic MDPs
334
335
396
337
would remove all information from preferences and therefore

---

340
We created 100 deterministic MDPs that instantiate variants of our delivery domain (see Section To create each MDP, we sampled from sets of possible widths, heights, and reward component values, and the resultant grid cells were randomly populated with destination, objects, and road surface types (see Appendix F:2 for details), Each segment the preference datasets for each MDP was generated by choosing a start state and three actions, all uniformly randomly. For a set number of preferences, each method had the same set of segment pairs in its preference dataset. Figure Shoos the percentage of MDPs in which each preference model results in near-optimal performance. The regret preference model outperforms the partial return model at every dataset size, both with and without noise. By Wilcoxon paired üigned-rank test on normalized mean returns, 0.05 for 86% of these comparisons and < 0.01 for 57% of them, as reported in Appendix F22
341
342
345
344
348
349
350
Further analyses can be found in Appendix 22 including with stochastic transitions, with different
segment lengths, and while artificially lowering the discount factor (as is common in deep RL. and recent work on deep reward learning from preferences).
353
6.3
Results from human preferences
of of partitions in which
performance is neur
100%
80%-
We randomly assign human preferences from our gath- ered dataset to different numbers of same-sized parti- tions, resulting in different training set sizes, and test
Regret
optimal
60%
Partial
355
40%
return
356
20%
357
each preference model on each partition. Figure 6 shows the results. With smaller training sets (20-100
358
36
91
900
1812
(100)
(50)
(20)
(10)
(2) (1)
359
partitions), the regret preference model results in near- optimal performance more often. With larger training sets (1 (1-10 partitions), both preference models always reach near-optimal return, but the mean return from the regret preference model is higher for all of these partitions except for 3 partitions in the 10-partition
Preferences per training set
(partitions)
Figure 6: Performance comparison over various amounts of human preferences. Each partition has the number of preferences shown one less.
362
363
384
365
test. Applying a Wilcoxon paired signed-rank test on normalized mean return each group with or
366
more partitions, p 0.05 for all numbers of partitions except 100 and p < 0.01 for 20 and 50 partitions.
367
7
Conclusion
Over numerous evaluations with human preferences, our proposed regret preference model (Pr (Pregret) shows improvements summarized below over the previous partial return preference model (Ps,). W hhen each preference model generates the preferences for its own infinite and exhaustive training set, we prove that Progret identifies the set optimal policies, whereas Pr,, is not guaranteed to do so without preference noise that reveals the proportions of rewards with respect each other. With finite training data of synthetic preferences, Pregret also empirically results in learned policies that tend to outperform those resulting from Pz.. This superior performance of Pregret is also seen with human preferences. In summary, our analyses suggest that regret preference models are more effective both descriptively with respect to human preferences and also normatively, as the model we want humans to follow we had the choice.
389
370
371
372
373
375
378
377
378
Independent of Pregret, this paper also reveals that segments' changes in state values provide informa- tion about human preferences that is not fully provided by partial return. More generally, we show that the choice of preference model impacts the performance of learned reward functions.
381
This study motivates several new directions for research. Future work could address any of the limitations detailed in Appendix A. Specifically, future work could further test the general superiority of P Pregret or apply it deep learning settings. Additionally, prescriptive methods could be developed via the user interface elsewhere nudge humans conform more to Pregret or other normatively appealing preference models. Lastly, subsequent efforts could seek preference models that even more effective with preferences from actual humans, now that this work has provided conclusive evidence that the choice of preference model is impactful.
382
383
384
385
387

---

References
388
[1]
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre. George Van Den Driessche, Julian Schrittwieser. Joannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go 2016.
390
391
with deep neural networks and tree search. Nature, 529(7587):84
392
Andrew W Senior, Richard Evans, John Jumper, James Kirkpatrick, Laurent Sifre, Tim Green, Chongli Qin. Augustin Zidek, Alexander WR Nelson, Alex Bridgland, et al. Improved protein structure prediction using potentials from deep learning. Nature, 577(7792):700-771 2020.
393
394
395
[3] Oriol Vinyals, Igor Babuschkin, Wajeiech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(77882:350--54, 2019.
396
387
396 [4] Mare G Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong, Marlos C Machado, Subhodeep Moitra, Sameera S Ponda, and Ziyu Wang. Autonomous navigation of stratospheric balloons using rein- forcement learning. Nature, 588(7886):777-8855
399
400
2020.
401
[5] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv: h9tp::9922.00 066800 2019.
402
403
404
[6] Jonas Degrave, Federico Felici. Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abhas Ahdolmaleki, Diego de Las Casas, al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897):414-419 2022.
405
407
[7] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safcty. arXiv preprint
408
2016.
409
[8]
W Bradley Knox, Alessandro Allievi, Holger Banzhaf, Felix Schmitt, and Peter Stone. Reward (mis)design for autonomous driving. arXiv preprint arXiv:2104. 13906, 2021.
410
Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforce- ment learning from human preferences. In Advances in Neural Information Processing Systems (NIPS).
412
413
pages 4299-4307, 2017.
[10] Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward leaming from human preferences and demonstrations atari. arXiv preprint 181 1.06521, 2018.
415
416
[11] Xiaofei Wang, Kimin Lee, Kourosh Hakhamaneshi, Pieter Abbeel, and Michael Laskin. Skill preferences: Learning extract and execute rohotic skills from human feedback. In Conference on Robot Learning. pages 1259-1268. PMI.R. R, 2022.
418
[12] Erdem BIyrk, Dylan P Losey, Malayandi Palan, Nicholas CI Landollf, Gleb Shevehuk, and Dorsa Sadigh. Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences. The International Journal of Robotics Research, page 0278364921 1041652, 2021.
420
421
[13] Dorsa Sadigh, Anca D Dragan, Shankar Sastry, and Sanjit A Seshia Active preference- prefeeencc-based learning of reward functions. Robotics: Science and Systems, 2017.
423
424
[14]
interaactvereiinooceenntt reinforcement learning via relabeling experience and unsupervised pre pre-training. arXiv preprint arXiv:2106.00001,, 2021.
Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble:
426
[15]
Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. B-pref:: Benchmarking preference-based
427
reinforcement learning. arXiv preprint arXiv:2111.00 1ete221111000.06 2021.
428
[16] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint ar00v:2203 0225500115000500 2022.
429
430
431
[17] R Duncan Luce. Individual choice behavior: A theoretical analysis. John Wiley, 1959.
[18] Roger N Shepard. Stimulus and response generalization: A stochastic model relating generalization to distance in psychological space. Psychometrika, 22(4):325-345, 1957.
433
434
[19] A.Y. Ng and S. Russell. Algorithms for inverse reinforcement learning. In Seventeenth International
Conference on Machine Learning (ICML), 2000.
436
[20] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforeement learning. In thenty-thrrd AAAI Conference on Artificial Intelligence, volume 8, pages 1433-1438, 2008.
437
439
[21] Dylan Hadfield-Menell Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca Dragan. Inverse reward design. In Advances in Neural Information Processing Syxtems (NIPS), pages 67744 2017.
440
441
[22] Stuart Russell and Peter Norvig. Artificial intelligence: modern approach. 2020.
[23]
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
10

---

443
[24]
André Barreto. Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado Van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. arXiv preprint 16K6:06606.05312,, 2016.
445
[25]
Riad Akrour, Mare Schoenauer, and Michele Scbag. Preference-basse policy learning. Joint European Conference on Machine Learning and Knowledge Discovery in Dutabases, pages 12-27. Springer, 2011.
446
447
[26] Daniel Brown, Russell Coleman, Ravi Srinivasan, and Scott Niekum. Safe imitation learning via fast bayesian reward inference from preferences. In International Conference on Machine Learning, pages 1165-1177.
448
2020.
450
[27]
Picter Abbecl and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, page 1, 2004.
451
452
[28]
Kuno Kim, Shivam Garg, Kirankumar Shiragur, and Stefano Ermon. Reward identification in inverse reinforcement learning. In International Conference on Machine Learning, pages 1496-550s PMLR, 2021.
453
454
[29] Saurabh Arora and Prashant Doshi, survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligenee299:1 297:103500, 2021.
455
456
[30] John Von Neumann and Oskar Morgenstem. Theory of games and econamic behavior. Princcton university
457
press, 1944.
[31] Yuchen Cui, Qiping Zhang, Alessandro Allievi, Peter Stone, Scott Niekum, and W Bradley Knox. The empathic framework for task learning from implicit human feedback. arXiv preprint arXiv:2009. 2020.
459
460
[32] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché- Buc, E. Fox, and R. Camett, editors, Advances in Neural Information Processing Systems 32, pages 8024 8035. Curran Associates, Inc., 2019.
462
463
484
486
467
[33] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer. R. Weiss, V. Dubourg. J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python, Journal of Machine Learning Research, 12:2825-2830, 2011.
469
Checklist
470
1. For all authors...
472
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
473
contributions and scope? [Yes]
474
(b) Did you describe the limitations of your work? [Yes] See Appendix A.
475
(c) Did you discuss any potential negative socictal impacts of your work? [Yes] See
476
Appendix A.2
477
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
478
2. If you are including theoretical results...
480
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Sections 3 Jand[ include all assumptions.
481
482
(b) Did you include complete proofs of all theoretical results? [Yes] See Section S and
483
Appendix C
484
3. If you ran experiments...
485
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [No] However, the learning code, the code for running experiments, the code and UI elements for gathering human preferences on Mechanical Turk, and the anonymized human preferences data will be opened. We are particularly excited to provide the first open dataset of human preferences over pairs of trajectory segments.
487
488
489
490
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Appendix
11

---

493
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] Error hars do not seem applicable to our plots, which do not show the exact data that we do statistical testing on. However, statistical significance testing was reported, Sections and (with pointer to the appendix for details).
484
495
496
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix F1
498
4. If you are using existing assets (e.g., code, data, models) curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Appendix D does so for visual assets used to visualize the delivery task.
502
(b) Did you mention the license the assets? [Yes] Appendix D mentions the license for
503
visual assets used visualize the delivery task.
504
(c) Did you include any new assets either in the supplemental material or as a URL? [No]
505
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [Yes]] See Appendix D
506
507
(c) Did you discuss whether the data you are using/curating contains personally identifiable
508
information or offensive content? [Yes] See Appendix D
509
5.
If you used crowdsourcing or conducted research with human subjects...
510
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [Yes] Section 1.1. includes a link to a video of a full experimental session (with an author acting as the subject).
511
512
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals. if applicabl?? [Yes] We discuss participant risks from our crowdsourced
515
study and provide link to the IRB approval in Appendix D
516
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent participant compensation? [Yes] See Appendix D
12


Reviews:
[{"review_id": "XFxlZNrtnhn", "reviewer": "Reviewer_ieTk", "summary": "The paper deals with the problem of modeling human preferences in a setting where they are presented with two segments and choose to compare the two. Importantly, the authors propose that people compare the segments in terms of their values (measured by a notion of regret) instead of the total reward achieved by each segment. A regret notion---the sum of the negative advantage function value of the segment---is used to summarize the value of a segment. The advantage function is defined under an optimal policy pi^* (optimal wrt the true reward r) and the reward of interest \\tilde{r}.", "questions": "Could the authors elaborate more on the setting when there are multiple optimal policies, how one should define regret in such cases, and how the results in Section 3 look like when one chooses different optimal policies to define ther regret?\n\n\n-----\n\nL63: typo in the mathematical expression", "limitations": "Yes, the authors have discussed the limitations and societal impact of their work in Appendix A. (When writing the above reviews, I have not read Appendix A of their paper yet.)", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: The paper provides a key insight that preferences are not just a function of the (instantaneous) reward of the segment of interest. Given that humans may care about the goal of a task, their preference towards different segments will necessarily be influenced by their long-term value, i.e., how these segments perform in terms of achieving the goal. The identification results presented in Section 3 nicely summarized the proposed insight. \n\n\nWeakness: The biggest weakness to me is how one can utilize this insight in practice. In particular, calculating the regret requires one to know the optimal policy under the true reward r (or at least the value and q-value function of that optimal policy under any reward \\tilde{r}). In general, even if one provides an optimal policy \\pi^*, the planning problem (and possibly learning problem when the transition matrix is unknown) will make the estimation of the regret very hard. As a related note, in the general framework, the regret definition under a particular optimal policy and L150 (\"And regret(\\sigma|r) > 0 if and only if \\sigma is suboptimal\") in the proof seem to only hold if one assumes there is one single optimal policy \\pi^*. What if there is a set of optimal policies under the true reward r? How should one adjust the regret notion for that?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DtNlCuPRJy", "reviewer": "Reviewer_SeBN", "summary": "The paper aims to learn the reward function from preferences between pairs of trajectory segments by introducing a new notion of the segment's regret that is based on the advantage function. Notable, they argue that the existing approach using the partial returns in the literature is flawed due to the lack of an identifiability property: the ability to recover the reward function underlying the preference dataset. As a result, the importance of using the segment's regret in learning the reward function from preference datasets is highlighted.\n", "questions": "Please address the weakenesses mentioned above.", "limitations": "The scalability of the proposed method to a more realistic environment is not demonstrated.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main contribution of the paper is to introduce a new preference model based on the advantage function. The derivation of the segment's regret as the advantage function is sound and reasonable.\n\nHowever, there are several weaknesses in the paper.\n\n1, if I am not mistaken, the proposed regret model is limited to the case of deterministic policies. How is it extended to stochastic policies?\n\n2, compared with existing approaches that use partial returns, computing the advantage function in the segment's regret is computationally expensive (as mentioned in Appendix A1). While the paper shows an approximation to reduce the computation in Section 6.1, it only works for the linear reward function. Hence, a more detailed discussion on the time complexity of the proposed regret vs. that of the partial returns is necessary, and the work needs to illustrate whether the proposed method is scalable to more practical problems, e.g., by including experiments with neural networks and larger state/action spaces.\n\n3, if I am not missing anything, Theorem 3.1 and 3.2 are not comparable. While both partial returns and the segment's regret use the logistic function (Equation 2 and Equation 5), there is a difference in the two theorems: in Theorem 3.1, if $regret(\\sigma1|\\tilde{r}) < regret(\\sigma_2|\\tilde{r})$, $P_{regret}(\\sigma_1 > \\sigma_2|\\tilde{r}) > 0.5$, in Theorem 3.2, if $\\Sigma_{\\sigma_1} \\tilde{r} > \\Sigma_{\\sigma_2} \\tilde{r}$, $P_{\\Sigma_r}(\\sigma_1 > \\sigma_2|\\tilde{r}) = 1$.\nWhich existing works assume/imply the latter assumption?\nFurthermore, it raises the question of whether $P_{\\Sigma_r}$ is indeed nonidentifiable if Theorem 3.2 has the same premise as Theorem 3.1.\n\n4, there is no comparison with IRL methods (which often rely on the value function or the Q function instead of partial returns): intuitively how they are different, and empirically how they are different (e.g., by experimental results).\n\n5, existing works (e.g., [9,10]), the reward function is define as a function of the state and the action or the state only (i.e., r(s,a), r(s)). Do these reward formulations affect the result in the paper?\n\n6, there is only a toy grid world experiment (a very small state and action space, and linear reward functions) which is quite limited.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "BpAFeMpOkZI", "reviewer": "Reviewer_xps1", "summary": "This paper studies the problem of learning from human preferences. A new preference model is proposed which compares the advantage on the optimal policy between two trajectory segments. Some theoretical results are given that show the method is consistent under some assumptions and that a popular alternative is potentially not consistent. An algorithm is proposed to make use of this model. A new dataset is collected for a simulated delivery problem where workers provided preference labels for segments given to them. The proposed algorithm and model were compared with a partial return model on both a synthetic dataset and the collected dataset.", "questions": "- Can the authors clarify what assumption is being made over the covariates in the dataset $D_{\\geq}$ in Definition 3.1?\n- In Section 4, it mentions that data was collected via two different methodologies. In the end, was data from only the second used to present the results in the end or was it a mix?\n- Can the authors provide more details about how the actual segments that were presented to the labelers were generated? In the appendix it just says that certain trajectories were favored, but it’s unclear to me what that means. Were these generated by demonstrations or an algorithm? Were they generated specifically to have good coverage over the state space?\n", "limitations": "See above discussion.", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The proposed preference model seems new to my knowledge and makes sense algorithmically.\n- The theory could be interesting although I have some reservations about correctness and the significance of one of the results.\n- The algorithm seems to perform significantly better than the partial return baseline in empirical evaluations.\n\nWeaknesses\n- The presentation could be improved. Generally speaking, there are many seemingly out of place paragraphs and sections that do not seem to advance the content of the paper. For example, Section 3 is about theoretical comparisons of the learning objectives but then Section 4 abruptly is about an experimental model, but we do not even know what the algorithm looks like or how we will use the models from Section 3 at this point. This is only explained in Section 6.\n- The description of the algorithm is very confusing and way too informal to the point where I am still unclear on what it is actually being done.  \n    - What are the input rewards and policies? Why can they be empty sometimes?\n    - Why would line 4 already estimate those? What procedure is being done to estimate these?\n    - How many feature functions are necessary?\n    - For the partial return algorithm, was the same framework used but the model just swapped out? How does this compare with past algorithms that use partial return? This seems important for the experimental comparison.\n- The discussion of related work is quite sparse and it is difficult to place this work in the literature as a result.\n- Theorem 3.1 (and Definition 3.1) is dubious. Implicitly there is an assumption that the dataset covers every $\\sigma_1$ and $\\sigma_2$ pair and infinite data for each one. Not just that it contains infinite data for some distribution over certain segments, which is what Definition 3.1 is ambiguous about. This only becomes apparent in the proof where this unstated fact is used. \n- I am also not convinced that Theorem 3.2 is all that significant. The setting is different from the preliminaries of the paper: to assume noiseless labels is to say that $P(\\cdot | r)$ is actually not in the class of models from Section 2.2 since the softmax cannot realize this model for finite rewards. The problem is therefore misspecified, and it’s not surprising that there is an identifiability issue, considering that the KL divergence cannot even be driven to zero if the true reward function is plugged in! Note that Theorem 3.1 crucially made use of the fact that the model is realizable to prove the positive result, so I feel that this is an unfair comparison.\n- The computational burden of solving potentially many MDPs to optimize the proposed preference model seems difficult to overcome.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2uZBzxzAyHj", "reviewer": "Reviewer_dwHc", "summary": "The paper studies the issues of reward functions. It considers alignment on the pairs of trajectory segments from human-generated preferences. The paper finds that it is flawed that human preferences are assumed to be informed by partial return. Previous works consider the sum of rewards over a segment. The paper provides an alternative preference model based on the regret of each segment.\n", "questions": "What is the segment’s desirability? \nFigure 1 is a little confusing.  In Figure 1, the right segment should have a higher sum of reward according to humans’ preference.\nHow to define a start state of a segment? And how to define an end state of a segment?\n\nRegret is computed based on a segment. The segment is also partial. \nWhat is the advantage of using regret? \nThe motivation of using desirability is not clear.\n", "limitations": "It would be better to provide more results and more details about experiments.\nThe paper needs to compare with other baselines, such as methods using (2).", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper addresses an interesting problem about human preferences. It shows that the partial return preference model can prefer suboptimal actions with lucky outcomes.\nThe paper also provides a method based on the regret of each segment, which is equivalent to the negated sum of an optimal policy’s advantage of each transition in the segment.\nThe paper also provides theoretical comparisons.\n\nThe main weakness is that the experimental results seem not to be enough. The proposed method does not provide comparison with previous methods from [9-16].\nThe environments are also limited. \nIt would be better to provide more results.\nAnother weakness is that the code is not opened. It would be difficult to reproduce the results by others.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Reject