Learning Reward Functions from Scale Feedback
Authors: Nils Wilde
Erdem Biyik
Dorsa Sadigh
Stephen L. Smith
Venue: CoRL2021 Poster (2021)
Keywords: HRI
reward learning
learning from choice
active learning

Abstract: Today's robots are increasingly interacting with people and need to efficiently learn inexperienced user's preferences. A common framework is to iteratively query the user about which of two presented robot trajectories they prefer. While this minimizes the users effort, a strict choice does not yield any information on how much one trajectory is preferred. We propose scale feedback, where the user utilizes a slider to give more nuanced information. We introduce a probabilistic model on how users would provide feedback and derive a learning framework for the robot. We demonstrate the performance benefit of slider feedback in simulations, and validate our approach in two user studies suggesting that scale feedback enables more effective learning in practice.

Learning Reward Functions from Scale Feedback
Nils i Erdem Biyak Dorsa Sadigh
Stephen L. Smith
Electrical and Computer Engineering, University of Waterloo
Electrical Engineering, Stanford University Computer Science, Stanford University
(nwilde, stephen smith @uwaterloo ca, (ebiyik, (doosa]sstaaa fod. edu Authors contributed equally.
Abstract: Today's robots are increasingly interacting with people and need to efficiently learn inexperienced user's s preferences. A common framework is iteratively query the user about which of two presented robot trajectories they prefer. While this minimizes the users effort, a strict choice does not yield any information on how much one trajectory is preferred. We propose scale feedback, where the user utilizes a slider to give more nuanced information. We introduce probabilistic model on how users would provide feedback and derive learning framework for the robot. We demonstrate the performance benefit of slider feed- back in simulations, and validate our approach in two user studies suggesting that scale feedback enables more effective learning in practice.
Keywords: HRI, reward learning, learning from choice, active learning
Introduction
While autonomous robots are able accomplish increasing variety of tasks, key challenge that still remains is how they should pursue and trade off between their goals. In recent years, there has been significant work on interactively learning user preferences of robot behaviors [1-15]. Usually, the user is provided with one or more robot trajectories. and is asked to provide feedback through pairwise comparisons [2-8], rankings of the trajectories [16, 17], or physical feedback [18, 12, 14]. The underlying reward function governing human preferences can then be learned through this implicit feedback. Specifically, one framework with minimal complexity for the user is learning from choice feedback [2-8], where the robot demonstrates two alternative trajectories for some task. The user then simply chooses their preferred behavior allowing the robot to infer an underlying reward function for the user preferences.
Choice feedback, although simple collect, is limiting in number of ways. Consider the example shown in Fig. 1, where a robot is tasked to serve a drink to customer. The customer might have different preferences over the type of drink have (milk, orange juice, or water), or the specifies of the trajectory the robot takes (e.g., if it goes over the stove or around it which can affect the temper- ature of the drink or the likelihood of the robot accidentally hitting the pan handle). strict choice feedback between two trajectories does not really capture these intricacies of human preferences. We thus need have more expressive way of collecting data from humans. Our key insight is that allowing users to provide a scaled approach expressive medium for learning from humans and capture nuances in their preferences.
slider (as shown in Fig. 1) can provide more
In this work, we propose scale feedback as new mode of interaction: Instead of a strict question on which of the two proposed trajectories the user prefers, we allow for more nuanced feedback using a slider bar. We design a Gaussian model for how users provide scale feedback, and learn a reward function capturing human preferences. Similar to prior work in robotics, we assume this reward is linear function of set of features [19, 11, 13, 71, where the main task of learning from scale feedback is recover the weights of this reward function. To learn in a efffccief manner, we actively generate our queries to the user, i.e., pairs of trajectories demonstrated to a user similar to Fig. 1, by optimizing two well-known objectives of information gain [4] and max regret [5].
We demonstrate the performance benefit of scale feedback over choice in a driving simulation. Further, investigate its practicality in
studies with the real robot experiment shown in Fig. 1. Our results suggest scale feedback leads to significant improvements in learning performance.
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Strongly
Fqual
Strongly Prefer Orange
Prefer Blue
Preference
Figure I: Scale feedback allows users to provide finely detailed comparisons between different options.
2 Related Work
Learning from human feedback is an important problem in developing interactive robots that work alongside humans. Researchers study learning from demonstrations [19-21], corrections [22, 23, 14], ordinal feedback [24, 8], rankings [16, 17, 25, 26), critiques [27, 28], and choices [2-5, 29].
While demonstrations usually very informative, they not always viable: Demonstrating the desired behavior might require a high level of expertise [30], or can be difficult in high-order systems [31-33]. Choice questions minimize interface complexity and mental effort for the user. However, when the user is indifferent towards both options, learning becomes difficult since users may become noisier their responses. Thus, [7, 6] investigate modifications of learning from choice where users can also answer About Equal. These two forms of choice feedback are usually referred to as strict and soft choice. When the user chooses the neutral answer, the robot learns to assign about equal reward to the presented trajectories. While choice feedback provides an easy medium for learning from humans, it provides at most one bit of information. Thus, to effectively learn from choice feedback, we often need to actively generate the queries made to humans. Previous work has investigated different auxiliary measures that are greedily optimized enable efficient learning, including expected volume removal [2], information gain [4], and regret [5].
In the proposed scale feedback framework, take the soft choice approach step further: In- stead of three discrete values for feedback (prefer A, prefer B, neutral) users give quasi-continuous feedback. This allows the user indicate by how much they prefer one option over the other.
Slider bars have been used robotics for tuning parameters [34]. More related to our work, Cabi et al. [35] proposed using them for reward sketching. Instead of assigning numerical preference between presented options, users continuously indicate the robot's progress towards some goal. However, this requires users to assign scores different parts of trajectories. Developing the scale feedback for preference-based learning, we retain the ease comparing trajectories.
Problem Formulation
We now introduce the notation
in this paper and formulate the learning problem.
Reward function. We consider the scenario where robot needs to customize its behavior to the vector of
preferences of user Alice. We assume Alice evaluates robot paths P E P based
features &² [ch (P).... ... (P)). Similar prior works in robotics [19, 11. 13, 7]. define
linear reward function that assigns a numerical value to path P by weighting a set of features:
r(P, w) &² w.
(1)
These features are usually provided by a domain expert incorporating the core factors that the reward needs to capture, e.g., collision with other objects, or distance to the goal. Further, the robot has access to a motion planner that finds an optimal path given a set of weights, i.e., the planner is (deterministic) function R + P where p(w) arg maxper r(P, w).
Regret. Similar [5], we define the regret between any two weights (w, w') as the difference in the reward w' assigns to the paths p(w) and p(w'):
(2) which quantifies the suboptimality when the true weights are w', but the path is optimized using w. Learning. Let un denote Alice's weights for the reward function. These weights are not known to the robot; the only information initially available is prior distribution PPw w²). The robot
R(w, w') ((p(ww)) w (((((w))) w',

---

leams w" by iteratively presenting her with two paths P and Q for K iterations. We extend the learning from choice framework, where users simply indicate the path they prefer, to a setting where they instead provide a more finely detailed scale feedback.
Definition (Scale feedback). Presented with two paths P and Q, Alice returns numerical feedback s E [-- If 3b = O, this means Alice has no preference between the paths, x = equals a strong preference for path P and & - 1 strong preference for path Q. From an interface design and expressiveness perspective, it is undesirable to have users give a numer- ical value for tb. Instead, they can express such feedback with slider bar with a more fine- grained set of options. An example is illustrated in Fig. 1. We let Dx- = {(Pi, Q1. (.).....PPrr Qx, 9x)) be the set of recorded user feedhack.
Performance Measures. Let i be the robot's estimate of w², and {(ts, w²) be a performance measure for the learning process. Previous works focused on the alignment of weights [2, 4], Alignment = w.ww" /||"||-||""| ll, measuring the cosine similarity of vectors w and w", i.e. how well the parameters of Alice's reward function are learned. Alternatively, Wilde et al. [5] proposed the relative error in cost. We adapt this as the .ewaadd = ((p(")))- //(pp(")) ")" measuring how much Alice likes the trajectory optimized for i compared to the one optimized for w".
Problem Statement. Let æ be an adaptive policy for designing queries Q), and let (x(TT unn) be the expected set of user feedback when a user w² is queried by æ for K iterations. Given a robot motion planner p. user with preferences w², and a budget of K rounds to query the user about their scale feedback on two presented paths, our goal is to find an adaptive policy ı that solves
max & (E [w Dx (T] w }] w")
4
Approach
We now briefly review learning from choice, and then extend the framework to scale feedback.
4.1
Choice Feedback
When presented with two paths P and Q, a user returns an ordering P ž Q (P is preferred) or P 3 Q (Q is preferred). In a noiseless setting, we have
r(P, w²) r r(Q w²) 20 = PEQ.
(4)
That is, the path P has a reward that is at least as high as that of Q with re- spect to the hidden true user weights w". Using r(P, w) = &" w, we notation and write (Ø - "-"" instead of r(P, w")- r(Q, w*). Equation (4) already con- tains observation model: If the user chooses path P, the robot can in- fer that P has higher reward with respect to w" This inequality de- fines halfspace A(P,Q) = tw (Φ" -6)) w" 2 0} containing all weights that are feasible given the observed user choice. Qk) to obtain the feasible set FR
W2
Feasible
Feasible
can tighten
Feasible Set
Set
Set
(4) Choice feedback.
(b) Scale feedback (c) Scale feedback v (0.1).
Figure 2: Different feasible sets learned from choice and scale feedback. Shown is the updated weightspace (green) after observ- ing user feedback for one (P,, Q) pair. If sb scale feedback learns a tighter halfspace; when vb E (0, 1) scale feedback learns an equality, ie.. a hyperplane.
Over k iterations, we can intersect the sets ((PI,Qi).....aa shown in Fig. 2(a). By definition, this feasible set is convex.
4.2 Scale Feedback
Scale feedback allows the robot to gain more information: the robot can also infer by how much the user prefers P, allowing for learning tighter feasible sets. We extend the model (4) and show how noiseless user would provide scale feedback and then study how a robot can learn from it.
Definition 2 (Maximum Reward Gap). Given user w". the maximum reward gap is
(5) We notice that the maximum reward gap cannot be computed, since w" is unknown the robot. Nevertheless, we can formulate the user choice model and then derive an observation model.
8 max r(P,, w²) r(Q, w = max (Φ" - -) w. P.QEP
PQQEP
User model. The maximum reward gap helps to define when noiseless user would indicate strong preference. We assume this occurs if and only if the difference in reward of and Q with respect to w" is at least a 8* for some 0 < a < 1. Here a" is a saturation parameter which governs
3

---

V
1.0
1.0--+
1
Sntmralion Sor
0.75
will
-18
(U5
11 8-4-
11
-1
-0.7
0.7 ((P.w")-r(Q.u) w*
0.25
0.1
0.8
6'
Absolutt s ider position (9
(a) User model for providing scale feedback (b) Example slider feedback for different a. The boxplots in- with a² = (blue) and a² = 0.7 (green). dicate the four quartiles of the absolute slider values.
Figure 3: Noiscless user model.
at what reward difference (w.r.t. to the maximum gap) the user's feedback gets saturated to a strong preference. For any other (P, Q) where ( ==1 scale v between -1 and 1, which leads to the following model.
[0, a²)²)) we assume the user to linearly
Definition 3 (Noiseless User Model), Presented with two paths P and Q, noiseless user with parameter a" E (0, 1] will always provide the following feedback:
1 if if((,w"") - rQQ, w0) 2 a"d -1 if w²) r(, w0) 2 a²d",
(6)
(r(Rtw")r(Q.ww ")/a""""" otherwise.
We illustrate the noiseless user model in Fig. 3a under different saturation parameters a". Fig. 3b, show simulated example: for fixed w" simulate how users with different values for a² would provide scale feedback to the same 20 queries. For larger a", they position the slider closer to the neutral position. Finally, we derive observation model for the noiseless user:
= r(P, r(Q,, w0) I "ha""" d" s E (-1,1) = r(P, w") r(Q, w²) = vra" 0", +b = r(P, w²) r(Q, w²) Z vaa 8".
Figures 2(b) and 2(c) illustrate the resulting feasible sets from (7). Moreover, notice the user- specific and unknown parameters a and 8 always appear as a product. Thus, we can introduce an auxiliary parameter = a to write (7) as [- o² - Φ0 [3, w*] $ 0. As the model remains linear, the notion of halfspaces and the feasible set F can be extended the augmented vector space.
4.3 Probabilistic User Feedback
In practice, users are often noisy; they might consider additional or slightly different features than the robot, not follow the linear reward function, or simply be uncertain in some answers. Since we cannot expect users always provide slider feedback following (6), we introduce probabilistic model where we add uncertainty to the placement of the slider.
Another practical limitation is the fact that we cannot collect truly continuous feedback from the users. Instead, the slider bar has step size c E (o, 1] such that the user provides feedback of the form ne for E Z and - < n I e- Note that + 0 retains the continuous scale feedback, whereas = gives the soft choice model where the feedback always 1-),
Definition 4 (Probabilistic User Model), Given a user w" and a query (P,Q), let & be the user feedback defined in the noiseless user model in (6). A probabilistic user using slider bar with step size of then provides feedback
# = round(( v + v,E)
(8)
where is zero-mean Gaussian noise, i.e., v N/(0, o²) with standard deviation a, and round(x, e) outputs ne closest to x such that n E Z 2n
Probabilistic Observation Model. Given the probabilistic user model, we now show how robot can infer about wn from scale feedback. In the noiseless case, user feedback defines a feasible set. For the probabilistic case, we instead derive distribution over w and Let 6(w)= ((a)==axp.eEp(0 & ))) similar to (5). Then for 0 a < 1, the belief is defined
fwwa v. P)Q)
if (-1,1).
f(w,a | 1. P,Q)
ft (w.a e P.
i 1,
(9)
(w.a b. P.Q)
if -11

---

where
if --,, v 2 -001 [(v((w)) w/ 0,
f(w,al 4, P.Q) &
Ottherwise.
1 - & -6 (v)(w) w/ > 0,
ft 1. P.Q)
(10)
otherwwis.
[ 1-, Φ² 1 [a((w), w/ S
(w 3. P.Q) x
otherwise.
Given noisy user feedback as in (8), we can define probabilistic density function f( (.). Together with (9) we derive a compound probability distribution
f(w,a μ. P.Q) = [" f(w, a b. P.Q)(( p)d0.
(11)
where we can write f(v (1) for v
E 1 1-1, as
①
+//2
if if E
f(b (1) x
M ptc/2 o
+
(/ーT-C/2
(12)
ute/2
1=
and [(b1))=00 for 1b & [-1,1] Here, T denotes the edf of standard normal distribution. Finally, k= and some prior f(w, at), the joint posterior is
given a sequence Dx = ((Pk, kx Qk 14x)
K
I(w,a a Dx) x f(w, a)) II [(w, a pas Pk, Qx).
(13)
k=1
Here, we can factor f(w, a) as (P)))(()) by assuming and are independent and we also have prior for a". We then take the expectation the posterior f(w, a Dx) ) as our learned user model.
5 Algorithm Design
We now outline the learning algorithm. Over K iterations: (i) the robot actively generates query (PR. Qx) given previous observations Da-1, (ii) the user provides feedback to the query in the form of the slider value μı (in the noiseless case, μt = thk), and (iii) the robot updates its dataset Dx using (13), After iteration K, the algorithm returns the expected weight i B/w DR].
5.1 Worst Case Error Bound
To compare scale feedback choice feedback, we establish a worst case bound on the performance measures for both frameworks. We introduce the worst-case error as the maximum negative per- formance measure, 1 E(w, w²). The constant in front ensures positive value, which we then discount with the posterior belief, given observations D:
Err (w", D) max f(w D)
(14)
This describes the worst w the robot could pick, discounted by the posterior distribution f learned from data D. In the noiseless setting, this simplifies to maxwer 1 E(w, w²).
Proposition 1 (Upper error bound). Let D denote the observation made from scale feedback and DC be the observation from choice feedback for the same set of queries. For any user weights w", it holds in the noiseless setting that Eax "wo" D < Err (w" DC).
The proof follows from the observation JSsslle C Jrhosice, i.e., scale feedback removes more volume from the weight set. Hence, the worst choice an estimate given observations is guaranteed to have a smaller worst case error when using scale feedback. The full proof is in Appendix B.
5.2 Active Query Generation
To learn w* efficiently, the robot chooses the query (P, Q) it presents to the user. While randomly selected queries often lead to some learning progress, actively designing a query can drastically improve learning when the number of iterations is limited. Two recent approaches for learning from choice are information gain [4] and max regret [5]. Information gain seeks to reduce robot's uncertainty over w while choosing queries that are easy answer for the user. Max regret, on the other hand, minimizes the maximum regret by showing mutual worst case paths, which also results in easy queries. We leverage both of these methods for our active query generation in scale feedback.
5

---

We start with the information gain. Let H denote Shannon's 's information entropy [36]. As the outcome of the query is yet unknown, greedy step takes the expectation over μ.:
max (ww a P. Q) - @ (ww, a Ju, P. (.Q.
(15)
Q.Q
We approximate the computation of entropy by summing over set 92 of M samples of (w, a) Thus, following the derivation in Biyik al. [4], the new query (P', Q) solves
f.
P( P. w. a)
M I(( P. Q, w.a)
max P.Q
Σ Σ
log2
(16)
M
E(war P(u P, Q, un, w')'""))
(1.,))
The max regret policy generates queries (P, () such that if the robot learned P but the user optimal solution would be Q is worst case. With a symmetric perspective over P and Q, have
max P(w a D2)P((W a Dx)) R(wP w + R(W?, wn)) w.PaPP 2002 .04
(17)
where Ii(,,-) -) is the reward difference defined in (2). By observing feedback to such queries it greed- ily improves the probabilistic worst case error. In contrast to the information gain approach, max- imum regret requires P and Q be optimal trajectories for some users (w ( a and (w ?)
On the other hand, maximum regret does not require a one-step look-ahead and thus no summation over potential feedback values μ., making it computationally lighter.
Equations (16) and (17) now give us two different policies for solving the initial problem (3). In the simulations, we compare how the performance of both benefits from scale feedback.
Information
MaxRegret
Random
Simulation Results
1.0-
We now present our main simulation results. Addi- tional results can be found in the Appendix.
Aligunens
0.6
Scale
Experiment Setup. We simulate the presented framework using the Driver experiment used in 12, 4-6]. We modify the setup by adding 6 new fea- tures, obtaining more challenging 10-dimensional problem (details on the features, as well as results for the original driver can be found in the Appendix). 71 distinct user preferences w" are drawn uniformly random, and each user is simulated with a" (.25, .7,, 1). making it 284 runs for each method. We set a = 0.1 for the noise level. We generate set of 200 distinct sample trajectories by drawing ran- dom weights w and then computing their optimal trajectories. The active query generation methods then optimize over this set. We evaluate learning using the alignment metric and the relative reward.
0.4
Choice
1.0
Releive Reward
0.9-
0.8
Scale
Choice
0.7
20
Iteration
Iteration
Iteration
Figure 4: Comparison of scale feedback and soft choice for different active query methods.
As a baseline we use soft choice (strict choice showed a slightly poorer performance). To ensure a fair comparison, we emulate soft choice by setting the step size to 1 and use the same noise model for both forms of feedback.
Fig. shows the alignment and relative reward for the driver experiment for information gain, max regret and random query generation. We observe that in all cases scale feedback sig- nificantly improves the performance over soft choice in both metries (p < .001 in all cases with two-sample (-test). When using the proposed scale feedhack, the alignment after 20 iterations im- proves from .77 to .66 for information gain, from .67 to .76 for max regret, and from .64 to .75 for random queries. The relative reward improves for information gain and .97 to 1, i.c., the learned solution is optimal. Both methods make most progress during the first 10 iterations. Random queries improve the final relative reward from .94 to .97. Overall, the simulation showcases that scale feedback improves learning, independent of the query selection method. For information gain and max regret, scale feedback allows for finding optimal solution, i.e., collecting 100% reward, within small budget of iterations. In Appendix D, we show additional simulation results for higher noise.
Results.
regret similarly from

---

7
User Study
Finally, we analyze the scale feedback in comparison with choice feedback and under different active querying methods with two user studies." In both studies, we used = 0.1 for scale queries.
Experiment Setup. We designed a serving task with a Fetch robot [37] as shown in Fig. 1, and generated dataset of 120 distinct trajectories. Human subjects were told they should train the robot to bring the drink to the customer in the manner they prefer, paying attention to the following five factors: the drink (out of 3 options) to be served, the orientation of the pan in front of the robot, moving the drink behind or over the pan, the maximum height of the path, and the speed. The subjects were also informed about the types of queries they will respond to.
Independent Variables. In the first experiment, we wanted to compare scale and soft choice under random querying, and scale under random and information gain querying. Hence, varied the query type and the querying algorithm among: (i) soft choice with random querying, (ii) scale with random querying, and (iii) scale with information gain querying. In the second experiment, we wanted to compare scale and soft choice under information gain querying, Hence, we employed: (i) soft choice with information gain querying, and (ii) scale with information gain querying. For all, we took a = 0.35 based on pilot trials with different users (see Appendix E).
We recruited 18 participants (5 female, 13 male, ages 20- 55) for the first, and 14 participants (5 female, 9 male, ages 20 - 56) for the second experiment. Due to the pandemic conditions, the subjects participated in the study remotely with online interface The study started with an instructions page with a two-question quiz to make sure the participants understood how use the interface. After reading the instructions, we had the subjects fill form where they indicated their preferences for each of the five individual factors described above, to encourage them to be consistent in their responses during the data collection.
Procedure.
in Fig. 1.
In the experiments, each participant responded to 10 queries generated with each of the algorithms. After each of these 10-query sets, they were shown the optimal trajectory from the dataset with re- spect to their learned reward function. The participants responded to a 5-point Likert scale survey (1-Strongly Disagree, 5 5-Strongly Agree) for this trajectory: "The displayed trajectory fits my pref- erences on the task." " We also collected scale feedback for 10 more mandomly-generatee validation in each experiment. We randomized the order of these sets (of 10 queries) prevent any bias. The interface provided a "Sync Videos" button to restart both videos for easier comparison.
queries for
Dependent Measures. As an objective measure of the learning performance, we calculated the log- likelihood of the validation set (of 10 scale queries") under the posterior f(w, a D) learned using the 10 queries generated via each algorithm, i.e., we calculated:
Log- Likelhood
log PPVaaliittionn D) log E [P( (Paaaaaiitioon [w)]
(18)
We also used the responses to the 5-point Likert scale survey questions to measure how well the learned rewards achieve the task. Finally, the users took post-experiment survey where they rated (from to 5) the easiness and expressiveness of soft choice and scale questions.
Hypotheses. We test the following hypotheses.
HI. Scale feedback leads to faster learning than soft choice feedback.
H2. Querying based on information gain accelerates learning compared to random querying.
H3. Users will prefer information gain over random querying in terms of the optimized trajectories.
H4. Users will prefer scale feedback over soft choice feedback in terms of the optimized trajectories.
H5. Users will rate the scale feedback as easy as soft choice feedback.
H6. Users will rate the scale feedback as expressive as soft choice feedback.
Results. We present results of the first and the second experiments in Figs. 5 and 6, respectively. It can be seen that the log-likelihood of the validation set after learning the reward function via scale feedback is higher than learning via soft choice feedback, under both random and information querying. Besides, information gain based query generation accelerates the learning and leads to higher log-likelihood values compared random querying. All of these comparisons are statistically significant with p < .001 spaired-sample t-test), so they strongly support HI and H2.
'We have IRB approval from a research compliance office under the protocol number IRB-52441. A sum- mary video at https: //tps:///ssess htts:/ssssssssssleeeeeeooo
wearnidd-eaannnnn--ee wwwssscallleeeeeeeeeeeeeeeeeeeee and the code at bttpps://g thub. wom./ttaaford-IIIAD wewtADDDewaadd-eaaaee scae--feedaaa.c......e...o..
We present results with a validation set that consists of both scale and soft choice feedback in Appendix F.
7

---

Scale Information
Seale
Seft Choice
Scale Random
Log Libeelhhood
33
Soft Choice Random
Rosinne
Railng
10
-50
-35
Scale
Scale Random
Solt Choice
Random
Easiness
Expresiiennes
(a)
(b)
(c)
Figure 5: All results are shown for the first experiment (mcan over 18 subjects).
-25
Scale Information
Soft Chrize
Soft Choice Information
Log Lifelllhnod
Hosting
40
Rasting
-45
Iteration
Information
Information
Eastness
Expresseveness
Figure All results shown for the second experiment (mean 14 subjects).
In Fig. 5(b), it can be seen active querying led to learning reward functions that better optimize tra- jectories compared to random querying this comparison was somewhat significant with p ss .05, supporting H3. fact, when we fit a Gaussian distribution to the ratings, we observe that it is 1.95 times likely to get better rating with information gain querying than random querying. Surpris- ingly, learning via soft choice achieved slightly higher reward than learning via scale when queries were randomly selected, and slightly lower reward when queries tion gain. However, these comparisons are not statistically significant. This indeed analogous to the relative reward comparisons in Fig. 4: more complex tasks might be needed to better analyze the difference between the two methods. Thus, we neither reject nor accept H4.
generated based informa-
Finally, the subjective results in Fig. 5(c) and 6(C) suggest that users find the soft choice feedback slightly, but consistently, easier than the scale feedback (p < .01), rejecting H5. This is not sur- prising. it is often easier to make pairwise comparison and the "About Equal" option in the soft choice questions makes them even easier [4]. On the other hand, there was no statistically significant difference in terms of expressiveness of scale and soft choice feedback, partially supporting H6. In summary, is interesting that our users perceived the soft choice as easier and even more expressive at times; even though quantitatively, the scale feedback significantly outperforms the soft choice.
8 Discussion
Summary. We proposed scale feedback for reward learning where users provide more nuanced feedback than choice. We introduced user model and showed how a robot can infer reward from noisy scale feedback. We adapted state-of-the-art query generation methods to accelerate learning. In simulations and a user study, scale feedback significantly improved learning. Users rank choice feedback slightly easier, but both forms of feedback equally expressive. However, the minor decrease ease of use out-weighted by strong improvement learning performance.
We proposed scale queries way to give nuanced feedback between two trajec- tories. is possible to extend them to n + trajectories, with specialized user interfaces that allow users to select point from n-simplex instead of a slider bar. Future work should investigate this and if users can still give reliable feedback to these more complex queries.
Future Work.
In our experiments, we used pre-computed trajectory set. Alternatives, e.g., optimizing queries over action sets in [2], or using planners as in [5], should be studied for real-time online learning systems. The high estimate of a in the user studies suggests the proposed probabilistic model may be inaccurate. Future work should refine the user model, including interactively learning a;; or fit new user model that does not necessarily adopt Gaussian noise. Surprisingly, users did not perceive scale feedback as more expressive. This could be addressed with improving interface design as well as designing a query generation method that actively exploits the slider's expressiveness.

---

Acknowledgments
This research is partially supported by the Natural Sciences and Engineering Research Council of Canada (NSERC). The authors would also like to acknowledge funding by NSF grants #1849952 and #1941722, FLI grant RFP2-000, and DARPA.
References
[1] H. J. Jeon, S. Milli, and A. D. Dragan. Reward-rations (implicit) choice: A unifying formalism for reward leamning. In Advances in Neural Information Processing Systems (NIPS), Dec. 2020.
[2] D. Sadigh, A. D. Dragan. S. S. Sastry, and S. A. Seshia. Active preference-based learning of reward functions. Proceedings of Robotics; Science and Systems (RSS), July 2017.
[3] E. Biyik, N. Huynh, M. J. Kachenderfer, and D. Sadigh. Active preference-based gaussian process gression for reward learning. Proceedings of Robotics: Science and Systems (RSS), July 2020.
[4] E. Biyik, M. Palan, N. C. Landolfi, D. Loscy, and D. Sadigh. Asking easy questions: user-friendly approach to active reward learning. Proceedings of the 3rd Conference on Robot Learning (CoRL), 2019.
[5] N. Wilde, D. Kulić, and S. L. Smith. Active preference learning using maximum regret. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10952-10959, 2020.
[6] C. Basu, M. Singhal, and A. D. Dragan, Learning from richer human guidance: Augmenting comparison- based learning with feature queries. In Proceedings of the 2018 ACM/IEEE International Conference on Human-Robb Interaction, pages 132 140, 2018.
[7] R. Holladay, S. Javdani, A. Dragan, and S. Srinivasa. Active comparison based learning incorporating uncertainty and noise. In RSS Workshop Model Learning for Hlmann-Robor Communication, 2016.
[8] K. Li, M. Tucker, F. Biyik, E. Novoseller, J. W. Burdick, Y. Sui, Sadigh, Y. Yue, and A. Ames. Roial: Region of interest active learning for characterizing exoskeleton gait preference landscapes. In International Conference on Robotics and Automation (ICRA). May 2021.
[9] A. Shah, S. Wadhwania, and J. Shah. Interactive robot training for non non-markov tasks. arXiv preprint arXiv:000.002332 2020.
[10] N. Wilde. D. Kulić, and S. L. Smith. Bayesian active learning for collaborative task specification using equivalence regions. IEEE RA-L, 1(2):169,,1698, Apr. 2019. ISSN 2377-3766.
[11] N. Wilde, A. Blidaru, S. L. Smith, and D. Kulić. Improving user specifications for robot behavior through active preference learning: Framework and evaluation. IJRR, 39(6):651 667, 2020.
[12] A. Bajesy, D. P. Losey, M. K. O'Malley, and A. D. Dragan. Learning rohot objectives from physical human interaction. In Conference on Robut Learning, pages 217-226. PMLR, 2017.
[13] M. Palan, N. C. Landolfi, G. Shevehuk. and D. Sadigh. Learning reward functions by integrating human demonstrations and preferences. In Proceedings of Robotics: Science and Systems (RSS), June 2019.
[14] M. Li, A. Canberk, D. P. Losey, and D. Sadigh. Learning human objectives from sequences of physical corrections. In International Conference on Robotics and Automation (ICRA). IEEE, 2021.
[15] M. Cakmak, S. S. Srinivasa, M. K. Lee, J. Forlizzi, and S. Kiesler. Human preferences for robot-human hand-over configurations. In 2011 IEEE/RSJ International Conference Intelligent Robots and Systems, pages 1986-1993, 1993, IEEE, 2011.
[16] V. Myers, E. Biyik, N. Anari, and D. Sadigh. Learning multimodal rewards from rankings. In Proceedings of the 5th Conference on Robot Learning (CoRL), Nov. 2021.
[17] D. Brown, W. Goo, Nagarajan, and S. Niekum. Extrapolating beyond suboptimal demonstrations via inverse reinforeement learning from observations. In International Conference on Machine Learning, pages 783-792. PMLR, 2019.
[18] M. Kollmitz, T. Koller, J. Boedecker, and W. Burgard. Learning human-aware robot navigation from physical interaction via inverse reinforcement learning. In 2020 JEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages
IEEE, 2020.

---

[19] P. Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference Machine learning, page 1. ACM, 2004.
[20] B. D. Zichart, A. L. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement learning. In Aaai, volume 8. pages 1433-1438, 1438. Chicago, IL, USA, 2008.
[21] D. S. González, O. Erkent, V. Romero-Cano, J. Dibangoye, and C. Laugier. Modeling driver behavior from demonstrations in dynamic environments using spatiotemporal lattices. In International Conference on Robotics and Automation (ICRA), pages 1-7. IEEE, 2018.
[22] D. P. Losey and M. K. O'Malley. Including uncertainty when learning from human corrections. In Proceedings of the 2nd Conference on Robot Learning (CoRL), pages 123-132. PMLR, 2018.
[23] J. Y. Zhang and D. Dragan. Learning from extrapolated corrections. In International Conference on Robotics and Automation (ICRA), pages 7034-7040, 2019.
[24] W. Chu, Z. Ghahramani, and C. K. Williams. Gaussian processes for ordinal regression. Journal of machine learning research, 6(7), 2005.
[25] D. S. Brown, W. Goo, and S. Nickum. Rettesshhnndeemmnsttttt imitation learning via automatically- ranked demonstrations. In Conference on Robot Learning, pages 330-359. PMLR, 2020.
[26] L. Chen, R. Paleja, and M. Gombolay. Learning from suboptimal demonstration via self-supervised reward regression. Proceedings of the 4th Conference on Robot Learning (CoRL., 2020.
[27] B. Argall, B. Browning, and M. Veloso. Learning by demonstration with critique from human teacher. In 2007 2nd ACM/EEE International Conference on Himman-Robul Interaction (HRI), pages 57-64. IEEE, 2007.
[28] Y. Cui and Niekum. Active reward learning from eritiques. In International Conference on Robotics and Automation (ICRA), pages 6907-6914 14, IEEF, 2018.
[29] C. Wirth, Akrour, Neumann, Fürnkranz, et al. A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research, 18(136):1-46, 2017.
[30] V. Villani, F. Pini, F. Leali, and C. Secchi. Survey on human- robot collaboration in industrial settings: Safety, intuitive interfaces and applications. Mechatronics, 55:248-266, 2018.
[31] B. Akgun, M. Cakmak, J. Yoo, and A. L. Thomaz. Trajectories and keyframes kinesthetic teach- ing: A human-robot interaction perspective. In ACM/EEE international conference on Human-Robol Interaction, pages 391-398. ACM, 2012.
[32] D. P. Loscy, K. Srinivasan, A. Mandlekar, A. Garg, and D. Sadigh. Controlling assistive robots with learned latent actions. In International Conference on Robotics and Automation (ICRA), pages 378-384. IEEE, 2020.
[33] B. Akgun, M. Cakmak, K. Jiang, and A, I.. Thomaz. Keyframe-bused learning from demonstration. International Journal of Social Robotics, 4(4):343-355 355, 2012.
[34] M. Racca, V. Kyrki, and M. Cakmak. Interactive tuning of robot program parameters via expected diver- gence maximization. In Proceedings of the 2020 ACM/IEEE International Conference on Hluman-Robar Interaction, pages 629-638, 2020.
[35] S. Cabi, S. G. Colmenarejo, A. Novikov, K. Konyushkova, S. Reed, R. Jeong, K. Zolna, Y. Aytar, D. Bud- den, M. Vecerik, et al. Scaling data-driven robotics with reward sketching and batch reinforcement learn- ing. In Proceedings of Robotics: Science and Systems (RSS), 2020.
[36] L. Wasserman, of Statistics: A Concise Course Statistical Inference. Springer Publishing Company, Incorporated, 2010. ISBN 1441923225, 9781441923226.
[37] Wise, M. Ferguson, D. King, E, Diehr, and D. Dymesich. Fetch and freight: Standard platforms for service robot applications. In Workshop on autonomous mobile service robots, 2016.
10


Reviews:
[{"review_id": "piZ0K7sBaYa", "reviewer": "Reviewer_pmpv", "summary": "This paper introduces the use of scale feedback to learn user reward functions expressed as a probabilistic model. The author empirically evaluate the advantages of their method in simulation and in a real robot experiment.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n======\n- the proposed method is intuitive and sound;\n- the lower bound on the worst case error is a nice result;\n- experimental evaluation is well described and informative.\n\nWeaknesses\n==========\n- I find the technical description of the method a bit hard to follow;", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I like how the paper motivates the approach and provides a solid theoretically sound algorithmic implementation. I am not an expert in this field of study, but I find the validity of this method quite intuitive. In fact, I find it weird that it has not been proposed in the past. I am not knowledgable of the related literature, so I will trust the opinion of other reviewers about the originality of this work. It’s also difficult for me to judge the impact of this work in the robotics community, but I definitely think it could be interesting to a group of researchers. Overall, I am convinced of the validity of this work thanks to the accurate mathematical formalization of the problem and the intuitive motivation of the proposed method, although I am not sure about its impact and originality.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Poor: Limited knowledge of the area"}, "extra_text": {"issues": "I would appreciate if the authors can help me understanding the impact and originality of their work. For example, why do you think this has not been proposed in the past? What are the major problems that other researchers may have encountered?"}}, {"review_id": "U06olQ-83h", "reviewer": "Reviewer_Z5iG", "summary": "This paper proposes to learn from a novel type of human feedback in the form of scale-based preferences, i.e. allow a human teacher to indicate how much they prefer one trajectory query over another using a slider bar. Such form of feedback provides more nuanced information compared to binary/ternary choice-based feedback as used in existing preference-based reward learning methods. The authors formulate the problem of learning from scale feedback under the preference learning framework, develop a probabilistic model to account for noisy user feedback and propose two active learning methods that generate queries to maximize information gain and max regret respectively. Experiment in a simulated driving domain demonstrates the advantage of using scale feedback over choice feedback. A user study with a robot performing a shelving task is also conducted to compare learning from scale-based feedback vs choice-based feedback with human inputs. The user study shows that, while scale feedback enables faster learning than choice feedback, users indicated that choice feedback is easier to use.   \n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Learning from human feedback is an important research topic that is relevant to many research areas in robot learning. Finding new modalities that a human could provide feedback enriches the ways users could communicate with robots. This paper proposes a novel type of human feedback that an agent could learn from and demonstrate its advantage over existing approaches. The authors also develop a probabilistic model to account for the noisiness in human feedback. The paper is well-written with clarity and the experimental results clearly shows the proposed method’s advantage over the choice-based baseline. \n\n\nFor studying methods that interact with human users, it is very important to also control/measure the human user’s experience. While the authors did a very good job at asking how the users would rank the easiness of the two approaches, it would be useful to also see some objective measures such as how much time it actually took for the user to answer these preference questions on average.\nOne other small critique I have about the user study design is that the robot’s trajectories are all pre-computed, which is ok for the purpose of the study but does not apply to real applications. It would be nice if the authors could also comment on how to address this limitation if this method were to be deployed on real systems.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I highly recommend the acceptance of this paper. This paper proposes to learn from a novel type of human feedback with active learning algorithms that improves efficiency over existing methods. The writing and presentation of the work is well-executed.  ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "The author should also comment on practical limitations of the method, and potential ways of extending this method to real-time interactive learning systems.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "BVi6tFVE3nJ", "reviewer": "Reviewer_cvuG", "summary": "Update: I sincerely appreciate the authors' thorough and thoughtful responses as well as updates to the research/paper itself. I have updated my review accordingly.\n\n======\nThis paper presents a framework for learning user reward functions for humanoid robotic tasks (and possibly other more general robotic applications) by using scale feedback in lieu of binary or ternary choices. The paper presents the framework in terms of a noiseless user model as well as a noisy user model, and it also provides theoretical results on error bounds for some of the framework. The paper presents an algorithm for query generation, and then evaluates the efficacy of the algorithm relative to several baselines, across two studies.  The paper is clearly written and structured well. \n\n", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper focuses on an interesting problem of understanding user preferences and then being able to formally encode this information in terms of learning a reward function. The paper is structured well, the prose is clear.\n\nMy main suggestion has to do with scope, and I am not sure if this is \"fair\" given that I would essentially be asking the authors to change their approach entirely. I question whether a slider bar (scale response) evaluating two different trajectories is even the right way to think about evaluating/understanding/learning/encoding user preferences. I have several questions about this in my detailed comments below. Reducing everything down to a binary choice does not seem like a great idea, as the paper points out well. However, turning this choice into a sliding scale does not seem to enhance thing much, and as the users note, this does not seem to be very expressive. While the results suggest that the method does indeed converge on a more accurate reward function, I think one of the promises (and challenges) of explicitly including humans involves the now-overused terms like \"explainability\" and \"transparency\" and \"trust\" and things of this ilk. If the method is really just about learning rewards, then I suppose this paper does well enough. Again, I recognize that this assessment might not be fair and I am trying to be helpful here, I promise. What is the purpose of bringing the human in the loop? Can the authors make this more clear, given the relatively simple task that was used for evaluation/experimentation? To me, the big topics involving humans and automation have to do with the buzzwords I mention above, not necessarily understanding user preferences for a task that (in my opinion) is amenable to just writing down a reward function in a straightforward way.\n\nFinally, I am not sure I agree with the conclusions about the efficacy of the approach. I might say it works in the sense that it is not worse than binary/ternary feedback, but that feels like a bit of a strawman argument.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Hopefully it is clear from my comments below that I am concerned about the approach and whether this results in a significant enough contribution to the robotics community. There are no major technical flaws that I can see, and like I said above, I think the paper is written well enough. I encourage the authors to think about the fact that (as they state) these problems are multi-objective, and have many qualitatively different classes of solutions. ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "I am not entirely clear what the first paragraph (lines 14-18) have to do with the paper's focus and contributions. Yes, robots need to trade off between goals and have competing objectives, but I believe the paper is about understanding user preferences. While these concepts are ultimately related in any interactive robotic environment, I do not understand this paragraph's purpose. \n\nRegarding scale feedback as a general notion, I have one general concern and question. Wouldn't it be more benefiical to compare more than 2 trajectories, as opposed to having a sliding scale between a binary choice? This latter approach still seems very simplistic and might not help a human think about all the amazing possibilities that a robot could identify. \n\nLine 87, delete comma after \"setting. (I think)\n\nHow senstive is the system's performance to the annotations on the slider bar? There are 3 annotations in the approach; what if there were 5? And are they anchored in any type of way, such as what is typically done in polling and the like?\n\nCan the authors provide more justification for the modeling assumptions in definition 4? In particular, why is user noise zero-mean Guassian? And why is it stationary? Why is this stationarity independent of what the actual choice psi is? For example, I imagine the choice 0 might generally be less certain than 1 or -1. Thoughts here? \n\nProposition 1 is well-reasoned. However, this seems like a bit of a \"strawman\". Of course if you give more options to non-noisy agent, any algorithm should do better both in the average case and worst case. The authors seem to imply this with the point that adding choices removes volume from the weight set. But what happens if the human is noisy?\n\nLine 204 states that max regret \"hand minimizes\" the maximum regret. Is this a typo, or a term I am unfamiliar with?\n\nLine 238-239: missing word: \"...make most progress *during* the first 10 iterations\"?\n\nFigure 5(b) is very interesting. The authors discuss this a bit in the body text, but I am not sure it is helpful. Can the authors at least propose a framework that would help (in future work) to support or invalidate H3? Is it as simple as doing more experiments? Or does the design of experiments actually need to be changed?\n\nI do not agree with the conclusion, in line 297, that the data partially supports H5. It either leads one to reject H5, or partially reject it, but not partially support. Furthermore, what does \"expressiveness\" even mean? Was this explained tot he user during the questioning?\n\nRegarding the discussion about expressiveness: returing to my earlier comment, I think the important factor to the user is the expressiveness of the *trajectory itself*. That is, do the presented trajectories really represent the tradeoffs I care about? This is why I had the earlier question about presenting multiple \"classes\" of trajectories. What is a user really expressing when they say they prefer one trajectory over another? There are so many different objectives (as the authors acknowledge), and collapsing things down to a single dimension - which a sliding bar does, no matter how small epsilon becomes - severely limits the user's ability to tell the machine about his/her preferences. \n\nIn short, I am skeptical about whether the slider even fundamentally allows the system to \"ask the right question\" to the users. Going from a binary (or 3-nary) choice to an infinite slider is fine (apparently, from the data) if one wants to accurately estimate a user's optimal policy. Therefore, who really cares about expressiveness? In the end, how much does a user's opinion of the interface matter, as long as the user can stay engaged to the point that a model can be learned? "}}, {"review_id": "3QCphS9DzIu", "reviewer": "Reviewer_2eoP", "summary": "I have updated my estimate of the technical quality from fair to good in response to further user studies and more detailed analysis of the data. Overall, I have upgraded my judgment from Strong reject to weak accept in response to this further evidence. I urge the authors to perform further experiments in order to confirm or reject their understanding of the discrepancy between log-likelihood and user ratings, and I personally still find it more likely that there is some problem in the log-likelihood evaluation. However, the collection of evidence is interesting enough that it ought to be published. The original review is below. \n\n===\n\nChoice feedback has been successful as an alternative way to specify the desired behavior of an RL agent. The key feature of choice feedback is that it is easier to correctly produce compared to optimal demonstrations or a manually specified reward function. However, choice feedback is easy to produce in part because it is an inherently low information communication mechanism: each binary choice by definition provides at most one bit of information.\n\nHumans can sometimes provide more information than this single bit when comparing two trajectories. For example, in prior literature there are methods which present the human with a third option to indicate that two trajectories are roughly as good as each other. This paper takes this insight to it's natural conclusion by allowing the user to specify a scalar strength indicating precisely how much they prefer one trajectory to another, including not at all. Whereas most (perfect, noiseless) choice feedback implicitly imposes a halfspace constraint on the reward function space (under known linear reward features), the paper shows that a scalar opinion reduces the set of possible reward values to a hyperplane in the reward space (reducing the total dimension of the feasible rewards by one). Humans are never actually capable of providing noiseless feedback, so the paper also develops a natural Gaussian noise model of the probability of a human reporting a certain preference strength under a given ground truth reward function, which they use to infer the true reward function in a Bayesian fashion.\n\nHuman feedback elicitation research often operates by picking a feedback modality to elicit from the human, a Bayesian model of how that feedback is produced, and then finally a criterion with which to determine the optimal query to ask at every timestep. This paper is not attempting to innovate on the latter, and so evaluates their method using simulated and real human data under two such criteria, maximizing expected information gain and minimizing the maximum regret, as well as a baseline where completely random questions are asked independent of the current belief distribution. Experiments are performed using all of these query generation methods under a binary choice feedback modality and the new scalar feedback modality in the simulated case, and a smaller subset of these experiments are performed on human data.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The idea to elicit scalar preferences from a human is a natural extension to existing successful ideas in choice feedback, and frankly I wish I had done these experiments myself.\n2. The paper is clearly written with excellent figures.\n3. The appendix includes a large number of supporting simulated human experiments.\n\nWeaknesses:\n1. The experimental results of the paper do not support the conclusions stated in the abstract and introduction.\n2. There is an important missing experiment from the human study.\n3. The practical choice of $\\sigma$ is an important complication of the method is not well discussed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "As the paper stands the experiments can only support the conclusion that the Gaussian noise model under scale feedback is not so incorrect as to be disastrous. There is no robust evidence to suggest that scale feedback is superior to choice feedback on human data. The main issue is that no choice experiment was performed using a state of the art active query criteria (e.g. infogain). This makes it impossible to make like to like comparisons between scale and choice feedback while using the techniques either are likely to be used with. The results reported from the experiments that were performed support neither the claim that humans find scalar feedback more expressive nor the claim that the learned reward function under scalar feedback better fits their preferences on the experimental task. In fact, on the latter measure, the evidence suggests that humans will likely prefer the reward function learned under choice feedback, and they prefer choice feedback when comparing like query generation to like. As the likelihood models are different for the scalar and choice feedback, log likelihood cannot be compared between feedback types, and so I cannot see it as an indication of learning good reward functions. In order to accept this paper I would need to see the missing infogain + choice human experiment and either\n\na) results indicating that scalar feedback robustly outperforms choice feedback\n\nb) if scalar feedback is not always better, a discussion of when/for what problems it is, or\n\nc) a more frank portrayal of the results, perhaps by saying that you can get to the same (or a slightly worse) reward function in many fewer queries to the human, or whatever conclusion the results of the final experiment support.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {"issues": "How exactly was the choice of $\\sigma=0.35$ made in the user study (as mentioned on line 255)? Shove this in the appendix if you have to, but this is an important part of your experimental method. Did you try using a per-user sigma value? In my experience different users will behave wildly differently in these kinds of settings. But perhaps there simply wasn't enough data per user to determine a meaningful $\\sigma$ value.\n\nOn line 287, tell me exactly what the p-value is. Given how small all of the other p values are, I'm not inclined to trust a \"tending towards significance\" conclusion. You can say that the study is underpowered and H4 is inconclusive as you do with H3 (for which I would also like a p-value for the hypothesis that you're exactly wrong). Alternatively, if you think the current results really do suggest there's something there that shouldn't be ignored, you can do a Bayesian analysis and report a (presumably) moderate but not overwhelming likelihood ratio, which would say what I imagine you're trying to say with \"$p \\approx 0.05$\" but in a principled way.\n\nThe choice of experiments in the user study is very confusing, and results in a weak baseline that makes it impossible to be certain that scale queries are robustly better than soft choice queries. In the prior work on the subject (I'm mostly familiar with Biyik and Sadigh's work) the rough ranking of the efficacy of active query generation criteria is infogain > regret > random. In some sense random queries are the obvious thing to do, and serve as a baseline for all other active methods. Why then is the only soft choice query baseline in the user study using questions generated randomly, instead of one of these better criteria? Due to this choice, the only apples-to-apples comparison we can make is between Scale - Random and Soft Choice - Random, leaving completely open the possibility that soft choice might outperform scale when both are using infogain generate queries. Even if this is not the case (i.e. scale infogain queries are the best), it is interesting to know how relatively important the active reward criteria is relative to the form of query. If soft choice infogain outperforms scale random queries, then we know it is most important to use infogain queries (or possibly some other non-random criteria), and that scale queries are a robust improvement, but perhaps a minor one. It is hard to tell from the figures in the paper (I would appreciate a table of the exact final alignment, relative reward, and log likelihood values of all runs in the appendix), but this is plausibly true in the simulated experiments as well. Alternatively, we mind find that scale random outperforms soft choice infogain, and that the choice of query type is more important. As it stands, there are many fundamental questions about your method that we cannot answer due to the choice of a weak baseline.\n\nThis may just be my familiarity with the area, but Proposition 1, while informative, also seems essentially trivial. It would significantly improve the paper if there were a similar theoretical result for the noisy case, even a noisy case where you have to assume \"too much\" about the noise model in order to get a result.\n\nSeveral plots in figures 10 and 11-13 make it appear as if choice queries are approaching the performance of scale queries. If this is the case scale queries may be much better for requiring less data, but it would be interesting to see where this saturation point is. Based on the runs where the two lines do meet, and due to your theory, it seems unlikely that choice ever surpasses scale, but this would be valuable information to have in future comparisons of the information value and human difficulty of different styles of query.\n\nThere are a couple of places in the introduction where I agree with the spirit of the claim but disagree in the specifics:\n\nLine 51: \"While demonstrations are usually most informative\" Demonstrations are often not more informative than preferences. In particular, optimal demonstrations are often good along all features, and so it can be in principle impossible to disambiguate between different relative weights between pairs of features which are highly correlated in optimal trajectories. To steal the example from your video, if we only see examples of passing the beverage that are fast and don't break anything on the table, it is in principle impossible to tell if the speed or the safety is the more important factor here.\n\nLine 58: \"it also only provides one bit of information\" choice queries provide AT MOST one bit of new information, they often provide much less, especially if the query being asked is essentially redundant considering prior queries. I personally find this is often the case when working with active preference elicitation algorithms. This only strengthens your point.\n\nThe following are grammatical or formatting issues that I expect can be fixed easily and are primarily for the author's use:\n\nLine 162: Please write out all of the dot products explicitly as equalities or inequalities. At the very least, place commas as separators between the terms, the spacing alone makes it hard to tell.\n\nLine 240: \"the simulation showcase that\" -> \"the simulation showcases that\"\n\nFigure 5: The lines below and above the figures are distracting"}}]

Author rebuttal:

Meta-review: Overall, the reviewers found that the paper addressed an important problem and was clearly written. The proposed method is a reasonable and intuitive solution to the problem and was supported by empirical evidence consisting of a user study. 

The paper would be improved by more discussion about the impact of the approach, for example, is using a slider to select preferences the right approach or could there be potentially better solutions? What are limitations of the approach? Furthermore, the approach was evaluated with pre-recorded robotic trajectories, and so discussion of how this might scale to more difficult settings would also help make the impact more clear. Finally, there were some concerns about the user study and the conclusions drawn in the paper, and so further evidence or discussion supporting the claims would improve the paper. 

Decision:
Thank you to the authors for taking time to address the reviewer's concerns. Overall, I already found the paper to be interesting and believe the paper will be greatly improved with the new updates and experiments.

Decision: Accept (Poster)