INQUIRE: INteractive Querying for User-aware Informative REasoning
Authors: Tesca Fitzgerald
Pallavi Koppol
Patrick Callaghan
Russell Quinlan Jun Hei Wong
Reid Simmons
Oliver Kroemer
Henny Admoni
Venue: CoRL 2022 Poster (2022)
Keywords: Active Learning
Learning from Demonstration
Human-Robot Interaction

Abstract: Research on Interactive Robot Learning has yielded several modalities for querying a human for training data, including demonstrations, preferences, and corrections. While prior work in this space has focused on optimizing the robot's queries within each interaction type, there has been little work on optimizing over the selection of the interaction type itself. We present INQUIRE, the first algorithm to implement and optimize over a generalized representation of information gain across multiple interaction types. Our evaluations show that INQUIRE can dynamically optimize its interaction type (and respective optimal query) based on its current learning status and the robot's state in the world, resulting in more robust performance across tasks in comparison to state-of-the art baseline methods. Additionally, INQUIRE allows for customizable cost metrics to bias its selection of interaction types, enabling this algorithm to be tailored to a robot's particular deployment domain and formulate cost-aware, informative queries.

INQUIRE: INteractive Querying for User-aware Informative REasoning
Tesca Fitzgerald".14
Pallavi Koppol
Patrick Callaghan² Oliver Kroemer² Henny Admoni
Russell Q. Wong
Reid Simmons
Yale University Carnegie Mellon University
tesca fitzgerald@yale edu,
{pkoppol, pcallagh, rqwong, rsimmons, okroemer, hadmoni (attoonn@@andde... Sandrew. cmu, edu
Abstract: Research on Interactive Robot Learning has yielded several modalities for querying human for training data, including demonstrations, preferences, and corrections. While prior work in this space has focused on optimizing the robot's s queries within each interaction type, there has been little work on optimizing over the selection of the interaction type itself. We present INQUIRE, the first algo- rithm to implement and optimize over generalized representation of information gain across multiple interaction types. Our evaluations show that INQUIRE can dynamically optimize its interaction type (and respective optimal query) based on its current learning status and the robot's state in the world, resulting in more ro- bust performance across tasks comparison to state-of- oftte-o--eearrr baseline methods.
Additionally, INQUIRE allows for customizable cost metrics to bias its selection of interaction types, enabling this algorithm be tailored to a robot's particular deployment domain and formulate cost-aware, informative queries.
Keywords: Active Learning, Learning from Demonstration, Human- Inter- action
Introduction
As we envision robots that adapt to novel tasks and environments after deployment, it is important to consider how they can efficiently obtain training data to address this novelty, Research in Interactive Robot Learning has yielded many effective methods for obtaining this training data via interaction between a robot and a human teacher. In a demonstration, the teacher provides the trajectory that the robot should take starting from particular state [1, 2]. In preference query, the teacher selects one trajectory from set candidates proposed by the robot [3, 4]. In response to a single trajectory proposed by the robot, the teacher can provide correction [5, 6] simply binary reward [7].
These interaction types differ according how the robot queries the teacher, how the teacher is constrained in providing feedback, how the robot should interpret the teacher's 's feedback as training data, and the physical and cognitive load imposed on the teacher [8]. Prior work Active Learning has investigated how to formulate informative queries by maximizing the expected information gain resulting from the teacher's feedback. However, barring a few exceptions (19, 10, 11)), prior work typically assumes that the robot uses a single interaction type for all queries. We expect that the op- timal interaction type depends on the robot's task knowledge (which changes over time), the robot's query state (i.e., the state from which it queries the teacher), and domain-specific considerations (e.g., the time or effort it takes a teacher to respond to queries) [12].
Our work is motivated by this question: How can a robot optimize both the type and content of its queries to a human teacher based on the information needs at any given moment? We intro- duce INQUIRE: robot learning system that performs this optimization by representing multiple interaction types single unified framework, enabling the robot to directly estimate and com- pare the expected information gain of its queries across multiple interaction types. We evaluated INQUIRE against two state-of-the-ard interactive learning methods that use a single or fixed pattern of interaction types. We analyzed the effect of domain on INQUIRE's performance and selection
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

of interaction types over time by simulating four domains with unique reward-learinng problems. We found that INQUIRE learned reward functions that were more accurate and resulted in better task performance than either baseline, with particular strength in accommodating low-information query states (i.e., repeated states in which the robot has already received feedback). Furthermore, we demonstrate how INQUIRE can incorporate cost metrics (representing physical or cognitive load on the teacher), optimizing queries over both the informativeness and ease of the teacher's 's responses.
2 Related Works
Learning from Demonstration involves interpreting human feedback training data using methods such as imitation learning [1] or inverse reinforcement learning [2]. However, there are many other forms that human feedback can take, including preferences [3, 4), labels [13], and corrections [6, 5]. These approaches optimize queries within single interaction type, typically by maximizing volume removal [14] or information gain resulting from the teacher's response to the query [3]. Other approaches such as min-max regret optimization have also been explored [15]. Prior work has also investigated the use of fixed strategies for selecting interaction types; for example, requesting fixed number of demonstrations before requesting preferences for the remaining queries [9, 16). [1]] incorporates more interaction types (demonstrations, labels, and feature queries) and contributes both rule-based and decision-theoreee from multiple feedback types include combining preferences with ordinal labels [17]. and using both demonstrations and rankings [18]. Furthermore, [19] presents software library for combining different preference feedback types and demonstrations.
strategies for query selection. Other methods for learning
Several attempts have been made to impose a unifying and consistent framework across multiple interaction types. [20] describes interactions in terms of the explicit and implicit information they convey; [8] surveys how different interaction types affect a teacher's ability to provide informa- tive feedback. Our work similarly contributes perspective on the unifying and differentiating features of interaction types: we propose a generalized framework for computing information gain across multiple interaction types. We focus on four interaction types corresponding to the archetypes (Showing, Categorizing. Sorting, and Evaluating) identified in [12]. and empirically show the effects of dynamically selecting interaction types robot learning.
3 Approach
We define query as a set of possible choices presented to the teacher, and feedback as the teacher's selected choice in response to a query. Our goal is to enable a robot to (1) efficiently query a teacher using multiple interaction types, and (2) learn from feedback obtained via these interactions. We ground this goal in the problem of learning a distribution W over feature weight vectors w E W, each resulting in linear reward function r(t) <(t) w, where o(t) is the feature vector of trajectory t. Thus, our goal translates into (1) selecting queries and interaction types that minimize uncertainty over W, and (2) updating W over feedback from multiple interaction types.
We present INQUIRE (Alg. 1), an algorithm comprised of three key steps for each query: (1) select- ing the optimal interaction type i and corresponding query 9f that maximizes the information gain over the weight distribution W (approximated as the sample set Q), (2) recording the teacher's 's re- sponse to that query (i.e., feedback) in a feedback set F, and (3) updating the weight distribution W such that it maximizes the likelihood of all feedback in To generalize across multiple interaction types, we must contend with the differing formulations of query and feedback corresponding to each type. We follow the framing presented in [8], where each interaction type consists a query space Q(s) (the set of possible queries from state s) and choice space C(q)) (the set of possible teacher feedback, i.c., the choices available to the teacher in response to a query q E Q3))). We assume the robot must query from whatever initial state it is placed in, and cannot optimize the state itself.
For demonstration, let T(s) represent the set of all possible trajectories originating from the initial state The robot (implicitly) enables the teacher to demonstrate any trajectory in this set, and thus its query space (+)) = (T()) (i.e., a single query consisting of the entire trajectory space). The teacher's choice space is C = Tss) (any trajectory within that space). For a preference, the robot queries the teacher with two trajectories q = (lo: L 1o,4 E T(s) who then chooses either to
or tj. The query space is Qss) = T(s) x T(s) and the teacher's 's choice space is C(g) {to, f.).

---

Table 1: Each interaction involves separate query spaces, choice spaces, and choice implications.
Query Space Qi(s)
Choice
Query
Choice Implication
Space
q E Qi(s)
CEC,(9) = (c*,
C.(g)
Demo
{T}
T
T
et T (
Pref.
Tx T
(to, (1), t1.11 ET (to.t1}
ct :t: c :91ct
Corr.
T
tET
T
ctit'e c::q
{0,1} c= et T C :q c ct 4 c : T/
Binary T
t TET
Algorithm INQUIRE Overview
Algorithm 2 INQUIRE Generate Query
Input:
Set of query states S
Input: (state), (interaction types), Ω (weight samples)
Parameters: K (# of queries), I (interaction
Output: Query q"
types)
1: T +- uniformly-sample harpalyyssmmllleaajcotrreeeee trajectoriies())
Output: Weight vector
2: Compute E (Er, Vt, t' E T, w E St}
(Eq. 4)
3: for each interaction type i Z do
x- 0)
Q +Q(ss)
(See Table (See Table 1)
2: Q <- M random initial weight vectors
C (Ci(g), Vq E
3: for K iterations do
Compute info gain matrix G from E
(Eq.
4:
next query state in S
7: + arg m Eeec,
G WER G
4. senerate.query(s. 92) (Alg. 2)
9 log(A.) EnEo G
g.c.0
6:
+ U [query. teacher(g))))
information gain > g" then
7:
Ω medate_weights((
10:
g" + 9
8: mean(
11:
(Store query with highest info. gain}
return
12: return q"
For correction, the robot executes one trajectory q T(s) which the teacher then modifies to preferable behavior. The agent's 's query space is Q(s) T(s) and the teacher's choice space is C(q) Ts)). For binary reward, the robot executes single trajectory q E (s), and the teacher indicates positive negative reward. The agent's query space is Q(s) T(s) and the teacher's choice space C(g) = (0, 1}.
The implication of the teacher's choice E C(q) is a set of accepted trajectories c+ and set of rejected trajectories c_, which we define in Table I and use later calculate information gain. Since the set of all possible trajectories originating from (represented by T(s)) is potentially infinite, we approximate
the I containing N trajectory samples originating from the state and consisting of randomly selected actions.
3.1 Query Optimization
When optimizing the agent's query, our goal is to greedily select one that maximizes the agent's s expected information gain over W after receiving any feedback from the choice set (summarized in Alg. 2). Selecting a query involves optimizing over information gain (IG) as follows:
q = arg max Ee(Ci(y) [IG(W] [c] (EQ,(s)
(1)
M P(clw) P(clw) log EweR P(clw")
Σ (EQ.(s) CEC,( WER
Σ
arg max
where Ω contains M samples of the distribution W. The expansion from Eq. 2 follows the derivation presented in [3]; see Appendix A. for intermediate steps. We adopt the commonly-used Boltzmann ratiioaal equation define P(clw):
P(clw) Elect (-)(())
(3)
Excatue e-o(fyw
3

---

where ((t) returns the feature trace of the trajectory t; that is, the sum over the feature vectors of all states visited in trajectory t.' Note that Eq. 3 reduces to Bayesian Inverse Reinforcement Learning [21] for each / E ct. [8 is parameter representing the expected optimality of the teacher's feedback with respect to w. We assign value of B 20 across all interaction types (selected through empirical evaluation).
To minimize the computational complexity of solving for Eq. we reformulate it as series of operations over a [Q] x |C| |S2] probability tensor P, where P Pq...w represents the probability (according to weight sample w E (2) that the teacher will select choice e in response to query q. To construct P, let E be a N M (i.e., [T] (T' x (92) tensor representing exponentiated rewards:
E e --(("")-
E + E 1e = )- + --(1))
(4)
All tensor transposes are performed over the first two axes. With E in hand, we next define the probability tensors of each interaction type as follows:
Pidemo)
Eg
ΣET
E
(since QI 1 for demonstrations)
(5)
IET
P(pref)
E 2 (E ET) (E+E+))))
(c E {0, 13 for prefs.)
(6)
9,0,W
2.20-91:44
(sorr
E 2 (E + qie,w
(7)
9,c,c
((bury) Ep ΣE" EE
E"
E. 2 ΣE" ET
(e E {0, for binary rewards)
(8)
9,6.
IET
where 2 represents an element wise division of two matrices (i.e., (AOB), Aij/B;; and a is normalization factor such that Σ. Pheer 1. For derivations, see Appendix A.3. The main effect of this formulation is that it enables tractable optimization over multiple interaction types by sharing common representation E. To solve for the optimal query of using interaction type use p()) to construct IQ) x |C| [S2] information gain tensor G
((() M
G(i)) 9,e,u g,0,W
p())
log
d.ew
of arg max
GO 9,0,u
Ewees ((O)
g,C,W!
We then solve for the optimal interaction type itself. To perform a cost-weighted optimization, with the aim of optimizing over both interaction cost and informativeness, A; may be set according to domain-specific cost factors (e.g., the time or mental load involved in answering a query) for each interaction type.? To perform unweighted optimization and maximize solely tiveness of each query, let A be a constant value over all interaction types i E I.
the informa-
arg max IE
log(>s) Σ C
(10)
We summarize this process Alg. 2.
3.2 Update Weights from Feedback
After presenting the optimal query to the teacher, the agent receives feedback and appends to feedback F-aa cumulative set that contains all feedback received by the agent thus far. Our goal is to then update the weight estimate such that maximizes the likelihood of all feedback in F:
Eiset B-P(t)--
w" = arg max II P(clw) = arg max II cEF
(11)
Eiectue @9-(())
OEF
'See Appendix for each domain's 's definition of Φ. "In our evaluations, we assign a cost of 20 to each demonstration, 15 to each correction, 10 to each prefer- and 5 each binary query.

---

Param, Estimation
Linear Lander
Lander
Pizza (Changing
(Changing
(Sunic
Quary
Selected interaction types without cost-weighting
Param, Estimation
Dyn System (Changing
Amangement
(Stdi
(Changing
changing
Quary
(b) Selected interaction types with cost-weighting
Figure 1: Heatmaps illustrating how INQUIRE selects different interaction types as it learns more over time. These selections differ when deriving unweighted (top) or cost-weighted (bottom) in- formation gain estimations. In the cost wesi-weghhted setting (bottom), INQUIRE selects more low-cost binary queries than it does in the unweighted setting (top).
We calculate the gradient over w by differentiating over its log-likelihood given F:
al(w) Owj
Exect 3 (b((1) @011)
Exectve- 3 3,0((1) e
Σ
(12)
Exeot @
Eisctue-e e3.o(t)
CEF
Exeatuar 01) 300000
Σ cEF
B (,((()))))
(iff let = 1)
(13)
ElEctue c.o(())
See Appendix A.4 for the full derivation. After receiving feedback from each query and updating F, we approximate Q by randomly initializing and then performing gradient ascent on each weight sample w E Ω.
4 Results
simulate four types of learning problems robotics using an oracle teacher to obtain controlled evaluations. The oracle teacher, similar to INQUIRE, requires its own set of trajectory samples 7". It then selects a response to query via one of three mechanisms: returning the highest-reward trajec- rejection sampling of trajectories followed
tory from its choice space (demonsttraiion//rreeere) by selection the trajectory with the highest reward-to to-dtttancce ratio from the queried trajectory (corrections), and returning whether query meets exceeds a reward threshold (binary reward). Implementation details can be found Appendix B.2.
The Parameter Estimation domain involves directly estimating randomly-initialin ground truth weight vector containing 8 parameters. The Linear Dynamical System domain, inspired by [3], simulates controls problem and involves learning 8 parameters. The Lunar Lander do- main [22] simulates a controls problem involving parameters. The Pizza Arrangement domain simulates preference-learning problem involving 4 parameters. Each domain (except for Param- static-state changing-state condition indicating whether the robot must formulate all queries from the same query state or not, respectively. For the full evaluation procedure and oracle implementation details for each domain see Appendix B.
eter Estimation)
4.1 INQUIRE Query Selection
We first analyze how INQUIRE selects queries. Figure 1 reflects the changes in interaction types selected by INQUIRE over time. Figure la first first reports these interaction selections an unweighted query optimization setting, where all interaction types are assumed to be equally costly. In the parameter optimization domain, INQUIRE requests corrections in the first 14- queries and then requests preferences as the remaining queries. Demonstrations were not enabled in this domain. In
5

---

Distance- Linear Dynamical System
Performance - Linear Dynamical System (Changing State)
Cost Linear Dynamical System (Chenging State)
(Changing Stale)
Dretance from
05
Dencedrlly
Deveret
Performancee
Dietance from
2.
Irdfoerness Dnly
01
30 A Queries
:50 Accumulated Query Cost
200
of Queries
(a) Linear Dynamical System
Distance -Lunar Lander
Performance- Lunar Lander
Cost Lunar Lander (Changing State)
[Changing State)
(Changing State)
0.5
-inary City
Drotance from WP
0.5
Yask Performance
00
pemtret
Pndoerees-Cb Crly
0.3
EIQUES
077
Diatence
0.8
02
of Queries
of Querles
Accumulated Query Cost
(b) Lunar Lander Task
Distance- Pizza Arrangement
Performance Pizza Arrangement
Cost -Piza Arrangement (Changing State)
[Changing State)
[Changing--o State)
0.5
-Crmecttnn Only
Distance wr
Teak purformance
Instoeross Dnly
0.3
PIQUISE
04
Queries
Accumulated Query
(c) Pizza Arrangement Task
Figure 2: Metrics for the changing state condition in which the robot's initial state changes with each query. Error bars/regions represent variance across multiple evaluation runs with randomized query states and initial weights. Cost metrics are cut off after 20 queries for the binary-only method (c) due to extensive computation times.
all other domains, INQUIRE requests a demonstration as its first query, then immediately switches requesting preferences for the remaining queries (occasionally alternating between preferences and demonstrations in the Lunar Lander domain).
After assigning different cost values to each interaction type, INQUIRE chooses more diverse inter- action types in order to maximize its information-to-ccn results in INQUIRE posing more binary queries due their relatively low cost. This pivot toward binary queries may occur at the start (as seen in the linear dynamical system), middle (as seen in the parameter estimation domain), or interspersed throughout the learning process (as seen in the lunar lander domain).
ratio. As shown in Figure 1b, this typically
4.2 Learning Performance
We now analyze the effect of INQUIRE's interaction type selections on its learning performance and compare to two types of baselines. The first, DemPref [9], learns from 3 demonstrations and then learns from preference queries by using a volume removal objective function. As our second baseline, compare INQUIRE against agents that use only form of interaction: demonstra- tions, preferences, corrections, or binary reward. Note that the preference-only agent is formulated according to [3] and thus represents this baseline method.
We first consider the changing-star state formulation of each domain, where the robot is presented with a new state for each query. Since the Parameter Estimation domain does not contain states, we exclude from this first set of results. Figure 2 illustrates this learning performance the Linear Dynamical System and Lunar Lander domains according to three key metrics. Distance measures the angular distance between the ground truth feature weights (w²) and the algorithm's 's estimated feature weight w after each query. Performance measures the task reward achieved using trajectory optimized

---

Distance- - Parameter Estimation
Performance Parameter Estimation (Static State)
Cost Parameter Estimation (Static State)
(Stalic State)
Dretance from we
0.
Incbreroes Coly eiqure
Dietance from
25
0-4
01
6.1
30 A Queries
100
:50
200
of Queries
Accumulated Query Cost
(a) Parameter Estimation Task
Distance- Linear Dynamical System
Performance - Linear Dynamical System
Cost Linear Dynamical System
(Static State)
(Static State)
(Static State)
0.5
-Biaay O:ty
066
Drotsnne from wP
Yask marformance
00
-hdoeereesCe Cnly
IIQUIE
0.7
Distence
0.8
02
6.6
of Queries
A Quertes
Accumulated Query Cost
(b) Lincar Dynamical System
Distance -unnar Lander
Performance Lander
Cost Lunar Lander (Static State)
(Static State)
(Static State)
0.5
-raaay Only
Comeeiton Only
Teak Parto Purformance
Distance from
Demiiee
0.4
Irdfocross Daly
0.3
0.
Queries
Querles
Accumulated Query
(c) Lunar Lander Task
Distance- Pizza Arrangement
Performance- Pizza Arrangement
Cost Pizza Arrangement (Static State)
(Static State)
(Static State)
0.5
-Garectoons oity
-Drrc Onty
Tassk performancs
Distince
0.4
88
0.1
20
15
100
:30
200
Queries
Queriee
Acoumulated Query Cost
(d) Pizza Arrangement Task
Figure 3: Metrics for the static state condition in which the robot is presented with the same state for all 20 queries. Error bars/regions represent variance across multiple evaluation runs with randomized query states and initial weights. Cost metrics are cut off after 20 queries for the binary-only method in (a) and (d) due to extensive computation times.
according to e (the algorithm's estimated feature weight after each query). Performance is scaled between 0-1, with 0 and I representing the worst and best possible task rewards according to w", respectively. Note that INQUIRE's distance and performance metrics are achieved in the unweighted condition. Cost-vs-Distance and the resulting distance between e and w' after each query. INQUIRE's s metrics in this graph are achieved in the cost- weightee condition.
measures the relationship between the cumulative cost of each query
Figure presents the same three metries for the static-state condition in which all 20 queries must be selected from the same initial state. Finally, we quantify these graphs by reporting the area-under- the-eurve (AUC) metrics for the distance, performance, and cost curves across all tasks. These metrics are available in Appendix C. The AUC metrics indicate that, compared to the baseline meth- ods, INQUIRE results in the best average learning performance (measured both by the distance and performance plots in Figures 2-3) across all domains and dominates learning performance in the static-state domains. INQUIRE also results in the best average distance-to-cost ratio across all domains.
7

---

5
Discussion
The results show the importance of dynamically selecting interaction types according the robot's current state. For example, demonstrations can be highly informative when provided in novel states, but when the robot may only query a teacher from a single state, multiple demonstrations are likely to be very similar (if not identical). As result, receiving multiple demonstrations in static query state uninformative. We see the benefits of dynamically selecting interaction types in Figure 3, where INQUIRE outperforms all single-interaction methods by optimizing both query type and content to maximize the informativeness of the query feedback.
INQUIRE selects the interaction type that, after receiving feedback, minimizes the entropy over its distribution of weight estimates W. This distribution thus serves as a representation of the robot's current model of the task reward. Figure la illustrates how INQUIRE changes the query type as it learns over time (represented by of queries). This particularly evident in the Parameter Estima- tion task, where the algorithm originally requests corrections until it has refined its model of the task reward to a point where preferences become more informative (after 14-18 queries). Overall, dynam- ically adapting to the robot's 's model of the task reward results in better performance than adopting fixed strategy for selecting interaction types (i.e., DemPref, which always requests demonstrations before selecting preferences).
An added benefit of INQUIRE is that it can incorporate cost metric identify cost-aware, infor- mative queries. The AUC metrics for the cost graphs indicates that INQUIRE selects queries that, on average, minimize the cost-to-distance ratio across all domains. We expect that this cost metric is domain-specific and can represent number of human factors that the algorithm should take into account (e.g., the effort involved for human to respond to each query type [8]). The cost metric used in our study thus serves as an example of how INQUIRE can factor in interaction costs.
6 Limitations
Our evaluation is performed using feedback from an optimal oracle. Real human feedback, however, is likely to be at least somewhat sub-optimal and its severity likely depends the interaction type. For example, a non-optimal demonstration may be one that is sufficient but not ideal for completing the task. In contrast, binary rewards offer only two feedback choices to the user, and thus optimal binary reward may indicate the opposite information from what the user intended to convey. These examples illustrate how non-optimal feedback may need to be handled differently depending on the interaction type, and thus, should affect INQUIRE's estimation of information gain. Future work should investigate setting separate values of B (see Eq. 3) for each interaction type, with the goal of reflecting interaction-specific expectations for sub-optimal feedback.
Furthermore, INQUIRE does not yet have the ability to select the state in which it queries the teacher. Prior work in Active Learning has shown that state selection can improve the informativeness of resulting demonstrations [23]. and we expect that optimizing over the query state in addition to query type and content would improve the performance of INQUIRE.
7 Conclusion
We introduced INQUIRE, algorithm enabling robot dynamically optimize its queries and interaction types according to its task knowledge and its state within the environment. We showed that using information gain to select not just optimal queries, but optimal interaction rypes, results in consistently high performance across multiple tasks and state configurations. Future work will include formal user studies to investigate our method's efficacy with people of varied skillsets and comfort with robots; incorporation of novel interaction types and other communication modalities; and alternative representations of the reward function and feature spaces. Moreover, we are excited at the possible extensions others might present by using our open-source framework' for evaluating and comparing active-learning agents across multiple environments and simulated teachers.
htttps///gitabb.cc///////aa////
8

---

Acknowledgments
The authors would like thank the members of HARP Lab, IAM Lab, and the anonymous reviewers for their valuable feedback. This work is supported in part by the Office Naval Research (N00014- 18- 18-1-2503).
References
[1] A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne. Imitation learning: survey of learning methods. ACM Comput. Surv, 50(2), apr 2017. ISSN 0360-0300. doi:10. 145/3054912. URL https://dos http:///dd...o/..../. 1145/3054912.
[2] A. Y. Ng. S. J. Russell, et al. Algorithms for inverse reinforcement learning. In /CML, vol- ume 1, page 2, 2000.
[3] E. Biyik, M. Palan, N. C. Landolfi, D. P. Losey, and D. Sadigh. Asking easy questions: A user-friendly approach to active reward learning. In Conference Robot Learning (CoRL), pages 1177-1190, 2020.
[4] C. Wirth, R. Akrour, G. Neumann, and Fürnkranz. A survey of preference-based reinforce- ment learning methods. The Journal of Machine Learning Research, 18( 18(1):495549999 2017.
[5] T. Fitzgerald, E. Short, A. Goel, and A. Thomaz. Human-guided trajectory adaptation for tool transfer. In Intl. Conf. on Autonomous Agents and MulliA Systems, pages 1350-1358, 2019.
[6] A. Bajesy, D. Losey, M. O'Malley, and A. Dragan. Learning robot objectives from physical human interaction. Proceedings of Machine Learning Research, 78:217-226, 2017.
[7] C. Celemin and J. Ruiz-del Solar. An interactive framework for learning continuous actions policies based corrective feedback. Journal of Intelligent & Robotic Sys., 95(1):77-97, 2019.
[8] Y. Cui, P. Koppol, H. Admoni, S. Nickum, R. Simmons, A. Steinfeld, and T. Fitzgerald. Un- derstanding the relationship between interactions and outcomes human-in-the-lon toop machine learning. International Joint Conference on Artificial Intelligence (ICCAI), 2021.
[9] M. Palan, N. C. Landolfi, G. Shevchuk, and D. Sadigh. Learning reward functions by integrat- ing human demonstrations and preferences. RSS, 2019.
[10] E. BIyIk, D. P. Losey, M. Palan, N. C. Landolfi, G. Shevchuk, and D. Sadigh. Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences. The International Journal of Robotics Research, 41(1):55-67, 67, 2022.
[11] K. Bullard, A. L. Thomaz, and S. Chernova. Towards intelligent arbitration of diverse active learning queries. In IROS, pages 6049-6056, 2018
[12] P. Koppol, H. Admoni, and R. G. Simmons. Interaction considerations in learning from hu- mans. IJCAL, pages 283-291, 2021.
[13] C. Daniel, M. Viering, J. Metz, O. Kroemer, and J. Peters. Active reward learning. In Robotics: Science and Systems, 2014.
[14] D. Sadigh, A. Dragan, S. Sastry, and S. A. Seshia. Active preference-based learning of reward functions. In RSS, 2017.
[15] N. Wilde, D. Kulić, and S. L. Smith. Active preference learning using maximum regret. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10952-10959, IEEE, 2020.
[16] B. Ibarz, J. Leike, T. Pohlen, G. Irving, S. Legg, and Amodei. Reward learning from human preferences and demonstrations in atari. NeurIPS, 31:8011-8023, 2018.

---

[17] K. Li, M. Tucker, E. Bıyık, E. Novoseller, J. W. Burdick, Y. Sui. D. Sudigh, Y. Yue, and A. D. Ames. Roial: Region of interest active learning for characterizing exoskeleton gait preference landscapes. In 2021 IEEE International Conference Robotics and Automation (ICRA), pages 3212-3218. IEFE, 2021.
[18] D. Brown, W. Goo, P. Nagarajan, and S. Nickum. Extrapolating beyond suboptimal demon- strations via inverse reinforcement learning from observations. In International conference machine learning, pages 783-792. PMLR, 2019.
[19] E. BIyIk,
A. Talati, and D. Sadigh. Aprel: A library for active preference-based reward learning algorithms. In 2022 17th ACM/IEEE International Conference on Eluman-Robot Interaction (HRI), pages 613-617. IEEE, 2022.
[20] H. Jeon, S. Milli, and A. D. Dragan. Reward-rational (implicit) choice: A unifying formalism for reward learning. NeurlPS, 2020.
[21] D. Ramachandran and E. Amir. Bayesian inverse reinforcement learning. In IJCAI, volume 7, pages 2586-2591, 2007.
2007.
[22] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym (2016). arXiv preprint arXiv:1606.01544 2016.
[23] M. S. Lee, H. Admoni, and R. Simmons. Machine teaching for human inverse reinforcement learning. Frontiers in Robotics and AI, 8: 188, 2021.
[24] Giorgino. Computing and visualizing dynamic time warping alignments in r: the dtw pack- age. Journal of statistical Software, 31:1-24, 2009.
10


Reviews:
[{"review_id": "ulGUnkPLxoG", "reviewer": "Reviewer_pgRm", "summary": "The paper presents a method for optimizing the information gained from interaction with a human teacher over several possible types of interaction that have been studied in prior works. The key contribution is an algorithm that uses greedy selection to determine the interaction type. This step is incorporated into an algorithm that builds on previous work in interactive robot learning to update the learner’s estimation of feature weights. An empirical evaluation of the algorithm shows how interaction types vary over the learning process, in different types of problem, and with non-uniform weighting on interaction types. Finally the manuscript demonstrates that  utilizing multi-type interactions generally outperforms a single interaction type methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "#### Strengths\n- The paper integrates several well-founded interaction methods into a uniform framework.\n- The empirical results of the interaction selection (figure 1) and its accompanying discussion are clear and support the central idea that combining interactions should lead to overall better performance by the learner.\n- In comparisons to learners using single types of interactions, the paper reports multiple performance measures (alignment with ground truth weight vectors and task reward).\n- Result show the algorithm presented generally outperforms comparison points in 3 of 4 tasks and has best mean learning performance of the 6 frameworks evaluated.\n\n#### Weaknesses\n- As discussed in the limitations section, an example of the algorithm implemented with a human user would be valuable. The paper does not discuss how the different interaction type requests would be communicated to the user. As this paper does seem to take this step closer to implementation ‘in the wild’, it would be helpful to address this.\n- The comparison experiments utilize at least 1 demonstration to initiate the learning algorithm. However, there is little discussion of what assumptions are made about the demonstrations. Prior work has shown that suboptimal (A. Colome and C. Torras, “Dual reps: A generalization of relative entropy policy search exploiting bad experiences,” Transactions on Robotics, vol. 33, no. 4, pp. 978–985, 2017.), failed demonstrations (K. Shiarlis, J. Messias, and S. Whiteson, “Inverse reinforcement learning from failure,” 2016; D. H. Grollman and A. G. Billard, “Robot learning from failed demonstrations,” Int. Journal on Social Robotics, vol. 4, no. 4, pp. 331–342, 2012.), and even intentionally bad demonstrations (A. Kalinowska, A. Prabhakar, K. Fitzsimons, and T.D. Murphey, “Ergodic imitation: Learning from what to do and what not to do,” ICRA, pp. 3648–3654, 2021.). Would the performance of the algorithm still hold if the demonstrations were suboptimal or even negative demonstrations?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper presents an integration of several interactive learning methods. The framework could easily be built upon by others in the robotics community to incorporate other interaction modalities and improve learning through HRI.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "No major or minor issues other than the weaknesses outline above.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}, {"review_id": "l_ggXaOzUlC", "reviewer": "Reviewer_rWdS", "summary": "This paper formulates a method for active reward learning from user feedback, where the agent’s goal is to learn the task reward function, while also having control over what type of feedback to query from the user. Four types of feedback are considered: demonstrations, preferences between two trajectories, trajectory corrections, and binary good/bad labels. The paper formulates the choice for feedback queries as an optimization to maximize information gain in the distribution over reward, which is estimated via a discrete set of weight hypotheses.\nResults are presented using a simulated oracle with access to a large set of demonstrations and their rewards to provide feedback using the available modalities. The paper shows very interesting results, where INQUIRE tends to prefer demonstrations initially, and later switches to preferences and other types of feedback, thereby outperforming a learning from demonstration while using less expensive type of supervision.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper looks at an interesting, relevant and under-studied problem.\n- The method is very clean and well motivated.\n- Paper is well written with clear problem formulation, method description, notation etc.\n- Extensive experiments show effectiveness of the method on an appropriate set of simulated tasks.\n\nWeaknesses:\n- It seems that the formulation assumes a reasonably low-dimensional reward, query, and answer space. I am not sure how well this method would scale to more complex high-dimensional tasks, such as manipulation.\n- Did not test on a real-robot problem.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper proposes a well-formulated method to a relevant and well-motivated problem. However, the experimental domains were somewhat simple which casts doubt on the applicability to more complex robotics tasks.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "Questions:\n- It seems a little unintuitive that in the unweighted case the agent often selects preference queries over demonstrations and corrections. Why are they considered more informative?\n- If the feedback provider was able to observe the behavior of the policy in practice over time, like a human bystander would, could they provide more informative demonstrations that target the current failure modes?\n- Would it make sense to compare to a baseline that emulates an online imitation learning algorithm such as DAgger? For example, by rolling in a policy that optimizes the current most likely weights, and then requesting a correction from some random starting state. It seems like one of the obvious approaches to building an online learning system in cases where trajectory-level supervision is possible.\n\nSuggestion:\n- Add some visualizations that better explain the specific tasks and give an intuitive understanding of complexity?"}}, {"review_id": "R4X07G5mv1s", "reviewer": "Reviewer_pRs8", "summary": "A lot of works studied robots learning from various forms of human feedback, e.g., comparison, demonstrations, corrections, etc. Additionally, a lot of effort has been put into how to actively query humans for a specific feedback type. This paper tries to unify different feedback types so that the robot can optimize for both what type of feedback to acquire and what query (of that feedback type) to make.\n\nThe paper is well-written and mostly easy to follow. Multiple experiments have been conducted in simulated domains with simulated oracles. Baseline methods from the literature has been used.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is trying to tackle a problem that has been discussed in the community for a relatively long time now.\n- The framework allows more feedback types than any other paper in this domain (except [15]).\n\nWeaknesses:\n- The paper is trying to solve this famous problem using the most obvious approach: let the system designer decide a cost value for each feedback type. However, the difficulty of the problem is already because it is difficult to come up with those cost values. In that sense, it is not really solving the problem. If we knew cost values, this approach would already be what we would use, because it is already pointed out by earlier work (e.g., [12] or \"Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferences\" by Biyik et al.)\n- Comparisons with baseline methods are useful and needed, but it does not really say much: the baselines are already expected to perform worse than the proposed approach because they do not have the same \"action space\" as the proposed approach. For example, one of the baselines is allowed to only make preference queries whereas the proposed approach can ask for demonstrations, corrections, and binary feedback in addition to preferences.\n- Equations 12 and 13 are confusing. Worse than that, I am concerned the paper makes a mathematical mistake while obtaining and using the set \\Omega. Please see my detailed comments below.\n\n\n\n\nBelow is my detailed review of the paper.\n\n\nA lot of works studied robots learning from various forms of human feedback, e.g., comparison, demonstrations, corrections, etc. Additionally, a lot of effort has been put into how to actively query humans for a specific feedback type. This paper tries to unify different feedback types so that the robot can optimize for both what type of feedback to acquire and what query (of that feedback type) to make. I am very familiar with the literature in this domain, and I know this has been an open problem for some time, which many researchers have been discussing. So I appreciate the authors' effort for trying to address this problem and the contribution.\n\nHowever, the paper is trying to tackle this problem through the most obvious / straightforward way: \"Let the system designer decide the cost for each particular feedback type, and then maximize informativeness under some cost constraint, or maximize the difference\". I think the difficulty of the problem is not in optimizing over different feedback types or learning from various forms of feedback. The difficulty is in coming up with the cost values. Hence, this paper is not really solving the problem. This is my most major concern about the paper.\n\nBelow are my other comments and criticisms:\n1- This is just formatting, but is the paper scanned from hard-copy papers? For some reason, I am not able to select texts in the PDF and some portions of the PDF have lower quality. This makes reviewing on a computer a little painful (and I don't want to print the paper for obvious reasons). Please upload the LaTeX output next time.\n2- In line 22 and in various other places, the paper uses the term \"binary feedback\" to refer to a feedback type where a human says \"successful\" or \"unsuccessful\" for a trajectory. However, \"binary feedback\" is a more general term than that. For example comparisons (or \"preferences\" as used by the paper) are also binary. I suggest using another term instead of \"binary feedback\". Similarly, preferences do not have to be binary: they can be, for example, rankings. But I know the term preference is usually used for pairwise comparisons, so this is okay.\n3- The second paragraph of the Introduction does not mention the mental or physical effort required from the teacher while talking about the differences between the feedback types. But this is really one of the main differences here. This is later discussed, but it should be mentioned here, too, because this is the first place where differences are discussed.\n4- The third paragraph says previous works typically do not work with multiple feedback types. However, [12] and the paper by Biyik et al. that I mentioned earlier handle preferences and demonstrations together.\n5- I do not understand what \"adapting to a state\" means in the last sentence of Introduction.\n6- Section 2 mentions volume removal and information gain based active querying techniques, which have really been the two most dominant techniques (even though recent works, e.g., [1], showed info gain is strictly better). But it would be nice to mention other alternatives too, e.g., max regret optimization. See \"Active preference learning using maximum regret\" by Wilde et al.\n7- Relatedly, in general, the paper is missing many important citations, e.g.,:\n7a) \"Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations\" by Brown et al., which also leverages both demonstrations and rankings.\n7b) \"ROIAL: Region of Interest Active Learning for Characterizing Exoskeleton Gait Preference Landscapes\" by Li et al., which combines preferences with ordinal labels.\n7c) \"APReL: A Library for Active Preference-based Reward Learning Algorithms\" by Biyik et al., which introduces a software library that combines multiple different preference feedback types and demonstrations.\n7d) \"Unified learning from demonstrations, corrections, and preferences during physical human-robot interaction\" by Mehta and Losey. The name should say it all.\n8- Line 88-98 introduce an implicit assumption that the initial state is fixed during a querying iteration, so the robot cannot choose which state the demonstration should start at. This is okay, but this should be clearly and explicitly stated.\n9- If the initial state is fixed, then there is no active querying in demonstrations and corrections. Is this true? If yes, then \"Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferences\" by Biyik et al. already proved that active feedback should be collected after non-active feedback. Why is this knowledge not utilized in selecting feedback types?\n10- It seems demonstrations and corrections are mathematically equivalent, as they are non-active. Is this correct?\n11- In Corr. row of Table 1 and in line 94, I believe \"$q$\" must be replaced with \"$t$\".\n12- Equation 3 reduces to Bayesian IRL, so that paper by Ramachandran and Amir should also be cited.\n13- The paper says $\\beta$ was set to 20. This is meaningful only if there is a magnitude constraint on $\\omega$. Otherwise, one can scale \\omega arbitrarily and $\\beta$ becomes meaningless. Please clarify any constraints on the magnitude of $\\omega$.\n14- The paper says $E$ is a matrix, but it is not. It is a tensor. Relatedly, what does the transpose of a tensor mean in Equation 6? Is it being transposed over the first two dimensions? Please clarify.\n15- Lines 115 through 126 (and equations 4-9) are just an implementation trick, not a novelty. Although I appreciate the fact that it is shared, I think it best fits to the Appendix.\n16- To calculate information gain, the framework uses a discrete set of trajectories of size $T$. For demonstrations (and corrections), this means as $T$ increases, the information gain value (which is approximated) will increase (as can be seen from Equation 9). So the information gain estimates are not unbiased, they are under-estimated. This should be discussed.\n17- In Equation 10, what's the motivation for using $1/\\log(\\lambda_i)$ instead of a simple additive or multiplicative term? Is this somehow more interpretable or intuitive? Please elaborate.\n18- Lines 136 and 137 are concerning. Do they mean the framework does not sample the set $\\Omega$ from the posterior over $\\omega$? The way the paper currently reads tells me that there is a bunch of samples, and then gradient ascent is applied to each sample until they converge (before the next query). What if they all go to the same local maximum (which is unlikely not to happen)? However, if the set $\\Omega$ is obtained in this way, then the samples in it are not really samples from the posterior. In turn, this makes the set useless for information gain computation, because its derivation is based on the fact that $\\Omega$ are the samples from the posterior.\n19- The paper should cite OpenAI Gym's paper for LunarLander. I am not familiar with the Pizza Arrangement domain, but if it is from a literature, that should also be cited.\n20- Experiments do not really say much, because of multiple reasons: (1) the baselines are already expected to perform worse than the proposed approach because they do not have the same \"action space\" as the proposed approach. For example, one of the baselines is allowed to only make preference queries whereas the proposed approach can ask for demonstrations, corrections, and binary feedback in addition to preferences. (2) The simulated humans are oracles. (3) The system has not been deployed on a real robotic system.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The problem is interesting, relevant and important. But the proposed framework is not addressing the main difficulty of the problem, it is introducing an obvious framework for combining different feedback types. Besides, the literature review in the paper is very limited. Finally, experiments are not realistic (e.g., oracle humans and simulated systems). All other concerns are relatively minor.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please see my comments in the \"Strengths and Weaknesses\" section. I moved everything there due to character limits.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}, {"review_id": "6ZEkqD0Vc4B", "reviewer": "Reviewer_DFDJ", "summary": "The paper tackles interactive querying across multiple human interaction modalities for reward learning. The INQUIRE approach selects queries to maximize a cost-weighted information gain about reward feature weights across demonstration, preference, correction, and binary feedback modalities. The paper compares their approach with baselines that use a single human interaction modality or a fixed pattern of interactions in simulated environments with an optimal oracle as a human proxy. The INQUIRE method is able to often outperform baseline methods, especially when the initial state is held static.\n", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The writing and figures are clear and concise.\n- The paper presents a clear description with matrix formulas of how the information gain across querying modalities is efficiently computed.\n- The paper presents clear evaluation metrics and visualizations (# of queries and cost of queries against task performance and feature learning error; querying modality by step).\n- The approach appears competitive with baselines in simulated environments.\n\nWeakness:\n- A weakness is the lack of real robot experiments and a lack of real human user studies—real humans can be significantly different from a Boltzmann-rational model and exhibit multimodal reward preferences. Further, the INQUIRE method performs best relative to baselines in - the more unrealistic static initial state settings.\n- It is unclear if INQUIRE is significantly better than baseline approaches.\n- It is assumed that “good” trajectories can be randomly sampled from the environment.\n- The corrections and binary human response models may be unrealistic (see below).\n- See other issues below.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper tackles the interesting problem of deciding not only which queries to make within a human interaction modality but also which modality to query. The paper solves this problem by elegantly extending past information-gain techniques to make cost-weighted comparisons between modalities. However, without analysis of statistical significance of simulation AUC results, real robot experiments, or human user studies it is difficult to assess how effective the INQUIRE approach is. Further, inconsistencies in the human response model for corrections and binary responses call into question the information gain computation used for these interaction modalities. The work could be significantly improved by (1) clarifying the response model for computing correction and binary response information gain, (2) conducting statistical analysis of the benefits of the approach, and (3) running experiments on real robots and/or with real human teachers.\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- Statistical significance of the AUC metric against baselines should be clearly analyzed.\n\n- It is unclear how corrections differ from demonstrations in terms of information—for both settings, the choice space is the full trajectory space regardless of query. So for the queries to have different value, the human response model would need to be different. Meanwhile, looking at Eqs. 7 and 55, the distribution p(c|w) does not seem to sum to 1, making it hard to understand what the intended response model is. Further, the response model probability p(c|w) for corrections in Eq. 7 doesn’t appear to depend on the similarity between the corrected trajectory and the queried trajectory, which is inconsistent with how the oracle selects trajectories (by maximizing a ratio between the value of a trajectory and it’s dissimilarity with the query). This response model seems very inconsistent with how trajectories would be selected by a human (a corrected trajectory one state away from the query is much more likely to be selected than a completely unrelated trajectory as a correction). The authors should clarify the human response model for computing information gain for corrections, and how this model causes it to differ for the information gain.\n\n- The binary response model seems to assign a very low probability to c=1 when there are multiple high-value trajectories in T. This is inconsistent with the oracle model, which responds with c=1 when a trajectory is in the top 75%, and could artificially deflate information gains.\n\n- Why is gradient ascent used to update the posterior Omega? Gradient ascent could bias the posterior to place undue weight on its modes. Past works approximating information gain / volume removal with posterior sampling have used Markov chain methods to approximate posterior sampling [1].\n\n- The authors should clarify if demonstrations and corrections need to be drawn from the randomly sampled pool T. A user making sufficiently fine corrections / demonstrations would usually want to construct a trajectory that doesn’t randomly lie within the sampled pool.\n\n- Why is the simulated oracle deterministic rather than Boltzmann-rational like the response model for computing information gain?\n\n\n[1] https://escholarship.org/content/qt88k894w7/qt88k894w7.pdf\n"}}]

Author rebuttal:

Decision: Accept (poster)