Eliciting Compatible Demonstrations for Multi-Human Imitation Learning
Authors: Kanishk Gandhi
Siddharth Karamcheti
Madeline Liao
Dorsa Sadigh
Venue: CoRL 2022 Poster (2022)
Keywords: Interactive Imitation Learning
Active Demonstration Elicitation
Human Robot Interaction

Abstract: Imitation learning from human-provided demonstrations is a strong approach for learning policies for robot manipulation. While the ideal dataset for imitation learning is homogenous and low-variance - reflecting a single, optimal method for performing a task - natural human behavior has a great deal of heterogeneity, with several optimal ways to demonstrate a task. This multimodality is inconsequential to human users, with task variations manifesting as subconscious choices; for example, reaching down, then across to grasp an object, versus reaching across, then down. Yet, this mismatch presents a problem for interactive imitation learning, where sequences of users improve on a policy by iteratively collecting new, possibly conflicting demonstrations. To combat this problem of demonstrator incompatibility, this work designs an approach for 1) measuring the compatibility of a new demonstration given a base policy, and 2) actively eliciting more compatible demonstrations from new users. Across two simulation tasks requiring long-horizon, dexterous manipulation and a real-world ``food plating'' task with a Franka Emika Panda arm, we show that we can both identify incompatible demonstrations via post-hoc filtering, and apply our compatibility measure to actively elicit compatible demonstrations from new users, leading to improved task success rates across simulated and real environments.

Eliciting Compatible Demonstrations for Multi-Human Imitation Learning
Kanishk Gandhi, Siddharth Karamcheti, Madeline Liao, Dorsa Sadigh Department of Computer Science, Stanford University (kanishk.gandn skaramcheti, madelineliao, dorsaj@stanford edu
Abstract: Imitation learning from human- human-provided demonstrations is a strong approach for learning policies for robot manipulation. While the ideal dataset for imitation learning is homogenous and low-variance - reflecting a single, optimal method for performing a task natural human behavior has a great deal of her- erogeneity, with several optimal ways to demonstrate a task. This multimodality is inconsequential to human users, with task variations manifesting scious choices; for example, reaching down, then across to grasp an ohject, versus reaching across, then down. Yet, this mismatch presents a problem for interactive imitation learning, where sequences of users improve on a policy by iteratively collecting new, possibly conflicting demonstrations. To combat this problem of demonstrator incompatibility, this work designs an approach for 1) measuring the compatibility of a new demonstration given base policy, and 2) actively eliciting more compatible demonstrations from new users. Across two simulation tasks requiring long-hon moo-zooiooo. dexterous manipulation and a real-world "food plating" task with a Franka Emika Panda arm, we show that we can both identify incompat- ible demonstrations via post-hoc filtering, and apply our compatibility measure actively elicit compatible demonstrations from new users, leading to improved task success rates across simulated and real environments.
subcon-
Keywords: Interactive Imitation Learning, Active Demonstration Elicitation, Human-Robot Interaction
1 Introduction
Interactive imitation learning [1, 3] from pool of human demonstrators is scalable approach for learning multi-task policies for robotic manipulation [4, 6]. Yet, such approaches have a critical problem, especially in the low-to-moderate data regime: data from multiple human demonstrators often have conflicting modes, where two users provide opposing behaviors for a single task behaviors that manifest subconscious, random choices. For example, consider the nut-on-peg on-peg task in Fig. 1: one user (in orange) approaches the nut by moving across the table, then down, while the other user (blue) reaches down, then across.
Training on aggregated batches of data series starting with a base policy, adding small amounts of data from new users, and retraining the policy after each batch - is common in interactive imitation formulations [2, 3]; unfortunately, when we add small number of conflicting demonstrations during the interaction phase, the retrained policy attempts to cover both the base demonstrations and the new set. This leads to incongruent overfitting, where a policy - even one equipped to learn multimodal behaviors [7, 8, 9] tries to fit the base set for most of trajectory, but overfits to the new set for small subset the state space, often with catastrophic failure modes.
To mitigate this problem, this work tackles two questions: 1) how can we measure the compatibiliry between new demonstrator and an existing policy, and 2) how can we use this measure to actively elicit better demonstrations from a new user?
While our approach for measuring and eliciting compatible demonstrations during online collection is novel, prior work has studied the impact of suboptimal demonstrations on learning. Most relevant, Mandlekar et al. [10] introduce RoboMimic, suite of simulated manipulation tasks that consist
Additional videos & results: hitpps//ssites httpps://sites..ooggl htt.a//ss..oogglee.oo//eee/////e wttp://wllittiie
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

of demonstrations collected from 6 humans of varying qualities (Worse, Okay, and Better). This work and recent followups [11] show how policies degrade in the presence of heterogenous data with multimodal behaviors, finding that training imitation learning policies on data from a single demonstrator can often exceed the performance of training on equivalent amounts of data from multiple demonstrators. A related body of work offline policy learning propose approaches for overcoming suboptimality by using extra annotations, usually in the form of rewards or returns for each demonstration. These methods seek to reweight [12, 13, 14], or directly filter out demonstrations [15, 16, 17]. Yet, these are all post-hoc approaches that operate after demonstrations have been collected. None of these prior approaches target the root cause: informing users of the compatibility of their demonstrations, as they collect data in the first place.
In contrast, our proposed approach targets this root cause by first learning a fine- Lne-grained mea- sure of demonstrator compatibility subject to an initial base policy, and then operationalizing this measure to inform users while they collect demonstrations. Our compatibility measure is derived by sorting individual state-action pairs the data based on their likelihood and nov- elty under the base policy. By plotting each state-action along 2D "map" with these two properties as the axes, are able to learn visual boundaries denoting "aligned and compatible" demonstrations (those with high likelihood and low novelty) to those that are grossly incom- patible (low likelihood and low novelty) an approach inspired by prior work in dataset inter- pretability (18, 19]. From these maps, iden- tify scoring function and set of thresholds to define our demonstrator compatibility measure, which we then cheaply and efficiently deploy during active data collection. Crucially, with this fine-graie fineinrained measure, users with "incom- patible" demonstrations are given rich feedback as to where in their demonstration the incom- patibility arose coupled with visualizations of compatible demonstrations from the base pol- icy's training set, guiding their behavior moving forward.
Figure I: Visualization of two users (orange and blue) inserting a nut onto round peg. Both demon- strators approach the round nut, pick it up and insert it into the peg in very different ways, in- troducing heterogenous data that is incompatible with the base policy, hurting performance when retrained.
Our results across two simulated tasks from prior work [10, 20] and real-world "food-plating" task with Franka Emika Panda arm show that our compatibility measure enables the elicitation of more compatible demonstrations demonstrations that, when added the base dataset, lead boost of up to 25% in success rate relative to 20% decrease in performance when collecting data naively, without feedback.
2 Related Work
Identifying & Learning from Diverse Demonstrations. Our definition of diversity subsumes both heterogeneity- where demonstrators capture multiple distinct, but equally optimal behaviors and suboptimality where users provide demonstrations that are poor relative to some objective reward function. On examining the effect of learning from these diverse demonstrations, Mandlekar et al. [10] found that imitation learning models trained on demonstrations from single "proficient" human achieved higher success rates than models trained on significantly larger dataset sourced from mixture of human demonstrators with varying optimality. Gopalan et al. [21] further show that without prompting users with videos or specific subtasks, only a third of demonstrators provide com- patible demonstrations. These works motivate our approach by showing that in naive demonstration collection, heterogeneity and suboptimality are prevalent; instead, we need a direct way to intervene during collection, actively inform users, and guide them to provide compatible demonstrations.

---

Other work studies learning from diverse demonstrations. Several approaches [7, 8, 9, 22, 23] model the multimodality of diverse demonstrations directly, usually by explicitly handling each mode of the data. Unfortunately, these approaches fail in the sequential, interactive imitation learning setting that we study, where we gradually introduce new modes and retrain after each collection. separate line of work tackles learning with suboptimal data through the use of offline reinforcement learning [12, 13, 14], filtered imitation [11, 15, 16, 17], or inverse reinforcement learning [24, 25, 26]. These methods use extra reward annotations for each demonstration, and generally try to reweight or selectively remove suboptimal data. Unlike these approaches, this work tackles the root cause of the problem - how to collect compatible demonstrations from humans in the first place, removing the need to filter out 10s 100s of painstakingly collected demonstrations.
Interactive Imitation Learning. Most of the work in interactive imitation learning attempts to address the problem of covariate shift endemic to policies trained via behavioral cloning. Ross et al. [1] introduce DAgger, method for iteratively collecting demonstrations by relabeling the states visited by policy interactively, teaching the policy to recover. Later work builds on DAgger, primarily focusing on reducing the number of expert labels requested during training. These variants identify different measures such as safety and uncertainty to use to query the expert limiting the amount of supervision user needs to provide [2. 20, 27, 28, 29, 30J. The focus of these works is on learning from a single user rather than from multiple users demonstrating heterogenous behaviors.
Our proposed approach presents novel interactive imitation learning method that learns to actively elicit demonstrations from a pool of multiple users during the course of data collection, with the express goal of guiding users towards compatible, optimal demonstrations.
3 Problem Setting
We consider sequential decision making tasks, modeled Markov Decision Process (MDP) with the following components: the state space S, action space A, transition function T, reward function R, and discount factor 7, In this work, we assume sparse rewards only provided on task completion. state s = (Ogrounded: Oproprio E is comprised of grounded observation (either coordinates/poses of objects in the environment, or an RGB visual observation) and the robot's 's proprioceptive state. An action a E A is a continuous vector of robot joint actions. We assume access to a small dataset of trajectories Drase
{m1, 72.-. TN) where each trajectory T; is sequence of states and actions, ar)] We train an initial policy Thase on this dataset via behavioral cloning.
= ((11, 41)
Our approach has two components: 1) developing a measure of the compatibility of new demonstra- tion with an existing base policy, and 2) building a method for actively eliciting compatible demon- strations from users. For the first component, we learn measure M(Drase base, (Snew, (new )) [0. that defines a compatibility at the granuality of a state-action pair. For the second component, we use our fine-graines measure M to provide rich feedback to users about their demonstrations, in addition to deciding whether to accept/reject new demonstration Tnow. The set of new demonstrations the user collects comprises D after each collection step, we train new policy Tnew on D U Dnew.
Under our definition of compatibility (and the measure M we derive), we hope to guide users to provide new datasets Dnew such that nnww will have as high of an expected return as possible, and minimally, a higher expected return than Thase- In other words, our definition of compatibility asks that new data should only help, not hurt performance relative to the initial set.
4
Learning to Measure Compatibility in Multi-Human Demonstrations
We first derive general compatibility measure M given base set of demonstrations, then evaluate our measure through a series of case studies grounded in real, user- providd demonstration data.
4.1 Estimating Compatibility and Identifying Good Demonstrations
An idealized compatibility measure M has one role estimating the performance of a policy Tusee that retrained on the union of known base dataset Druse and new dataset Dnew- Crucially, M needs to operate at granular level (ideally at the level of individual state-action pairs), without incurring the cost of retraining and evaluating Thase on the new dataset. Phrased this way, there is a clear connection to pool-based active learning [3 J. informing a choice of a plausible metrics that could help predict downstream success. While many metrics could work, we choose two easy-to-comput metrics that lend themselves well to interpretability: the likelihood of actions anew in Dnew under Tbase measured
3

---

High
High
NOVELTY
NOVELTY
61 Rejected
128 Rejected
High
LIKELIHOOD
High
LIKELHOOOD
Low
(a) Compatible Operator
(b) Incompatible Operator
Figure 2: Plots, "maps" of the likelihood and novelty values for demonstrations recorded by the blue compatible operator (a), and the orange incompatible operator with respect to the base policy Thase- The gray border indicates a filter with our chosen thresholds (λ, n)) The orange operator performs far more low-likelihood actions in familiar states, earning lower compatibility score.
by the negative mean squared error between actions predicted by Thase (enew) and Onew well as the novelty of a given state, measured by the standard deviation in the predicted actions under the base policy. To turn these two metrics into single measure, we borrow from interpretability and active learning literature [18, 19]; we plot, or "map" on a 2D plane the likelihood and novelty values for each trajectory in dataset, with the goal of identifying two thresholds (X, 17) on likelihood and novelty thresholds that allow us to identify a given trajectory based on its compatibility.
The final piece is setting the thresholds for novelty (1) and likelihood (A). While this can be done in many ways often just by looking at the "mapped" dataset on the 2D plane [18] - in this work, we assume the ability to obtain or collect a handful of priori incompatible demonstrations that we can use to build contrast sets to regress a threshold. We define compatibility M(Drases (Snew anew )) [0, as parameterized by thresholds T)):
min
(Thar (Seew)
SaveetyySSeevi <
M
otherwise.
A state-action pair is compatible (M(Dowee (Snew; Onew )) 1) if the likelihood of Onew under the base policy is high, or the novelty of Snew 17, incompatibility defined smoothly on the inverse interval with a state-action pair being completely incompatible if (MMDrusee (Snew, anew )) = 0) when (Thase (Nnew) - (new)² )2 X. Intuitively, this measure mirrors what one expects when collecting new data in states where the policy is confident about what actions to take, reject other actions, and new states where the policy is uncertain, more, diverse data good. The following subsection validates this measure against demonstration data collected by real users series of tasks, showing how we can use M as filter to boost policy performance the presence of a large dataset of heterogenous demonstrations.
4.2 Case Studies and Experiments
To validate our measure M, we consider three tasks in simulation, each with a set of demonstrations collected by multiple operators. For each task, choose one operator initialize the base set of demos Drese and use trajectories from another operator to form Dnew. For more detailed summary of the tasks datasets, and policy training procedure, please consult the Supplementary Material.
Square Nut [10]: The goal the task to place a square nut onto a square peg. We use the environment and demonstrations collected by Mandlekar et al. [10], consisting of 200 demonstrations from proficient operator, and 50 demonstrations each from 6 operators of varying qualities. We initialize D with 50 demonstrations from the proficient operator, and use demonstrations from each of the 4 more efficient operators as the different Dnew.
Round Nut [10, 20]: Similar to the prior task, the goal is to place a round nut onto round peg (as Fig. 1). For this task, we collected 30 demonstrations from a proficient operator and 30 demonstrations each from 3 other operators where the data provided for one operator is taken

---

Square Nut
Round Nut
Hammer
Placement
Operator
Naive
M-Filtered
Naive
M-Filtered
Naive
M-Filtered
Base Operator
38.7 (2.1)
13.3 (2.3)
24.7 (6.1)
Operator
54.3 (1.5)
61.0
26.7 (11.7)
(12.)
38,0 (2.0)
39.7 (4.6)
Operator
40.3 (5.1)
(2.0)
22.0
(5.0)
33.3 (3.1)
32.7 (6.4)
Operator 3
37.3 (2.1)
42.7 (0.6)
17.3 (4.6)
18.0 (13.9)
8.0 (0.0)
12.0 (0.0)
Operator 27.3 (3.5)
37.3 (2.1)
7.3 (4.6)
13.3 (1.2)
4.0 (0.0)
4.0 (0.0)
Table I: Success rates (mean/std across 3 training runs) for policies trained on Dnew from different operators. M-Filtered denotes filtering by compatibility scores.
directly from Hoque et al. [20]. We initialize Dosse using the proficient operator's demonstrations, and we randomly sample 10 demonstrations from each operator get different Duew-
Hammer Placement [32]: Here the goal to open a drawer and place a hammer inside (see Fig. 3). We collected 25 demonstrations from a proficient operator to initialize the Druse and 5 demonstrations from each of other operators to get different Deew-
Policy Architecture, Training, and Evaluation. We train an ensemble of 5 MLP policies using coordinate-based state observations and the robot's proprioceptive state to predict 7-DoF joint actions. Models are trained for 1000 epochs, and evaluated every 200 epochs; we select our policy by taking the checkpoint with the highest success rate averaged over 50 rollouts.
Naïve Training. To establish baseline performance, we compare the performance of policies trained on different Drew given by different operators identify meaningful markers. The results in Table 1 show a large difference in the success rate of Tnew depending on which operator provided Dnew- For example, on Square Nut, sampling Dew from Operator (the same operator who provided Drase) leads to an improvement of 15.6% over the base policy, while sampling from Operator 4 leads reduction of 11.4%. Similarly, Operator (again, the operator who provided Doase) in Round Nut (the blue line, Fig. 1) almost doubles the performance of the base policy while Operator 4 (orange line in Fig. 1) halves it. general. training on data with incompatible demonstrations can degrade policy performance by seemingly arbitrary amounts.
Case Study:
Compatibility-Drive Filtering. Finally, evaluate the ability of our measure M to identify compatible demonstrations from fixed pools of demonstration data. We first note the result of "mapping" the various demonstrators to provide some additional visual intuition, grounded in the case study results; Fig. 2 shows the 2D "maps" for Operator (orange) and Operator 4 (blue). These plots (see the Supplemental Material for plots of all operators), present a key insight: incompatible operators provide more unlikely actions for familiar, or less-novel, states. Closing the loop and using our measure M to filter out incompatible demonstrations, we see strong results: Table shows that removing incompatible samples leads to performance that is at least as good, if not much better than the base policy. For instance, we see an improvement of 6.7% for Operator (the most compatible), on the Square Nut task and of 10% for Operator 4 (the least compatible).
Experiment:
5 Actively Eliciting Compatible Demonstrations
Our pipeline starts traditionally: operators are provided a task specification (e.g.. "place the square nut on the square peg") and are given three episodes of practice with the robot. We then enter our active elicitation process (see Fig. 3 for full overview of our interface). First, is prompting; each operator is shown 5 rollouts from Dosse to understand the style of the base operator. Second is demonstration, where the operator is tasked with providing demonstration similar to that which they were just shown. Critically, as they record their demonstration, we use our measure M to provide online feedback a green indicator for compatible actions, and red indicator for incompatible actions.
After demonstrating, begins the feedback phase; if the number of incompatible aate-action pairs (Nnew; (new)) pairs with M(Drases (Snew; (new) )) 0 exceeds 5% of the accepted demonstration length, then the demonstration rejected. Rather than a binary decision, we take the opportunity to provide rich feedback: using simple sliding window, we identify 3 candidates with the highest average incompatibility from the rejected demonstration. We show this to the user, along with a demonstration from Drase retrieved based on state- stiieaiim to the 3 candidates. With this contrastive
5

---

(c) Corrective feedbback demonstation rejected
(a) Operators ure shown five demoestrations from the initial set of demonstrations that the policy was trained on.
Incompatible parts trajectory replayedd
(b) The operators receive oaline visual feedhack on compatibility of their demonstration with the buse policy (green comparible and red is not).
(ii) Expert demo with simmlar states played back
Figure 3: The three phases our active elicitation interface spanning the initial prompting phase (a), the subsequent demonstration phase with live feedback (b), and finally, the feedback phase (c).
feedback, we find that users are able to more effectively pinpoint where in the demonstration they went wrong, and learn how to provide more compatible demonstrations moving forward. We continue this process until desired number demonstrations have been recorded. Finally, Thase is updated to Tnew every time batch of demonstrations is collected from demonstrator.
5.1 User Studies Simulation
We build browser interface for users to provide teleoperated demonstrations for two tasks in simulation: Round Nut and Hammer Placement. We conduct user studies in two conditions, naive and informed. Under the naive condition, users get practice the controls during practice episodes, before collection demonstrations without any additional prompting. Under the informed condition, users are provided the active elicitation interface above.
Round Nut. Drase consists of 30 trajectories collected by a proficient operator. The users are randomly assigned to one of the two conditions, and are asked collect 10 successful demonstrations Dnow. We conduct user studies for this task with n = 16 participants.
Dowe consists of 25 trajectories collected by a proficient operator. For this task, we perform a longitudinal study with n = participants, where users are first asked to collect demonstrations in the naive condition and then collect demonstrations in the informed condition. This allows us to measure the effect of the interface in eliciting demonstrations within subjects.
Hammer Placement.
Results. For both tasks, we find that policies trained on demonstrations collected via active elicitation consistently outperform those from the naive condition. Our studies indicate that naive participants find it difficult to provide compatible demonstrations, worsening the base policy average. In contrast, informed demonstrators improve the base policy, beating the naive condition by 6. 6.1% on Round Nut and 11% on Harmer Placement. From our longitudinal studies the latter, find that informed demonstrations beat naive demonstrations from the same user for 3/4 operators, clearly indicating that active elicitation is a strong approach for interactive imitation. We also find that the length trajectories recorded by informed demonstrators is closer to those the base demonstrator (see Supplementary). To test if the high frequency feedback from interface discouraged users from providing corrections, we also perform experiments with collecting interactive, human-gated (Hg-DAgger [2]) demonstrations, Informed data collection outperforms naive data collection for out of 4 users even in this setting and informed users intervene at about the same rate as informed ones (see supplementary for details).
5.2 User Studies Real Robot
Food Plating. As depicted in Fig. 4, the task requires the Franka Emika Panda arm to lift the pink pan from "stove" and guide it over plate, where it is responsible for transferring fried egg

---

Task
Base
Naïve
Naïve Filtered
Informed
Round Nut
13.3 (2.3)
9.6 (4.6)
9.7 (4.2)
15.7 (6.0)
Hammer Placement
24.7 (6.1)
20.8 (15.7)
22.0 (15.5)
31.8 (16.3)
[Real] Food Plating
60.0
30.0 (17.3)
85.0 (9.6)
Table 2: Success rates (mean/std across users) for the user studies evaluating both naive and informed demonstration collection against base users.
from the pan the plate without dropping it on the table. The location of the plate varies across episodes, and the robot has to learn to move and tilt the pan in a way that the egg gently slides onto the plate. Druse consists of 15 demonstrations collected by single, proficient operator that maintains a consistent strategy of lifting the pan and tilting it forward ("vertically plating") to transfer the egg.
Policy. We train ensemble of 5 policies that predict 7-DoF joint actions from visual observations (RGB images) and proprioceptive states. We use ResNet-34 backbone pretrained on ImageNet [33] to encode the visual observations, followed by single MLP to predict actions from the concatenated ResNet embedding and proprioceptive state (see the Supplementary Material for additional details). We select our final model based on the best validation loss (MSE) after 20 epochs of training.
Experimental Setup. We build an analog of our browser interface for active elicitation in a console UI. Unlike simulation, users provide demonstrations kinesthetically by physically moving the robot arm. Before collecting demonstrations, users are time to explore the controls of the robot and practice moving it. We conduct within-subjects study with 4 users who first perform the naive collection of 5 demonstrations, then collect 5 demonstrations using the active elicitation interface.
Results. Training on demonstrations collected with active elicitation outperform the base policy by 25% and the naive condition by 55%. Additionally, similar to simulation, we see that naive demonstration collection often hurts policy performance.
6 Discussion
Driving this work has been a simple story; in $4 we performed thorough analysis and set of experiments on pre-existing datasets show that heterogeneity is a very real problem in multi- human imitation learning datasets, one that can harm policy performance in arbitrary ways. Through this section, building up to $4.1 we were able to profile these existing datasets, and regress a straightforward compatibility measure M that hinged on two attainable features of the base policy the likelihood and novelty of a new state-action pair under the existing model parameters. Given these case studies and our proposed measure, we turned to active elicitation in $5, building an interface to guide users towards producing more compatible demonstrations, as depicted in Fig. 3.
On our simulated tasks (Table 2), we showed that an uninformed collection procedure no prompting, no fine-graines feedback -cads unequivocally worse performance than the base policy downstream, due to the introduced heterogeneous data. However, with our active elicitation and prompting behavior, we see the opposite- more data leads significant improvements over the base policy, by broad margin. This story is only further confirmed with our experiments on a real robot, as we had users provide demonstrations kinesthetically, showing more extreme results. On the seemingly simple "food plating" task, uninformed demonstration collection led to broad spectrum of behaviors (e.g.. the sideways plating depicted in Fig. vs. the idealized vertical plating of the base policy) driving the success rate down almost 30% relative to the base policy. In contrast, the same users with just 1-2 episodes of feedback - were able to learn to provide compatible demonstrations, with their "informed" demonstrations leading an absolute policy success rate of 85%. Not only is the active elicitation procedure highly eifective, but it is fundamentally cheap, generalizable across various control schemes (e.g., the discrete keyboard control of simulation vs. the kinesthetic teaching in the real world), and is broadly applicable across various types of manipulation tasks.
Limitations. We make several assumptions in this work that may impede future applications. Our biggest assumption is that the preferences reflected in the initial dataset are the "ground-truthh" to aspire to. In settings where users have specific, conflicting preferences about how to accomplish task, our approach falls short. Consider the age-old question of when to pour milk into a cereal bowl- - namely, before or after the cereal has made to the bowl. Under approach, given base dataset
7

---

egg
(a) Successful rullout of policy trained on demonstrations from an informed operator
(b) Failed rollout of policy trained on demonstrations from a naive operator
Figure 4: Evaluation trajectories for the Informed (a) & Naive (b) conditions. In (b), the user provides conflicting demos (a "sideways tilt," moving laterally and rotating sideways) compared to the initial demos ("vertical plating"). The resulting policy moves sideways and forward before failing.
of cereal being poured into the bowl first, every subsequent user trying to teach our system despite their preferences will be guided towards "cereal-first" approach. This paper focuses on tasks that do not have this quality tasks where the order in which one picks up a nut, or opens a cupboard is inconsequential but we hope that future work expands on our initial formulation, and allows for cases where users with strong preferences can override the default demonstration set. In cases where users do have strong preferences, we are instead able to allow users make an informed choice about whether they want to spend the time collection more demonstrations to reflect their preferences. Compared to the status quo where users just don't 't know their demonstrations are incompatible in the first place, our approach provides them with more feedback to make an informed choice when it comes to demonstration collection.
The second limitation stems from our assumption of access to the initial dataset of demonstrations for training the base policy. Access this data is necessary for policy retraining, well as for developing our compatibility measure M. One could imagine scenarios where (due to privacy or release feasibility) this initial dataset cannot be made available. While we could obtain reasonable proxies to use in our method (e.g.. by rolling out several times in an environment to collect new dataset), performance of the learned policy would likely suffer. Luckily, methods for continual learning and model interpretability reaching within the black-box are growing, presenting starting point for how to proceed in cases where initial datasets are not available.
Looking Forward. A pivotal idea of this work breaking the traditional influence pattern in human-robot interaction, where human influences a robot (by providing demonstrations, feedback, preferences). There is power in going the other way allowing the robot to guide and prompt humans to the type of data needed to improve. Future work will build upon our Hg-DAgger results, refining our interface for active elicitation for more responsive interactive imitation algorithms. Other work might consider language, providing users with higher-level feedback ("no, reach across first!") for more effective guidance.
Conclusion
We introduce two-phase approach for multi-human interactive imitation learning that 1) derives a fine-grained measure of demonstrator compatibility with an existing policy, and 2) develops an interface for actively eliciting compatible demonstrations from new users. Through our user studies on two diverse simulation environments, and a real-world Franka Emika Panda the importance of demonstrator compatibility. We show the inefficiency of naive, unprompted demonstration collection and more importantly, we show that spending time to prompt users with fine- grained feedback about the demonstrations they provide is extremely useful, boosting the performance of the task policy by up to 25% absolute success rate from just demonstrations.
demonstrate

---

Acknowledgments
This project sponsored by NSF Awards 19417222. 2132847, and 2218760, the Office of Naval Research (ONR), and the Air Force Office of Scientific Research (AFOSR). Toyota Research Institute ("TRI") additionally provided funded funds to support this work. Siddharth Karamcheti is further grateful to be supported by the Open Philanthropy Project Al Fellowship.
References
[1] S. Ross, G. Gordon, and A. Bagnell. reduction of imitation learning and structured prediction to no-regret online learning. In Artificial Intelligence and Statistics (AISIATS), 2011.
[2] M. Kelly, C. Sidrane, K. Driggs- Cangpbell and M. J. Kochenderfer. HG-DAgger: Interactive imitation learning with human experts. In International Conference on Roboties and Automation (ICRA), pages 8077-8083, 2019.
[3] J. Spencer, S. Choudhury, M. Barnes, M. Schmittle, M. Chiang, P. J. Ramadge, and S. S. Srinivasa. Learning from interventions: Human-robot interaction as both explicit and implicit feedback. In Robotics: Science and Systems (RSS), 2020.
[4] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. BC-Z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning (CoRL), 2021.
[5] Ebert, Y. Yang, K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and S. Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. In Robotics:
Science and Systems (RSS), 2022.
[6] M. Ahn, Brohan, Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan,
K. Hausman, A. Herzog, D. Ho. J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, C. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, S. Levine, Y. Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao, K. Rao, J. Rettinghouse, D. M. Reyes, P. Sermanet, N. Sievers, and M. Yan. Do as I can, not as I say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691. 2022.
K. Jeffrey, S. Jesmonth, N. J. Joshi,
Tan, A. Toshev, V. Vanhoucke, F. Xia, Xiao, Xu, S. Xu,
[7] C. M. Bishop. Mixture density networks. In NCRG Report, 1994.
[8] C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet. Learning latent plans from play. arXiv preprint arXiv:1903.019773 2019.
[9] R. Florence, C. Lynch, A. Zeng, O. Ramirez, A. Wahid, L. Downs, A. S. Wong, J. Lee,
I. Mordatch, and J. Tompson. Implicit behavioral cloning. In Conference on Robot Learning (CoRL), 2021.
[10] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, Fei-Fei, S. Savarese, Y. Zhu, and R. Martín-Martin. What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning (CoRL),, 2021.
[11] M. Beliaev, A. Shih, S. Ermon, D. Sadigh, and R. Pedarsani. Imitation learning by estimating expertise of demonstrators. In International Conference on Machine Learning (ICML), 2022.
[12] X. B. Peng, A. Kumar, G. H. Zhang, and S. Levine. Advantage- weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:abx//110.00177 2019..
[13] Z. Wang, A. Novikov, K. Zolna, J. T. Springenberg, S. E. Reed, B. Shahriari, N. Siegel, J. Merel, C. Gulcehre, N. M. O. Heess, and N. de Freitas. Critic regularized regression. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[14] M. Yang, S. Levine, and O. Nachum. Trail: Near-optimal imitation learning with suboptimal data. In International Conference on Learning Representations (ICLR), 2022.
[15] S. Emmons, B. Eysenbach, I. Kostrikov, and S. Levine. Rvs: What is essential for offline RL via supervised learning? In International Conference on Learning Representations (ICLR), 2022.

---

[16] J. Grigsby and Y. Qi. A closer look at advantage filtteee behavioral cloning in high-noise noise datasets. arXiv preprint 110.2211000449,,, 2021.
[17] Zhang, Z. Cao, D. Sadigh, and Y. Sui. Confidence-awarc imitation learning from demonstra- tions with varying optimality. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[18] S. Swayamdipta, R. Schwartz, N. Lourie, Y. Wang. H. Hajishirzi, N. A. Smith, and Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Empirical Methods in Natural Language Processing (EMNLP), 2020.
[19] S. Karamcheti, R. Krishna, L. Fei-Fei, and C. D. Manning, Mind your outliers! investigating the negative impact of outliers on active learning for visual question answering. In Association for Computational Linguistics (ACL), 2021.
[20] R. Hoque, A. Balakrishna, E. R. Novoseller, A. Wilcox, D. S. Brown, and K. Goldberg. ThriftyDAgger: Budget-aware novelty and risk gating for interactive imitation learning. In Conference Robot Learning (CoRL), 2021.
[21] N. Gopalan, N. Moorman, M. Natarajan, M. C. Gombolay, and Georgia. Negative result for learning from demonstration: Challenges for end-users teaching robots with task and motion planning abstractions. In Robotics: Science and Systems (RSS), 2022.
[22] K. Hausman, Y. Chebotar, S. Schaal, G. Sukhatme, and J. J. Lim. Multi-modal imitation learning from unstructured demonstrations using generative adversarial nets. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
[23] A. Kuefler and M. Kochenderfer. Burn-in demonstrations for multi-modal imitation learning. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2018.
[24] D. S. Brown, W. Goo, P. Nagarajan, and S. Niekum. Extrapolating beyond suboptimal demon- strations via inverse reinforcement learning from observations. In International Conference on Machine Learning (ICML), 2019.
[25] L. Chen, Paleja, and M. C. Gombolay. Learning from suboptimal demonstration via self-supervised reward regression. In Conference on Robot Learning (CoRL), 2020.
[26] V. Myers, E. Biyik, N. Anari, and D. Sadigh. Learning multimodal rewards from rankings. In Conference on Robot Learning (CoRL), 2021.
[27] B. Kim and J. Pineau. Maximum mean discrepancy imitation learning. In Robotics: Science and Systems (RSS), 2013.
[28] K. Menda, K. Driggs-Campbell and M. Kochenderfer. Ensembledagger: A Bayesian approach to safe imitation learning. In International Conference on Intelligent Robots and Systems (IROS), pages 5041-5048, 2019.
[29] Zhang and K. Cho. Query-efficient imitation learning for end-to-ent autonomous driving. In Association for the Advancement of Artificial Intelligence (AAAI), 2017.
[30] R. Hoque, A. Balakrishna, C. Putterman, M. Luo, D. S. Brown, D. Seita, B. Thananjeyan, E. R. Novoseller, and K. Goldberg. LazyDAgger: Reducing context switching in interactive imitation learning. IEEE 17th International Conference on Automation Science and Engineering (CASE), 3:502-509, 1092,509,400 202.
[31] B. Settles. Active learning literature survey. Technical report, University of Wisconsin, Madison, 2009.
[32] Y. Zhu, Stone, and Y. Zhu. Bottom-up skill discovery from unsegmented demonstrations for long-horizon robot manipulation. IEEE Robotics and Automation Letters (RA-LJ, 144126-4133 2022.
[33] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Computer Vision and Pattern Recognition (CVPR), 2016.
10

---

[34] L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization, arXiv preprint arXiv: 2016.
[35] N. Srivastava, G. Hinton, A. Krizhevsky, Sutskever, and R. Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. Journal of Machine Learning Research (JMLR), ((1):19229-158 2014.
[36] D. Kingma and J. Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations (ICLR), 2015.
[37] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735-1780, 1997.
[38] Y. Lin, A. S. Wang, G. Sutanto, A. Rai, and F. Meier. Polymetis. https://facebokkesserch. 2021.
[39] D. Hendrycks and Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv: 160::1660 2016.
[40] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019.
[41] S. G. Hart. NASA task load index (nasa-tix); 20 years later. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, volume 50, pages 904-908, 2006.


Reviews:
[{"review_id": "xYXO94qc1ol", "reviewer": "Reviewer_hqZQ", "summary": "This paper proposes an interactive method to inglobate new demonstrations into an existing dataset and policy in such a way that it is not detrimental to the existing policy. The authors propose to do so by measuring how similar and compatible the new demonstration is with the existing policy, and discarding it in case the algorithm finds it too dissimilar. They also extend this technique to provide online feedback to a human operator on how to better record the demonstration.\nThe method is based on the assumption that, in a state already visited, the new demonstration should be close to the existing policy. In a novel state, however, new data is always good. The algorithm therefore computes at each step a novelty and consistency measure based on the existing policy and dataset.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I believe the problem this paper tackles is important, and methods to have interactive learning between robots and human may be beneficial. Although in the future we may want to learn complex policies from heterogeneous data, at the moment robot learning can't scale datawise as other fields, and therefore these techniques are still fundamental.\nStrengths:\n- Interesting problem, tackled with a simple but well designed approach. \n- The experiments section is well written, and indeed shows the effect of an incompatible additional dataset, showing that in this scenarios more data is not always good. Real robot experiments also further prove the effectiveness of the algorithm.\n- The interactive algotithm they propose is interesting and potentially helpful especially at larger scale.\nWeaknesses:\n- Tasks are still relatively simple and do not involve the use of longer horizon skills.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I suggest to accept the paper. The method presented is intuitive but effective and can be beneficial also in larger scale settings. While the tasks do not show particularly complex skills, I believe they still prove the improvements that the algorithm brings, also in the real world.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "No particular issue. I would suggest for future extensions to also test it on more challenging, longer horizon tasks.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "TPJjg3GSyOf", "reviewer": "Reviewer_Esih", "summary": "The paper proposes an interesting paradigm in which, to improve the consistency among the multiple demonstrations of a task, a measure of compatibility is used to guide the demonstrator. This guidance results in a consistent set of demonstrations, which in turn results in increased success rates. As a result, by eliminating the heterogeneity in the demonstration set, the learning is more efficient since multi-modal models are not required, and at the same time, tasks become more prone to succeed. The method was demonstrated in simulation on a simple task of inserting a nut onto a peg, and in a real robot on a task of placing food on a plate.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is clearly written and easy to follow. It has a straightforward structure. There are certainly merits of the idea of decreasing heterogeneity of the task and the paper motivates it quite well. Regarding the weaknesses, please, refer to the section on issues.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The experiments are too simple and do not seem to justify the complexity of the method. As it is, the narrative of the paper clashes with a body of literature on multi-modal learning but does not convincingly justify the paper's choice of approaching demonstration via loss of heterogeneity. I think there are many ways to improve the paper (see issues section) but this would require significant changes which are more appropriate for a re-submission.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "One of the main issues with this paper is that it is easy to counterargue two of the main claims on which the contribution of the paper hinges on. First, that by suppressing heterogeneity the method could be potentially moving itself towards collecting a poor, local minima prone dataset of demonstrations. Many works try to do the opposite, which is to learn the richness of human variability via multi-modal models. Second, if the system is already so informed to the point it is capable of measuring the demonstration adequacy and providing feedback, then what is the actual reason to collect more demonstrations in the first place?\n\nTo improve upon the first issue, the paper could leverage insights from the literature on, for example, virtual fixtures and virtual guides [r1, r2]. In virtual guides, experiments motivate well the fact that although users may provide multi-modal demonstrations, it is clear that only a single mode is allowed, for example, in surgical tasks.\nThe tasks used in the current paper could be reformulated such that it becomes more evident the need for uni-modality. For example, a more convincing story could be written by changing the sentence from \"This multimodality is inconsequential to human users, with task variations manifesting as subconscious choices; for example, reaching down, then across to grasp an object, versus reaching across, then down.\" to something along the lines \"Although non-trained users may find natural to demonstrate a task in a variety of ways, there are certain tasks for which a single pattern is more preferable. Such patterns can be captured initially by expert demonstrations which are then used to provide the metric M which will guide subsequent users on how to demonstrate.\"\n\nTo improve upon the second, it would be useful to show why the data used to compute the likelihood in M is not enough to solve the task. For example, would it be possible to show that the current demonstrations to place the egg on the plate are too uncertain and thus more demonstrations are required? I think a look into active learning on robot demonstrations [e.g. r3] can provide some interesting insights to motivate the need for demonstrations.\n\nMinor comment\n======================\nUnder excessive correction, will users become less prone to demonstrate? Wouldn't it be better to let them demonstrate freely, and then use M to give single final feedback on the demonstration over the entire episode, rather than provide high-frequency feedback at each time step?\n\n[r1] https://www.telerobotics.utah.edu/uploads/Main/Abbott_RoboticsResearch07.pdf\n[r2] https://hal.archives-ouvertes.fr/hal-02158827/file/main_final.pdf\n[r3] https://proceedings.mlr.press/v78/maeda17a.html"}}, {"review_id": "ApR2TPLLyb9", "reviewer": "Reviewer_tzdZ", "summary": "This paper presents an approach for measuring and actively soliciting \"compatible demonstrations\" in interactive imitation learning (IL) from multiple humans, where compatibility is defined relative to the behavior of the initial demonstrator. As opposed to prior work that handles the issue of suboptimal and multimodal humans in IL in a post-hoc manner (after the dataset has been collected), this work proposes to influence the data collection process itself toward the collection of more compatible demonstrations. They define a novel compatibility metric over state-action pairs as a synthesis of (1) action likelihood and (2) state novelty relative to the initial policy, and they apply this metric to both filtering data and actively soliciting compatible data. In user studies in simulation tasks and a real food-plating task, they show that the active method achieves higher success rates over a naive approach in all tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper tackles an important problem in imitation learning, i.e., that of multimodal and suboptimal human demonstrations, especially in the multiple human setting. The high-level approach is compelling and novel: instead of the conventional approach of processing human data after it has been collected, the robot can actively influence the human to provide the data in a compatible manner. This is certainly a promising and intuitive strategy for mitigating the compatible demonstration problem. The paper is very well-written, clear, and easy to follow, and the figures are good for providing intuition as well. The experiments are convincing and include both real user studies and real robot experiments.\n\nMy main reservations are with the technical aspects of the approach. Firstly, although the overall framework of interactively querying for compatible demonstrations is novel, the actual mechanism for doing so is not particularly novel. Specifically, they propose using the likelihood of actions and the novelty of states under the base policy, neither of which is a novel metric. Secondly, the authors do not provide sufficient technical detail. I was not able to find the details of the likelihood and novelty computations in the main text or supplemental material: I’m assuming novelty is computed as the policy output variance across the ensemble of neural networks, but what does “likelihood” mean? Why does it only take the action as input, when likelihood seems to be a state-dependent measure (a particular action can be more likely at a given state than another one)? Furthermore, how exactly are the thresholds $\\lambda$ and $\\eta$ calculated, and what are the exact values they take in the experiments? Clarifications here would be appreciated. Thirdly, I have some reservations regarding the overall soundness of the approach. (1) In interactive IL, the policy is iteratively improved by new intervention data. Measuring compatibility against the base policy $\\pi_{base}$ seems to prevent future policies from significantly outperforming the base policy, which runs counter to the objective of interactive IL. Is the $\\pi_{base}$ here iteratively updated to $\\pi_{new}$ when computing compatibility in later rounds of interactive learning? (2) Currently, high novelty demonstrations are deemed compatible, but this seems flawed - if multiple operators are providing demonstrations in novel regions of the state space, aren’t they free to provide demonstrations that are incompatible with each other?\n\nSome more notes:\n- Line 95: “these approaches fail in the sequential, interactive imitation learning setting that we study.” Is there justification or a source for this claim? I don’t readily see why an approach like Implicit BC wouldn’t work on the aggregated dataset $D_{new}$.\n- Line 212: an incompatible state-action pair is said to have $\\mathcal{M} = 0$. Isn’t $\\mathcal{M}$ a continuous measure that’s unlikely to be exactly 0?\n- Line 216: The feedback phase uses state-similarity to retrieve relevant snippets of demonstrations. How is this done in image space? L2 pixel distance? \n- The active informed algorithm makes several modifications compared to the naive baseline, but is only compared in a composite manner. It would be good to include some ablations or additional baselines for more convincing experimental results. For instance, experiments with only the prompting phase (showing operators the base demonstrations) to see if the demonstration and feedback phases are helpful.\n- The real results section is quite short with only two quantitative numbers reported. It would be good to see more detailed metrics here (e.g., standard deviation or other quantitative results). Also, justification or intuition on why the performance gap is so much larger in the real experiment compared to the simulation experiments.\n\n**Post Revisions:** The reviewers have adequately addressed most of my concerns and I have raised my assessment from Weak Reject to Weak Accept. However, some remaining concerns are that the method lacks algorithmic novelty and the real experimental results are a bit lacking.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Originally was a Weak Reject but after revisions I modify my recommendation to a Weak Accept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "1. Add more technical details to the text or appendix, especially on the computation of the likelihood metric.\n2. Add more justification for the chosen compatibility metric, particularly how to address the high novelty yet incompatible case above.\n3. Add a “Prompt-Only” baseline or similar baseline for the experiments with active solicitation.\n4. Add more quantitative results from the real robot experiment data."}}]

Author rebuttal:

Decision: Accept (poster)