Reciprocal MIND MELD: Improving Learning From Demonstration via Personalized, Reciprocal Teaching
Authors: Mariah L Schrum
Erin Hedlund-Botti
Matthew Gombolay
Venue: CoRL 2022 Poster (2022)
Keywords: meta-learning
personalization
imitation learning

Abstract: Endowing robots with the ability to learn novel tasks via demonstrations will increase the accessibility of robots for non-expert, non-roboticists. However, research has shown that humans can be poor teachers, making it difficult for robots to effectively learn from humans. If the robot could instruct humans how to provide better demonstrations, then humans might be able to effectively teach a broader range of novel, out-of-distribution tasks. In this work, we introduce Reciprocal MIND MELD, a framework in which the robot learns the way in which a demonstrator is suboptimal and utilizes this information to provide feedback to the demonstrator to improve upon their demonstrations. We additionally develop an Embedding Predictor Network which learns to predict the demonstrator’s suboptimality online without the need for optimal labels. In a series of human-subject experiments in a driving simulator domain, we demonstrate that robotic feedback can effectively improve human demonstrations in two dimensions of suboptimality (p < .001) and that robotic feedback translates into better learning outcomes for a robotic agent on novel tasks (p = .045).

Reciprocal MIND MELD: Improving Learning From Demonstration via Personalized, Reciprocal Teaching
Mariah L., Schrum, Erin Hedlund-Botte Matthew C. Gomoblay
Institute for Robotics and Intelligent Machines
Georgia Institute of Technology euchrum3@gatcch..om ehedlund66@ galtlundd@@@atthh...ammm
gatech. edu, matthew.gombolly waclaayy@cc..gteoe. @ow.gatch..cccccu
Endowing robots with the ability to learn novel tasks via demonstra- tions will increase the accessibility of robots for non-expert, non-roboticists. How- ever, research has shown that humans can be poor teachers, making difficult for robots to effectively learn from humans. If the robot could instruct humans how to provide better demonstrations, then humans might be able to effectively teach broader range of novel, distrribbiiii tasks. In this work, introduce Re- ciprocal MIND MELD, a framework in which the robot learns the way in which demonstrator is suboptimal and utilizes this information to provide feedback the demonstrator to improve upon their demonstrations. We additionally develop Embedding Predictor Network which learns predict the demonstrator's sub- optimality online without the need for optimal labels. In a series of human-subjec experiments in driving simulator domain, we demonstrate that robotic feedback can effectively improve human demonstrations in two dimensions of suboptimal- ity (p .001) and that robotic feedback translates into better learning outcomes for robotic agent on novel tasks (p- .045).
Abstract:
Keywords: meta-learning, personalization, imitation learning
1
Introduction
When an individual purchases an in-home cleaning robot, the robot will have to taught many novel tasks over an extended period of time. The user may have to teach the robot how to move dishes from the dishwasher to the proper location in the cabinets or how to wash the windows and take out the trash. Simply ree-prorramming these tasks may not be an adequate solution as different users may have differing preferences for how their robot should operate. Therefore, to effectively meet the needs of the end-user, the robot must be capable of successfully learning new tasks quickly via demonstration. Prior work has shown that robots learn well via robot-centric (RC) learning from demonstration (LfD), compared to human-centri LfD [1]. In RC LfD, the human demonstrator provides corrective feedback at each timestep while the robot is rolling out its current policy, which helps the robot learn how recover from mistakes. While RC LfD works well when the demonstrations are high quality, prior work has also shown that humans find RC LfD unintuitive and tend to provide low-quality corrective demonstrations [2, 3, 4, 5]. Such suboptimality, if left uncorrected, is likely to hinder the robot's ability to learn from end-users.
While several approaches have attempted to improve upon teacher's ability provide high quality demonstrations via tutorials and videos [6. 7, 8). prior work has primarily focused on correcting for suboptimality after-the ffter-the-fee fact rather than directly improving teaching abilities. For example, prior work introduced MIND MELD [9]. MIND MELD meta-learns a personalized embedding deseribing the way in which demonstrator is suboptimal in providing feedback in RC LiD via calibration tasks. The calibration tasks are a curated, pre-defined set of policy rollouts with known optimal demonstrations and meant capture the way in which demonstrator is suboptimal. However, we hypothesize that correcting for suboptimal demonstrations under-the-hood as MIND MELD does may not be the best long-term strategy because doing so may 1) contribute end-users' lack of functional understanding, 2) reinforce suboptimal tendencies, and 3) result in poor performance on out-of-distribution tasks and novel robotic platforms [2, 10, 11].
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Study 1:
Sludy 2:
Study 3:
robotic
Yes
demonstrators benefit from feedback along multiple
Yes
recharacterize
robatic feedback
feedback augment teaching skill?
teacher without ground truth knowledge?
improve learning
dlimensions simultaneously?
novel tasks?
b) Characterize Demonstrator
e)
Recharacterize Demonstrator
a)
Calibration Tasks
Novel Task
c) Robotic
f)
Feedback
Robotic Feedback
deleged
Figure 1:
This figure illustrates overview of our methodology and study designs. Figs 1a, 1b, and Ic show the methodology for Studies 1 and 2 and Figs 1d, le, and If the methodology for Study 3.
Consequently, there is need for a framework that can coach demonstrators to become better teach- ers. To solve this problem, we propose Reciprocal Mutual Information Driven Meta-Learning from Demonstration (Reciprocal MIND MELD), Reciprocal MIND MELD is based upon the MIND MELD framework but is meant to guide the human demonstrator to proactively improve their feed- back. Reciprocal MIND MELD differs from MIND MELD in three significant ways. First, Recip- rocal MIND MELD learns a semantically meaningful personalized embedding via calibration tasks that describes the way in which a demonstrator suboptimal (Fig. lb and e). Second, based upon this personalized embedding, Reciprocal MIND MELD provides robotic feedback to the demonstra- tor to improve their teaching abilities (Fig. Ic and f) and consequently improve learning outcomes for the agent rather than correcting for suboptimality retroactively. Third, we introduce an Embed- ding Predictor Network (EPN) which dynamically updates the demonstrator's personalized embed- ding by estimating its new location (Fig. le), thus eliminating the need to repeat the calibration tasks. In our work, contribute the following:
1. We propose Reciprocal MIND MELD, novel method for providing feedback to demon- strators improve their teaching abilities via a personalized embedding.
2. We develop EPN to dynamically update the demonstrator's personalized embedding
without the need to repeat the mmeeccosumiigg calibration tasks (Fig. Ic).
3. We demonstrate that Reciprocal MIND MELD can improve an individual's demonstrations
(p < ,001), accurately estimate a demonstrator's new embedding (p .002), and improve
learning outcomes of the robot (p .045) in a driving simulator domain.
2 Related Work
Researchers are increasingly designing new algorithms to learn from suboptimal demonstrations [12, 13, 14, 15, 16, 17] as well make LfD more user-friendly [3, 18, 19, 20]. In Chen et al., the authors introduce SSRR which improves upon an agent's ability to learn from suboptimal demonstrations by characterizing the relationship between noise and performance [21]. Brown al. [16, 22] and Myers et al. [23] improve upon the ability to learn from suboptimal demonstrations by learning reward function from ranked set of demonstrations. Schrum et al. introduced MIND MELD [5, 9] which meta-learns a personalized embedding describing a teacher's suboptimal tendencies and was shown outperform prior work in LID.
Several approaches have also investigated how best to provide feedback to a demonstrator to im- prove their demonstrations [6, 7, 8, 24, 25]. Cakmak and Takayama conducted study investigating several modalities for communicating improvements to demonstrator. The authors found instruc- tional videos to be the best modality for improving teaching [6]. Sena et al. investigated video feedback with and without rule guidance and found that both modalities produced better results than no feedback [7]. A more extensive discussion of related work can be found in the Appendix.
3 Preliminaries
Reciprocal MIND MELD inspired by the MIND MELD architecture demonstrated in previous work [5]. The objective of MIND MELD is to learn personalized embedding to describe the way

---

in which demonstrator is suboptimal in an RC LfD paradigm. where the demonstrator provides corrective feedback to the robot. MIND MELD then utilizes this embedding to map a demonstra- tor's suboptimal demonstrations to demonstrations closer to optimal. The MIND MELD architecture (shown in gray in Fig. 2) is trained via calibration tasks, which are used to learn the mapping (fo. (p)
Eg,, and qa) from suboptimal labels, & to better labels, ((P) and learn the personalized
embedding, w()) representing an individual demonstrator. The calibration tasks consist of a set of pre-recorded policy rollouts with known optimal labels. Participants provide corrective demon- strations to the robot during these rollouts direct the robot to a goal. MIND MELD learns to map the participant's corrective labels to higher-quality labels while simultaneously inferring the personalized embedding, w(P), representig an individual, p's, suboptimal style. To ensure that w(() can represent various and distinct feedback styles, MIND) MELD maximizes lower bound on mu- tual information between the way in which a demonstrator is suboptimal and w(P) via variational inference [26]. Additional details can be found in the Appendix.
While prior work demonstrated that MIND MELD is capable of improving upon suboptimal demonstrations, MIND MELD suffers from several key limitations: 1) MIND MELD cor- rects for suboptimality under-the-hood and does not convey to the demonstrator how best to improve their suboptimal ten- dencies, and 2) MIND MELD assumes that demonstrators are static (i.e., the way in which they are suboptimal does not change over time). Reciprocal MIND MELD overcomes these limitations by 1) providing actionable robotic feedback to the demonstrator to improve upon the quality of their demonstra- tions and 2) dynamically updating the estimate of their person alized embedding online via our EPN in order to account for changes in suboptimal tendencies and teaching ability.
@(P) (2lo),
)
9
fo
(
t
t w(P)
Exi
- pv
Figure 2: MIND MELD archi-
tecture [9] (gray) and additional network head, pe, (blue) for learn- ing a semantically meaningful embedding space (see Section 4.1).
Driving Simulator Domain keeping with prior work [9]. utilize a driving simulator domain based on the high-fidelity physics simulator, Airsim, and an Xbox steering wheel to evaluate Reciprocal MIND MELD. Driving simulators allow researchers to study novel algorithms in an environment that is safe for human subjects. In this domain, participants are tasked with teaching a car to drive from a start location to a goal in various environments while avoiding obstacles. The action space consists of the position of the wheel (-540 to 540%), and the state space consists of images, position, velocity, and acceleration. Feedback is provided to demonstrators via verbal instructions.
Methodology
Because humans have a greater ability generalize novel tasks and domains than machine- learning algorithm [11], our objective is to provide demonstrators with knowledge about how to improve their demonstrations rather than correcting suboptimality under-the-hood. We propose an approach to reason about a demonstrator's embedding and provide robotic feedback derived from their embedding that intended to improve upon their demonstration abilities. In keeping with prior work [9], we investigate the abilities of our approach in driving simulator domain. We break the problem of improving upon a demonstrator's teaching abilities into three research questions.
RQ1: Can robotic feedback improve upon a demonstrator's teaching abilities?
RQ2: What is the best method to provide robotic feedback to improve teaching abilities?
RQ3: Does robotic feedback result in improved learning outcomes on novel tasks and over time?
4.1 Semantically Meaningful Embedding Space
Prior work [5, 9] has illustrated that MIND MELD learns embeddings that correlate with suboptimal tendencies and that demonstrators tend to over-/under-correct and provide anticipatory/delayed feedback driving simulator domain. We note that domain expertise is required to determine these dimensions of suboptimality. This suboptimality is related to the unintuitive nature of RC LiD as well as the correspondence problem (4, 27] which arises from differences in embodiment between humans and robots. These suboptimal tendencies are unrelated to the specific task itself, but are
3

---

related to the task specifications (e.g., providing corrective feedback via steering wheel). While there may be additional dimensions of suboptimality depending on the robotic domain, we focus (o/u) and anticipatory/delayed (a/d) dimensions, as
our investigation on the over-/under-correciing these were determined in prior work to be principle dimensions of suboptimality [9]. We posit that these two dimensions will be common across RC LfD paradigms which require continuous control input and plan to test this hypothesis in future work. Our goal is to learn a semantically meaningful embedding space (i.e., space that can be translated into actionable feedback) and then utilize the location of the demonstrator's 's embedding within the embedding space to provide robotic feedback.
To learn semantically meaningful embedding space whose dimensions reflect suboptimal tendencies, we add an additional network head, Ps: (w(()) = (P),, (Fig. 2, shown in blue) to the MIND MELD architecture to esti- mate the suboptimal tendency, m()) (i.e., the magnitude
Over come-coorcoor dellyyd
Over Junder boundary
Over-corretoor anticipatory
Art drteeratn//eeeeleeeeee boundary
1.50
Embedding 2imension
1.00
by which the demonstrator over-/under-correctt and is
0.50
anticipatory/dellyedd. We utilize a mean squared error
(MSE) loss, L(ti, w) = & E, (l))(w()) n | to train the network to predict the suboptimal tendency, m/(P) given the personalized embedding. This loss helps to ensure that the dimensions of the embedding space are semantically meaningful and can therefore be translated into actionable robotic feedback. Under IRB approval, we leverage the calibration dataset collected Schrum et al. [9] to learn a semantically meaningful embedding space. This dataset consists of 76 participants who provided demonstrations on a set of calibration tasks. The suboptimal magnitude, mi(p), is determined via dynamic time warping (DTW) [28] between the participants' feedback and optimal labels from the calibration tasks. Because MIND MELD outputs the difference between the participant's s corrective label and the optimal label, the perfect demonstrator's embedding, w", is defined the embedding which minimizes the output vol))) where of the MIND MELD architecture, w* = argmin (P)) Et,p Sel EEy (a ((t--ant w()))
0.00
Q3
Q4
-0.50
-0.25
0.25
0.75
1.25
Embedding Dimension
Figure 3: The learned embedding space and decision boundaries. Q1-Q4 indi- cate quartiles 1-4 for the o/u dimension.
(p) Ar Ar is a sequence of demonstrations.
Our next objective is to determine the semantically meaningful dimensions of the embedding space. We train a support vector machine (SVM) with a linear kernel to learn the decision boundaries which best separate the demonstrators into their respective suboptimal categories (o/u and a/d). The SVM training labels are determined via DTW between the participant labels and the optimal labels from the calibration tasks. We utilize an SVM to learn the decision boundaries so that we can add the additional constraint that the classifier must pass through the point representing the perfect demonstrator, w". The distance between the embedding and the decision boundary along the suboptimal dimension determines the magnitude by which the demonstrator is suboptimal.
Fig. 3 depicts our embedding space with linear classifiers separating over-correctors from under- correctors and delayed from anticipatory. The size of the point represents the magnitude by which the demonstrator is suboptimal in the o/u dimension as determined by DTW. The plot illustrates that demonstrators who are more suboptimal in o/u (as represented by larger points) are farther from the o/u decision boundary, supporting our hypothesis that distance from the decision boundary can be used to measure the degree of suboptimality. To further support our claim, we apply Spearman's correlation and find that distance from the decision boundary strongly correlates with magnitude of suboptimality in both the o/u (p- .84, p < .001) and in a/d dimensions (p = .93. p < .001).
4.2 Robotic Feedback
To determine the feedback the robot should provide, we calculate the distance, 6, along the semantically meaningful dimension between the personalized embedding, w(()), and perfect embedding, w², as shown in Fig. Ic and f. In the driving domain, we are interested in e 6/u and (0)
(4)
aadd which define the distance between the demonstrator's embedding and the hypothetical perfect demonstrator's embedding the o/u dimension and the a/d dimension respectively after the 7th round of feedback. In our framework, the feedback is proportional to the distance from w".

---

W (p)
Actual Under
0:155
9 6¹
0.150
0.0
Actual
02
h
a
0.15
0.100
0.30
02
0.77
W (p)
d
03
0.050
8.08
0:1
Q4
8
0:1
Under
Predicted
Over
Delayed
Predicted
Anticip.
(p)
12
m
to
(a) Embedding Predictor Network
(h) Overr/under-corrcctingg
(c) Anticipatory/delayed
Figure 4: Fig. 4a illustrates our EPN architecture. Fig. 4b and 4c show the confusion matrices for predicting the quartile that the embedding falls within on holdout test tasks.
To convert oo/u into actionable and intelligible robotic feedback, we discretize the range of 6/ by splitting the embeddings from the previously collected calibration participants into quartiles as shown in Fig. 3. Our objective to move a participant's embedding so that they are in the range denoting the 25% of calibration participants who the least suboptimal (i.c., quartile one). Partici- pants who fall a quartile farther from the decision boundary receive feedback proportional to their quartile. For example, if a participant falls in the fourth quartile in the o/u dimension, the robot will instruct the participant to turn the wheel a lot less compared to slightly less in the second quartile. A table showing the feedback for each quartile and dimension can be found in Appendix Table 1.
4.3 Online Embedding Estimate
To determine if additional feedback should be provided to the demonstrator and if so, the form of the feedback, we must update our estimate of (P) after each iteration of robotic feedback. One option to update our estimate of the embedding is have the demonstrator redo the calibration tasks. However, doing so is time consuming and increases the workload of the demonstrator.
Instead, we propose to dynamically update the embedding online using LSTM-based architecture which extracts salient features from the demonstrations to estimate the personalized embedding rather than relying on calibration tasks which require known, optimal labels. For example, the velocity and magnitude with which the demonstrator turns the steering wheel are two salient features which inform the estimate of the new embedding. We call this network the Embedding Predictor Network (EPN) (Fig. 4a). The input to the EPN is the set of new demonstrations, To:m the (p)
demonstrator's 's previous embeddings, W and the robotic feedback that previously provided to the demonstrator, T0:-11 The output of the EPN is an estimate of the new personalized embed- ding, (0 This network utilizes two LSTM subnetworks, h& and dg, the output of which is then fed into subnetwork, 9ø', made up of linear layers with ReLU activations. The inputs to he are wor- (p) and 00 Each trajectory, T!(P) is fed into LSTM subnetwork, dg. We then average across the outputs of de and feed the result into gø which produces embedding estimate, w!!"). We choose to average across the outputs of de so that our network agnostic to the number of trajectory inputs.
We train our EPN on the data collected in Studies 1 and 2 as described in Section 5. Fig. 4b and 4c show confusion matrices depicting the ability of the network accurately predict the quartile of suboptimality in the o/u dimension and the a/d dimension respectively on holdout test tasks.
5 Human-Subjects Studies, Results, and Discussion
To determine if Reciprocal MIND MELD is able to improve upon demonstrator's 's ability to provide high-quality demonstrations, we conduct three human-subjects studies. The objective of Study is to determine if we are able to shift demonstrator's embedding via verbal robotic feedback in the o/u dimension (RQ1). In Study 2, we investigate if, and how best, we can shift demonstrator's embedding in two dimensions (RQ2). In Study 3, we determine if 1) robotic feedback derived from
5

---

Feedback
18
16
Corge Gimonoe
14
Owigg
Found
Raand
Round
(a) Study
(b) Study 2
(c) Study 3
Figure 5: Fig. 5a and 5b show the difference between the embedding distance at round and the embedding distance at round one for Study 1 and Study 2 respectively. Fig. 5c shows the final distance from the goal for the robot after each round of Study 3.
EPN rather than the calibration tasks is good metric of teacher suboptimality and 2) if robotic feedback improves teaching outcomes over time (RQ3). During each study, we employed surveys measure how robotic feedback altered participants subjective attitude towards each agent. In our analysis, we check parametric models for normality and homoscedasticity. Model details, tests for assumptions, and additional results are in the Appendix.
5.1 Study (RQ1)
Our objective in Study to demonstrate that robotic feedback can effectively modulate partic- ipant's teaching. In this study, start by investigating feedback only in the o/u dimension. After completing pre-study surveys, participants complete four rounds of the calibration tasks to measure how their embedding is changing. Participants receive robotic feedback between each round and complete trust [29] and fluency [30] surveys to determine their subjective perceptions of the robot.
Conditions: In the Cooperative condition, the robot provides feedback to improve the demonstra- tor's teaching. the Adversarial condition, the robot provides feedback to make the participant worse demonstrator. In the None condition, the participant does not receive any feedback.
Results: We recruited 27 participants (Mean age 24.15, SD 3.4; 37.0% Female). Fig. 5a shows the change in the distance (c.) -//,) in the o/u dimension between round one and rounds one through four. We plot 6o/u 6o/u to show how participants change irrespective of their initial (1)
(2) (1)
teaching skill. We find that the distance at round one, caju* is significantly greater from the distance, "o/u" at round four in Cooperative (x²(1) = 5.44, p .020) and significantly less in Adversarial (F(1,8) 20.1, p .002).
We additionally find that Adversarial results in the embedding shifting sigmificantly farther from the perfect embedding between rounds one to four (F 24) = 20.2. p < .001) compared to Cooperative (p < .001) and None (p = .014). Cooperative shifts the embedding significantly closer to the perfect embedding (p .009) compared to None. Together, these findings indicate that our approach capable of modulating teaching style in either direction along the suboptimal dimension. Further, the results in None shows that participants repeated interactions. Interestingly, we find that participants become significantly worse in the a/d dimension when they only receive feedback in the o/u dimension. We also find that participants' trust increased significantly more (2, 24) = 5.15, p = .014) in Cooperative compared to Adversarial (p .020) and None (p .038). Additionally, we find positive change in fluency (F(2, 24) 5.10, p = .014) in Cooperative compared to Adversarial (p = .017). Takeaway: Robotic feedback can effectively improve participant's teaching abilities in a driving simulator domain.
not simply improving due to
5.2 Study (RQ2)
In Study we next determine how best to provide robotic feedback to both prevent cognitive over- load and efficiently improve upon a participant's teaching abilities. Our study design follows the same procedure as Study 1, in which participants complete five rounds of the calibration tasks and receive robotic feedback between each round.

---

Conditions: In Sinultaneous, the robot provides feedback related to both the o/u and the a/d dimen- sions. In Greedy, the robot only provides feedback related to the condition in which the participant is worst (i.e., farthest from w²). In None, the participant receives no feedback.
Results: We recruited 39 participants (Mean age = 22.46, SD = 3.3; 38.5% Female). Fig. 5b shows the overall change in the distance (c
Co/uta/d co/uta/d) in the two dimensions of suboptimality
(1)
between round one and rounds one through five. We find that the distance at round one, fo/w+a/d" is
(1)
significantly greater from the distance, (5)
fo/uta/d' at round five in Simultaneous (F(1,12) 22.3),
P < .001). We next compare Aoo/uo/dd across conditions (F(2, 36) = 3.77, = .033). We find that Simultaneous results in the embedding shifting significantly closer the perfect embedding between rounds one to five compared to None (p = .034). We do not find significance between None and Greedy or Simultaneous and Greedy. Participants trust (2 36) = 3.81, p- = .032)
and team fluency (2, = 7.23, = .002) significantly increased in Simultancous compared to None (p .029, p .002 respectively). Lastly, although the result is not significant, we find that participant's understanding [31] of the robot increased more in the Simultaneous (M = 0.61,
SD 0.62) condition compared to None (M- 0.14, SD 0.69) and Greedy (M .15,
SD = 0.58). Takeaway: Providing feedback in both dimensions simultaneously produces better results for both objective and subjective metries.
5.3 Study 3 (RQ3)
In Study 3, we aim to show that our approach and the results from Study 1 and 2 translate to improved learning outcomes for an LiD agent on novel tasks. Participants first complete the calibration tasks to obtain an initial estimate of their embedding, w/)), and determine 6%/u and a//d' Next, the robot provides feedback to the participant intended to improve their demonstrations in both the o/u and the a/d dimensions given our positive findings for the Simultaneous condition in Study 2. Participants then train the robot for three rounds three different novel environments (i.c., new start and goal locations) for six demonstrations each. Between each environment, we estimate the participant's s (e)
(4) new embedding, w via the EPN, and calculate 6o/a and <add after each round, i E (1, 3). The robot provides robotic feedback based upon the new estimate of the participant's 's embedding derived (4)
from the EPN. At the end of the study, the participants redo the calibration tasks to determine 60/1
(4) and "a/d" By redoing the calibration tasks, we are able to obtain ground truth estimate of how the quality of their demonstrations has changed over the course of the study.
Conditions: In Feedback, the robot provides feedback to the participant about their demonstrations. In No Feedback, the robot still interacts with the participant but does not provide feedback.
Results: We recruited 60 participants (Mean age = 21.9. SD = 2.89; 28.3% Female). Fig. 5c shows the robot's final distance from the goal for Feedback and No Feedback for rounds 1-3. Participants in Feedback achieve a lower final distance to the goal in round one despite starting off as worse
demonstrators on average, as measured via the initial calibration tasks (Mean co/u-ajd in Feedback:
(0)
0.93, Mean c oou+aajd in No Feedback: 0.89). Additionally, the final distance of the robot improves over the rounds in Feedback whereas in No Feedback, the robot improves slightly then gets worse in the final round. In round three, the robot achieves significantly lower final distance from the goal (Z- 2.0, p .045) compared to No Feedback.
To determine the embedding as estimated by the EPN is a good metric of performance, we com-
(t)
pute the correlation between the distance, "o/ictafd" of the estimated embedding from the perfect embedding and performance, measured by the average distance from the goal for each round, i. Fig. 6 shows a significant correlation (p- = -23, p = .002) between emhedding distance and perfor- mance, suggesting that the embedding estimated by the EPN is a good measure of suboptimality.
Next, we investigate the overall change in the quality of the participants' demonstrations as measured via the first set of calibration tasks (conducted at the beginning) and last set (conducted at the end). Fig. 7 shows the change, (0) - (1), for the o/u dimension and the a/d dimension. We find that participants became significantly better Feedback (I(52) = 2.62, p = .006) compared to No Feedback in the o/u dimension. While we do not find significance in a/d, we do find that participants
7

---

0.8
0.2
Feedback
Colataid
0.7
pa.23
No Feedback
0.6
0.0
-0.1
Average
Over-/Under-Correciing Anticipatory/Delayed
Figure 6: Correlation between the embedding distance, (8)
Figure 7: Change in Co/u. and ca/d between first and last calibration tasks.
"o/uta/d' and distance from the goal.
improve in Feedback whereas they become worse in No Feedback in this dimension. Lastly, we investigate Feedback versus No Feedback in terms of subjective metrics. Feedback significantly increases trust (Z = -2334, == .019) and decreases workload [32] (t(58.0) = -.779, p = .039) compared No Feedback. Takeaway: Feedback derived from our EPN improved participant teaching and resulted in better learning outcomes for the robot in novel tasks.
5.4 Discussion and Limitations
In Study demonstrated
shift demonstrator's embedding both farther from and closer to the perfect embedding depending on whether the demonstrator received feedback from an Ad- versarial or Cooperative robot respectively (p .001) (RQ1). In Study 2, found that pro- viding feedback intended to improve upon both dimensions of suboptimality simultaneously is the best strategy and does not cause participants suffer from undue level of cognitive overload (p < .001) (RQ2). Studies and 2 present strong evidence that robotic feedback is capable of improving upon demonstration quality, suggesting that robot will learn better from a teacher who has received robotic feedback. In Study 3, we test this hypothesis and investigate the abilities of our EPN to update our estimate of the participant's embedding during novel tasks. We found that final distance of the robot from the goal improves as the demonstrator receives more feedback about their demonstrations (p .045). Overall, we demonstrated that robotic feedback derived from our Reciprocal MIND MELD architecture results in better learning outcomes for robot in driving simulator domain.
Limitations: A limitation of Reciprocal MIND MELD is that domain knowledge is required to determine the dimensions of suboptimality. However, robotic domains share many similarities in terms of the control interfaces and the potential for suboptimality, suggesting that the dimensions in one domain will likely be similar in others. In this work, we investigate verbal feedback improve upon demonstration quality. However, prior work has suggested that alternative methods of provid- ing feedback may be more effective improving teaching abilities [6]. We leave future work an investigation of the best modality for providing demonstrator feedback in the context of Reciprocal MIND MELD. Furthermore, in this work, we only investigate two dimensions of suboptimality in driving simulator domain. We plan to investigate, in future work, Reciprocal MIND MELD's 's abil- ity to generalize to additional dimensions of suboptimality in other domains. Lastly, our population consisted mostly of college aged students. In future work, we propose to sample from more diverse participant pool.
6 Conclusion
We introduce Reciprocal MIND MELD, a novel LíD framework for providing robotic feedback to a human demonstrator based upon personalized embedding to improve suboptimal teaching tendencies. We demonstrate our approach in a series of three human-subjed" experiments in which we show that robotic feedback can improve upon the quality of a teacher's 's demonstrations, providing feedback in multiple dimensions simultaneously results in improved learning outcomes for robot. Additionally, we show that our Embedding Predictor Network capable of accurately estimating the updated personalized embedding online, thus enabling continuous feedback to be provided the demonstrator.
most effective method, and robotic feedback

---

7 Acknowledgements
This work was supported by the Georgia Institute of Technology State Funding, NASA Early Career Fellowship under grant 60HQTRR990XX0-11ECFFBB MIT Lincoln Laboratory under grant PA8702-15-D-0001, gift from Konica Minolta, and the National Science Foundation under grants 1545287 and 20-604.
MIT
References
[1] S. Ross, G. J. Gordon, and J. A. Bagnell. No-Regret Reductions for Imitation Learning and Structured Prediction. 14th Internation Conference on Artificial Intelligence and Statistics, Fort Lauderdale, FL. 2011.
[2] S. Amershi, M. Cakmak, W. B. Knox, and T. Kulesza. Power the people: The role of humans interactive machine learning. Al Magazine, 35(4):105-120, 2014. ISSN 07384602. 1oi:10. 1609///m....33550 v35i4.2513
[3] Spencer, Choudhury, M. Barnes, M. Schmittle, Chiang. P. Ramadge, and Srinivasa. Learning from Interventions: Human-robot interaction as both explicit and implicit feedback. Science and Systems XVI, 2020. doi:10.
Robotics:
[4] M. Laskey, C. Chuck, Lee, Mahler, S. Krishnan, K. Jamieson, A. Dragan, and K. Goldberg. Comparing human-centric and robot-centric sampling for robot deep learning from demon- strations. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 358-365, 2017. doi: 100.1100//CAA.211......9
[5] M. Schrum, E. Hedlund, and M. Gombolay. Improving Robot-Centric Learning from Demonstration via Personalized Embeddings. arXiv, 2021. 2110.03134,
[6] M. Cakmak and L.. Takayama. Teaching people how teach robots: The effect of instruc- tional materials and dialog design. In ACM/EEE International Conference on Human-Robot Interaction, pages 431-438. IEEE Computer Society, 2014. ISBN 9781450326582. doi: 10. 11144/2255963.2255567777
[7] A. Sena and M. Howard. Quantifying teaching behavior in robot learning from demonstration. International Journal of Robotics Research, 39:54-72, 2020. ISSN 17413176. doi:10.1177/ 0278364919884622
[8] M. Cakmak and M. Lopes. Algorithmic and human teaching of sequential decision tasks. In Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence, AAAI'12, page 1536-1542. AAAI Press, 2012.
[9] M. Schrum, Hedlund-Botti, N. Moorman, and M. C. Gombolay. MIND MELD: Per- sonalized Meta-Learning for Robot-Centric Imitation Learning. In Proceedings of the 2022 ACM/IEEE International Conference on Sluman-Robar Interaction, HRI '22 pages 157-165. IEEE Press, 2022.
[10] G. Troisi. Humanization builds trust: the effect of human-like chatbots on the willingness to disclose personal information online. PhD thesis, Luiss University. Department of Business and Management, 2021.
[11] R. Geirhos, C. R. M. Temme, Rauber, Schütt, M. Bethge, and F. Wichmann. Gener- alisation in humans and deep neural networks. Conference on Neural Information Processing Systems, 2018
[12] K. Menda, K. Driggs-Camphell, and M. J. Kochenderfer. Ensembledagger: A bayesian ap- proach safe imitation learning. In 20/9 IEFF/RSI International Conference Intelligent Robots and Systems (IROS), pages 5041-5048, 2019. doi: 10.:10...11//SSS..0 10011009//0004007.0.1111 29 8968287.
[13] M. Kaiser, H. Friedrich, and R. Dillmann. Obtaining good performance from bad teacher. In International Conference on Machine Learning, Workshop on Programming by Demonstra- tion, 1995.

---

[14] B. Burchfiel, C. Tomasi, and R. Parr. Distance minimization for reward learning from scored trajectories. Proceedings of the AAAI Conference on Artificial Intelligence, 30, 2016. doi: (0.1609/2aii.v33 v3011.1011.
[15] M. Valko, M. Ghavamzadeh, and A. Lazaric. Semi Semi-supervised apprenticeship learning. In M. P. Deisenroth, C. Szepesvári, and J. Peters, editors, Proceedings of the Tenth European Workshop on Reinforcement Learning, volume 24 of Proceedings of Machine Learning Research, pages 131-142, Edinburgh, Scotland, 2013. PMIR.
[16] D. Brown, W. Goo, P. Nagarajan, and S. Niekum. Extrapolating beyond suboptimal demonstra- tions via inverse reinforeement learning from observations. In K. Chaudhuri and Salakhut- dinov, editors, Proceedings of the 36th International Conference on Machine Learning, vol- ume 97 of Proceedings of Machine Learning Research, pages 783-792. PMIR, 2019.
[17] D. H. Grollman, Aude, and G. Billard. Robot learning from failed demonstrations. Interna- tional Journal of Social Robotics, 4(4):331-342, 2012.
[18] H. He, H. Daumé, and J. Eisner. Imitation learning by coaching. In Proceedings of the 25th International Conference on Neural Information Processing Systems Volume 2, NIPS' 12, page 3149-3157, Red Hook, NY, USA, 2012. Curran Associates Inc.
[19] M. Kelly, C. Sidrane, K. Drigge-Campbell, and M. J. Kochenderfer. HG-DAgger: Interac- tive imitation learning with human experts. 2019 International Conference Robotics and Automation (ICRA), pages 8077-8083, 2019.
[20] M. Laskey. S. Staszak, W. Y. Hsieh, J. Mahler, F. T. Pokorny, A. D. Dragan, and K. Goldberg. SHIV: Reducing supervisor burden in DAgger using support vectors for efficient learning from demonstrations in high dimensional state spaces. Proceedings IEEE International Conference on Robotics and Automation, 46944466 2016. ISSN 10504729. doi:10. 109/CCA.. 2016.7487167.
[21] L. Chen, R. R. Paleja, and M. C. Gombolay. Learning from suboptimal demonstration via self-supervised reward regression, In Conference on Robot Learning, 2020.
[22] D. S. Brown, W. Goo, and S. Niekum. automatically-ranked demonstrations. In L. P. Kaelbling, D. Kragic, and K. Sugiura, ed- itors, Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pages 330-359, PMLR, 30 Oct-01 Nov 2020. URL https: //proceeding, mlr,
senter-than-demonstteee imitation learning via
html.
[23] V. Myers, E. Biyik, N. Anari, and D. Sadigh. Learning multimodal rewards from rankings. In A. Faust, Hsu, and G. Neumann, editors, Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 342-352, PMLR, 2022.
[24] X. Zhu. Machine teaching: An inverse problem to machine learning and an approach toward optimal education. AAAI Conference Artificial Intelligence, 2015. URL www. aaai org.
[25] M. Gombolay, R. Jensen, Stigile. S.-H. Son, and Shah. Learning tutor from expert demonstrators via apprenticeship scheduling, AAAI Workshop, 03 2017.
[26] X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel. InfoGAN: In- terpretable representation learning by information maximizing generative adversarial nets. In Proceedings the 30th International Conference Neural Information Processing Sys- tems, NIPS' 16, page 2180-2188, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.
[27] A. Alissandrakis, C. L. Nehaniv, and K. Dautenhahn. Solving the correspondence problem in robotic imitation across embodiments: synchrony, perception and culture in artifacts, page 249-274. Cambridge University Press, 2007. doi:1001017/CB0977800 1489808.017
[28] Salvador and Chan. Toward accurate dynamic time warping in linear time and space. Intell. Data Anal., 11(5):611-580 2007. ISSN 1088-467X.
10

---

[29] J.-Y. Jian, A. Bisantz, and C. Drury. Foundations for an Empirically Determined Scale of Trust in Automated Systems. International Journal of Cognitive Ergonomics. 4:53-71, 2000. doi: 10.00// 1ett:00227//1532555000 13377561JCC6000 04.
[30] G. Hoffman. Evaluating Fluency in Human-Robot Collaboration. IEEE Transactions Stchhn-Machine Systems, 49(3):209 218, 2019. ISSN 21682291. 161110.1109/THMS.20011 2904558.
[31] N. L. Robinson, T.-N. Hicks, G. Suddrey, and D. J. Kavanagh. The robot self-efficacy scale: Robot self-efficacy. likability and willingness to interact increases after a robot-delivered tuto- rial. In 2020 29th IEEE International Conference Robot and Human Interactive Communi- cation (RO-MAN), pages 272-277, 2020. doi: 1090000- SAA.0AMAAN09962220022255555
[32] S. G. Hart and L. E. Staveland, Development of nasa-tix (task load index): Results of empirical and theoretical research. In P. Hancock and N. Meshkati, editors, Human Mental Workload, volume 52 of Advances in Psychology, pages 139-183. worh-HHolland, 1988. doi:https:/doi. org/10. 101 6/S01 (6-4415508))6386666


Reviews:
[{"review_id": "p7Moxw1desBW", "reviewer": "Reviewer_w2jn", "summary": "The paper proposes learning a model for human suboptimality in giving demonstrations that the robot can use to give the person feedback on how to improve their suboptimality. The robot then updates its estimate of the person’s suboptimality and personalizes its feedback based on it with a novel Embedding Predictor Network (LSTM-based model). The paper presents 3 studies that look at whether robot feedback helps users improve the quality of their demonstrations along one or two dimensions of suboptimality, and whether that leads to improved robot learning.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I found the studies to be overall interesting and well-conducted, with large sample sizes and rigorous ANOVA tests and post-hoc analyses. The overall framework is also interesting, very relevant to improving robot learning, and well-motivated.\n\nThe central weakness of this paper is that the writing is confusing:\n- The abstract and introduction are very unclear in terms of contributions. It was not clear until very late that the calibration tasks were a way to extract ground truth, and because of this, it wasn’t clear that the EPN network was meant to substitute for that until very late too (study 3), so for a majority of the introduction and abstract I did not understand what was a contribution vs what was pre-existing from MIND MELD. I would suggest very clearly stating how EPN is explicitly meant to replace the time-consuming calibration, and describe the 3 study hypotheses from the get go.\n- The paper places a few key components in the appendix, which interrupts the flow of the paper and makes it not self-contained. Crucial details (like the entire related work section, the network architecture from MIND MELD, the simulator driving domain used for experiments) are omitted and instead made a part of the appendix. One should be able to understand the key contents of the paper without having to look at the appendix.\n- As it is currently written, the method section seems too dependent on certain experimental details: the domain is driving, there are exactly 2 dimensions of suboptimality, etc. Because of this, it’s unclear how much this method can generalize beyond driving or beyond these two hand-chosen suboptimalities.\n- The metrics for validating the studies were unclear: what was the purpose of looking at participant trust and fluency? Why did we look at correlation as a metric for EPN feedback being a good measurement of personalized embeddings instead of just verifying against collected ground truth calibration values? Relatedly, the description of the results can be a bit dry at times (many ANOVA numbers and p-values), and I would have preferred to have more intuition and explanations of what the results mean.\n\nSmaller comments:\n- The newline between Abstract and the contents of the abstract is unusual, as is having a citation in the abstract.\n- Abstract: what is a “personalized embedding”? The term hasn’t been introduced prior to here, so it’s confusing what this sentence means.\n- “Prior work has shown that robots learn well via robot-centric learning from demonstration (LfD) [2] because the robot can effectively learn how to recover from mistakes” → this sentence doesn’t make sense: how does LfD help the robot recover from mistakes?\n- Fig. 1 is difficult to understand without a more descriptive caption.\n- Did not understand Fig. 3b and c.\n- Why did you choose the over/under-correcting dimension specifically for Study 1? Given Fig. 6 results, it's unclear how Study 1 would have panned out if anticipatory/delayed were studied instead.\n- I don’t understand the last paragraph of Study 1: it seems statistically wrong to compare who has the lower p-value and make statements about that; instead, we should directly compare the two delta values (delta_cooperative vs delta_adversarial).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I thought the technical work itself was well done and the studies were interesting and well-analyzed, but the writing was very confusing and it's unclear how much this work generalizes beyond the 2 hand-defined dimensions of suboptimality. For this reason, I am learning weak reject, but I am very willing to have my opinion swayed.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "The writing has considerable flaws that I believe need to be resolved for acceptance. Evaluating the method on only 2 dimensions of suboptimality is underwhelming, so I would like to either see more dimensions or a convincing explanation for why these 2 dimensions can cover a lot of cases of human suboptimality."}}, {"review_id": "bfcpTVRUcBA", "reviewer": "Reviewer_oD66", "summary": "This work builds on the MIND MELD framework, which improves human-in-the-loop robot learning by having the robot directly model and compensate for the ways that a user's corrections of robot behavior are sub-optimal along two pre-defined dimensions.  In this work, the authors show that telling a user about the dimension of their suboptimality, with feedback tied to their level of suboptimality, can change the user's feedback to be closer to optimal and that the MIND MELD architecture can be extended to compensate for those changes.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This paper shows that users can be trained to give better feedback, which is likely to be of interest to human-in-the-loop learning researchers\n- The authors present three studies to support each component of their work\n- The paper provides an example of how a learning algorithm can compensate for changing feedback, which is an important problem in human-in-the-loop learning\n\nWeaknesses:\n- The paper is very difficult to understand on its own; too much material has been cut and moved to the appendix (perhaps from a longer submission?).  For example, the task and evaluation environment really do need to be described in the paper, not just the appendix.\n- The contributions of the work are over-stated relative to what was actually done (the results are relevant to robotics but no real robot was involved; the \"novel\" tasks are the same task with a new goal location; it's not clear whether the \"robotic\" feedback was perceived as coming from the robot; the results are described as applying to suboptimality in general but only two dimensions of suboptimality are studied, and those dimensions are very task-specific; the authors state that feedback improved participant teaching, even though it only improved in one dimension, suggesting that there might be generalization problems)\n- The paper does not provide strong evidence that the methods described will generalize to frameworks other than MIND MELD or tasks other than simulated driving.\n\n\nQuestions:\n- How are the ground truth labels for the SVM classifier of suboptimality obtained? Why does the proposed method result in semantically meaningful labels? Would this method still result in semantically meaningful labels in another higher-dimensional task?\n- What are the calibration tasks?\n- How is feedback \"provided\" to users? Audio? Text?\n- How were participants recruited? Why is the gender balance so off for all of the studies?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, this paper provides a minor contribution to the field of human-in-the-loop learning with a robotics-relevant validation, but no demonstration on an actual robot.  The most interesting result here is that feedback from a learning system can shift the distribution of demonstrations/corrections even over the course of a single interaction, and that the demonstrations can be made \"better\" from the robot's perspective.  Many of the technical results are an extension to an existing framework; it's not clear how they would apply to other learning algorithms or frameworks.  I'm a little concerned about how much the authors over-generalize their claims throughout the paper -- for example, \"we demonstrate that robotic feedback can effectively improve human demonstrations (p < .001)\" (13-14) and \"We demonstrate that Reciprocal MIND MELD can improve an individual’s demonstrations across multiple dimensions of suboptimality\" (58-59) -- but in all three studies improvements are only shown for the over-/under-correcting dimension, not the the anticipatory/delayed dimension.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "I would love to see a revision that is a little more accurate in its claims -- for example, contribution 3 in the introduction should read: \"We validate Reciprocal MIND MELD in a simulated driving task and show that it can improve demonstrators' performance in the anticipatory/delayed dimension of suboptimality [...]\"\n\nThe limitations section could speak more accurately to the potential for generalization (or not).  For example, in the general case, there is a chicken-and-egg problem with determining suboptimality of demonstrations: the learner needs to know what it looks like to do the task correctly in order to know whether the demonstration is good, but if you have a robust model of a good demonstration, you have a robust model of the task, in which case you don't actually need to learn.\n"}}, {"review_id": "Zh3zFhzAixc", "reviewer": "Reviewer_QQMM", "summary": "The authors extend a learning from demonstration approach to provide interpretible feedback on the quality of demonstrations. They show that this feedback does indeed impact how users give demonstrations, that it's possible to estimate how this feedback is impacting the quality of subsequent demonstrations online, and ultimately that users can internalize this feedback to give more useful demonstrations in different task variations.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n* The authors take a more user-centered approach than is common in learning from demonstration work. The position that algorithmic improvements which better learn from suboptimal demonstrations may cause misunderstandings in users is interesting and deserves to be carefully considered.\n\n* It’s powerful to be able provide online feedback to demonstrators and update feedback after each demonstration. This is a worthy goal to strive towards, and the careful and well reported user studies give indication for the value that such a capability can offer.\n\n* I found the paper well structured and easy to read. Figure 1 is an especially helpful roadmap for the paper.\n\n\nWeaknesses\n\n* It’s unclear when or whether the assumption of a structured understanding of suboptimality makes sense in learning from demonstration. In what setting would the template of this approach be applicable? \n    * If the user is demonstrating a novel task because the robot does not yet possess the capability to carry out the task and because programming the task is not practical, then the agent does not have access to a notion of optimality. If it did, it would already know how to do the task. If someone provides a corrective label, how could you draw any conclusion about its suboptimality? This is a problem that the authors inherit from the MIND MELD approach which they extend (which, according to the paper, requires “access to a distribution of calibration tasks from which we can obtain the optimal, ground truth labels”), and one that they exacerbate by making the notion of an embedded representation of demonstrator “suboptimality” core to their approach.\n    * If the user is demonstrating a task for which the robot is already equipped with the constituent skills because they need the robot to do things according to their preferences, then LfD is not a particularly desirable interface to strive for. It’s more natural to think of the problem as one of task specification, where language is perhaps more suited. “Cups go in this cabinet, glasses over here.”\n    * The authors posit that a generic model of demonstrator characteristics may be possible because robot domains share “many similarities in terms of the control interfaces and the potential for suboptimality” (305). Control interfaces being bad and leading to systematic failings in human provided demonstrations suggests alternative, more effective interventions like improving interfaces or providing instruction and training on the interfaces.\n    * In essence, there is a fundamental circularity to the assumptions being made which the authors need to clarify and rationalize.\n\n* The domain and tasks used aren’t illustrated outside of one in Figure 1, which makes it harder to assess the flexibility of the method.\n    * The authors describe running participants through calibration, then providing them “novel environments (i.e., new start and goal locations)” (252) to evaluate the feedback approach with. Readers would benefit from seeing what the different instances look like.\n\n* Connection to the concept of reciprocal teaching from the education world doesn’t bear out. The purpose of reciprocal teaching is to engage students more deeply in something that they’re learning by having them externalize it. So, if they’re learning reading comprehension, they externalize strategies for comprehension by engaging in dialogues with their peers that materialize comprehension strategies like attempting to predict what’s going to happen. The main point is that this externalization is helpful (because ensures they are following particular strategies and the teacher can listen and intervene). A side effect is that their peers can also learn from them, instead of always learning directly from the teacher.\n    * In this paper, the externalization does not directly aide the agent’s learning process. Rather, it (hopefully) changes how the teacher teaches.\n    * Reciprocal teaching would be a model where the agent attempts to provide feedback to other learners, then the teacher could look at the demonstrations the agent is giving and infer something about how it needs to be corrected. Perhaps as a side effect, there would be some utility to whatever the other agents learn.\n    * As it stands, the proposed method is more like “teacher evaluation” MIND MELD", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The way the authors have designed their approach appears to make assumptions that limit its application to problems in the conventional LfD setting. The user studies presented are interesting, but the authors do not provide a persuasive case that the way in which they achieved online feedback would have been meaningfully superior to, for instance, Wizard of Oz-ing the feedback for the sole purpose of studying the interaction questions. I hope the authors can clarify the key assumptions and better illustrate the utility of their approach in a revised version of the text.\n\n**Update after rebuttal:**\n\nThe authors' substantial textual revisions have made the paper more transparent. With some of the key assumptions and limitation established early in the paper, I think readers will be better able to assess the approach and form assessments of how such an approach might or might not work in other circumstances, so I have updated my recommendation to weak accept.\n\nAdditional feedback regarding \"Reciprocal Teaching\":\nWith the explicit connection to reciprocal teaching removed from the text, I think the paper is no longer in danger of being misleading in this connection. With the term now only appearing in the title, there is some chance that a reader aware of the term may stumble for a moment. I want to provide more clarification from this perspective so you can assess whether to change the title.\n\nThe Sullivan and Brown paper you reference starts out with an example of reciprocal teaching transcribed from a classroom interaction. A student is leading a discussion on a reading passage with their peers, posing questions, summarizing, and otherwise demonstrating that they are using reading comprehension strategies that they were taught. The teacher provides encouragement and intervenes when the leader gets stuck. The way in which \"Reciprocal Teaching\" is reciprocal is that both parties (students and teachers) engage in teaching. The reason the parallel breaks down here is that the students **teach each other**. They do not teach the teacher strategies for comprehending text, the teacher already knows these. The teacher roams around between groups, listening to hear whether students are actually engaging in the strategies with their peers. The students benefit from seeing their peers model the strategies, but the key is ultimately that, if they aren't getting it, the act of attempting to externalize the strategies and failing provides the teacher the opportunity to give instruction.\n\nIn your approach, the \"student\" (machine learner) instructs the \"teacher\" (human) how to teach better. This doesn't provide \"opportunity to give instruction,\" it tells the human to teach differently. It's more akin to a teacher evaluation form given half way through a college class. \"It's hard to follow your lecture because you put too much on the slides. Maybe you should do the derivations on the board so we can follow the steps\".", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "* Clarify reasoning for the utility of the approach given the requirement for knowledge of optimal labels for calibration tasks inherited from MIND MELD, and further requirement to have knowledge of structure of suboptimality to apply their extension.\n\n* Consider providing visualizations of domains in the main document (as I believe they're important for readers' understanding), but at least as a supplement.\n\n* The work shares some motivations with an older line of work in providing “teaching guidance,” which you should consider relating to. This work provided generic advice to demonstrators/teachers about how best to teach a particular machine learner. An interesting aspect of the teaching guidance formulation is that it doesn’t assume knowledge of the specifics of a domain, rather it leverages heuristics or in some cases optimal teaching algorithms and attempts to give demonstrators generic advice. Results showed that even this level of unpersonalized advice could help people provide better instruction to classifiers and to inverse reinforcement learners.\n\n  * Eliciting good teaching from humans for machine learners. Maya Cakmak, Andrea L. Thomaz. http://dx.doi.org/10.1016/j.artint.2014.08.005\n  * Algorithmic and Human Teaching of Sequential Decision Tasks. Maya Cakmak, Manuel Lopes. https://www.aaai.org/ocs/index.php/AAAI/AAAI12/paper/download/4954/5298\n  * Machine Teaching: An Inverse Problem to Machine Learning and an Approach Toward Optimal Education. Xiaojin Zhu. https://ojs.aaai.org/index.php/AAAI/article/download/9761/9620\n\n* Line 126: It isn't immediately clear where the \"suboptimal categories\" come from. It should be explicit that they come from domain expertise of the referenced work.\n\n* Double check references. Many references incomplete or contain wrong links. 6, 12, 14, 22\n\n* Better contextualize and make explicit how the algorithm relates to reciprocal teaching.\n"}}, {"review_id": "Nfy5PglQLop", "reviewer": "Reviewer_tR3N", "summary": "Authors propose an extension to MIND MELD which aims at improving the human's demonstrations via tailored feedback as opposed to optimising suboptimal demonstrations after the fact. They provide an application specific method for achieving this which is validated extensively by carrying out several experiments with a large number of human participants.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Innovative idea to provide automatic tailored feedback and focus on improving demonstrations which is a largely unaddressed problem in the literature.\n- Method is thoroughly evaluated using a large number of human participants and well justified by addressing each of the research questions with focused experiments.\n- Paper overall well written and easy to read.\n- EPN is interesting and could be applied to other problems.\n\nWeaknesses:\n- Relatively simple extension to existing work.\n- Meaningful embeddings may not always be simple mappings such as over/under steering. Mapping between corrective actions and meaningful embeddings is simply proportional to the control error. Method is very application specific.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Although the proposed method is application specific, the main value in the paper is validating that providing tailored feedback to the demonstrator improves learning performance. This is valuable to the learning from demonstration community since it rationalises a promising direction for future research.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Too much reliance on appendices and prior work. Should have minimum to understand content in main document. For example, the additional network head in 3.1 is difficult to understand without prior knowledge of MIND MELD. More details on/ example of a calibration task would be helpful. In section 4, at least a high level description of the robot task should be provided as well as how feedback is given. Related worked should be in main document."}}]

Author rebuttal:

Decision: Accept (poster)