Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes learning a model for human suboptimality in giving demonstrations that the robot can use to give the person feedback on how to improve their suboptimality. The robot then updates its estimate of the person’s suboptimality and personalizes its feedback based on it with a novel Embedding Predictor Network (LSTM-based model). The paper presents 3 studies that look at whether robot feedback helps users improve the quality of their demonstrations along one or two dimensions of suboptimality, and whether that leads to improved robot learning.

strengths_and_weaknesses: I found the studies to be overall interesting and well-conducted, with large sample sizes and rigorous ANOVA tests and post-hoc analyses. The overall framework is also interesting, very relevant to improving robot learning, and well-motivated.

The central weakness of this paper is that the writing is confusing:
- The abstract and introduction are very unclear in terms of contributions. It was not clear until very late that the calibration tasks were a way to extract ground truth, and because of this, it wasn’t clear that the EPN network was meant to substitute for that until very late too (study 3), so for a majority of the introduction and abstract I did not understand what was a contribution vs what was pre-existing from MIND MELD. I would suggest very clearly stating how EPN is explicitly meant to replace the time-consuming calibration, and describe the 3 study hypotheses from the get go.
- The paper places a few key components in the appendix, which interrupts the flow of the paper and makes it not self-contained. Crucial details (like the entire related work section, the network architecture from MIND MELD, the simulator driving domain used for experiments) are omitted and instead made a part of the appendix. One should be able to understand the key contents of the paper without having to look at the appendix.
- As it is currently written, the method section seems too dependent on certain experimental details: the domain is driving, there are exactly 2 dimensions of suboptimality, etc. Because of this, it’s unclear how much this method can generalize beyond driving or beyond these two hand-chosen suboptimalities.
- The metrics for validating the studies were unclear: what was the purpose of looking at participant trust and fluency? Why did we look at correlation as a metric for EPN feedback being a good measurement of personalized embeddings instead of just verifying against collected ground truth calibration values? Relatedly, the description of the results can be a bit dry at times (many ANOVA numbers and p-values), and I would have preferred to have more intuition and explanations of what the results mean.

Smaller comments:
- The newline between Abstract and the contents of the abstract is unusual, as is having a citation in the abstract.
- Abstract: what is a “personalized embedding”? The term hasn’t been introduced prior to here, so it’s confusing what this sentence means.
- “Prior work has shown that robots learn well via robot-centric learning from demonstration (LfD) [2] because the robot can effectively learn how to recover from mistakes” → this sentence doesn’t make sense: how does LfD help the robot recover from mistakes?
- Fig. 1 is difficult to understand without a more descriptive caption.
- Did not understand Fig. 3b and c.
- Why did you choose the over/under-correcting dimension specifically for Study 1? Given Fig. 6 results, it's unclear how Study 1 would have panned out if anticipatory/delayed were studied instead.
- I don’t understand the last paragraph of Study 1: it seems statistically wrong to compare who has the lower p-value and make statements about that; instead, we should directly compare the two delta values (delta_cooperative vs delta_adversarial).

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: I thought the technical work itself was well done and the studies were interesting and well-analyzed, but the writing was very confusing and it's unclear how much this work generalizes beyond the 2 hand-defined dimensions of suboptimality. For this reason, I am learning weak reject, but I am very willing to have my opinion swayed.
issues: The writing has considerable flaws that I believe need to be resolved for acceptance. Evaluating the method on only 2 dimensions of suboptimality is underwhelming, so I would like to either see more dimensions or a convincing explanation for why these 2 dimensions can cover a lot of cases of human suboptimality.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors extend a learning from demonstration approach to provide interpretible feedback on the quality of demonstrations. They show that this feedback does indeed impact how users give demonstrations, that it's possible to estimate how this feedback is impacting the quality of subsequent demonstrations online, and ultimately that users can internalize this feedback to give more useful demonstrations in different task variations.
strengths_and_weaknesses: Strengths

* The authors take a more user-centered approach than is common in learning from demonstration work. The position that algorithmic improvements which better learn from suboptimal demonstrations may cause misunderstandings in users is interesting and deserves to be carefully considered.

* It’s powerful to be able provide online feedback to demonstrators and update feedback after each demonstration. This is a worthy goal to strive towards, and the careful and well reported user studies give indication for the value that such a capability can offer.

* I found the paper well structured and easy to read. Figure 1 is an especially helpful roadmap for the paper.


Weaknesses

* It’s unclear when or whether the assumption of a structured understanding of suboptimality makes sense in learning from demonstration. In what setting would the template of this approach be applicable? 
    * If the user is demonstrating a novel task because the robot does not yet possess the capability to carry out the task and because programming the task is not practical, then the agent does not have access to a notion of optimality. If it did, it would already know how to do the task. If someone provides a corrective label, how could you draw any conclusion about its suboptimality? This is a problem that the authors inherit from the MIND MELD approach which they extend (which, according to the paper, requires “access to a distribution of calibration tasks from which we can obtain the optimal, ground truth labels”), and one that they exacerbate by making the notion of an embedded representation of demonstrator “suboptimality” core to their approach.
    * If the user is demonstrating a task for which the robot is already equipped with the constituent skills because they need the robot to do things according to their preferences, then LfD is not a particularly desirable interface to strive for. It’s more natural to think of the problem as one of task specification, where language is perhaps more suited. “Cups go in this cabinet, glasses over here.”
    * The authors posit that a generic model of demonstrator characteristics may be possible because robot domains share “many similarities in terms of the control interfaces and the potential for suboptimality” (305). Control interfaces being bad and leading to systematic failings in human provided demonstrations suggests alternative, more effective interventions like improving interfaces or providing instruction and training on the interfaces.
    * In essence, there is a fundamental circularity to the assumptions being made which the authors need to clarify and rationalize.

* The domain and tasks used aren’t illustrated outside of one in Figure 1, which makes it harder to assess the flexibility of the method.
    * The authors describe running participants through calibration, then providing them “novel environments (i.e., new start and goal locations)” (252) to evaluate the feedback approach with. Readers would benefit from seeing what the different instances look like.

* Connection to the concept of reciprocal teaching from the education world doesn’t bear out. The purpose of reciprocal teaching is to engage students more deeply in something that they’re learning by having them externalize it. So, if they’re learning reading comprehension, they externalize strategies for comprehension by engaging in dialogues with their peers that materialize comprehension strategies like attempting to predict what’s going to happen. The main point is that this externalization is helpful (because ensures they are following particular strategies and the teacher can listen and intervene). A side effect is that their peers can also learn from them, instead of always learning directly from the teacher.
    * In this paper, the externalization does not directly aide the agent’s learning process. Rather, it (hopefully) changes how the teacher teaches.
    * Reciprocal teaching would be a model where the agent attempts to provide feedback to other learners, then the teacher could look at the demonstrations the agent is giving and infer something about how it needs to be corrected. Perhaps as a side effect, there would be some utility to whatever the other agents learn.
    * As it stands, the proposed method is more like “teacher evaluation” MIND MELD
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The way the authors have designed their approach appears to make assumptions that limit its application to problems in the conventional LfD setting. The user studies presented are interesting, but the authors do not provide a persuasive case that the way in which they achieved online feedback would have been meaningfully superior to, for instance, Wizard of Oz-ing the feedback for the sole purpose of studying the interaction questions. I hope the authors can clarify the key assumptions and better illustrate the utility of their approach in a revised version of the text.

**Update after rebuttal:**

The authors' substantial textual revisions have made the paper more transparent. With some of the key assumptions and limitation established early in the paper, I think readers will be better able to assess the approach and form assessments of how such an approach might or might not work in other circumstances, so I have updated my recommendation to weak accept.

Additional feedback regarding "Reciprocal Teaching":
With the explicit connection to reciprocal teaching removed from the text, I think the paper is no longer in danger of being misleading in this connection. With the term now only appearing in the title, there is some chance that a reader aware of the term may stumble for a moment. I want to provide more clarification from this perspective so you can assess whether to change the title.

The Sullivan and Brown paper you reference starts out with an example of reciprocal teaching transcribed from a classroom interaction. A student is leading a discussion on a reading passage with their peers, posing questions, summarizing, and otherwise demonstrating that they are using reading comprehension strategies that they were taught. The teacher provides encouragement and intervenes when the leader gets stuck. The way in which "Reciprocal Teaching" is reciprocal is that both parties (students and teachers) engage in teaching. The reason the parallel breaks down here is that the students **teach each other**. They do not teach the teacher strategies for comprehending text, the teacher already knows these. The teacher roams around between groups, listening to hear whether students are actually engaging in the strategies with their peers. The students benefit from seeing their peers model the strategies, but the key is ultimately that, if they aren't getting it, the act of attempting to externalize the strategies and failing provides the teacher the opportunity to give instruction.

In your approach, the "student" (machine learner) instructs the "teacher" (human) how to teach better. This doesn't provide "opportunity to give instruction," it tells the human to teach differently. It's more akin to a teacher evaluation form given half way through a college class. "It's hard to follow your lecture because you put too much on the slides. Maybe you should do the derivations on the board so we can follow the steps".
issues: * Clarify reasoning for the utility of the approach given the requirement for knowledge of optimal labels for calibration tasks inherited from MIND MELD, and further requirement to have knowledge of structure of suboptimality to apply their extension.

* Consider providing visualizations of domains in the main document (as I believe they're important for readers' understanding), but at least as a supplement.

* The work shares some motivations with an older line of work in providing “teaching guidance,” which you should consider relating to. This work provided generic advice to demonstrators/teachers about how best to teach a particular machine learner. An interesting aspect of the teaching guidance formulation is that it doesn’t assume knowledge of the specifics of a domain, rather it leverages heuristics or in some cases optimal teaching algorithms and attempts to give demonstrators generic advice. Results showed that even this level of unpersonalized advice could help people provide better instruction to classifiers and to inverse reinforcement learners.

  * Eliciting good teaching from humans for machine learners. Maya Cakmak, Andrea L. Thomaz. http://dx.doi.org/10.1016/j.artint.2014.08.005
  * Algorithmic and Human Teaching of Sequential Decision Tasks. Maya Cakmak, Manuel Lopes. https://www.aaai.org/ocs/index.php/AAAI/AAAI12/paper/download/4954/5298
  * Machine Teaching: An Inverse Problem to Machine Learning and an Approach Toward Optimal Education. Xiaojin Zhu. https://ojs.aaai.org/index.php/AAAI/article/download/9761/9620

* Line 126: It isn't immediately clear where the "suboptimal categories" come from. It should be explicit that they come from domain expertise of the referenced work.

* Double check references. Many references incomplete or contain wrong links. 6, 12, 14, 22

* Better contextualize and make explicit how the algorithm relates to reciprocal teaching.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work builds on the MIND MELD framework, which improves human-in-the-loop robot learning by having the robot directly model and compensate for the ways that a user's corrections of robot behavior are sub-optimal along two pre-defined dimensions.  In this work, the authors show that telling a user about the dimension of their suboptimality, with feedback tied to their level of suboptimality, can change the user's feedback to be closer to optimal and that the MIND MELD architecture can be extended to compensate for those changes.
strengths_and_weaknesses: Strengths:
- This paper shows that users can be trained to give better feedback, which is likely to be of interest to human-in-the-loop learning researchers
- The authors present three studies to support each component of their work
- The paper provides an example of how a learning algorithm can compensate for changing feedback, which is an important problem in human-in-the-loop learning

Weaknesses:
- The paper is very difficult to understand on its own; too much material has been cut and moved to the appendix (perhaps from a longer submission?).  For example, the task and evaluation environment really do need to be described in the paper, not just the appendix.
- The contributions of the work are over-stated relative to what was actually done (the results are relevant to robotics but no real robot was involved; the "novel" tasks are the same task with a new goal location; it's not clear whether the "robotic" feedback was perceived as coming from the robot; the results are described as applying to suboptimality in general but only two dimensions of suboptimality are studied, and those dimensions are very task-specific; the authors state that feedback improved participant teaching, even though it only improved in one dimension, suggesting that there might be generalization problems)
- The paper does not provide strong evidence that the methods described will generalize to frameworks other than MIND MELD or tasks other than simulated driving.


Questions:
- How are the ground truth labels for the SVM classifier of suboptimality obtained? Why does the proposed method result in semantically meaningful labels? Would this method still result in semantically meaningful labels in another higher-dimensional task?
- What are the calibration tasks?
- How is feedback "provided" to users? Audio? Text?
- How were participants recruited? Why is the gender balance so off for all of the studies?

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Overall, this paper provides a minor contribution to the field of human-in-the-loop learning with a robotics-relevant validation, but no demonstration on an actual robot.  The most interesting result here is that feedback from a learning system can shift the distribution of demonstrations/corrections even over the course of a single interaction, and that the demonstrations can be made "better" from the robot's perspective.  Many of the technical results are an extension to an existing framework; it's not clear how they would apply to other learning algorithms or frameworks.  I'm a little concerned about how much the authors over-generalize their claims throughout the paper -- for example, "we demonstrate that robotic feedback can effectively improve human demonstrations (p < .001)" (13-14) and "We demonstrate that Reciprocal MIND MELD can improve an individual’s demonstrations across multiple dimensions of suboptimality" (58-59) -- but in all three studies improvements are only shown for the over-/under-correcting dimension, not the the anticipatory/delayed dimension.
issues: I would love to see a revision that is a little more accurate in its claims -- for example, contribution 3 in the introduction should read: "We validate Reciprocal MIND MELD in a simulated driving task and show that it can improve demonstrators' performance in the anticipatory/delayed dimension of suboptimality [...]"

The limitations section could speak more accurately to the potential for generalization (or not).  For example, in the general case, there is a chicken-and-egg problem with determining suboptimality of demonstrations: the learner needs to know what it looks like to do the task correctly in order to know whether the demonstration is good, but if you have a robust model of a good demonstration, you have a robust model of the task, in which case you don't actually need to learn.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: Authors propose an extension to MIND MELD which aims at improving the human's demonstrations via tailored feedback as opposed to optimising suboptimal demonstrations after the fact. They provide an application specific method for achieving this which is validated extensively by carrying out several experiments with a large number of human participants.
strengths_and_weaknesses: Strengths:
- Innovative idea to provide automatic tailored feedback and focus on improving demonstrations which is a largely unaddressed problem in the literature.
- Method is thoroughly evaluated using a large number of human participants and well justified by addressing each of the research questions with focused experiments.
- Paper overall well written and easy to read.
- EPN is interesting and could be applied to other problems.

Weaknesses:
- Relatively simple extension to existing work.
- Meaningful embeddings may not always be simple mappings such as over/under steering. Mapping between corrective actions and meaningful embeddings is simply proportional to the control error. Method is very application specific.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Although the proposed method is application specific, the main value in the paper is validating that providing tailored feedback to the demonstrator improves learning performance. This is valuable to the learning from demonstration community since it rationalises a promising direction for future research.
issues: Too much reliance on appendices and prior work. Should have minimum to understand content in main document. For example, the additional network head in 3.1 is difficult to understand without prior knowledge of MIND MELD. More details on/ example of a calibration task would be helpful. In section 4, at least a high level description of the robot task should be provided as well as how feedback is given. Related worked should be in main document.
review_title: null

Meta-review: metareview: ### Strengths
+ interesting topic
+ extensive and thorough experiments
+ potentially very useful and powerful

### Weaknesses
- doubts about assumption of a structured understanding of suboptimality and connection to reciprocal teaching
- key components in appendix
- method and experiments are too interwoven, it is not clear how/if the method will generalize
- contributions not accurately represented, and not clear from the start
- questions about only evaluating 2 dimensions of suboptimality
- domain and tasks need to be explained better

### Summary
A promising paper that will need a quite substantial revision. it will be tricky to make this paper work with the CoRL format/page limit (key components in appendix, might fit a longer journal format better)

### After rebuttal and discussion
The rebuttal and revision clarified the assumptions and limitations. The re-write significantly improved the paper, please also see the "Update after rebuttal" in the review of QQMM.


confidence: 5: The area chair is absolutely certain
Decision: Accept (Poster)