Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Strong Reject: I recommend rejecting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This work proposes a framework for using robot experts to teach human users to optimize collective rewards in collaborative tasks. This type of machine teaching is novel, in that in this scenario, the robot is the agent with expert knowledge that is to be transferred to the human, who is assumed to be initially inexperienced in the given task.

The proposed framework combines elements of active teaching — making assumptions that the robot teacher has full knowledge of the human’s optimal policy and can select meaningful states to use to teach the human — as well as elements of hierarchy and item response theory to decompose the behavior of the human into a set of a priori defined “sub-skills” that the robot teacher can decide between to teach the student.

The results show that student-aware teachers that reason over sub-skills are slightly better or on par with “fully assistive” teachers, and are slightly better than “random” teachers.
strengths_and_weaknesses: I think the motivation of this work is nice — I strongly believe that as our agents become more capable, they will play a huge role in teaching humans new skills; I think developing general frameworks to that end is immensely useful.

---

Unfortunately, I have some serious concerns with this work, stemming from the results and the initial setup and assumptions of the framework.

Regarding the results — I find it hard to justify the differences between the “fully assistive” baseline and the “student aware” (proposed approach) in both Figures 4 and 5. When it comes to a question of reward, the student aware baseline is not ever significantly better than the fully assistive strategy, and the graphs that do show a significant improvement (e.g., Figure 4b) are estimated over metrics such as “percentage of students who found the efficient strategy” — where efficient is defined as a factor of the underlying hierarchy that is only exposed to the student-aware agent! Furthermore, the “fully assistive” baseline is rather weak (and already hugely performant) — most “assistive” baselines for these types of dynamic/adaptive tasks look at annealing the amount of assistance that the robot gives over time either with a fixed, or learned schedule... approaches that seem like they’d significantly beat the student-aware policy by a sizable amount, given these results (and those in the appendix). I strongly believe this work needs to compare against these simple heuristic baselines to justify the complexity of the proposed approach.

The approach itself also gives me cause for concern — there are several complex moving parts, without meaningful justification of why each component is necessary. To start with, Definition 2 around the teaching task makes a *huge assumption* that the robot teacher has full knowledge of the oracle human policy, and is working to guide the human to that policy as fast as possible... however, it feels strange to me that there’s only “one” optimal policy that does not take into account the preferences of the user at all — indeed, this seems to be a huge hurdle for collaborative leader/follower tasks like those studied in this work.

The second major concern I have with the approach is the formulation of the teaching task as having the robot pick from a set of pre-defined skills, via Item Response Theory; Section 4.2 states that the sub-skills K is defined a priori (for example, in the OverCooked Task the skills are “putting ingredients into the pot”, and “delivering the soup” and for the real robot task, following a similar 2 sub-skill “divide-and-conquer” strategy). Furthermore, the parameters for the Item Response Theory component that the teacher uses to figure out what skills to teach the robot are not well-defined; 4.2 states that the difficulty parameter Beta and the Smoothness parameters Lambda are fixed constants... but Algorithm 1 indicates that these are all learned in tandem? How does this happen — none of the details are here, and a naive optimization of these parameters could lead to arbitrary outcomes.

robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Based on the lack of motivation and underspecification of the core algorithmic components of this work, the limited evaluation with lukewarm results, and the huge assumptions made about the knowledge of the robot teacher and tasks, I would advocate that the paper be rejected at this time.
issues: It seems that as a whole, the key component of this work that makes it better than random approaches or fully assistive approaches is the partitioning into sub-skills — not any of the added complexity of the item response theory or knowledge tracing components. The punchline of this paper seems like it can be crystallized as “in collaborative tasks with clear divide-and-conquer strategies, we can teach humans efficiently by teaching them to exploit one of the strategies, while the robot does the other.” This is not general to even normal collaborative HRI tasks (consider assistive feeding or collaborative construction), let alone broader tasks. The assumption that we can divide up a task into a small K <= 2 number of sub-skills is too strong, and coupled with the limited evaluation, and poorly justified algorithmic complexity, raise many issues I hope the authors can address in future submissions.

Minimally, I’d love to see a stronger set of baselines (with various heuristic “fully assistive” teachers), a set of ablations or simplification of the current algorithm for teaching, and more robust evaluation metrics.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work first proposed a conceptual framework for the cooperative robot teaching where a robot(autonomous agent) is the teacher, and a human user is the student. The proposed framework allows the robot teacher to adaptively select actions in order to accelerate the student's learning process. The authors then implemented a version of the framework and evaluated the system in two simple games with 20 and 16 human participants respectively. In their implementation, a simple one-parameter model and the Wiener process was used to model the learner's state and the knowledge acquisition process. Maximum a Posteriori Estimation was used to control the robot by learning three critical parameters (i.e. task difficulty, student proficiency, and the learning "smoothness") from experience. The user study provided some evident showing the proposed method works better than a fully assistive and a random teacher baseline. Yet, the experiments only provided very limited amount of conclusive results.
strengths_and_weaknesses: Strengths: This work presented an abstract framework for cooperative robot teaching, which specifically model the human learning process and allows the robot to select teaching strategy adaptively. The idea of robot teaching/coaching has been explored by a few existing HRI works, but has not yet been extensively studied. Therefore, the proposed framework provides an interesting research direction for the future works. The authors also provided a functional implementation of the frame, which can be used as a baseline system for the future works. The paper is well written, the ideas and concepts are clearly explained, making it very easy to follow.

Weaknesses: The proposed framework was tested in two simple environments with very small action, state, and parameter space. Applying this framework to a more difficult task can be very challenging, which is the major limitation of this work. Because accurately initializing the parameters and estimating the hidden human state can be drastically more difficult and data demanding as the task complexity increases. Moreover, the reviewer has questions and major concerns over the experiments:

What is the definition of "efficient strategy" in the two games and how was it identified during the experiments?

It is not clear to the reviewer what exactly do the figure 4b and the discuss on the percentage of reward(line 278 ~ 280) are trying to show and how do they support the argument about the fully-assistive teacher impedes learning.

In figure 4.a., any insights about why the students perform significantly worse in evaluation than in training in the fully-assistive scenario? With increased experience, one would expect a similar or better performance in simple task learning.

The claim in line 285, "random teacher is incapable of teaching a truly cooperative task", is not supported by the experiment. As the author stated, the Duo Ball Maze task requires at least some degree of cooperation between the two players to successfully finish the task. This means that the random agent is more of a disturbance generator instead of a cooperative helper. Drawing conclusions about if the human users have learned COOPERATIVE skills by directly comparing these two settings is not sufficient. Instead, a better experiment could be training the users via the two different teachers, and evaluating the users' skills with another unseen baseline cooperative robot.

Similarly, the statement in line 298, "Such a strategy…, therefore resulting in a poorer performance score", suffers from the same problem where the human performance of different settings are not directly comparable when evaluated by different robot partners.

For figure 6, how were the line plots created? And can you explain why are these individuals representative enough to speak for the whole population? Could this figure be replaced by some statistics that summarize the entire group?

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper presented a framework that allows a robot to assist human skills learning via cooperative HRI. The proposed idea is interesting, the system was well implemented, and the experiments have sufficient amount of human samples. Yet, the experimental results provided in the paper do not fully support some of the important claims. The conclusive results that can be draw from the current experiments are relatively weak.

I appreciate the authors' detailed responses. The biggest concern of mine about insufficient experimental setup appeared to be a misunderstanding on the experimental procedure. I am changing my recommendation to weak accept.
issues: Questions and concerns are described in the weaknesses section.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper tackles an interesting problem on enabling robot teachers to teach human students in a multi-agent cooperative setting through interaction. The authors formalize the problem, and propose a general conceptual framework along with one simplified instantiation with it. The method is evaluated in two tasks, overcooked in simulation and duo maze ball with robots
strengths_and_weaknesses: Strengths:
- The paper is well written. The motivation is very clear and straightforward. Method description is very detailed. Although it has a lot of notations since it formalizes the problem, they are all very clear. 
- The problem itself is novel and interesting. The paper focuses on a novel setting where the robot teaches the human partner to cooperate through interaction. 
- Although the specific instantiation of the framework makes certain assumptions and simplifies the problem, it manages to model and reason over the adaptive human, which inherently is challenging itself. 


Weaknesses:
- More reference is needed for motivation the problem. Even though the motivation is straightforward, adding more reference to support this might be better. Some examples are: 
* In the introduction, “the learner can no longer learn to cooperate by simply observing the demonstration” what are the drawbacks of directly learning from multi-agent demonstrations? 
* In the introduction, “human normally learns through interacting and practicing the skills with representative teachers, rather than exhaustively collecting demonstrations to cover all possible behaviors of the partners” Any literature to support this claim as well?
- Adding explicit statement of all the assumptions earlier might be clearer in the first place. E.g. the robot (teacher) assumes full knowledge about the task, sub-skills decomposition etc. 
- It somehow feels like the framework described in Section 3 is too general while the instantiation in Section 4 simplifies too much by contrast. I am not against a general setup and do appreciate that this paper tries to formalize the problem and propose a general framework. Maybe adding some discussion on potential instantiation/more application of this framework in appendix would make it stronger. 


robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall, this paper is well written, looks at a novel problem and makes a solid first attempt in solving this by proposing a general conceptual framework and a practical instantiation.  I recommend for acceptance.

issues: - Typo: figure 2 caption: ‘represnet’ should be ‘represent’

For further issues, please refer to the weakness.
review_title: null

Meta-review: metareview: Post discussion review
----------------------

The rebuttal does a good job of taking the reviewer comments seriously and using them to improve the paper.  The authors even went so far as to upload a revised manuscript with their changes. Additionally, they also engaged in a series of detailed discussions with the most critical reviewer. 

The additions and edits in the new manuscript really help clarify some of the main assumptions and highlight the value of the work. I believe that while there may be valid discussions to be had about certain limitations or assumptions, this work still has merit to the field.



Original review
---------------


This paper presents a framework for machine teaching of cooperative tasks. It includes evaluations in simulation (Overcooked) and on a physical robot (Duo Ball Maze). 


Strengths
* Reviewers agreed that this area of machine teaching is underexplored and that the work addresses an important and valuable task. 

* Reviewers noted that the paper is clear and well explained. 

* There are some interesting insights in this work, including that people with fully assistive robots are disincentivized to explore ("lazy student effect"), that people will learn from a random robot in tasks that are semi-independent but not in fully cooperative tasks.


Weaknesses
* Reviewers were concerned about significant assumptions that are required for the model to work. For example, the framework relies on the robot knowing the correct human policy and having full knowledge of the task. These seem like they would quickly be violated in real world settings. It would help to understand how the given approach would transfer to more realistic and complex situations, or at least in what settings these assumptions could hold.

* Reviewers critiqued the evaluations, noting that the tasks were simple and the results were only slightly better for the proposed framework. Reviewers suggested alternative baselines for both simulated and real world tasks. They also posed a number of clarification questions about how the results are presented.

* In general, the paper might benefit from being clearer about the limitations of this approach and how they might be mitigated in the future, rather than trying to unequivocally outperform baselines with the current implementation. 
confidence: 3: The area chair is somewhat confident
Decision: Accept (Poster)