COACH: Cooperative Robot Teaching
Authors: Cunjun Yu
Yiqing Xu
Linfeng Li
David Hsu
Venue: CoRL 2022 Poster (2022)
Keywords: Robot Teaching
Human-Robot Interaction

Abstract: Knowledge and skills can transfer from human teachers to human students. However, such direct transfer is often not scalable for physical tasks, as they require one-to-one interaction, and human teachers are not available in sufficient numbers.  Machine learning enables robots to become experts and play the role of teachers to help in this situation.  In this work, we formalize cooperative robot teaching as a Markov game, consisting of four key elements: the target task, the student model, the teacher model, and the interactive teaching-learning process.  Under a moderate assumption, the Markov game reduces to a partially observable Markov decision process, with an efficient approximate solution. We illustrate our approach on two cooperative tasks, one in a simulated video game and one with a real robot.

COACH: Cooperative Robot Teaching
Cunjun Yu'
Yiqing Xu'
Linfeng Li David Hs
School of Computing
Smart Systems Insitute
National University of Singapore
Knowledge and skills can transfer from human teachers to human stu- dents. However, such direct transfer often not scalable for physical tasks, as they require one-to-one interaction, and human teachers are not available in sufficient numbers. Machine learning enables robots to become experts and play the role of teachers to help in this situation. In this work, we formalize cooperative robot teaching as a Markov game, consisting of four key elements: the target task, the student model, the teacher model, and the interactive Under a moderate assumption, the Markov game reduces to a partially observable Markov decision process, with an efficient approximate solution. We illustrate our approach on two cooperative tasks, one simulated video game and one with a real robot.
Abstract:
Baaching-learning process.
Keywords: Robot Teaching, Cluman-Robo Interaction
Introduction
How do we teach humans to re-orientate a table jointly or play tennis? Humans often learn by practicing the skills with teachers or partners [1, 2, 3J. This mode of learning is, however, difficult to scale up, as it requires one-to-one interaction and there are not sufficient human teachers [4], With advances in machine learning, robots can not only master complex tasks [5, 6, 7] but also collaborate with humans and adapt to human behaviors [8, 9, 10]. In this work, we aim to create robot teachers for physical tasks, thus scaling up teaching and providing learning opportunities to many even when human teachers are not available.
Specifically, we propose Cooperative rOhot teACHing (COACH), a robot teaching framework to teach humans cooperative skills for wwo-player physical tasks through interaction. We assume the robot teacher has full knowledge of the task, specifically, a set of optimal policies. The objective is to teach the student optimal policy as fast as possible. See Fig. for an illustration. COACH treats the teaching task as a two-player Markov game for a target task. One player is the robot teacher, and the other is the human student. Under suitable student learning model, COACH transforms the game into a partially observable Markov decision process (POMDP). The POMDP solution enables the robot teacher to adapt to the different behaviors, according to the history of interactions.
One key challenge of COACH to represent the student's knowledge of the target skills and learn- ing behaviors. First, we leverage item response theory (IRT), well-established framework for educational assessment [11]. IRT provides simplified parametric models that capture the student's s knowledge level with respect to the task difficulty in a small number of parameters. COACH treats these parameters as latent variables in the teaching POMDP and learns them from human-robs in- teraction data by solving the POMDP. Next, to teach complex skills, we draw insights from student- centered learning [12] and human-robot cross-training [13]. We decompose a complex target skill into a set of sub-skills, based on the student's potential roles in the target task. With this compact, decomposed skill representation, we naturally obtain a partially assistive robot teaching curriculum to facilitate learning: the human student Icarns the sub-skills one at time, and the robot teacher assists with the sub-skills not yet learned, to complete the target task. While the robot assists the human in the teaching task, its behavior differs from those in common collaborative human-robot interaction tasks [14, 15J. There, the primary objective is to complete the task, and the robot fully assistive: if the human does not perform, the robot then tries to complete the task on its own, if
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Target Task
Teaching Task
human player
human player 2
robot teacher
human student
O
C 0
Figure 1. Cooperative robot teaching. In the target task (left), two human players jointly reorient table, for example. In the corresponding teaching task (right), the robot teacher interacts with the human student and teaches cooperative skills so that the student learns cooperate with partners with varying capabiliitess or preferences the target task.
possible. In the teaching task, the robot is partially assistive and usually avoids assisting with the specific sub-skill to be learned, in order encourage student exploration and learning.
As a first attempt, we conducted human-subject experiments on two challenging human-robor col- laboration tasks, Overcooked-A Al and Cooperative Ball Maze Fii.. 2). Our results show that COACH enables the robot teacher to model and reason over adaptive human students in cooperative teaching. Also, a fully-assistive teacher may impede student learning, and a partially assistive teacher indeed motivates the student to explore new strategies.
2 Related Work
Assistance in HRL. One major aspect of HRI is how the robot could assist humans with a hidden human objective [14, 15]. The objective of the robot is to infer the human's intention and learns to assist the human. In its simplest form, the action selection and human intention inference are separated [16, 17, 18]. A decision-theoretic framework, assistant POMDP, is developed to capture the general notion of assistance in HRI [19]. The robot integrates the reward learning and control modules to perform sophisticated reasoning over human feedback [20, 21]. However, these two approaches neglect human leaming/adaptationn and may hinder humans from improving their skills. Our work focuses on how to generate behaviors that facilitate human learning during interactions. Collaboration in HRI. Another important aspect of HRI is to model interactions as the collabo- ration between the human and the robot [22], for which the human and the robot share the same unknown objective. However, the joint optimal policy, e.g. rotating the table counter-clockwise, to both agents in the first place. Their interaction is mutually adaptative [23, 24, 25]. Particularly, as pointed out in [10], if one side is only aware of partial information about the task, the optimal policy pair naturally induces the behavior of active teaching, active learning, and efficient communication between the robot and human. In this work, we focus on the following setting: given that the robot teacher knows the optimal policy, how to carry active teaching.
Teaching Algorithm for Algorithms. Teaching for algorithms aims to facilitate the learning of the algorithm by choosing generating training samples. Various teaching techniques including curriculum learning [26] and machine teaching [27, 28, 29, 30, 31] have been effectively applied supervised learning and semi-supervised learning problems. Similar ideas are further extended to train reinforcement learning agents to learn complex skills, e.g., generate training environment for reinforcement learning [32, 33, 34], choose various demonstrations [35] or learn to decompose the skill [36, 37]. Teaching in cooperative multi-agent RL allows agents to simultaneously become teachers and students for each other [3, 38, 39]. However, such approaches generally require rel- atively more data for training and to some extent the controlled learning behavior of the learner. Transfer of these approaches human learning promising but difficult.
Teaching Algorithm for Human. Despite the aforementioned practical challenges, some algo- rithms have been successfully deployed for human learning. Attempts on teaching the crowd on classification concepts prove to be successful [40, 41, 42, 43, 44]. While humans can learn con- cepts from visual or verbal examples, complex skills like motor control skills can hardly mastered through these signals. Recently, skill discovery techniques in reinforcement learning have been in-

---

troduced to generate a curriculum based on skill decomposition and facilitate humans to learn motor control skills [45]. It focuses on how to adaptively decompose the skill into learnable sub-skills for human to practice on its own and achieves promising results. Here, we seek to automate the teaching process for humans cooperate in a physical task and provide framework for this teaching mode, e.g., table co-reorientation.
3 Cooperative Robot Teaching
We identify four key elements in COACH: (1) the target task, (2) the student learning model, (3) the teacher model, and (4) the interactive teaching-leaman process.
Target task. In this work, we focus on teaching in a (wo-player cooperative task, which we call it the larget task.
Definition
The target lask is two-player www-playercopperrtiie cooperative Markov game M -05 At, A2,,A2,,5 R,,)
between two agents, 1 and 2, where
S is a
target task states;
A is a set of actions for agent 1;
A is set of actions for agent 2:
Ts'l(s. aa,,"",,,, ,a²) is a conditional probability function on the next target task state E S,
given the current state S and both agents' actions a' A' and a² E A2:
Ris, a²,a²) is target task reward function that maps the target task state and players' actions to a real number; 7 is a discount factor.
At each step t, agent 1 and 2 both observe the current task state and select their respective actions at x² and af 72, where T' the policy of agent for i 1,2, They then receive joint reward = R(st, at, a²). The next state is updated as St+1 ~ T(s+11 8t; af))
Given the definition of the target task, first answer how represent the knowledge/skills. In this work, we choose to represent a skill by the optimal policy o² to the target task. The optimal policy maximizes the expected cumulative reward when the agent cooperating with given partner. For example, in the table co-reorientation task, the agent needs to learn to deal with either stubborn or adaptive partners. We recognize that there are other ways represent knowledge/skills,, such as a set of demonstrations and the ground-truth reward function. However, such representations are indirectly linked with the skill's performance; therefore, evaluating its proficiency is more obscured. We choose the optimal policy as the representation since it can be directly optimized over and evaluated.
Student. The student policy is non-stationary since it will improve along with teaching. We model this evolutionary behavior with tuple of student policy and updating function (o, U). The student policy represents the student knowledge state. will take in the current target task state s as input and output the student's action. The updating function U models how the student changes its policy after each teaching step.
Teacher. We define the teacher knowledgeable agent (expert) who knows a set of optimal policies d* for a target task. The teacher aims to acquire a teaching policy я that can teach any of E x* to the student effectively. In this general setting, the choice of the optimal student policy 6² depends on the capability, preference, and current knowledge level of the student. A principled approach to selecting the optimal student policy needs to consider the student's preference, his/her update model for the knowledge level, and an estimate of his/her current capability. In this paper, assume that we have an oracle choose the optimal student policy d' E Φ to teach, such that this policy o" matches the preference of the student. The teacher can be described by a tuple of an optimal target task policy and the corresponding teaching policy, (ø" ))).
Interactive teaching-learning. We now refer to the robot as the teacher and the human as the student. In the target task, the teacher knows the target task's optimal policy o² while the student does not. The teacher's goal is to act the most informative way so that the student learns o* fastest. The choice of ø* should account for the student's preferences. To embed the objective of teaching and distinguish it from the Target Task, we define it as the Teaching Task in the following way:
3

---

Definition 2 Given target task M = ".,A',A,,7 R. student (6. (). and an optimal
policy 6- for the target task, the teaching task is a POMDP M' (S, O, Z, R. 5) for the leacher, where
S is of teaching states: (8, Q) for target task state E S and student policy o;
A is set of actions: A = A A
T s s, a) a conditional probability functioo on the next state s' E S. given the current
state E S and teacher's is action â E A;
(s, r). for target task state s and target task reward ry Z( ō a, 5) is conditional probability function on the observation ō E O. given teacher's ss and current state $ S;
0 a set of observations:
action
R(s. a. 3') is teaching reward function that maps current state 3 E S, teacher's action A, and next state 8 S real number measuring the effectiveness of teaching;
7 is discount factor.
The objective of the teaching task is to derive a teaching policy x², enabling students to learn d for the target task fastest. More specifically, the teacher actions ā E A. Given the student policy & and the update function U, the student would learn through interaction: Φ++1 = U(),,-) The goal of the teaching task is find teaching policy x² that allows Φn + 6* as fast as possible.
influence the student through interactive
Next, introduce our choice of the teaching policy n², the update function U, and the reward function R. To devise a swwdent-aware teaching strategy, apart from the current state st and the target policy 6², our x" also takes the history of observation is h4 = [(50: ro).... (s). The action of the teacher can be sampled from the policy, i.c., ā ~ xt (āt h St, 6²)) The student updates & with any arbitrary iterative functions conditioned the history of interactions: Φ++1 (6,, h2). Moreover, to incentivize the teacher to speed up the teaching process, introduce step-wise teaching cost the teacher c. C(s,, @e) penalize unnecessary teaching actions. To this end, we define the reward function as
input. The history of observation
(1) where 3 = (st, 3' (1 d(+1), w is the weighting factor to trade-off the teaching cost and
R(3, oc, D,C, w) D( o²) - D(
teaching efficiency, and D can be any reasonable distance measure between two policies, e-g., initial state value in the target task. The solution to the POMDP M is a teaching policy x" that maximizes the expected sum of rewards Ea,- EE 5" R(s, Ce 8.)].
4 Method
In this section, we provide solution that grounds all the ingredients in the concep- tual framework of COACH. The main spirit of our solution to parameterize students' knowledge state with IRT and decompose complex tasks into a set of role-based inde- pendent skills. This enables easier evalua- tion of the students' proficiencies and pro- vides ground to derive the partially assis- tive interaction mode. To begin with, we
Algorithm 1 Approximated Solution to the Teaching Task
Require: Maximum Interactions I., Predefined Inter- actions N.
for E do:
2: Randomly initialize À and at 8, and X O for i 1, .... N do:
3:
4:
X.add(e;)
5: end for
6:
end for
7:
for i
2, do:
first define the action space, A.
8:
for k E A do:
9:
Learn İ and au,, , from X
10: end for
4.1 Actions
11: 6- Action selection from and a. 3
12: 0. + Performance measure from interactions
The action space is constructed through sub-skill decomposition. Sub-skills de- composition well-studied for single- agent tasks [46, 47, 48]. However, extending the same idea to multi-agent setting is still chal- lenging since task completion relies on the interaction among multiple parties. We observe that in multi-agent game, the task naturally comprises several roles, of which each agent takes a subset. The well-established leader-follower model is a particular choice of role-based skill decomposi- tion [49, 50, 51, 52]. Therefore in our work, we propose to decompose skills based on role alloca-
13: Xvadd(v:)
14:
end for

---

tion. We divide the skill into K independent teachable sub-skills according to the student's 's potential roles the task. The teacher's action space A = { : k E Z, 0 k < K} consists of teaching each sub-skill. Such decomposition of skills naturally leads to partially assistive mode of interaction.
4.2 States
The state space is constructed with Item Response Theory (IRT) [11]. IRT provides a parametric form to represent students' skill levels. Given the limited interactions, we adopted the simplest form, the paramaraeteer logistic model (IPL), to model human skills. In the IPL model, each sub- skill & E A is assigned a parameter 3 representing the difficulty, and a parameter a called the proficiency representing student's knowledge state. The probability that student has mastered sub-skill is given by P(k) == aaa 3*), where a is the sigmoid function. Hence, instead of representing the state with the student's policy é. we use (a, 3) to represent the hidden state. That is, for & E S, $ = (s, (a, 3) ). where (a: 3) is hidden. For each student and each &s E A, we
assume that changes over time while does not.
4.3 Transitions
The transition model consists of two main parts, the target task transition model T,, and the student's update function U. While the former one is known to the teacher, we need to make assumptions about the latter one. Since we define the state space over the student's proficiency a in Sec 4.2, the transition model is also constructed over the proficiency. Following the previous work on online estimation of student proficiency [53, 54], for each sub-skill, we model the student's proficiencies
over time Wiener process: U(ar4a/lau) = exp (oL)))-a)))) where At refers to the step
interval and À is parameter controlling the "smoothness" with which student's proficiency varies over time. For each student and for each E A, assume À does not change over time and is learned for each sub-skill respectively. To this end, we construct the transition model in the POMDP as T (T, U}, where T is the transition function in the target task.
4.4 Observations
The observation is composed of the target task state and the reward received, (s, Recall that in Sec 4.1, we define the action as choosing one sub-skill to train the student, which is macro- action. For teaching sub-skill k; we redefine the observation as the ratio between the target task R(a.a.=))
rewards achieved by the student's current and optimal policies: action generated by the optimal policy 6* given the same the target task state s. Since all the sub- skills are treated equally, we will omit the index k for simplicity in the following discussion. As a result, for ō E Ô, = (s,u). 2). Unlike the binary response in conventional knowledge tracing, the response u we have is continuous and we assume the teacher will only teach one sub-skill at construct the observation model: ((c] P(k)) = (1 P(k))=-r, where & is the sub-skill being taught when is observed. As result, the observation model can defined as Z (T, Z). where an identity mapping for the observable target task state, T(s)
R(s.a.a")) where a" is the
time. Thus, we use the continuous Bernoulli distribution
4.5 Reward
The distance between the student's policy and the optimal policy can be represented using P(k). We represent the distance as the average one minus master probabilities of each sub-skill: D(o, o*) =
o I-((k) There are other ways to specify the goal according to the decomposition of the skill, K
e.g. weakest multiply [55]. We choose the sum due to our independence assumption on sub- skills. In this work, we assume the cost is uniform, thus, given a finite horizon of interactions, maximizing the reward function defined in Equation (1) is equivalent to maximizing R(3, ā,, 3")
>k Pt 1(k)- Pi(k) where (kk) o(a - -*)).
K
5

---

4.6 Model Learning and Decision Making
We use the student's performance during the interactions estimate both À and ar, B. Parameters for each sub-skill are learned separately, thus, we omit & for simplicity. Let U1:t denote sequences of student's performance measure against the optimal policy up to step We have the posterior P(X, 04t, 3/u1=)) x (( 0t, B) P(X, Ot, 04, B). The conditional probability of the observation and current proficiency can obtained by integrating out all the previous proficiencies. The likelihood can approximated through P(v Y ar 3) R II P(ve] A, at', [av[lo ( An ap- proximation of the log posterior over the student's current proficiency given previous responses can be derived to learn the parameters À and or,, B. Following [53, 54J. we employ maximum a posteri- ori estimation (MAP) to learn these parameters. Given the estimation of current state using the past history, we use one-step look- -ahead to reduce the impact of the inaccuracy in the transition function. At timestep t, the teacher's 's action is given
/ U(o"=alat) dayt1 - Pa(k).
ā++1 arg max
(2)
Кеλ
In practice, the student is asked to perform each sub-skill for few interactions to initialize the parameters.
4.7 Training on Sub-skills
Our overall strategy for training students each sub-skill is to diversify scenarios the student would encounter during training. Training students on sub-skills naturally leads to a partially assistive partner unlearned sub-skills, which allows the student to explore the sub-skill freely. We adopt an intuitive assumption: an agent learns cooperation betler with a diverse group of partners. Such a teaching strategy is effective when dealing with synthetic students [56, 57J. The student could learn from a diverse set of partially assistive partners or learn to cope with them by acquiring new skills.
5
Experiments
We carried out two human-subject experiments to demonstrate how COACH works, one in simu- lation (Overcooked-AI [58]) and the other with real robot (Cooperative Ball Maze). Experiment setups are shown in Figure 2. We investigated the teaching performances three types of teachers: the fully-assistive teacher who performs optimally concerning the student's initial capability, the student-aware teacher who behaves according to our teaching strategy, and the random teacher. The random teacher in the Cooperative Ball Maze experiment chooses sub-skills randomly, while the random teacher in the Overcooked-A experiment executes actions randomly.
5.1 Setups
Overcooked-AL Overcooked-AI is a benchmark environment for fully cooperative human-Al task performance and has become a well-established domain for studying coordination [59, 60, 61, 62]. The goal of the game is to cook and deliver much soup possible in limited time. We decompose the policy into two sub-skills:
putting ingredients in the pot and delivering the soup. To put ingredients in the pot, there exists efficient strategy
pass the in- gredient through the middle table. In brief, time
rather than picking up one onion and putting them into the pot, the efficient strategy 1) put multiple onions on the mid- dle table; 2) go to the pot: 3) pick up onions from the middle table; 4) put them into the pot. The overall idea is to reduce the num- ber of movements needed deliver the same amount of ingredients. We recruited N=20 (8 females and 12 males) participants and randomly assigned them into three groups, each with a different teaching strategy. Students are trained with different teachers and are evaluated with common unseen partner. We emulate the human partner
(b)
Figure 2. Experiment setups. (a) Overcooked-AI lay- out: human participants control the "cher" and the robot controls the "robot". (b) The real robot setup of Cooperative Ball Maze with a simplified setting.

---

1.0
Train
0.75
Reward
Eival
Percemage
Parcemage
0.50
0.5
0.25
Stadent Aware
Fully Assistive
Random
0.00
Student
Pully Assistive
Random
0.0
Student
Fully Assistive
Random
Aware
Aware
(a)
(b)
(C)
Figure 3. Results of the Overcooked-Al experiment. (a) Rewards achieved together by the human-robot
pairs during training and evaluation. The eπrrr bars correspond the 95% confidence intervals (95%CT) The student-aware teacher oulperformed the fully assistive and the random teachers in terms of the evaluation reward (with one-sided p-values 0.001 0.03). (b) Percentage of students who found the efficient strat- egy. None of the students are aware of this strategy at the beginning of the training. (c) Percentage of reward achieved by the human participants during training.
1.0
0.4
Leading
Performance
Improvement
0.3
0.6
0.4
0.2-
Lending Following
0.2
0.0
0.5 Following
1.0
0.0
Student
Fully
Random
Student Aware
Fully Assistive
Random
Aware
(a)
(b)
(c)
Figure 4. Results of the Cooperative Ball Maze experiment. Evaluation of performances of the sub- skills of participants. The marker styles correspond the sub-skill preferences of the participants. (b) Evaluation performances. The error bars correspond the 95%CIs. (c) Improvements after 20 interactions. The error bars correspond to the 95%CIs. The students improve more under student-an student-aware aware teachers than both fully-assistive and random teachers (with one-sided p-values 0.069 and 0.039).
evaluation using a trained model. Each participant was trained for games and then evaluated for game.
Cooperative Ball Maze. The Cooperative Ball Maze game requires coordination from both the robot and the human. Each party will hold one side of the maze board and tilt it to move the ball out from one the two exits. We define two sub-skills leading the rotation and following the rotation. We recruited N=21 (10 females and 11 males) participants to carry human human-subject experiments. The participants were first evaluated the two sub-skills, then trained for 20 interactions, and finally evaluated in the two sub-skills again. Details can be found in the supplementary materials.
5.2 Results
A fully-assistive teacher impedes human's acquisition of skills. the Overcooked-Al experiment shown in Figure 3(a), we observe that the students trained with fully-assistive teacher perform worse than the students with a random teacher: it seems that student becomes "lazy" and free rides the teacher when the teacher unilaterally adapts to the student and performs optimally. We further investigate the learning pattern the "lazy student" problem and find out that this "laziness" does not lie in the student's 's reluctance to lake actions, but rather in the lack of motivation to explore and improve. In Figure 3(c), we show the percentage reward achieved by student Overcooked- Al during training. Compared with the student-aware counterpart, the percentage of reward achieved by humans is similar. However, only 17% of the participants of the group find out the efficient strategy (Figure 3(b)), which is crucial to achieving high scores in the evaluation.
Partially assistive or random partner motivates students explore new strategies. By leaving some/all work to the student, partially assistive and random teachers both motivate the student to acquire new skills. This shown in Figure 3(b) that most of the students under these two teachers can find out the efficient strategy in Overcooked-AL. However, their performance and the robustness of the learned strategies differ significantly. Though multiple explanations could account for we hypothesize the student under the random teacher learns a single fixed strategy to finish the task alone (Figure 3())). Such strategy that completes task alone cannot utilize the possibly helpful inputs from the partner, therefore resulting in poorer performance score.
7

---

An individualized curriculum should be designed for the student. In the post-experiment survey of Cooperative Ball Maze, we asked the participants "which mode of the robot is easier to cooperate with?". Out of the 21 participants, 4 participants preferred to follow the robot and 17 participants preferred to lead the robot. Moreover, as we evaluated the student performance with partners of different sub-skills, we found that the student performances were consistent with their declared preferences (Figure 4(a)). That is to say, the student may have a bias over which strategy to ac- quire, and tailoring the teaching curriculum to focus on that specific strategy is efficient and more intuitive to the student. As demonstrated in Figure 5(a), after the first 6 trials that estimated the stu- dent's 's proficiency for each sub-skill, the teacher found out this student improved more as the leader, therefore, the teacher allocated 10 trials to perfect the leading sub-skills and only 4 trials for fol- lowing. Moreover, one participant in the random teacher group responded "the robot leading mode is too difficult and I gave up". This demonstrates the importance of an individualized curriculum: though there are multiple equally optimal strategies, the individual may have strong preferences, and teaching a non-preferable strategy will discourage the student from learning anything at all. We refer the readers to the Appendix for the complete data of all participants.
0.8
0.8
0.6
0.6
6 Limitation
0.4
0.4
0.2
0.2
Decomposition into sub-skills. For many tasks, it
0.8
08-
is not easy to identify distinct roles to fulfill the local-independence criteria of sub-skills. We man- ually decompose the skill into a few sub-skills ac- cording to the role of the student. Often, such a de- composition may not be possible or requires care- ful design. We can mitigate this problem with re- cent progress skill decomposition in single-agent task [45] and role-based task decomposition in multi- agent tasks [63]. However, it still demands much more effort to verify their efficacy with a real human on weal-world tasks.
0.4
0.6
04
0.6
0.2
02-
0.0
(a)
(b)
Figure 5. Sub-skill performances (vertical axis) with respect to training progress (hor- izontal axis) of two example participants trained by the student-aware teacher. Dots represent the raw scores and lines represent the smoothed scores. The top and bottom figures correspond to leading and follow- ing sub-skills respectively. (a) Participant 4. The student improved more when trained in the leading sub-skill. (b) Participant 6. The student improved more when trained in the following sub-skill.
Teacher's Knowledge. In the definition of the teach- ing task, we assume the teacher has full knowledge of optimal policies. However, it can be hard for the robot to know the oracle human policy beforehand. To make the conceptual framework practical, we need to relax the requirement on the teacher's prior knowledge. In our implementation, we reduce such assumption by approximating the distance to the optimal policy by the difference in performances. There can be cases where the optimal perfor- mance is hard know or such relaxation results in severe information loss. We need more insights on tasks to make the framework practical.
Curriculum design. In this work, we only design the curriculum over different sub-skills. However, during our experiment, we observe that humans show various responses to the same sub-skill of different difficulties. One specific finding is that people may give up learning when the task becomes too difficult. As a result, a finer-grained curriculum on the sub-skill training shall be generated to further facilitate human learning.
7 Conclusion
In this work, we propose a conceptual framework, Cooperative Robot Teaching, that enables robots to teach humans cooperative tasks. We show that, by abstracting a teaching task over the original duo cooperative task, the robot can learn to act as a specialized teacher to humans. To be more specific, we model the teaching task as a POMDP with hidden student policy and propose a partially assistive teaching curriculum to support human learning. We believe that robot teaching fills in the gap in the bilateral knowledge transfer HRI: unlike other HRI tasks where the humans instruct the robots how to behave, the role is reversed and robots try to instill the knowledge back into humans. Despite the challenges that lie ahead, we believe that robot teaching has great potential and is a necessary step forward to bring robots closer to our daily life.
8

---

Acknowledgements. This research is supported in part by the National Research Foundation, Sin- gapore under its Medium Sized Centre Program, Center for Advanced Robotics Technology Inno- vation (CARTIN), and Al Singapore Programme (AISG Award No: 101---hD-2022-0-022-52 and AISG2-PhD-2201--00 -000000))), and by the Science and Engineering Research Council, Ageney of Science, Technology and Research, Singapore, under the National Robotics Program (Grant No. 192 25 00054).
References
[1] R. M. Gillies. Cooperative learning: Review of research and practice. Australian Journal of Teacher Education, 41:39-54, 2016.
[2] J. A. Ross and E. M. Smyth. Differentiating cooperative learning to meet the needs of gifted learners: A case for transformational leadership. Journal for the Education the Gifted, 19: 63-82, 1995.
[3] S. Omidshafiei, D. K. Kim, M. Liu, G. Tesauro, M. Riemer, C. Amato, M. Campbell, and J. P. How. Learning teach in cooperative multiagent reinforcement learning. AAAI Conference on Artificial Intelligence, 2019.
[4] E. García and E. Weiss. The teacher shortage real, large and growing. and worse than we thought. the first report in" the perfect storm in the teacher labor market" series. Economic Policy Institute, 2019.
[5] D. Silver, A. Huang, C. Maddison, A. Guez, L.. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalch- brenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529:484-503, 2016.
[6] A. Coates, P. Abbeel, and A. Y. Ng. Apprenticeship learning for helicopter control. Commu- nications of the ACM, 52(7):97-105, 2009.
[7] Suárez-Ruiz, X. Zhou, and Q--C. Pham. Can robots assemble ikea chair? Robotics, 3(17), 2018.
Science
[8] S. Nikolaidis and J. Shah. Human-robot tross-training: Computational formulation, modeling and evaluation of human team training strategy. In ACM/IÉEE International Conference Homman-RRbbol Interaction, 2013.
[9] M. Chen, H. Soh, D. Hsu, S. Nikolaidis, and S. Srinivasa. Trust-aware decision making for human-robot collaboration: Model learning and planning. ACM Transactions on Human-Robot Interaction, 1-2); 2020.
[10] D. HadnelldMeenell, S. Russell, P. Abbeel, and A. Dragan. Cooperative inverse reinforcement learning. In Advances in Neural Information Processing Systems, 2016.
[11] R. Hambleton and H. Swaminathan. Item Response Theory: Principles and Applications. Evaluation in education and human services. 2013.
[12] L. Jones. The Student-centered Classroom. 2007.
[13] S. Nikolaidis and Shah. Human-robot cross training: Computational formulation, modeling and evaluation of human team training strategy. ACM/IEEE International Conference Human- Human-Robot Interaction, pages 33-40, 2013.
[14] A. D. Dragan and S. Srinivasa. policy-blending formalism for shared control. Interna- tional Journal of Robotics Research, 32(7):790-80) 2013.
[15] S. Reddy, A. D. Dragan, and S. Levine. Shared autonomy via deep reinforcement learning, In Robotics: Science and Systems, 2018
[16] Leike, D. Krueger, T. Everitt M. Martic, V. Maini, and Legg. Scalable agent alignment via reward modeling: research direction. CoRR, 2018.

---

[17] H. J. Jeon, S. Milli, and A. Dragan. Reward-rations (implicit) choice: A unifying formalism for reward learning. Advances in Neural Information Processing Systems, 2020.
[18] F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, 2017.
[19] A. Fern, S. Natarajan, K. Judah, and P. Tadepalli. A Journal of Artificial Intelligence Research, 50:71-104, 2014.
model of assistance.
[20] R. Shah, P. Freire, N. Alex, R. Freedman, D. Krasheninnikov, L. Chan, M. D. Dennis, P. Abbeel, A. Dragan, and S. Russell. Benefits of assistance over reward learning, 2021.
[21] O. Macindoe, L. Pack Kaelbling, and T. Lozano-Pérez. Pomcop: Belief space planning for sidekicks in cooperative games. AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, 2021.
[22] B. J. Grosz and S. Kraus. Collaborative plans for complex group action. Artificial Intelligncce, (6(2):269-357, 1996.
[23] Nikolaidis, A. Kuznetsov, D. Hsu, and S. Srinivasa. Formalizing human-robot mutual adap- tation via a bounded memory based model. In ACM/EEEE International Conference on Human Robot Interaction, 2016.
[24] S. Nikolaidis, D. Hsu, and S. Srinivasa. Human-robot mutual adaptation in collaborative tasks: Models and experiments. International Journal of Robotics Research, (6(5-7))618-66) 2017.
[25] S. Nikolaidis, Y. X. Zhu, D. Hsu, and S. Srinivasa. Human-robot mutual adaptation in shared autonomy. In ACM/IEEE International Conference on Himann-Robo Interaction, 2017.
[26] Y. Bengio, J. Louradour, R. Collobert, and J. Weston. Curriculum learning. In International Conference on Machine Learning, 2009.
[27] X. Zhu. Machine teaching: An inverse problem to machine learning and an approach toward optimal education. AAAI Conference on Artificial Intelligence, 2015.
[28] J. Liu, X. Zhu, and H. Ohannessian. The teaching dimension of linear learners. In International Conference Machine Learning, 2016.
[29] Mei and X. Zhu. Using machine teaching identify optimal training-ser set attacks on machine learners. In AAAI Conference on Artificial Intelligence, 2015.
[30] F. Khan, B. Mutlu, and J. Zhu. How do humans teach: On curriculum learning and teaching dimension. In Advances in Neural Information Processing Systems, 2011.
[31] W. Liu, B. Dai, A. Humayun, C. Tay, C. Yu, L. B. Smith, J. M. Rehg, and L. Song. Iterative machine teaching. In International Conference on Machine Learning, 2017.
[32] I. Gur, N. Jaques, Y. Miao, J. Choi, M. Tiwari, H. Lee, and A. Faust. Environment Genera- tion for Zero-Shot Compositional Reinforcement Learning. Advances in Neural Information Processing Systems, 2022.
[33] R. Portelas, C. Colas, K. Hofmann, and P.-Y. Oudeyer. Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environments. Annual Conference on Robot Learning, 2019.
[34] M. Fontaine*, Y.-C. Hsu* Y. Zhang B. Tjanaka, and S. Nikolaidis. On the Importance of Environments in Human- Coordination. Robotics: Science and Systems, 2021.
[35] D. S. Brown and S. Niekum. Machine teaching for inverse reinforcement learning: Algorithms and applications. AAAI Conference on Artificial Intelligence, 2019.
[36] B. Gerkey and M. J. Matarić. formal analysis and taxonomy of task allocation in multi- robot systems. The International Journal of Robotics Research, 23(9):939999 9544, 2004.
10

---

[37] J. P.. Gonzälez-Breee and J. Mostow. What and when do students learn? fully data-driven joint estimation of cognitive and student models. In Educational Data Mining, 2013.
[38] D. K. Kim, M. Liu, S. Omidshafiei, S. Lopez-Cot, M. Riemer, G. Habibi, G. Tesauro, S. Mourad, M. Campbell, and J. P. How. Learning hierarchical teaching policies for coop- erative agents. International Joint Conference on Autonomous Agents and Mulliagent Systems, 2020.
[39] N. Jaques, A. Lazaridou, E. Hughes, C. Gulcehre, P. A. Ortega, D. J. Strouse. J. Z. Leibo, and N. de Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. International Conference on Machine Learning, 2019.
[40] A. N. Rafferty, E. Brunskill, T. L. Griffiths, and P. Shafto. Faster teaching by pomdp planning. In Artificial Intelligence in Education, pages 280-287, 2011.
[41] A. Singla, I. Bogunovic, G. Bartók, A. Karbasi, and A. Krause. Near-optimally teaching the crowd classify. International Conference on Machine Learning. 2014.
[42] S. Zilles, S. Lange, R. Holte, and M. Zinkevich. Models of cooperative teaching and learning. Journal of Machine Learning Research, 12:349-384, 2011.
[43] T. Doliwa, H. U. Simon, and S. Zilles. Recursive teaching dimension, learning complexity, and maximum classes. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2010.
[44] O. Mac Aodha, S. Su, Y. Chen, P. Perona, and Y. Yue. Teaching categories to human learners with visual explanations. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[45] M. Srivastava, E. Biyik, S. Mirchandani, N. Goodman, and D. Sadigh. Assistive teaching of motor control tasks to humans. In Advances in Neural Information Processing Systems, 2022.
[46] A. Graves. Supervised Sequence Labelling with Recurrent Neural Networks. Studies in com- putational intelligence. 2012.
[47] K. Shiarlis, M. Wulfmeier, S. Salter, S. Whiteson, and I. Posner. TACO: Learning task decom- position via temporal alignment for control. In International Conference on Machine Learning, 2018.
[48] т. Kipf, Y. Li, H. Dai, V. Zambaldi, A. Sanchez-Gonzalez, E. Grefenstette, P. Kohli, and P. Battaglia. Compile: Compositional imitation learning and execution. In International Con- ference on Machine Learning, 2019.
[49] P. Evrard and A. Kheddar, Homotopy switching model for dyad haptic interaction in physical colàâlaborative tasks. In Joint Euro Haptics conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2009.
[50] A. Kheddar. Human-robot haptic joint actions is an equal control-sharing approach possible?
In International Conference on Human System Interactions, pages 268-273, 2011.
[51] N. Jarrassé, T. Charalambous, and E. Burdet. A framework to describe, analyze and generate interactive motor behaviors. Plos One, 1111):1-13, 2012.
[52] A. Mörtl, M. Lawitzky, A. Kucukyilmaz, M. Sezgin, C. Basdogan, and S. Hirche. The role of roles: Physical cooperation between humans and robots. The International Journal of Robotics Research,
2012.
[53] C. Ekanadham and Y. Karklin. T-skirt: Online estimation of student proficiency in adaptive learning system. Machine Learning for Education Workshop at ICMI., 2017.
[54] K. H. Wilson, Y. Karklin, B. Han, and C. Ekanadham. Back to the basics: Bayesian extensions of irt outperform neural networks for proficiency estimation. In Educational Data Mining, 2016.
11

---

[55] J. Gonzalez- Bonzalez-Brenes, Y. Huang, and P. Brusilovsky. General features in knowledge tracing to model multiple subskills, temporal item response theory, and expert knowledge. Educational Data Mining, 2014.
[56] A. Lupu, B. Cui, H. Hu, and J. Foerster. Trajectory diversity for zero-shot coordination. In International Conference on Machine Learning, 2021.
[57] R. Zhao, J. Song, H. Haifeng, Y. Gao, Y. Wu, Z. Sun, and Y. Wei. Maximum Entropy Popula- tion Based Training for Shro Human-AI Coordination. CoRR, 2021.
[58] M. Carroll, R. Shah, M. K. Ho, I. Griffiths. S. A. Seshia, Abbeel, and A. D. Dragan. On the utility of learning about humans for human-ai coordination. In Advances in Neural Information Processing Systems, 2019.
[59] Knott, M. Carroll, S. Devlin, K. Ciosek, Hofmann, D. Dragan, and Shah. Evaluat- ing the robustness of collaborative agents. In International Joint Conference on Autonomous Agents and Multiagent Systems, 2021.
[60] R. Charakom, P. Manoonpong, and N. Dilokthanakul. Investigating partner diversification methods in cooperative multi-agent deep reinforcement learning. In Iconip, 2020.
[61] Nalepka, J. Gregory-Dunsmore, J. Simpson, G. Patil, and M. Richardson. Interaction flex- ibility in artificial agents teaming with humans. In Annual Meeting of the Cognilive Science Society, 2021.
[62] B. Sarkar, A. Talati, A. Shih, and S. Dorsa. Pantheonrl: A marl library for dynamic training interactions. In AAAI Conference on Artificial Intelligence (Demo Track), 2022.
[63] T. Wang, T. Gupta, A. Mahajan, B. Peng. S. Whiteson, and C. Zhang. (RODE}: Learning roles to decompose multi-agent tasks. International Conference on Learning Representations, 2021.
12


Reviews:
[{"review_id": "x2EclEEh_49", "reviewer": "Reviewer_aCDa", "summary": "This work proposes a framework for using robot experts to teach human users to optimize collective rewards in collaborative tasks. This type of machine teaching is novel, in that in this scenario, the robot is the agent with expert knowledge that is to be transferred to the human, who is assumed to be initially inexperienced in the given task.\n\nThe proposed framework combines elements of active teaching — making assumptions that the robot teacher has full knowledge of the human’s optimal policy and can select meaningful states to use to teach the human — as well as elements of hierarchy and item response theory to decompose the behavior of the human into a set of a priori defined “sub-skills” that the robot teacher can decide between to teach the student.\n\nThe results show that student-aware teachers that reason over sub-skills are slightly better or on par with “fully assistive” teachers, and are slightly better than “random” teachers.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think the motivation of this work is nice — I strongly believe that as our agents become more capable, they will play a huge role in teaching humans new skills; I think developing general frameworks to that end is immensely useful.\n\n---\n\nUnfortunately, I have some serious concerns with this work, stemming from the results and the initial setup and assumptions of the framework.\n\nRegarding the results — I find it hard to justify the differences between the “fully assistive” baseline and the “student aware” (proposed approach) in both Figures 4 and 5. When it comes to a question of reward, the student aware baseline is not ever significantly better than the fully assistive strategy, and the graphs that do show a significant improvement (e.g., Figure 4b) are estimated over metrics such as “percentage of students who found the efficient strategy” — where efficient is defined as a factor of the underlying hierarchy that is only exposed to the student-aware agent! Furthermore, the “fully assistive” baseline is rather weak (and already hugely performant) — most “assistive” baselines for these types of dynamic/adaptive tasks look at annealing the amount of assistance that the robot gives over time either with a fixed, or learned schedule... approaches that seem like they’d significantly beat the student-aware policy by a sizable amount, given these results (and those in the appendix). I strongly believe this work needs to compare against these simple heuristic baselines to justify the complexity of the proposed approach.\n\nThe approach itself also gives me cause for concern — there are several complex moving parts, without meaningful justification of why each component is necessary. To start with, Definition 2 around the teaching task makes a *huge assumption* that the robot teacher has full knowledge of the oracle human policy, and is working to guide the human to that policy as fast as possible... however, it feels strange to me that there’s only “one” optimal policy that does not take into account the preferences of the user at all — indeed, this seems to be a huge hurdle for collaborative leader/follower tasks like those studied in this work.\n\nThe second major concern I have with the approach is the formulation of the teaching task as having the robot pick from a set of pre-defined skills, via Item Response Theory; Section 4.2 states that the sub-skills K is defined a priori (for example, in the OverCooked Task the skills are “putting ingredients into the pot”, and “delivering the soup” and for the real robot task, following a similar 2 sub-skill “divide-and-conquer” strategy). Furthermore, the parameters for the Item Response Theory component that the teacher uses to figure out what skills to teach the robot are not well-defined; 4.2 states that the difficulty parameter Beta and the Smoothness parameters Lambda are fixed constants... but Algorithm 1 indicates that these are all learned in tandem? How does this happen — none of the details are here, and a naive optimization of these parameters could lead to arbitrary outcomes.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Based on the lack of motivation and underspecification of the core algorithmic components of this work, the limited evaluation with lukewarm results, and the huge assumptions made about the knowledge of the robot teacher and tasks, I would advocate that the paper be rejected at this time.", "technical_quality": "Poor: Lack of required evidence to support the claims.", "clarity_of_presentation": "Poor: Difficult to follow and understand.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "It seems that as a whole, the key component of this work that makes it better than random approaches or fully assistive approaches is the partitioning into sub-skills — not any of the added complexity of the item response theory or knowledge tracing components. The punchline of this paper seems like it can be crystallized as “in collaborative tasks with clear divide-and-conquer strategies, we can teach humans efficiently by teaching them to exploit one of the strategies, while the robot does the other.” This is not general to even normal collaborative HRI tasks (consider assistive feeding or collaborative construction), let alone broader tasks. The assumption that we can divide up a task into a small K <= 2 number of sub-skills is too strong, and coupled with the limited evaluation, and poorly justified algorithmic complexity, raise many issues I hope the authors can address in future submissions.\n\nMinimally, I’d love to see a stronger set of baselines (with various heuristic “fully assistive” teachers), a set of ablations or simplification of the current algorithm for teaching, and more robust evaluation metrics."}}, {"review_id": "BFDjbvMUP_u", "reviewer": "Reviewer_GwM5", "summary": "This paper tackles an interesting problem on enabling robot teachers to teach human students in a multi-agent cooperative setting through interaction. The authors formalize the problem, and propose a general conceptual framework along with one simplified instantiation with it. The method is evaluated in two tasks, overcooked in simulation and duo maze ball with robots", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well written. The motivation is very clear and straightforward. Method description is very detailed. Although it has a lot of notations since it formalizes the problem, they are all very clear. \n- The problem itself is novel and interesting. The paper focuses on a novel setting where the robot teaches the human partner to cooperate through interaction. \n- Although the specific instantiation of the framework makes certain assumptions and simplifies the problem, it manages to model and reason over the adaptive human, which inherently is challenging itself. \n\n\nWeaknesses:\n- More reference is needed for motivation the problem. Even though the motivation is straightforward, adding more reference to support this might be better. Some examples are: \n* In the introduction, “the learner can no longer learn to cooperate by simply observing the demonstration” what are the drawbacks of directly learning from multi-agent demonstrations? \n* In the introduction, “human normally learns through interacting and practicing the skills with representative teachers, rather than exhaustively collecting demonstrations to cover all possible behaviors of the partners” Any literature to support this claim as well?\n- Adding explicit statement of all the assumptions earlier might be clearer in the first place. E.g. the robot (teacher) assumes full knowledge about the task, sub-skills decomposition etc. \n- It somehow feels like the framework described in Section 3 is too general while the instantiation in Section 4 simplifies too much by contrast. I am not against a general setup and do appreciate that this paper tries to formalize the problem and propose a general framework. Maybe adding some discussion on potential instantiation/more application of this framework in appendix would make it stronger. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, this paper is well written, looks at a novel problem and makes a solid first attempt in solving this by proposing a general conceptual framework and a practical instantiation.  I recommend for acceptance.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "- Typo: figure 2 caption: ‘represnet’ should be ‘represent’\n\nFor further issues, please refer to the weakness.\n", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "2aJUl4D6oGZ", "reviewer": "Reviewer_thAt", "summary": "This work first proposed a conceptual framework for the cooperative robot teaching where a robot(autonomous agent) is the teacher, and a human user is the student. The proposed framework allows the robot teacher to adaptively select actions in order to accelerate the student's learning process. The authors then implemented a version of the framework and evaluated the system in two simple games with 20 and 16 human participants respectively. In their implementation, a simple one-parameter model and the Wiener process was used to model the learner's state and the knowledge acquisition process. Maximum a Posteriori Estimation was used to control the robot by learning three critical parameters (i.e. task difficulty, student proficiency, and the learning \"smoothness\") from experience. The user study provided some evident showing the proposed method works better than a fully assistive and a random teacher baseline. Yet, the experiments only provided very limited amount of conclusive results.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: This work presented an abstract framework for cooperative robot teaching, which specifically model the human learning process and allows the robot to select teaching strategy adaptively. The idea of robot teaching/coaching has been explored by a few existing HRI works, but has not yet been extensively studied. Therefore, the proposed framework provides an interesting research direction for the future works. The authors also provided a functional implementation of the frame, which can be used as a baseline system for the future works. The paper is well written, the ideas and concepts are clearly explained, making it very easy to follow.\n\nWeaknesses: The proposed framework was tested in two simple environments with very small action, state, and parameter space. Applying this framework to a more difficult task can be very challenging, which is the major limitation of this work. Because accurately initializing the parameters and estimating the hidden human state can be drastically more difficult and data demanding as the task complexity increases. Moreover, the reviewer has questions and major concerns over the experiments:\n\nWhat is the definition of \"efficient strategy\" in the two games and how was it identified during the experiments?\n\nIt is not clear to the reviewer what exactly do the figure 4b and the discuss on the percentage of reward(line 278 ~ 280) are trying to show and how do they support the argument about the fully-assistive teacher impedes learning.\n\nIn figure 4.a., any insights about why the students perform significantly worse in evaluation than in training in the fully-assistive scenario? With increased experience, one would expect a similar or better performance in simple task learning.\n\nThe claim in line 285, \"random teacher is incapable of teaching a truly cooperative task\", is not supported by the experiment. As the author stated, the Duo Ball Maze task requires at least some degree of cooperation between the two players to successfully finish the task. This means that the random agent is more of a disturbance generator instead of a cooperative helper. Drawing conclusions about if the human users have learned COOPERATIVE skills by directly comparing these two settings is not sufficient. Instead, a better experiment could be training the users via the two different teachers, and evaluating the users' skills with another unseen baseline cooperative robot.\n\nSimilarly, the statement in line 298, \"Such a strategy…, therefore resulting in a poorer performance score\", suffers from the same problem where the human performance of different settings are not directly comparable when evaluated by different robot partners.\n\nFor figure 6, how were the line plots created? And can you explain why are these individuals representative enough to speak for the whole population? Could this figure be replaced by some statistics that summarize the entire group?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper presented a framework that allows a robot to assist human skills learning via cooperative HRI. The proposed idea is interesting, the system was well implemented, and the experiments have sufficient amount of human samples. Yet, the experimental results provided in the paper do not fully support some of the important claims. The conclusive results that can be draw from the current experiments are relatively weak.\n\nI appreciate the authors' detailed responses. The biggest concern of mine about insufficient experimental setup appeared to be a misunderstanding on the experimental procedure. I am changing my recommendation to weak accept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Questions and concerns are described in the weaknesses section.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)