Assistive Teaching of Motor Control Tasks to Humans
Authors: Megha Srivastava
Erdem Biyik
Suvir Mirchandani
Noah Goodman
Dorsa Sadigh
Venue: NeurIPS 2022 Accept (2022)
Keywords: teaching
shared autonomy
human-AI interaction
education
reinforcement learning

Abstract: Recent works on shared autonomy and assistive-AI technologies, such as assistive robotic teleoperation, seek to model and help human users with limited ability in a fixed task. However, these approaches often fail to account for humans' ability to adapt and eventually learn how to execute a control task themselves. Furthermore, in applications where it may be desirable for a human to intervene, these methods may have inhibited their ability to learn how to succeed with full self-control. In this paper, we focus on the problem of assistive teaching of motor control tasks such as parking a car or landing an aircraft. Despite their ubiquitous role in humans' daily activities and occupations, motor tasks are rarely taught in a uniform way due to their high complexity and variance. We propose an AI-assisted teaching algorithm that leverages skill discovery methods from reinforcement learning (RL) literature to (i) break down any motor control task into teachable skills, (ii) construct novel drill sequences, and (iii) individualize curricula to students with different capabilities. Through an extensive mix of synthetic and user studies on two motor control tasks - parking a car with a joystick and writing  characters from the Balinese alphabet - we show that assisted teaching with skills improve student performance by around 40% compared to practicing full trajectories without skills, and practicing with individualized drills can result in up to 25% further improvement.

Assistive Teaching of Motor Control Tasks to Humans
Megha Srivastava Stanford University megha@cs megha@cs.stanfore stanford.edu
Erdem Bıyık UC Berkeley ebiysk@berkeleey edu
Suvir Mirchandani
Stanford University suvirêcs. stanford, ierffordddddi edu
Noah D. Goodman Stanford University
Dorsa Sadigh
Stanford University dorsa@cs stanf edu
ngoodman@stant own.ord.cco edu
Abstract
Recent works on shared autonomy and assistive-Al technologies, such as assistive robot teleoperation, seek to model and help human users with limited ability in a fixed task. However, these approaches often fail to account for humans' ability to adapt and eventually learn how to execute a control task themselves. Furthermore, in applications where it may be desirable for human to intervene, these methods may inhibit their ability to learn how to succeed with full self-control. this paper, focus the problem of assistive teaching of motor control tasks such parking a car or landing an aircraft. Despite their ubiquitous role in humans' daily activities and occupations, motor tasks rarely taught in uniform way due their high complexity and variance. We propose an Al-assisted teaching algorithm that leverages skill discovery methods from reinforcement learning (RI.) (i) break down any motor control task into teachable skills, (ii) construct novel drill sequences, and (iii) individualize curricula students with different capabiliiiies. Through an extensive mix of synthetic and user studies on two motor control tasks- parking a car with a joystick and writing characters from the Balinese alphabet-we show that assisted teaching with skills improves student performance by around 40% compared to practicing full trajectories without skills, and practicing with individualized drills can result in up to 25% further improvement.
1 Introduction
Imagine novice human is tasked with operating a machine with challenging or unfamiliar controls, such landing aireraft or parking an oversized vehicle with novel transmission. A fully au- tonomous form of assistance, such as self-driving car, would aid this user by entirely replacing their control, seeking interventions only when necessary. Shared autonomous systems, such as those used in assistive robot teleoperation [1, 2, 3]. would seek to model the human's 's goals and capabilities, and provide assistance in the form of corrections [4] or simplified control spaces [3]. However, neither form of assistance would actually help the user learn how to successfully operate the machine, and may even serve as erutch that prevents their ability to ever perform the task independently.
Although humans can learn a variety of complex motor control tasks (e.g.. driving a car, or operating surgical robots), they often rely the assistance of specialized teachers. Receiving fine-grained instruction from these specialized teachers can often be non-uniform, costly, and limited by their availability. In this work, we interested in more accessible and efficient approach: develop an Al-assisted teaching algorithm capable of teaching motor control tasks in an individualized manner. While several works have leveraged AI techniques to aid instruction in traditional education tasks such as arithmetic [5, 6, 7] and foreign language learning [8, 9], motor control tasks introduce
wish to
Our source code available at hatpps:/ggicaaa..co//tttrrrrt cow/Stanford ILIAD/teaching.
36th Conference on Neural Information Processing Systems (NeurIPS 2022).

---

several unique challenges, such as the high complexity of the input and output space for motor control tasks (e.g., controlling high degree of freedom robotic arm with 6 degrees of freedom joystick or the continuous output space of trajectories in driving), as well as the need for generalization across different scenarios in these high-dimensional and complex spaces.
To address these challenges, our insight is to utilize compositional motor skills for teaching: skills are more manageable to teach, and can be used to construct novel compositional drills for an individual. We take inspiration from how specialized human teachers teach motor control tasks. For example, a piano teacher may assign a student structured exercises (e.g., musical scales or arpeggios) to build up their general technique. Further, they may guide the student to break down complex musical phrases into more manageable chunks, repeatedly practice those, and then compose them together [10].
However, such fine-grained teaching often requires access to a specialized teacher capable of identi- fying skills and creating individualized drills based a student's unique expertise. This is not only expensive, but can be highly suboptimal for many motor control tasks where a teacher may not have full visibility into the student's various inputs for a task (e.g. how much the student pushed the brake pedal), but only observability of the final behavior of the system (e.g. how close the vehicle is to the curbside), leading to less meaningful feedback. Can we leverage the rich action data provided by students learning motor control tasks to develop more efficient and reliable individualized instruction?
Our key insight is to leverage automated skill discovery methods from the RL literature to break down motor control tasks performed by an expert into teachable skills. We then identify which of the expert's skills are skills that the student is struggling with, and construct novel drills that focus on those particular skills to individualize their curriculum. Our contributions include: 1) Developing an algorithm identify which skills of a motor control task student struggles with, 2) Automatically creating individualized drills based on students' expertise in various skills, and 3) Empirically validating the helpfulness of our Al-assisted teaching framework in two different motor control tasks: controlling a simulated vehicle to park with an unintuitive joystick, and writing Balinese words using a computer mouse trackpad. Our results show that assisted teaching with skills improve student performance by around 40% compared to practicing full trajectories without skills, and practicing with individualized drills can result up to 25% further improvement.
2 Related work
Skill discovery in Reinforcement Learning. A large body of work has studied how to leverage behavioral primitives, or skills in RL. Hicrarchical RL approaches [11, 12] use skills as temporal abstractions accelerate learning [13, 14, 15]. However, hand-designing skills can be difficult, especially in continuous spaces. Several works proposed methods to automatically discover skills [16, 17, 18, 19, 20]. Works in unsupervised skill discovery derive task-agnoste skills directly from an environment, without a reward function [21, 22, 23, 24, 25, 26]. Skills can also be learned from a set of tasks [27] or demonstrations [28, 29]; e.g., ComplLE [28] learns to segment and reconstruct given trajectories, using learned segment encodings to represent skills. Rather than using discovered skills to aid the learning of automated agents (e.g., via RL or planning [23, 24, 25]), we investigate the novel application of discovered skills to teaching humans.
Assistive human-Al systems. Human-Al collaborative systems have the potential augment a human's 's individual capabilities. In the paradigm of shared autonomy [30], an AI assistant modulates a user's observation [31] or input on tasks such as robot teleoperation [1, 2, 3]. Unlike our work, these works generally assume the human's 's policy is fixed, and do not directly aim for a human to learn from the AI agent [4]. Bragg et al. [32] improve both collaborative performance and user learning in shared autonomy framework, but they rely on simulated users in their evaluation. We present technique to support human learning to improve individual proficiency, and rigorously evaluate on
real users.
Al-assisted teaching. A distinct line of work has focused on developing algorithms for helping humans learn. Knowledge tracing attempts to model student's 's knowledge state in terms of skills relevant to a set of tasks, yet have generally required manual skill annotations [33] or large-scale student data [34]. Instructional sequencing uses model of student learning to select a sequence of instructional activities to maximize learning, but these approaches have generally focused on generating curricula for domains like language learning, arithmetic. and logic [35]. Similarly, machine
teaching [36] explores the problem of selecting optimal training data for a student with a known
2

---

model of learning, and has been effective in discrete domains with structured knowledge such as mathematics and visual classification [37].
We build on the idea that a student's proficiency can be represented in terms of different skills, and an Al assistant adapt currieulum accordingly. However, to the best of our knowledge, we are the first to explore how to teach motor control tasks informed by student actions. This setup has unique challenges: it less obvious how delineate skills, how to handle student feedback (in the form of continuous trajectories versus binary correct/incorrect responses), and how to measure a student's proficiency at a skill. By leveraging advances from skill discovery in RL, we do not require expert annotation of skills nor large-scale student data.
Motor learning in humans. Motor skill learning in psychology refers to the acquisition of skilled actions as result of practice or experience [38, 39]. Our approach is informed by several theories in motor learning and specifically the common insights that human motor control operates hierarchically, and that direct practice of individual skills generally aids skill learning [38, 40].
3 Formulation
In this section, we formalize the problem of Student-Aware Al-Assisted Teaching. We go over preliminaries and formalize skill discovery, and then define the problem statement leading to our 3-step approach of teaching complex motor control tasks: (i) distilling tasks into underlying skills, (ii) identifying the student's proficiency, and (iii) creating novel and individualized curricula.
3.1 Preliminaries
We formalize the target motor control task as a standard Markov decision process (MDP) (S. A. p. f, R, R") with finite horizon T, where P is the initial state distribution and deter- ministic transition function. In our formulation, R is a random variable for reward functions such that each r ~
a function r S x A -> R. Given an initial state su ~ p and a reward function r ~ R (both of which are revealed at the beginning of the task), the goal is to maximize the cumula- tive reward over the horizon. We call any (sq. pair a scenario & drawn from a set of all possible scenarios E. We assume access to set of expert demonstrations, e.g., demonstrations drawn from pre-trained RL agent that optimizes the cumulative reward for any given scenario.
An inexperienced human would lack the expertise to perform optimally in a given scenario &. Our goal develop teaching algorithms that leverage expert demonstrations to improve student performance. For example, we want human unfamiliar with Balinese to learn to write any Balinese word, which would correspond to one r. For this. we develop an algorithm that uses skill discovery methods to automatically create individualized drills based on the discovered skills and the student performance.
Skill discovery. Humans often break down complex tasks into relevant skills (i.e., compositional action sequences that lead to high performance)-wher learning how to write, we first master how to write letters, and then compose them to write words and sentences. We define skill latent variable m that lies in some finite discrete latent space M. Each skill is an embedding that corresponds an action sequence A = (a1, 022 .......... .., (((aa) optimal some state 8 S under some reward r~ R: Es E 3r: P(R r) 0, A (ai,
+r(f(s, (a 0z ... a App.... E arg max (r(s, ((s,aa)+((f(s.))) +...) (1)
a1.ag...... "(.
and |A| E [Mmin, H m,xs) where Hmin and Hmux < T are hyperparameters.
In many tasks, is difficult to enumerate all possible skills, and it is often not clear when a skill starts and ends trajectory. In this work, we use skill discovery algorithms automatically extract skills relevant to a task, and then teach the student those generated skills. While our teaching method is agnostic to the skill discovery algorithm, we use ComplLE [28] as it learns skills directly from expert trajectories. Our formulation assumes access to expert trajectories, so it would be unnecessary to learn an expert policy in parallel with the skill discovery process, as several other methods do.
We use the trained ComplL. as a function: SKILLEXTRACTOR(T) where T is a trajectory. It outputs a sequence of skills M = (m,, mb....) and boundaries B = (16, b...., by N.n)) such that 0= bi < <... < b% =T. dividing into Noeg segments. Actions in 7 between steps [b}- b3) correspond to skill m E M. Effectively, ComplLE learns to divide trajectories into segments, and cluster segments with similar actions into skills. We refer to [28] and the Appendix for more details.
3

---

3.2 Problem statement
Having formalized SKILLEEXTRACTE which, given trajectory 7, outputs M and B", we now present our problem statement for Al-assisted teaching of motor control tasks.
What should the student do? Given set of expert demonstrations in scenarios E CE, where each { E is sampled from (p. R), we would ideally teach the student to eventually have a policy n²: S + A that minimizes the difference between their behavior and the expert demonstrations:
π
arg mig
Σ _EE ((E) TE) R. arg min Σ CEE C(TE,
where T denotes the student trajectory performed by policy x in scenario g. Similarly, 7ē denotes the expert trajectory in 5.2 C is a task-dependent loss function. While it is a straightforward goal, it is not clear how to teach a human to optimize this objective to achieve expert-like trajectories for any g.
What should the teacher do? We can formulate the teaching problem as a partially observable MDP (POMDP), where the state of this POMDP is the stu- dent's policy (initially To), and the teacher's actions guide the student to the policy π². The teacher's 's ob- servations would be student behavior expressed by their trajectories, or T TL for some & for state т at teaching round k. The teacher can then select actions (e.g. practice scenarios) this POMDP. A teaching action at round & moves the student from TT to Th+11
πt
Optimal Teaching Action
T
Figure I: An optimal teacher should select teach- ing action (e.g. which practice scenario to provide) based on each student's 's individual policy to help guide them towards the expert policy x².
giving the teacher a reward of EçEE P(E) (C(TE: +5") L ++1)) i.e., the decrease in the
loss value according Eq. (2). Thus, the teacher should take the optimal teaching actions that maxi- mize its cumulative reward over teaching rounds, whose global optimum would move the student to T (see Fig. 1).
Unfortunately, we lack computational models of human learning necessary for making the POMDP's state transition function (i.e., how a student updates its policy from n& to T&+1 for any teaching action the teacher could take) tractable for motor control tasks. Recent works exploring data-driven frameworks for knowledge tracing [34] and RL-based education [6] usually require large amounts of data for simpler problems, and are thus not scalable to motor control tasks with large trajectory spaces.
Approach overview. Our insight is that once we identify important skills of a motor control task, an AI assisted teaching framework can efficiently leverage the skills teach the task. The S SKILLEXTRACTOR function allows us to identify the skills used in given trajectory. Applying it to all expert demonstrations in scenarios 3 C E gives us a set of skills relevant in the task. Say we label the expert demonstration of for each scenario & E E with skills and their intervals: (MG, B SKILLEXTRACTOR(?? We would then want the student to perform each relevant skill as close as possible to the expert to solve the following optimization:
arg min Eees P(E) Emens
(T T(") ara min EeS EEEEsEEEnNN Cm (T(TT:T)) (3)
where L. is some loss function that compares expert and student trajectories in terms of the skill m. The intuition behind this objective is two-fold: (i) we want the student to perform the same set of skills as the expert for any { E E (this is why we sum over m E M2), and (ii) we want them to perform each skill as close as possible to how the expert performs it. The underlying assumption is that student will gain expertise if they perform the same skills in the most similar way to the expert.
Overall, (3) is simpler objective as the space of action sequences for each skill is much smaller than the space of trajectories or scenarios. We can now try to teach small number of skills to the student instead of unrealistically trying to match them with the expert in the large trajectory space.
Only teaching popular skills of a task may suffer from the problem that the student does not learn how to compose different skills to achieve the target task. We thus propose to create novel drills, which consist of repetitions of multiple skills in single trajectory. Drills help the students (i) learn how to connect action sequences for different skills, and (ii) develop muscle memory, which is
FFor concision, we assume single expert demonstration rā and most single student trajectory r for any & = Without this assumption, would take expectations over expert and student trajectories for each

---

Extract Skills from Expert Demonstrations
Select Scenarios with Diverse Skills Er
Identify Individual Student Skill Expertise
Automatically Create Individual Practice Drills On
" 7 Molnngao Byy
-nA D LAAAL
~
o prseepaammmooo
< x - Py
(n² RR, nR", 12, RR", .21)
Figure 2: Overview of our approach. We train a ComplLE model over expert demonstrations (e.g.. handwritten Balinese text) our SKILLEXTRACTOR function, which we use to select diverse set of scenarios (e.g. words) covering many skills. After a student provides trajectories for these scenarios, we use SKIL SEILL..XTRAATTI again to identify their individual expertise for each skill, which informs the set of drills we provide them to practice.
crucial in motor control tasks, for the action sequences of each skill. The generated drills can also be personalized based on what skills each student is struggling with. For instance, as in Fig. 1, two different students (in green and blue) might have different policies with different skill sets. Therefore, teacher should generate individualized drills informed by the student's ' initial level of expertise.
To efficiently solve (3), we propose student-aware teaching approach that follows three steps: 1) Diverse scenario selection: Identify scenarios that yield a high diversity of popular skills when an expert controls the system, 2) Expertise estimation: Have the student control the system under these scenarios to estimate their expertise in different skills, and 3) Individualized drill generation: Based the estimates, create individualized drills have the student practice the skills that they struggling with. In the subsequent section, we describe our procedure for each of these steps.
4 Student-aware assisted teaching
Algorithm Diverse Scenario Selection
1) Diverse scenario selection. We select diverse scenarios that cover many different skills in or- der flexibly estimate students' expertise and in- dividualize our teaching strategy. For example, in English, we may have expert demonstrations of writing the words (scenarios) "expert", "per- son", "skills", from which we would select "per- son" cover more diverse letter strokes (skills). Formally, we treat the problem of choosing sce- narios with diverse skills as a maximum set coverage problem, which we solve using a simple greedy algorithm (see Algorithm 1), outputting set of scenarios that led the expert to demonstrate variety of skills.
Input: Skill labels M for each & E E Input: Number of scenarios to be selected N E Meavered <- 0, 0
for i i= 1,,, N do
<" f- arg max (2 (Moverrd M
E =
Meovared + Meovered U ME
return =
2) Identifying individual expertise in skills. To efficiently teach complex tasks, we seek to esti- mate student's expertise in different skills, such as identifying which strokes they struggle with when writing Balinese words. Requiring di- verse set of skills, the scenarios E' we select en- able us to infer student's expertise efficiently: we ask the student to perform the task in those see- narios, and based on their data, construct labels vector E that quantifies the student's expertise Em for each skill m M.
Algorithm Individual Expertise Identification
Input: Selected scenarios E C E Input: Skill labels ME for each & E Input: Student demonstration Tद for each E
I: Em + 0 for Vm E M
for & Ex do
Mt. B <- SKILLEXTTRATTOR
j 5-
for m E M do
if m & M then
Fm + Fm + rr((7))5
The discrepancy between skills used by the stu- dent and the expert provides a useful signal to estimate the student's 's expertise. Intuitively, we expect the student to demonstrate similar skills to the expert if they are proficient in those skills, while, otherwise, they might rely on simpler or incorrect set of skills. For example, student driver who lacks the expertise of reverse driving might attempt to park a vehicle by only driving forward. We therefore utilize the expert demonstrations in the same scenarios the student performed the task by extracting the skills the student used via SKILLEXTRACTOR and comparing them with the expert skills in those scenarios. The student's expertise in skill is then: Em Σeez: And , TT), where T5 is the student trajectory in &, and Am is function that computes the discrepancy between the two trajectories in terms of m.
j -+j+1
return E
5

---

In our implementation. if a skill is used only by the expert, we assume the student did not attempt or failed that skill. We increase the discrepancy based on how far in the trajectory they failed. We assign higher penalty to failures in the beginning of the scenario for two reasons: 1) Skills relevant early in the task are often more important: other skills would not even be needed without first achieving these skills. 2) Failing at early skills might mean the student never had a chance or need to perform the later skills due to compounding errors, so our uncertainty about the later skills is higher. Thus, we decide to be more conservative about the later skills. Specifically, we use:
TT(())/5 if m & ME and m. T m.
(m TE
(4)
otherwise.
Here rg is the reward function under scenario &, which we assume to be non-ponn non-positive." 3 We increase the discrepancy if a skill is only performed by the expert, and we discount this value over time (j in Eq. (4)). We also scale discrepancy with rę(Tz) so that student trajectories with lower reward will be penalized more in terms of expertise. Alg. 2 presents the pseudocode for this step. In applications where the reward function is not readily available, one could design the skill-discrepancy function Am independent of the rewards by using a divergence metric between expert and student trajectories.
3) Creating individualized drills. Referring to Eq. (3), a student should master the skills they struggle with to get better at the overall task. Therefore, the teacher should first identify those skills, and then have the student practice them. For example, if a teacher thinks a student who attempted to write the word "person" struggled with writing the letter "p", they could provide the word "puppet" as a drill for the student. Using the expertise vector F. for a student, which is the output of Alg. 2, we can select the skills m² with low Fm as the skills that the student needs to practice.
Given skill m", the student should ideally prac- tice it in the context of skills that frequently go together with m². rather than in isolation or with arbitrary skills. Our approach uses the expert demonstrations to identify frequent skill sequences containing skill m². We create drills consisting of these sequences that enable us to have the student practice (i) the target skill m", and (ii) how skill m connects with other skills that co-exist in various scenarios. Each drill re- peats such a sequence Neep times. A natural ap- proach is to iterate over all n-grams, borrowing terminology from natural language processing, of skills from the expert demonstrations. Within each n r-gram, we check if m² exists to identify the skills that often co-occur with m". Mathe- matically, frequenc of an n n-gram of skills æ is (where is an indicator for the particular n-gram .r):
Algorithm 3 Individualized Drill Creation
Input: Individual expertise vector E
Input: Number of skills to create drills for Nusgot
Input: Hyperparameters n, Naiills NNop
Input: Expert demonstration T for each & E Input: Skill labels Mē for each & E
I: Compute f(x for all n-grams x
b Eq. (5)
2: for m + arg set Nage minm Em do > see f
3:
X + argset Nam. maxx3m* f(x)
> see
4: Dm + UnEX i=l
Nep + mEn Tm > see t
5: return D
i arg set, min. E... returns the set of n distinct indices that individually minimize Em.
i Thi is the segment of an expert demonstration from in- terval [bj_1, bz") ) such that mj = m. The operator @ concatenates the actions of trajectory segments.
(5) We can then ask the student to practice the Naalls n-grams involving the target skill m with the highest frequency.
f(x) -
I (x = (m; (m; m mm411....... m
CEE
1-1
Next, we formalize a drill as a repetition of the skills in the n-grams. This ensures the student gets enough practice of skill m² with the most common skill sequence containing m². We then project this constructed skill sequence back to the trajectory space by again using the expert demonstrations. While different segments of the expert demonstrations may correspond to the same skill, we randomly select one of those segments to project back to the trajectory space." Alg. presents the pseudocode. It returns set of drills Dm for every target skill m Though these drills can be used in several ways, e.g., explaining them to a student, here we let the student observe and then imitate each drill trajectory.
3This is a mild assumption, since we can always subtract a constant off'stt from any finite reward function. "We set the initial state of each drill as the initial state of the first trajectory segment in the drill. Since the transition function/ is deterministic, the initial state and action sequence specify a full trajectory.

---

5 Experiments
Environments. We consider two motor control tasks: parking a car in simulation with a joystick controller (PARKING), and writing Balinese characters (WRITING) using a computer mouse"
WRITING: We introduce novel task of writing Balinese character sequences" from the Omniglot dataset [41], which contains action sequences for 1623 characters spanning 50 alphabets provided by crowdworkers from Amazon Mechanical Turk. We sample Balinese characters, introduce a connector character """, and create goal sequences of up to 8 characters. An agent in this environment operates 2D (r-y) continuous state and action space, and receives reward corresponding to the degree of overlap between the agent's s and the gold trajectory. We train ComplLE with 1000 random sequences based on trajectories provided by human contributors to the Omniglot dataset.
We use the Parking environment from HighwayEnv [42], goal-based continuous control task where an agent must park a car in specified parking spot by controlling its heading and acceleration (2D action space). The 6D state space corresponds to the car's 's position, velocity, and heading and at each time step, and the agent receives a reward representing how close the current state is to the goal state. Scenarios differ across initial heading angles and goal parking spots, resulting in diverse set of skills needed. We train ComplLE with rollouts across random scenarios from an optimal RL agent trained for 10 epochs using the StableBaselines [43] implementation of Soft Actor-Critic.
PARKING:
For both tasks, reward is negative with an optimal value of O, but reported with positive offset. Fur- ther details, as well as necessary pre- pre-processing steps for both environments and all hyperparameters used for training ComplLE models as part of
are in the Appendix.
Drill Creation. We create one drill for each latent skill identified by SKILLEXTRACTOR for both PARKING (n = 3, Nepp 1, Nurget 7, Nanills 1) and WRITING (n = Nep = 3, Nurget 8, Naaills = 1) tasks. Fig. 6 shows examples of these drills.
User Study. We design web applications for our user study for both environments. For PARKING, students see the environment rendered on the interface, and control the car's steering and acceleration with a 2D joystick. For WRITING, students use their computer mouse or trackpad to write on an HTML5 canvas, with the goal sequence of Balinese characters displayed underneath. Students in both environments follow a sequence of pre-test scenarios, practice sessions (including skills or drill-based practice), and evaluation scenarios. We recruit students through the Prolific platform, and include further information about pay rate, average experiment times, and task interfaces in the Appendix.
5.1 Results
Does individualization help synthetic students? To demonstrate our full approach, we first show results for synthetic "students" in the PARKING environment. We train two different synthetic students using standard behavior cloning on goal-conditioned (s, a) tuples from expert demonstrations:
1. Half-trained:: We train a 4-layer feed-forward neural network for only 50 epochs, resulting in around 50% increase in Mean Squared Error than a student trained for 400 epochs.
2. Reversing Difficulty: We train 4-layer feed-forward neural network for 400 epochs, but with only 20% of the data containing reverse acceleration actions, to simulate difficulty with reversing.
Next, following the approach described in Sec. 3, we create pool of 25 scenarios covering a diverse set of skills returned by our trained SKILLEXTRACTOR, We then collect rollouts from each synthetic student across all scenarios, use Alg. 2 to identify which 3 skills each synthetic student has least expertise in, and select drills from our offline drill dataset that target these skills. Using an equal dataset size, we compare and report average student reward across 15 different evaluation sets, where the student is fine-tuned on (s, a) sampled from these individualized drills (Ind. Drills) vs. random drills (Drills) vs. entire (s, a) trajectories from the original expert demonstrations (Full Trajectory).
Our results in Fig. 3 show that for the synthetic Half-trained IL Student, fine-tuning on data from both Ind. Drills and randomly selected Drills significantly outperforms the Full Trajectory setting (p .05. Wilcoxon signed-rank test), while the difference between Ind. Drills and Drills is not significant. This may be due to balf-trained IL agent not having low "expertise" in any particular region, but more generally benefiting from training further on good quality data. Interestingly, for
'Note that the skills extracted are specific to operating a joystick/pen with a computer mouse. 'We select Balinese because, in comparison to other scripts available in the Omniglot datuset, the writing trajectories were consistent across users. Assisted teaching when there are multiple ways of performing a task important future direction.
7

---

the Student with Reverse Action Difficulty, Ind. Drills significantly outperforms both randomly selected Drills and Full Trajectory settings, but randomly selecting Drills slightly underperforms Full Trajectory, perhaps due to several drills not containing as many reverse actions as in the full- trajectory data. These results suggest that while drills can benefit both types of synthetic students in general, due to being composed of very common skill sequences, the synthetic student designed to lack a certain kind of "skill" is more likely to strongly benefit from individualization. This presents promising signal that our CompILE-based SKILLEXTRACTIN intuitions for what skills are necessary for the PARKING task. However, our eventual goal is to teach humans motor control tasks, so next test the usefulness of our approach with human students.
learns skills that align with our
Does assisted skill-based practice improve human learning outcomes? Before constructing and individualizing drills, it important to verify whether the skills returned by SKIIL.Ex- TRACTOR-the building blocks of our approach- do indeed benefit student performance, as well as in comparison to simpler heuristics. For example, CompILE, covery methods, take as input the expected number of skills K per expert demonstration. A simpler Time Heuristic approach could instead be to assume that skills are temporally ordered and have equal length, and just split each expert demonstration into K equally-sized segments. If such heuristic sufficed, then it may not be necessary to leverage more complex skill discovery algorithms. We therefore run user studies for both PARKING and W WRITING environments comparing three different settings:
Full Trajectory
Random Drills
Individual Dellls
Reward
well other skill dis-
Half Trained Reversing Dificultyy Figure 3: The effect of Ind. Drills and random Drills for synthetic students.
1. Skills: Each user practices trajectories corresponding to the 3 most common skills returned by
SKILLEXTRACTOR based on CompILE parameterized with K, for 3 sessions each.
2. Time Heuristic: We temporally split each trajectory into K intervals of equal length, and each user practices 3 different intervals for sessions each.
3. Full Trajectory: Each user practices 3 different full trajectories of an expert demonstration (e.g. parking a car from the start state in PARKING, an entire character sequence in WRITING).
We set K 3 for PARKING and K = 8 for WRITING, For fair comparison, the number of time-steps users are able to practice in the full trajectory and time-heuristic modes is roughly equivalent. Because our users may differ widely in prior experience (e.g. video game player may be more familiar with the joystick in PARKING), we measure and report Reward Improvement, or the difference in average reward across 5 random evaluation scenarios and 2 random pre-test scenarios.
Our results in Fig. 4 show significantly (p < .05) higher reward improvement for students in both WRITING (N = 25) and PARKING (N 20) in the Skills setting vs.
PARKING
ED WRITING
à RRward
200
100
Full Trajectory setting. This suggests that teaching students the necessary skills for a motor control task is extremely beneficial to improve learning. However, comparison between the Skills and Time Heuristic is interestingly inconsistent between environ- ments, where Time Heuristic significantly leads to worse student learning outcomes in PARKING, but not for WRITING. One possible expla- nation might be that individual skills in WRITING, such as characters or certain curves, might be semantically intuitive their own, and therefore even if Time Heuristic presents different characters as the same overall "skill", there is less confusion for users. On the other hand, because of the chal- lenging dynamics in PARKING, user may rely more on practicing skill trajectories that have clearly similar actions. Because heuristics are often wwvironment-specifi and complex, we view these results as encouraging sign of the suitability of intelligent skill-discovery methods for teaching humans.
Skills Time Heuristic No Tral Figure 4: Students receive significantly higher reward im- provement when practicing with the Skills setting than Full Trajectory. A simple Time Heuristic results in inconsistent performance across environments, demonstrating the neces- sity for a more reliable way to identifying skills for teaching.
Skille
Time Heuristic
No Traj.
Finally, we ask all participants in a post-study survey "What else would have been helpful for you to learn how to write the characters or how to park car?". One user for the WRITING task who was assigned practice without skills responded "Separating the characters" " Responses from users who were assigned skill-based practice included "Practicing string of characters" and "More time and more repetition". ". These responses motivate our next set of experiments on the effect of drills.
8

---

Does practice with novel drill sequences outperform skill-based practice? Though our previous results demonstrate skill-fooe foll-foo practice leads to stronger learning outcomes in both environments, we hypothesize it is equally important to learn how to combine skills that often co-occur via drills. For example, qualitative look the skills in Fig. suggests some of the discovered skills consist of few actions, such as short dot in WRITING or only a few time steps in PARKING. On their own these may not useful skills to practice, but may be important to teach in the context of other skills.
To compare with our individualization
PARKING
ED WRITING
method, we create a fixed set of 5 pre- test scenarios covering a diverse set of skills returned by EXILTEXTR for both environments. We select 2 random drills, and compare the change in student reward when practicing with drills (Drills) vs. practicing the same underlying skill se- quences used to composed the drills as sep- arate practice sessions (Skills). Overall, as shown in Fig. 5, we find no statistically sig- nificant difference in reward improvement when students practice repetitive drill se- quences composed of multiple skills (Blue bar) vs. skills in separation (Pink bar). Possible explanations include the increased number of prac- tice sessions in the Skills setting (because we keep the number total time-steps equivalent across settings) mitigating benefits of repetition practicing skill transitions in Drills. However, next ask whether we may observe benefit from drills we were to individualize them to students.
A Bewward
200
Helyfa Sifflllusss
Skills
Dells Ind. Drills
Skills
Drills Ind. Drils
Figure 5: Students marginally improve reward more when practicing with Ind. Drills than Skills for both tasks. In WRITING, students significantly prefer Ind. Drills over both random Drills and Skills, yet this does not hold in PARKING, where targeted practice of reversing may frustrate students.
Does individualizing drills improve student learning outcomes? For each student, run ComplLE-based SKILLEXTR ACTOR immediately after receiving all pre-test student trajectories. Fig. 6 demonstrates the top 2 skills identified as "low expertise" across all students for both PARKING (N = 20) and WRITING (N = 25) environments. SKILLEXTRACTOR combined with Alg. 2 is indeed able to identify wide set of low-expertise skills in student trajectories, ranging from more popular skills (e.g. "horn" shape in WRITING) to rarely difficult skills (c.g. steering the car forward in straight line in PARKING). Importantly, this is despite the likely strong differences between human student and expert trajectories, emphasizing the viability of our approach to teaching humans.
Next, we select unique drills targeting each identified skill, and provide them to students for practice before measuring reward improvement on an evaluation set. Fig. 5 shows individualized drills (Ind. Drills) lead to stronger reward im- provements for students than Skills with marginal significance (p .06 WRITING, Re .09 PARK- ING), supporting our hypothesis that individual- izing drills leads better learning outcomes. We visualize all student trajectories before and after practicing Ind. Drills in the Appendix.
Drills
Finally, we ask students to rate the helpfulness of practice sessions from 1-7 (1 least helpful), and show in Fig. for WRITING, students strongly preferred practice sessions from Ind. Drills over Drills and Skills. However, despite the higher average reward improvement for Ind. Drills in PARKING, students who participated in Skills practice rated the helpfulness of their sessions significantly higher. Though there may exist many explanations for this result, including dent preference for shorter sequences, we observe in evaluation rounds that users who received Ind. Drills were around twice as likely to try reversing into parking spot than other students (53% vs. 27 %)). This is likely due our individualization method providing drills with reversing skills, which
O
Drills
MAKAAMA
Figure Alg. 2 identifies wide range skills as low expertise across students for both environments. Reverse actions are identified as the most common skills for students to improve on in PARKING, while common curved-shapes identified for WRITING.

---

were more common targeted skills (see Fig. 6). Crucially, in these scenarios the expert action is also to reverse, suggesting that while individualization may create a more frustrating learning experience, it may still help students achieve stronger learning outcomes by teaching more challenging skills.
6
Discussion
Our work is a first step towards building Al-assisted systems to more efficiently teach humans motor control tasks. However, there exist few important limitations, which we now outline below.
6.1
Extracting "Human Teachable" Skills
A possible limitation of our work is whether skills extracted via SKILLEXTRACTOR are suitable for teaching human learners. While one benefit of our framework is its agnosticism to the type of the expert, we could imagine that using an automated RL-trained expert, for example, might result in teaching skills that are unsuitable for human learners. Addressing this requires stronger cognitive models of human motor learning in order to iterate different types of skill decomposition and optimize for some notion of "teachability." We could consider adapting existing methods used to model learning over time in other education domains (e.g. mathematic)), such Deep Knowledge Tracing ([34) and Item Response Theory, which typically represent questions as discrete items or with simple features, and student responses binary (correct incorrect). Unfortunately, adapting such methods to handle the rich information in student trajectories is quite unstable, likely due to high dimension and scale (e.g. > IK timesteps). Furthermore, student response variation for the same scenario is lot higher for motor tasks than in other tasks such as in mathematics, where within a short period a student likely answers the same question the same way. Finally, such approaches require large amounts of student data (on the scale of thousands). Further discussion, as well as results demonstrating the unreliability of asking human experts identify skills instead, is in Appendix Section E.
6.2 Accounting for Student & Expert Multimodality
One important aspect of many motor control tasks is multimodality there may exist many optimal ways to complete the task. Although one can naively extend our approach to handle this by collecting demonstrations from diverse set of experts and identifying which expert to "match" a student with, understanding the different control preferences present in a motor task is complex. For example, in writing, while different stroke orders for the same character clearly constitute different modes, there exist more subtle differences, such as the degree of rotation used, or the sharpness of the trajectory. These may occur due to physical preferences or conditions such as arthritis, but are hard to automatically distinguish as separate modes versus noise. Cleanly defining the distinct modes of a control task likely requires strong domain knowledge combined with large amounts of data across diverse population. Furthermore, inferring which mode to teach a novice student is challenging, as pre-test trajectories only provide limited amount of information. In traditional education settings, we often ask students questions such as whether one is left or right handed, and use that to inform how we teach. However, there may exist more obscure forms of preferences that require longer interaction with a student to elicit, and identifying such preferences is an important direction for future work.
6.3 Potential Failure Cases
Although we provide a general framework for assistive teaching of any arbitrary motor control task, including those that lack many expert instructors like teleoperating a novel surgical robot system, there are some settings that may be incompatible with our approach. These include situations where important skills do not appear in observed trajectories (e.g. the positioning of a hand on a ball before pitching in baseball), or there exists extreme stochasticity such that the same actions lead drastically different outcomes, resulting in unintuitive dynamics for learners. Furthermore, in some tasks learning may not be feasible just from practice, but requires stronger forms of teacher-student interaction, such as physical guidance. We believe understanding the limits of demonstration- based teaching for motor tasks necessitate further insights from several areas within the broader NeurIPS research community, including cognitive science, reinforcement learning, and social aspects of machine learning.
7 Acknowledgements
We thank all anonymous reviewers for their valuable feedback. We acknowledge support from Point72, Ford, AFOSR, and NSF Awards #2218760, #2132847, and #2006388. Megha Srivastava was also supported by the NSF Graduate Research Fellowship Program under Grant No. DGE-1656518.
10

---

References
[1] Shervin Javdani, Henny Admoni, Stefania Pellegrinelli, Siddhartha S Srinivasa, and Andrew Bagnell. Shared autonomy via hindsight optimization for teleoperation and teaming. The International Journal of Robotics Research (IJRR), 37(7):717-742, 2018.
[2] Hong Jun Jcon, Dylan P Losey, and Dorsa Sadigh. Shared autonomy with Iearned latent actions. Robotics: Science and Systems (RSS), 2020.
[3] Dylan Losey, Hong Jun Jeon, Mengxi Li, Krishnan Srinivasan, Ajay Mandlekar, Animesh Garg, Jeannette Bohg, and Dorsa Sadigh. Learning latent actions control assistive robots. In Journal of Autonomous Robots (AURO), 2021.
[4] Siddharth Reddy, Anca Dragan, and Sergey Levine. Shared autonomy via deep reinforcement learning. Robotics: Science and Systems (RSS), 2018.
[5] Anna N. Rafferty, Emma Brunskill, Thomas I. Griffiths, and Patrick Shafto. Faster teaching via pomdp planning. Cognitive Science, 40 6: 1-11990-322 2016.
[6] Jonathan Bassen, Bharathan Balaji, Michael Schaarschmidt, Candace Thille, Jay Painter, Dawn Zimmaro, Alex Games, Ethan Fast, and John C Mitchell. Reinforcement learning for the adaptive scheduling of educational activities. Conference on Human Factors Computing Systems (CHI), 2020.
[7] Gabriel Poesia, WenXin Dong, and Noah Goodman. Contrastive reinforcement learning symbolic reasoning domains. Advances in Neural Information Processing Systems (NeurIPS). 2021.
[8] Megha Srivastava and Noah Goodman. Question generation for adaptive education. Association for Computational Linguistics (ACL), 2021.
[9] Tong Mu, Shuhan Wang, Erik Andersen, and Emma Brunskill. Automatic adaptive sequencing in webgame. International Conference on Intelligent Tutoring Systems, 2021.
[10] Denes Agay Teaching Piano: A Comprehensive Guide and Reference Book for the Instructor. Music Sales Amer, 1982.
[11] Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: framework for temporal abstraction reinforcement learning. Artificial Intelligence, 11((1-2):18122 181-21,, 1999.
[12] Andrew Barto and Sridhar Mahadevan. Recent advances hierarchical reinforeement learning, Discrete Event Dynamic Systems, 13(1):41-77,2000
[13] Amy McGovern and Richard S Sutton. Macro-actions in reinforcement learning: An empirical analysis. Computer Science Department Faculty Publication Series, page 15, 1998.
[14] Ronald Parr and Stuart Russell. Reinforcement learning with hicrarchies of machines. Advances in Neural Information Processing Systems (NIPS), 1997.
[15] Thomas G Dietterich. Hicrarchical reinforcement learning with the MAXQ value function decomposition. Journal of Artificial Intelligence Research, 13:227-303, 2000.
[16] Stefan Schaal, Peters, Jun Nakanishi, and Auke Ijspeert. Learning movement primitives. Robotics Research. The Eleventh International Symposium, pages 561-572, 2005.
[17] Christian Daniel, Gerhard Neumann, and Jan Peters. Autonomous reinforcement learning with hierarchical reps. IEEE International Joint Conference on Neural Networks (IJCNN), 2013.
[18] Pravesh Ranchod, Benjamin Rosman, and George Konidaris. Nonparametric bayesian reward segmentation for skill discovery using inverse reinforcement learning. IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), 2015.
[19] Pierre-Lar Bacon, Jean Harb, and Doina Precup. The option-critic architecture. AAAI Conference on Artificial Intelligence, 2017.
[20] Akhil Bagaria and George Konidaris. Option discovery using deep skill chaining. International Conference on Learning Representations (ICLR), 2019.
[21] Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint a11vv.611.07507. 2016.

---

[22] Carlos Florensa. Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical reinforcement learning. International Conference Learning Representations (ICLR),
[23] Joshua Achiam, Harrison Edwards, Dario Amodei, and Picter Abheel. Variational option discovery algorithms. arXiv preprint arXiv: 10299, 2018.
[24] Benjamin Eysenbach, Abhishek Cupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. International Conference on Learning Representations (ICLR), 2019.
[25] Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. International Conference on Learning Representations, 2020.
[26] Victor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giró-i Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. International Conference on Machine Learning (ICML), 2020.
[27] Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. Meta learning shared hierarchies. International Conference on Learning Representations (ICLR), 2018.
[28] Thomas Kipf. Yujia Li, Hanjun Dai, Vinicius Zambaldi, Alvaro Sanchez- -Gonzazz.. Edward Grefenstette, Pushmeet Kohli, and Peter Battaglia. CompILE: Compositional imitation learning and execution. Interna- tional Conference on Machine Learning (ICML), 2019.
[29] Tammay Shankar, Shubham Tulsiani, Lerrel Pinto, and Abhinav Gupta. Discovering motor programs by recomposing demonstrations. International Conference on Learning Representations (ICLR). 2019.
[30] Anca D Dragan and Siddhartha S Srinivasa. A policy-blending formalism for shared control. The International Journal of Robotics Research (IJRR), 32(7):790-805, 2013.
[31] Siddharth Reddy, Sergey Levine, and Anca Dragan. Assisted perception: Optimizing observations to communicate state. In Conference on Robot Learning, pages 748 764. PMLR, 2021.
[32] Jonathan Bragg and Emma Brunskill. Fake it till you make it: Learning-compatible performance support. Conference on Uncertainty in Artificial Intelligence (UAI), 2019.
[33] Albert T Corbett and John R Anderson. Knowledge tracing: Modeling the acquisition of procedural knowledge. User modeling and user-adapted interaction, 4(4):253-278, 1994.
[34] Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J Guibas, and Jascha Sohl-Dickstein. Deep knowledge tracing. Advances in Neural Information Processing Systems (NIPS), 2015.
[35] Shayan Doroudi, Vincent Aleven, and Emma Brunskill. Where's 's reward? a review of reinforcement learning for instructional sequencing. International Journal of Artifical Intelligence in Education, 29:568- 620, 2019.
[36] Xiaojin Zhu. Machine teaching: An inverse problem to machine learning and an approach toward optimal education. AAAI Conference on Artificial Intelligence, 2015.
[37] Oisin Mac Aodha, Shihan Su, Yuxin Chen, Pictro Perona, and Yisong Yue. Teaching categories human learners with visual explanations. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 3820-3828, 2018.
[38] Natalic Krahe. Motor Skill Learning: A Review of Trends and Theories. Caroll College, 1999.
[39] Anne Shumway-Cood and Marjorie H Woollacott. Motor control: Theory and practical applications. Lippincqut Williams & Wilkins, 1995.
[40] David A Rosenbaum. Human motor control. Elsevier, 2010.
[41] Brenden Lake, Ruslan Salakhutdinov. and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266):133--- 332-13388, 2015.
[42] Edouard Leurent. An environment for autonomous driving decision-making. https: btttpa//giithul com/ eleurent www.eeatttthhhhhh-eeeeeee env, 2018
[43] Ashley Hill, Antonin Raffin, Maximilian Emestus, Adam Gleave, Anssi Kancrvisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines. https://giithen com/hill- aaa1laa/aaa htt///abbbbbblllineeeeeeea baselines, 2018.
12

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] . You are strongly encouraged to include a justification to your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes] Included in supplement.
Did you include the license the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We provide
a detailed Ethics statement in the Appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL)? [Yes] In Appendix
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] In Appendix
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] In Appendix
4. If you using existing assets (e.g., code, data, models) or curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [N/A]
(c) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [Yes] In Appendix
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [Yes] Full details in Appendix.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [Yes] In Appendix.
13


Reviews:
[{"review_id": "e30RzdoGpqN", "reviewer": "Reviewer_uRWk", "summary": "\nThis paper proposes a framework that allows an AI agent to teach the human to do motor tasks, e.g., using a joystick for parking a car in simulation, or writing letters in an unfamiliar language. This framework's key advantage is that it will decompose the motor task into multiple skills, and compose those skills into individualized drills for the human student to practice.\n\nThe method works as follows:\n1. Assuming that there are expert demonstrations of the given task. Then, the system first calls SKILLEXTRACTOR to extract skills from the demonstration. Each skill is represented as a sequence of actions and an interval where the skill is chosen to be executed.\n2. Diverse scenario selection: choose a set of scenarios that can cover all skills via a greedy algorithm.\n3. Identifying individual expertise in skills: ask the student to perform the task in those scenarios, and estimate her expertise.\n4. Creating individualized drills: from step 3, identify the skills with low student expertise. Then, from the expert demonstration, identify N_drills n-grams involving the target skill with high frequency among all expert demonstrations.\nThen, the student will practice each drill for N_rep times.\n\nResults:\n* Simulated student\n  * Conclusion: Individualized drills (the proposed method) improve the total reward (statistically significant).\n    * Compare among 3 methods\n      * Ind.Drills: Individualized drills (the proposed method).\n      * Drills: random drills.\n      * FullTrajectory: the entire trajectory of expert demonstration.\n    * Case 1: Simulated student as a half-trained behavior cloning agent\n      * Ind.Drills > FullTrajectory's total reward (statistically significant).\n      * Drills > FullTrajectory's total reward (statistically significant).\n    * Case 2: Simulated student as a behavior cloning agent with reverse action difficulty\n      * Ind.Drills > FullTrajectory's total reward (statistically significant).\n      * Ind.Drills > Drills's total reward (statistically significant).\n\n* Human student\n  1. Conclusion: skill by SKILLEXTRACTOR improves the reward improvement (statistically significant).\n    * Compare among 3 methods\n      * Skills: SKILLEXTRACTOR (used by the proposed method).\n      * TimeHeuristic: split the trajectory into skills evenly.\n      * FullTrajectory: entire expert demonstration as a skill to practice.\n    * Task 1: Parking\n      * Skills > FullTrajectory's reward improvement (statistically significant).\n      * Skills > TimeHeuristic's reward improvement (statistically significant).\n    * Task 2: Writing\n      * Skills > FullTrajectory's reward improvement (statistically significant).\n  2. Conclusion: practicing with drills vs skills is not statistically significant.\n    * Task 1: Parking\n    * Task 2: Writing\n  3. Conclusion: Individualized drills (the proposed method) improve the reward improvement (only marginal significance), and improves the subjective helpfulness (statistically significant)\n      * Ind.Drills: Individualized drills (the proposed method).\n      * Drills: random drills's reward improvement.\n      * Skills: practice with skills's reward improvement.\n    * Task 1: Parking:\n      * Ind.Drills > Skills's reward improvement (only marginal significance)\n    * Task 2: Writing\n      * Ind.Drills > Skills's reward improvement (only marginal significance)\n\n    * Task 1: Parking\n      * Ind.Drills > Skills's helpfulness (statistically significant).\n    * Task 2: Writing\n      * Ind.Drills > Drills's helpfulness (statistically significant).\n      * Ind.Drills > Skills's helpfulness (statistically significant).", "questions": "\n* Line 122: latent space M. Is the concept of latent space used in this paper? Latent space reminds me of VAE or graphical models. I was confused when seeing this in line 122.\n* Line 314: does full trajectory mean that the student will just practice on the original expert demonstration?\n", "limitations": "N.A. All technical limitations are in `# Strengths And Weaknesses`.\n", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n* Strength\n  * Very interesting and novel problem formulation.\n  * Online simulation-based user study in 2 tasks.\n\n* Weakness\n  * The proposed method assumes that the skill extracted by SKILLEXTRACTOR matches the skills that the student would need. However, in reality, each student could have a different set of skills extracted.\n  * The writing could be easier to follow by reducing the notations, especially those notations with many super / sub-scripts.\n  * User study with real human successfully shows that skill extracted by SKILLEXTRACTOR (used in the proposed method) improves the reward improvement with statistical significance. However, the user study failed to show that the proposed individualized drills improve the reward improvement (only statistically marginal significance).\n  * Minor: Line 145: the paragraph about the POMDP approach could be shortened, because the paper later says that POMDP is intractable and proposes another approach. So the POMDP approach is not very relevant to the paper.\n  * Minor: Fig.4: label \"No Skills\" should be \"Full Trajectory\"?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WA_WVraOb2", "reviewer": "Reviewer_W6Mp", "summary": "The paper introduced an assistive teaching system for improving human skills on continuous control tasks. The core idea of the paper is to decompose a complex task into multiple reusable skills, assess the human skill level for each decomposed skill, and compose new personalized drills that help improve the weakest skills of the user. The system is evaluated on two tasks: parking a car with joystick control, and writing characters from the Balinese alphabet. The authors tested the proposed method on both synthetic learners as well as real human subjects in a user study. The experimental results shows that the method achieve improved learning performance in both cases for both tasks.", "questions": "1. During the evaluation phase, if a user finds a different sequence of operations from the expertise but still accomplishes the task (for example in the parking task there might be multiple, equally good solutions), would the proposed system penalize that and try to make the user follow the expert trajectory?\n\n2. What do the learned skills look like for both tasks? It would be helpful to have some form of visualization for both expert trajectories and student trajectories to better understand the discovered skills and the evaluation process. In addition, for the writing task, can we use each letter as a skill?\n\n3. In the synthetic student case, how many epochs were used during fine-tuning? Why are half-trained ones so much worse than the reverse difficulty ones even after fine-tuning?\n\n4. In line 342, the Skills approach picks the 3 most common skills, while for writing 8 skills were discovered. Does this mean 5 of them are not seen by the user in this version of the method?\n\n", "limitations": "The authors have addressed some limitations of the work. One additional discussion point that would be helpful is some additional example problems where the proposed method would succeed and fail, which would not only provide insights on when the method is applicable, but also help better judge the potential impact of the work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The idea of leveraging skill discovery for teaching assistance is interesting and novel in my knowledge. \n2. The proposed method that combines skill discovery, personalized evaluation and drill design, is reasonable and well elaborated.\n3. The experiments support the proposed method in improving human learning performance.\n\nWeaknesses:\n1. One assumption that the method is making is that the decomposed skills can be acquired through demonstration, which circumvents the challenges of modeling human learning dynamics as described in the ‘What should the teacher do’ section instead of solving it directly. As such the method may not work for more complex tasks such as performing stunts where individual skills can also be difficult to acquire.\n2. The demonstrated tasks do not require complex and reactive decision making, compared to something like a real-time strategy game. It’s not clear for more complex problems like those whether the skill discovery technique can still find reasonable low-level skills and whether the pipeline can scale accordingly.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PajdJkYSqK4", "reviewer": "Reviewer_XUpi", "summary": "This paper presents an approach to providing training instruction for motor control tasks automatically by extracting skills from expert trajectories and combining them into drills based on user expertise. The authors use an existing framework for skill extraction and their main formal contribution is an approach for user evaluation and for individualized training curriculum construction. The paper evaluates different training scenarios with synthetic and real human users to analyze and demonstrate the benefits of their training curricula.", "questions": "\nIn the formalism, one question that remains is related to the limitations introduced by the underlying deterministic model.\nWhy is a deterministic skill model sufficing ?\n\nIn the experiments, a few issues would be important:\nDoes the training curriculum yield optimal behavior eventually or does it lead to an intermediate “plateau” where presented skills and drills do not address areas where the user could further improve ?\nHow realistic was the underlying synthetic agent model to capture human skill acquisition ?\n\n", "limitations": "While the paper discusses some limitations and social considerations, some of the underlying assumptions and resulting limitations are not fully discussed. Among these are the deterministic skill model and the used synthetic agents for the first set of experiments.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The application of AI to individualized training of human motor skills is an interesting application with many potential uses. The paper is well written and clearly presents the concepts and algorithms used. The experiments are intuitively presented and show important results for understanding of the effects on human learning.\nThe main weaknesses of the paper some potentially limiting assumptions that should be discussed more. For example, the target motor control task is defined as an MDP with deterministic transitions. Some discussion why deterministic transitions are sufficient here or what limitations this introduces would be useful. \nAnother issue is that in equation 5 the used indicator function II is not formally introduced.\nIn the experimental results, it would have been nice to also see some analysis of the effect of the amount of training on performance. In particular in the synthetic agent experiments, partially trained agents don’t seem to achieve high performance even after training. Would the proposed training approach actually get them to high performance or do they “plateau” ? In the human experiments, how close to the expert do users get ? This is hard to assess as no optimal performance reward estimate is provided.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "97M-RaGtGw5", "reviewer": "Reviewer_4JuP", "summary": "The paper presented an AI-assisted teaching algorithm that leverages skill discovery methods from reinforcement learning. The key idea of the proposed method is to decompose the skill to be taught into a few teachable sub-skills. By combining sub-skills into drill sequences, the algorithm then generates individualized curricula for different students.", "questions": "Questions:\n\n1. I am not familiar with Balinese. However, I would like to ask: does the learned sub-skill have any semantic meaning? For example, in many writing systems, there are prescribed strokes or components in a character. Will CompILE be able to learn meaningful sub-skills in that case?\n\n2. In the abstract, the paper mentions ‘individualize curricula to students with different capabilities’. What do the capabilities mean? Is it the current skill level of the student? In that case, is the training drill only generated based on the student’s initial performance? Is this caused by the lack of a human learning model?\n\n3. In the supplementary materials, in figure 7, the performance improvement on writing before and after training is very subtle but the improvement over reward is significant, which is quite confusing to me. \n\nOverall, I still think the paper is interesting and well-written. Therefore, I lean towards acceptance for now. ", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The motivation of the paper is valid. I think the decomposition of a complex task into teachable skills is a simple and efficient strategy.\n\n2. The paper is well-written.\n\n3. The experiments are extensive. Both the PARKING and WRITING experiments are interesting. \n\n4. Extending the skill decomposition in RL literature to human learning seems promising.\n\n5. The code is provided for reproducibility. \n\nWeaknesses:\n\n1. My major concern is the discrepancy between the proposed approach and the target task (motor control). Particularly, I think the proposed approach does not capture the key challenge of teaching motor control skills. It seems that the success of the proposed method solely depends on human learning but ignores that humans may not be able to learn motor control skills without proper assistance. Let me explain. One of the characteristics of motor control skills is that it normally involves physical contact with the environment. In my opinion, to distinguish a teaching algorithm for a motor control task from the conceptual counterpart (e.g. learn classification), it is important to embed the notion of such physical interactions. For example, designing a practice sequence may not suffice in many rehabilitation tasks.  Without this, I don’t see a significant advancement over the existing Automatic Curriculum Learning (ACL) approaches made by the paper. \n\n2. I think the paper lacks an analysis of why the learned teachable skills from reinforcement learning literature can be directly applied to human learning, which makes the result less convincing. \n\n3. The paper investigates two tasks, PARKING, and WRITING. However, for both these tasks, the decomposition into teachable sub-skills can both be done by an expert. The approach will be more valuable if the author could show how the learned teachable skills compare with expert-defined teachable skills.  \n\n4. In this paper, the objective is to align students’ trajectory with expert demonstrations. From my perspective, this is not a good way to specify the learning objective, especially for complicated tasks with multiple optimal solutions. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept