Curriculum Design for Teaching via Demonstrations: Theory and Applications

1 Introduction . Imitation learning is a paradigm in which a learner acquires a new set of skills by imitating a teacher ’ s behavior . The importance of imitation learning is realized in real-world applications where the desired behavior can not be explicitly defined but can be demonstrated easily . These applications include the settings involving both human-to-machine interaction [ 1–4 ] , and human-to-human interaction [ 5 , 6 ] . The two most popular approaches to imitation learning are Behavioral Cloning ( BC ) [ 7 ] and Inverse Reinforcement Learning ( IRL ) [ 8 ] . BC algorithms aim to directly match the behavior of the teacher using supervised learning methods . IRL algorithms operate in a two-step approach : first , a reward function explaining the teacher ’ s behavior is inferred ; then , the learner adopts a policy corresponding to the inferred reward . In the literature , imitation learning has been extensively studied from the learner ’ s point of view to design efficient learning algorithms [ 9–15 ] . However , much less work is done from the teacher ’ s point of view to reduce the number of demonstrations required to achieve the learning objective . In this paper , we focus on the problem of Teaching via Demonstrations ( TvD ) , where a helpful teacher assists the imitation learner in converging quickly by designing a personalized curriculum [ 16–20 ] . Despite a substantial amount of work on curriculum design for reinforcement learning agents [ 21– 27 ] , curriculum design for imitation learning agents is much less investigated . Prior work on curriculum design for IRL learners has focused on two concrete settings : noninteractive and interactive . In the non-interactive setting [ 17 , 18 ] , the teacher provides a near-optimal set of demonstrations as a single batch . These curriculum strategies do not incorporate 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . any feedback from the learner , hence unable to adapt the teaching to the learner ’ s progress . In the interactive setting [ 28 ] , the teacher can leverage the learner ’ s progress to adaptively choose the next demonstrations to accelerate the learning process . However , the existing state-of-the-art work [ 28 ] has proposed interactive curriculum algorithms that are based on learning dynamics of a specific IRL learner model ( i.e. , the learner ’ s gradient update rule ) ; see further discussion in Section 1.1 . In contrast , we focus on designing an interactive curriculum algorithm with theoretical guarantees that is agnostic to the learner ’ s dynamics . This will enable the algorithm to be applicable for a broad range of learner models , and in practical settings where the learner ’ s internal model is unknown ( such as tutoring systems with human learners ) . A detailed comparison between our curriculum algorithm and the prior state-of-the-art algorithms from [ 18 , 28 ] is presented in Section 1.1 . Our approach is motivated by works on curriculum design for supervised learning and reinforcement learning algorithms that use a ranking over the training examples using a difficulty score [ 29–35 ] . In particular , our work is inspired by theoretical results on curriculum learning for linear regression models [ 32 ] . We define difficulty scores for any demonstration based on the teacher ’ s optimal policy and the learner ’ s current policy . We then study the differential effect of the difficulty scores on the learning progress for two popular imitation learners : Maximum Causal Entropy Inverse Reinforcement Learning ( MaxEnt-IRL ) [ 10 ] and Cross-Entropy loss-based Behavioral Cloning ( CrossEntBC ) [ 36 ] . Our main contributions are as follows:1 1 . Our analysis for both MaxEnt-IRL and CrossEnt-BC learners leads to a unified curriculum strategy , i.e. , a preference ranking over demonstrations . This ranking is obtained based on the ratio between the demonstration ’ s likelihood under the teacher ’ s optimal policy and the learner ’ s current policy . Experiments on a synthetic car driving environment validate our curriculum strategy . 2 . For the MaxEnt-IRL learner , we prove that our curriculum strategy achieves a linear convergence rate ( under certain mild technical conditions ) , notably without requiring access to the learner ’ s dynamics . 3 . We adapt our curriculum strategy to the learner-centric setting where a teacher agent is not present through the use of task-specific difficulty scores . As a proof of concept , we show that our strategy accelerates the learning process in synthetic navigation-based environments . 1.1 Comparison to Existing Approaches on Curriculum Design for Imitation Learning . In the non-interactive setting , [ 18 ] have proposed a batch teaching algorithm ( SCOT ) by showing that the teaching problem can be formulated as a set cover problem . In contrast , our algorithm is interactive in nature and hence , can leverage the learner ’ s progress ( see experimental results in Section 5 ) . In the interactive setting , [ 28 ] have proposed the Omniscient algorithm ( OMN ) based on the iterative machine teaching ( IMT ) framework [ 37 ] . Their algorithm obtains strong convergence guarantees for the MaxEnt-IRL learner model ; however , requires exact knowledge of the learner ’ s dynamics ( i.e , the learner ’ s update rule ) . Our algorithm on the other hand is agnostic to the learner ’ s dynamics and is applicable to a broader family of learner models ( see Sections 4 and 5 ) . Also for the interactive setting , [ 28 ] have proposed the Blackbox algorithm ( BBOX ) as a heuristic to apply the OMN algorithm when the learner ’ s dynamics are unknown—this makes the BBOX algorithm more widely applicable than OMN . However , this heuristic algorithm is still based on the gradient functional form of the linear MaxEnt-IRL learner model ( see Footnote 2 ) , and does not provide any convergence guarantees . In contrast , our algorithm is derived independent of any specific learner model and we provide a theoretical analysis of our algorithm for different learner models ( see Theorems 1 , 2 , and 3 ) . Another crucial difference is that the BBOX algorithm requires access to the true reward function of the environment , which precludes it from being applied to learnercentric settings where no teacher agent is present . In comparison , our algorithm is applicable to learner-centric settings ( see experimental results in Section 6 ) . 1.2 Additional Related Work on Curriculum Design and Teaching . Curriculum design . Curriculum design for supervised learning settings has been extensively studied in the literature . Early works present the idea of designing a curriculum comprising of tasks with increasing difficulty to train a machine learning model [ 29–31 ] . However , these approaches require 1Github repo : https : //github.com/adishs/neurips2021_curriculum-teaching-demonstrations_code . task-specific knowledge for designing heuristic difficulty measures . Recent works have tackled the problem of automating curriculum design [ 38 , 39 ] . There is also an increasing interest in theoretically analyzing the impact of a curriculum ( ordering ) of training tasks on the convergence of supervised learner models [ 32 , 40 , 41 ] . In particular , our work builds on the idea of difficulty scores of the training examples studied in [ 32 ] . The existing results on curriculum design for sequential decision-making settings are mostly empirical in nature . Similar to the supervised learning settings , the focus on curriculum design for reinforcement learning settings has been shifted from hand-crafted approaches [ 34 , 35 ] to automatic methods [ 21–23 , 25 ] . We refer the reader to a recent survey [ 42 ] on curriculum design for reinforcement learning . The curriculum learning paradigm has also been studied in psychology literature [ 43–47 ] . One key aspect in these works has been to design algorithms that account for the pedagogical intentions of a teacher , which often aims to explicitly demonstrate specific skills rather than just provide an optimal demonstration for a task . We see our work as complementary to these . Machine teaching . The algorithmic teaching problem considers the interaction between a teacher and a learner where the teacher ’ s objective is to find an optimal training sequence to steer the learner towards a desired goal [ 37 , 48–50 ] . Most of the work in machine teaching for supervised learning settings is on batch teaching where the teacher provides a batch of teaching examples at once without any adaptation . The question of how a teacher should adaptively select teaching examples for a learner has been addressed recently in supervised learning settings [ 51–56 ] . Furthermore , [ 16–18 , 28 , 57 , 58 ] have studied algorithmic teaching for sequential decision-making tasks . In particular , [ 17 , 18 ] have proposed batch teaching algorithms for an IRL agent , where the teacher decides the entire set of demonstrations to provide to the learner before any interaction . These teaching algorithms do not leverage any feedback from the learner . In contrast , as discussed in Section 1.1 , [ 28 ] have proposed interactive teaching algorithms ( namely OMN and BBOX ) for an IRL agent , where the teacher takes into account how the learner progresses . The works of [ 57 , 58 ] are complementary to ours and study algorithmic teaching when the learner has a differet worldview than the teacher or has its own specific preferences . 2 Formal Problem Setup . Here , we formalize our problem setting which is based on prior work on sequential teaching [ 28 , 37 ] . Environment . We consider an environment defined as a Markov Decision Process ( MDP ) M : = ( S , A , T , γ , P0 , RE ) , where the state and action spaces are denoted by S and A , respectively . T : S × S × A → [ 0 , 1 ] is the transition dynamics , γ is the discounting factor , and P0 : S → [ 0 , 1 ] is an initial distribution over states S. A policy π : S × A → [ 0 , 1 ] is a mapping from a state to a probability distribution over actions . The underlying reward function is given by RE : S ×A → R. Teacher-learner interaction . We consider a setting with two agents : a teacher and a sequential learner . The teacher has access to the full MDPM and has a target policy πE ( e.g. , a near-optimal policy w.r.t . RE ) . The learner knows the MDPM but not the reward function RE , i.e. , has only access to M \ RE . The teacher ’ s goal is to provide an informative sequence of demonstrations to teach the policy πE to the learner . Here , a teacher ’ s demonstration ξ = { ( sξτ , a ξ τ ) } τ=0,1 , ... is obtained by first choosing an initial state sξ0 ∈ S ( where P0 ( sξ0 ) > 0 ) and then choosing a trajectory , sequence of state-action pairs , obtained by executing the policy πE in the MDPM . The interaction between the teacher and the learner is formally described in Algorithm 1 . For simplicity , we assume that the teacher directly observes the learner ’ s policy πLt at any time t. In practice , the teacher could approximately infer the policy πLt by probing the learner and using Monte Carlo methods . Generic learner model . Here , we describe a generic learner update rule for Algorithm 1 . Let Θ ⊆ Rd be a parameter space . The learner searches for a policy in the following parameterized policy space : ΠΘ : = { πθ : S ×A → [ 0 , 1 ] , where θ ∈ Θ } . For the policy search , the learner sequentially minimizes a loss function ` that depends on the policy parameter θ and the demonstration ξ = { ( sξτ , a ξ τ ) } τ provided by the teacher . More concretely , we consider ` ( ξ , θ ) : = − logP ( ξ|θ ) , where P ( ξ|θ ) = P0 ( sξ0 ) · ∏ τ πθ ( aξτ |sξτ ) · T ( sξτ+1|sξτ , aξτ ) is the likelihood ( probability ) of the demon- Algorithm 1 Teacher-Learner Interaction 1 : Initialization : Initial knowledge of learner πL1 . 2 : for t = 1 , 2 , . . . do 3 : Teacher observes the learner ’ s current policy πLt . 4 : Teacher provides demonstration ξt to the learner . 5 : Learner updates its policy to πLt+1 using ξt . stration ξ under policy πθ in the MDP M. At time t , upon receiving a demonstration ξt provided by the teacher , the learner performs the following online projected gradient descent update : θt+1 ← ProjΘ [ θt − ηtgt ] , where ηt is the learning rate , and gt = [ ∇θ ` ( ξt , θ ) ] θ=θt . Note that the parameter θ1 reflects the initial knowledge of the learner . Given the learner ’ s current parameter θt at time t , the learner ’ s policy is defined as πLt : = πθt . Teaching objective . For any policy π , the value ( total expected reward ) of π in the MDPM is defined as V π : = ∑ s , a ∑∞ τ=0 γ τ ·P { Sτ = s | π , M } ·π ( a | s ) ·RE ( s , a ) , where P { Sτ = s | π , M } denotes the probability of visiting the state s after τ steps by following the policy π . Let πL denote the learner ’ s final policy at the end of teaching . The performance of the policy πL ( w.r.t . πE ) inM can be evaluated via ∣∣∣V πE − V πL∣∣∣ [ 9 , 59 ] . The teaching objective is to ensure that the learner ’ s final policy -approximates the teacher ’ s policy , i.e. , ∣∣∣V πE − V πL ∣∣∣ ≤ . The teacher aims to provide an optimized sequence of demonstrations { ξt } t=1,2 , ... to the learner to achieve the teaching objective . The teacher ’ s performance is then measured by the number of demonstrations required to achieve this objective . Based on existing work [ 28 , 37 ] , we assume that ∃ θ∗ ∈ Θ such that πE = πθ∗ ( we refer to θ∗ as the target teaching parameter ) . Similar to [ 28 ] , we assume that a smoothness condition holds in the policy parameter space : |V πθ − V πθ′ | ≤ O ( f ( ‖θ − θ′‖ ) ) ∀θ , θ′ ∈ Θ . Then , the teaching objective in terms of V π convergence can be reduced to the convergence in the parameter space , i.e. , we can focus on the quantity ‖θ∗ − θt‖ .

Summary: This paper considers the problem of designing a curriculum of trajectories to teach an imitation learning agent. The authors propose to sequentially give trajectories that are both difficult (low probability) for the current learner and easy (high probability) for the demonstrator. Some theoretical justification is provided for why this approach works as well as some empirical results on simple domains. In general this paper seems novel, but there are many areas that deserve more clarification and it is unclear how far this idea can be taken given the strong assumptions.