Curriculum Design for Teaching via Demonstrations: Theory and Applications
Authors: Gaurav Raju Yengera
Rati Devidze
Parameswaran Kamalaruban
Adish Singla
Venue: NeurIPS 2021 Poster (2021)
Keywords: imitation learning
teaching via demonstrations
curriculum design
inverse reinforcement learning

Abstract: We consider the problem of teaching via demonstrations in sequential decision-making settings. In particular, we study how to design a personalized curriculum over demonstrations to speed up the learner's convergence. We provide a unified curriculum strategy for two popular learner models: Maximum Causal Entropy Inverse Reinforcement Learning (MaxEnt-IRL) and Cross-Entropy Behavioral Cloning (CrossEnt-BC). Our unified strategy induces a ranking over demonstrations based on a notion of difficulty scores computed w.r.t. the teacher's optimal policy and the learner's current policy. Compared to the state of the art, our strategy doesn't require access to the learner's internal dynamics and still enjoys similar convergence guarantees under mild technical conditions. Furthermore, we adapt our curriculum strategy to the setting where no teacher agent is present using task-specific difficulty scores. Experiments on a synthetic car driving environment and navigation-based environments demonstrate the effectiveness of our curriculum strategy.

Curriculum Design for Teaching via Demonstrations: Theory and Applications
Gaurav Yengera 1.2
Rati Devidze' rdevidze@ mppisss.rrr
Parameswaran Kamalaruban'
Adish Singla
gyengera mpp-sws.rrg
kparameswaran @ eturing.ac.uk
adishs @ mpi-swss.rg
Max Planck Institute for Software Systems (MPI-SWS), Saarbrucken, Germany Saarland University, Saarland Informatics Campus (SIC), Saarbrucken, Germany The Alan Turing Institute, London, UK
Abstract
We consider the problem of teaching via demonstrations in sequential decision- study how to design a personalized curriculum
making settings. In partieular, over demonstrations to speed up the learner's convergence. We provide a unified curriculum strategy for two popular learner models: Maximum Causal Entropy Inverse Reinforcement Learning (MaxEnt-IRL) and Cross-Entropy Behavioral Cloning (CrossEnt-BC)) Our unified strategy induces a ranking over demonstra- tions based on a notion of difficulty scores computed w.r.t the teacher's 's optimal policy and the learner's current policy. Compared the state of the art, our strategy doesn't it require access to the learner's internal dynamics and still enjoys similar convergence guarantees under mild technical conditions. Furthermore. we adapt our curriculum strategy to the setting where no teacher agent is present using task-specific difficulty scores. Experiments on a synthetic car driving environment and navigation-based environments demonstrate the effectiveness of our curriculum strategy.
Introduction
Imitation learning is paradigm in which learner acquires teacher's behavior. The importance of imitation learning is realized in real-world applications where the desired behavior cannot be explicitly defined but can be demonstrated easily. These applications include the settings involving both human-to-machine interaction [1-4], and human-to-human in- teraction [5, 6]. The two most popular approaches imitation learning are Behavioral Cloning (BC) [7] and Inverse Reinforcement Learning (IRL) [8]. BC algorithms aim to directly match the behavior of the teacher using supervised learning methods. IRL algorithms operate two-step approach: first, reward function explaining the teacher's behavior is inferred; then, the learner adopts policy corresponding to the inferred reward.
set of skills by imitating
In the literature, imitation learning has been extensively studied from the learner's point of view to design efficient learning algorithms [9-15]. However, much less work is done from the teacher's point of view to reduce the number of demonstrations required to achieve the learning objective. In this paper, focus the problem of Teaching via Demonstrations (TvD), where helpful teacher assists the imitation learner in converging quickly by designing a personalized curriculum [16-20]. Despite a substantial amount of work on curriculum design for reinforcement learning agents [21- 27], curriculum design for imitation learning agents is much less investigated.
Prior work curriculum design for IRL learners has focused on two concrete settings: non- interactive and interactive. In the non-interactive setting [17, 18), the teacher provides a near-optimal set of demonstrations a single batch. These curriculum strategies do not incorporate
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

any feedback from the learner, hence unable to adapt the teaching to the learner's progress. In the interactive setting [28]. the teacher can leverage the learner's progress to adaptively choose the next demonstrations to accelerate the learning process. However, the existing state-of-the-art work [28] has proposed interactive curriculum algorithms that are based on learning dynamics of a specific IRL. learner model (i.e., the learner's gradient update rule); see further discussion in Section 1.1. In contrast, we focus on designing an interactive curriculum algorithm with theoretical guarantees that is agnostic to the learner's dynamies. This will enable the algorithm to be applicable for broad range of learner models, and in practical settings where the learner's internal model is unknown (such as tutoring systems with human learners). A detailed comparison between our curriculum algorithm and the prior state-of of-the-art algorithms from [18, 28] is presented in Section 1.1.
Our approach is motivated by works on curriculum design for supervised learning and reinforcement learning algorithms that use ranking over the training examples using a difficulty score [29-35]. In particular, our work is inspired by theoretical results on curriculum learning for linear regression models [32]. We define difficulty scores for any demonstration based on the teacher's optimal policy and the learner's current policy. We then study the differential effect of the difficulty scores on the learning progress for two popular imitation learners: Maximum Causal Entropy Inverse Reinforce- ment Learning (MaxEnt-IRL) [10] and Cross-Entropy loss-based Behavioral Cloning (CrossEnt- BC) [36]. Our main contributions are as follows:'
1. Our analysis for both MaxEnt-IRL and CrossEnt- BC learners leads to unified curriculum strat- egy, i.e., a preference ranking over demonstrations. This ranking is obtained based on the ratio between the demonstration's 's likelihood under the teacher's optimal policy and the learner's s cur- rent policy. Experiments on a synthetic car driving environment validate our curriculum strategy.
2. For the MaxEnt-IRL learner, we prove that our curriculum strategy achieves a linear convergence rate (under certain mild technical conditions), notably without requiring access to the learner's dynamics.
3. We adapt curriculum strategy to the learner-centric setting where a teacher agent is not present through the use of task-specific difficulty scores. As a proof of concept, we show that our strategy accelerates the learning process in synthetic navigation- based environments.
1.1 Comparison to Existing Approaches on Curriculum Design for Imitation Learning
In the -nniiterractive setting. [18] have proposed batch teaching algorithm (SCOT) by showing that the teaching problem can be formulated as a set cover problem. In contrast, our algorithm is interac- tive in nature and hence, can leverage the learner's progress (see experimental results in Section 5).
In the interactive setting, [28] have proposed the Omniscient algorithm (OMN) based on the iterative machine teaching (IMT) framework [37]. Their algorithm obtains strong convergence guarantees for the MaxEnt-IRL learner model; however, requires exact knowledge of the learner's 's dynamics (i.e, the learner's 's update rule). Our algorithm on the other hand is agnostic the learner's dynamics and is applicable to a broader family of learner models (see Sections 4 and 5).
Also for the interactive setting, [28] have proposed the Blackbox algorithm (BBox) heuristic to apply the OMN algorithm when the learner's ' dynamics are unknown thi makes the BBox al- gorithm more widely applicable than OMN. However, this heuristic algorithm is still based on the gradient functional form of the linear MaxEnt-IRLL learner model (see Footnote 2), and does not pro- vide any convergence guarantees. In contrast, our algorithm is derived independent of any specific learner model and we provide a theoretical analysis of our algorithm for different learner models (see Theorems 2, and 3). Another crucial difference is that the BBox algorithm requires access to the true reward function of the environment, which precludes it from being applied to learner- centric settings where no teacher agent is present. In comparison, our algorithm is applicable to leamer-centric settings (see experimental results in Section 6).
1.2 Additional Related Work on Curriculum Design and Teaching
Curriculum design. Curriculum design for supervised learning settings has been extensively stud- ied in the literature. Early works present the idea of designing a curriculum comprising of tasks with increasing difficulty train machine learning model [29-31]. However, these approaches require
Github repo: http:://gghhubb.oommmddda//////p http:///hcccotppoo0ooooocouuauaa seaching-demoostratiiee code.

---

task-specific knowledge for designing heuristic difficulty measures. Recent works have tackled the problem of automating curriculum design [38, 39j. There is also an increasing interest in theo- retically analyzing the impact of curriculum (ordering) of training tasks on the convergence of supervised learner models [32, 40, 41]. In particular, our work builds on the idea of difficulty scores of the training examples studied in [32].
The existing results on curriculum design for sequential maasiion-mainng settings are mostly em- pirical in nature. Similar to the supervised learning settings, the focus on curriculum design for reinforcement learning settings has been shifted from hand-crafted approaches [34, 35] to auto- matic methods [21-23, 25]. We refer the reader to a recent survey [42] on curriculum design for reinforcement learning. The curriculum learning paradigm has also been studied in psychology lit- erature [43-47]. One key aspect in these works has been to design algorithms that account for the pedagogical intentions of teacher, which often aims to explicitly demonstrate specific skills rather than just provide an optimal demonstration for a task. We see our work as complementary to these.
Machine teaching. The algorithmic teaching problem considers the interaction between a teacher and learner where the teacher's objective is to find an optimal training sequence to steer the learner towards a desired goal [37, 48-50J. Most of the work in machine teaching for supervised learning settings is on batch teaching where the teacher provides a batch of teaching examples at once without any adaptation. The question of how a teacher should adaptively select teaching examples for a learner has been addressed recently in supervised learning settings [51-56].
Furthermore, [16-18, 28, 57, 58] have studied algorithmic teaching for sequential decision-making tasks. In particular, [17, 18] have proposed batch teaching algorithms for an IRL. agent, where the teacher decides the entire set of demonstrations to provide to the learner before any interaction. These teaching algorithms do not leverage any feedback from the learner. In contrast, as discussed in Section 1.1, [28] have proposed interactive teaching algorithms (namely OMN and BBOX) for an IRL agent, where the teacher takes into account how the learner progresses. The works of [57, 58] are complementary to ours and study algorithmic teaching when the learner has a differet worldview than the teacher or has its own specific preferences.
2 Formal Problem Setup
Here, we formalize our problem setting which is based on prior work on sequential teaching [28, 37].
Environment. We consider an environment defined as a Markov Decision Process (MDP) M
(S,A,T Po, R²), where the state and action spaces are denoted by S and A. respectively. T SxSxA-1 [0, 1] is the transition dynamics, 7 is the discounting factor, and Po : S + [0, 1] is an initial distribution over states S. A policy π S x A + [0, 1] is mapping from state to probability distribution over actions. The underlying reward function is given by R : x A R.
Teacher-learner interaction. We consider setting with two agents: teacher and sequential learner. The teacher has access to the full MDP M and has a target policy F. (e.g., a optimol policy w.r.t. RE). The learner knows the MDP M but not the reward function R i.e., has only access to M \ R The teacher's goal is to provide an informative sequence of demonstrations to teach the policy π to the learner. Here, a teacher's demonstration =
is obtained by first choosing an initial state si E (where Po(s)) > 0) and then choosing a trajectory, sequence of state-action pairs, obtained by executing the policy x in the MDP M. The interaction between the teacher and the learner is formally described in Algorithm 1. For simplicity, that the teacher directly observes the learner's policy of at any time f. In practice, the teacher could approximately infer the policy oे by probing the learner and using Monte Carlo methods.
assume
Generic learner model. Here, we describe a generic learner update rule for Algorithm 1. Let θ C Rdd be a parameter space. The learner searches for a policy in the following parameterized policy space: Hl := {To- S x A - [0, 1], where 0 E }). For the policy search, the learner se- quentially minimizes a loss function that depends on the policy parameter and the demonstration & = (s$$,af) provided by the teacher. More concretely, we consider é ((g, 9) == - log IP (5 (0),
where P (E(0) = (o)))) II. To (aflsf) T ( (6f42lof,05 af) is the likelihood (probability) of the demon-
3

---

Algorithm Teacher-Learner Interaction
1: Initialization: Initial knowledge of learner af-
2: for I ===, =,,,2... 1, do
3: Teacher observes the learner's current policy x..
4: Teacher provides demonstration & to the learner.
5: Learner updates its policy to M/41 using &
stration & under policy we in the MDP M. At time upon receiving demonstration & pro- vided by the teacher, the learner performs the following online projected gradient descent update: 8.+1 < Proje [0, g. where nu is the learning rate, and 9r [V (Ee, f) Note that the parameter 01 reflects the initial knowledge of the learner. Given the learner's 's current parameter 0, at time t, the learner's policy is defined as oT. := Te.
Teaching objective. For any policy v, the value (total expected reward) of in the MDP M is de- fined as V : E.,a ?
n. M.-J (a s)-Re (s,a), where {S, 8 T. M} denotes the probability of visiting the state s after r steps by following the policy т. Let T² denote the learner's final policy the end of teaching. The performance of the policy x² (w.r.t T in M
can be evaluated via V va 59]. The teaching objective is to ensure that the learner's final
policy e-approximates the teacher's policy, i.e.. |V=*' v* S c. The teacher aims to provide an optimized sequence of demonstrations {E: 1-1, to the learner to achieve the teaching objective. The teacher's 's performance is then measured by the number of demonstrations required to achieve this objective. Based on existing work [28. 37], we assume that 0 O such that T kg- (we refer to H* as the target teaching parameter). Similar to [28]. we assume that smoothness condi- tion holds in the policy parameter space: | V S (f (Il@ 0'll)) V8, 8' E O. Then, the teaching objective in terms of V convergence can be reduced to the convergence in the parameter space, i.e., we can focus on the quantity | 0z|l-
Curriculum Design using Difficulty Scores
In this section, we introduce our curriculum strategy which is based on the concept of difficulty scores and is agnostic to the dynamics of the learner.
Difficulty scores. We begin by assigning a difficulty score Vo (E) for any demonstration & w.r.t a parameterized policy Ty the MDP M. Inspired by difficulty scores for supervised learning algorithms [32]. we consider difficulty score which is directly proportional to the loss function L, i... V (E) x g (l (E, O)), for a monotonically increasing function g. Setting g(.) = exp(-) leads to Ve (E) II. (a55 for MDPs with deterministic transition dynamics. Based on this insight, we define the following difficulty score which we use throughout our work.
Definition 1. The difficulty score of a demonstration & w.r.t the policy na in the MDP M given 1
by Vo (E) 11. 10)
Intuitively, the difficulty score of a demonstration & w.r.t. an agent's policy is inversely proportional to the preference of the agent to follow the demonstration. Demonstrations with a higher likelihood under the agent's policy (higher preference) have lower difficulty score and vice versa. With the above definition, the difficulty scores for any demonstration & w.r.t. the teacher's and learner's policies (at any time f) are respectively given by E (E) == Vg (E) and / (E) Vo, (E).
General curriculum strategy. Our curriculum strategy picks the next demonstration & to provide to the learner based on preference ranking induced by the teacher's 's and leamer's difficulty scores. The difficulty score of a demonstration w.r.t. the teacher and learner (at any time t) is denoted by y E and of respectively. Specifically, our curriculum strategy is given by:
V (E) +- arg max WE (E)
(1)

---

Teacher-centric and learner-centric settings. In the teacher- setting formalized in Section 2, our curriculum strategy utilizes the difficulty scores induced by the learner's current policy af and the teacher's policy 5 From Eq. (1) and Definition 1, we obtain the following teacher-centric curriculum strategy: & < arg maxg II, (aaSle
Additionally, we also consider the learner-centric setting where teacher agent is not present and the target policy = is unknown. Here, the learner can benefit from designing a self-curriculum (i.e., automatically ordering demonstrations) based on its current policy x5- We adapt our curriculum strategy to this setting by utilizing task-specific domain knowledge to define the teacher's 's difficulty score F for any demonstration &. From Eq. (l), given the learner's current policy Th and the teacher's difficulty score (E), the learner-centric curriculum strategy is given as follows: & arg max; I[.
1
Note that our curriculum strategy only requires access to the learner's and teacher's policies (at and x z) and does not depend on the learner's internal dynamics (i.e, its update rule mentioned in Section 2). This makes our approach more widely applicable to practical applications where often possible infer an agent's policy, but the internal update rule is unknown.
4 Theoretical Analysis of Our Curriculum Strategy
In this section, we present the theoretical analysis of our curriculum strategy for two popular learner models, namely, MaxEnt-IRL and CrossEnt-BC. For our analysis, we consider the teacher-centric setting as introduced in Section 2. Our curriculum strategy obtains preference ranking over the demonstrations provide to the learner based on the difficulty scores (see Definition 1). To this end, we analyze the relationship between the difficulty scores (w.r.t. the teacher and the learner) of the provided demonstration and the teaching objective (convergence towards the target teaching parameter 8*)) during each sequential update step of the leamnrr.
Given two difficulty values E.E th E IR we define the feasible set of demonstrations at time t as Dr (N v²²) == {5: : F (E) = &b and ! (E) = = This set contains all demonstrations for which the teacher's difficulty score di (E) is equal to the value eh and the learner's 's difficulty score T (E) is equal to the value v1 Let A, (ME vil) denote the expected convergence rate of the teaching objective at time t. given difficulty values uhth and vh::
A, (et (² == Ee fe "=1l0000='00'0> - 0,112 110 - (
(2)
where the expectation is w.r.t. the uniform distribution over the set Dj (w v²) Below, we analyse the differential effect of F and dh on A, (v/ )) i..., the effect of picking demonstrations with higher or lower difficulty scores on the learning progress.
4.1 Analysis for MaxEnt-IRI. Learner
Here, we consider the popular MaxEnt-IRL leameer model [10, 59, 60] in an MDP M with deter- ministic transition dynamies, i.e., T
A - (0, 1}. The MaxEnt-IRI. learner model uses a parametric reward function Rg S A + R where 0 Rd is parameter. The reward function Rs also depends on feature mapping ø : S a feature vector á (s,a) a) E Rd For our theoretical analysis, we consider Rg with a linear form, i.c., Re (s, a) = (0, ó a)) and d d'. In our experiments, we go beyond these simplifications and consider environments with stochastic transition dynamics and non-linear reward functions.
R= which encodes each state-action pair (s, a) by
Under the MaxEnt- IRL learner model, the parametric policy takes the following soft- Bellman form: To(als) = exp (Qa a) (s)),, where Vo(s) log E. exp Qa(s, a) and Qa(s, a) Ro a) + E. T(s'ls, Vs().. For any given 8, the corresponding policy πа can be efficiently computed via the Soft- Value-Iteration procedure with reward Ro (see [59, Algorithm. 9.1]). For the above setting and given parameter 0, the probability distribution IP (g(0) over the demonstration & takes
the closed-form P (E]0) (xp((),n)))
where غي := ΣT 7T ( ( af) and Z (0) is normalization factor. Then, at time t, the gradient of the MaxEnt-IRL learner is given by gt = uu+,, where
2(0)
5

---

=-==E.. 2T= T (P. = T. M x (a s) (s, a) is the feature expectation vector of policy T. We note that our curriculum strategy in Eq. (1) is not using knowledge of 9t-
For the MaxEnt-IRI. learner, we obtain the following theorem, which shows the differential effect of the difficulty scores (W.r.t the teacher and the learner) on the expected rate of convergence of the teaching objective A. ,²)). . We note that [32] obtained similar results for linear regression learner models the supervised learning setting.
Theorem 1. Assume that тソ is sufficiently small for all s.t. tt llg:ll < 2 0z 9t)1, where 9t is the gradient of the MaxEnt-IRL. learner. Then, the MaxEnt-IRL learner, the expected convergence rate of the teaching objective DS (v,, 1b"1) ) is:
monotonically decreasing with value E/e, i.e.
0A, 0y
0, and
monotonically increasing with value wh i.e. By > 0.
0s,
Theorem 1 suggests that choosing demonstrations with lower difficulty score the teacher's policy and higher difficulty score w.r.t the learner's policy would lead to faster convergence. Our curriculum strategy in Eq. (1) induces a preference ranking over demonstrations that aligns with these insights of Theorem 1. Furthermore, the following theorem states that the particular form of combining the two difficulty scores used in curriculum strategy, Eq. (1), achieves linear convergence to the teaching objective. This is similar to the state-of-the-ar OMN algorithm based on the IMT framework for sequential learners [28, 37]. Importantly, unlike the OMN algorithm, our curriculum strategy does not rely specifics of the learner model when selecting demonstrations.
Theorem 2. Consider Algorithm with the MaxEnt-IRR learner and our curriculum strategy in Eq. (1). Then, the teaching objective ||0- - 0,1l S. e is achieved in = Oloog 2) iterations.
In the above theorem, the constant terms suppressed by the (()) notation depend on the learning rate the learner (1))), the distance between the learner's initial parameter/knowledgge and the target
teaching parameter (l10* 01 []), and the richness of the set of demonstrations obtained by executing the policy π E in the MDP M. The richness notion is formally discussed in the Appendix.
4.2 Analysis for CrossEnt-BC Learner
Next, we consider the CrossEnt-Be learner model [15, 36]. In this case, the learner's parametric exp(Ha(e.a))
policy takes the following softmax form: TT(aas) = Σ. (xp"_oee.a))) where Hg : S x A + R a parametric scoring function that depends on the parameter 0 E Rdd and a feature mapping 6 : S x A + Rd For our theoretical analysis, we consider a linear scoring function Hg of the form Ho (s,a) (0, & (s, a)} (with d'). Then, at time step the gradient gr of the CrossEnt- BC learner is given by: 9r Erz (Ba-Ro, (-left [o(s5 a)] $( a.. In the experiments, we also consider non-linear scoring functions parameterized by neural networks.
Similar Theorem 1, obtain the following theorem for the CrossEnt-BC. learner, which also justifies our curriculum strategy in Eq. (1).
Theorem 3. Assume that The is sufficiently small for all s.s. Th l| < (00 9.)l where 9t is the gradient of the CrossEnt-Be learner. Then, for the CrossEnt-BB BC learner. the expected convergence rate of the teaching objective DS (b=,,b") ..""). after first-order approximation, is:
monotonically decreasing with "F i.e.,
0d. < 0, and
monotonically increasing with b² i.e.,
0. Dut
0A.
We note that the proof of Theorem relies on the first-order Taylor approximation of the term Σ, log E., exp (Hg (sfr,aa)) around 0g (detailed in the Appendix). Due this approximation, it is more challenging to obtain convergence result analogous to Theorem 2.
Experimental Evaluation: Teacher-Centric Setting
Inspired by the works of [28, 61, 62], we evaluate the performance of our curriculum strategy, Eq. (1), in a synthetic car driving environment on MaxEnt-IRL. and CrossEnt-BC learners. In particular, we consider the environment of [28] and the teacher-centric setting of Section 2.

---

(*(s)
stone
gras
grass
-0.5
-5
ped
-10
car-front
-2
TTOV
86
ped-front
-5
poãce
HOV
+1
police 0
Figure I: Car environment with 8 different types of tasks. Arrows represent the path taken by teacher's policy.
Figure 2: Environment features
Φ (s) and reward weights
Car driving setup. Fig. illustrates synthetic car driving environment consisting of different types of tasks, denoted as TO, T1, ..... T7. Each type is associated with different driving skills. For instance, TO corresponds to a basic setup representing traffic-free highway. T1 represents crowded highway. T2 has stones on the right lane, whereas T3 has mix of both cars and stones. Similarly, T4 has grass on the right lane, and T5 has mix of both grass and cars. T6 and T7 introduce more complex features such pedestrians, police, and HOV (high occupancy vehicles). The agent starts navigating from an initial state at the bottom of the left lane of each task, and the goal is to reach the top of lane while avoiding cars, stones, and other obstacles. The agent's 's action space is given by A = {left, straight, right). Action left steers the agent to the left of the current lane. If the agent is already in the leftmost lane when taking action left, then the lane is randomly chosen with uniform probability. We define similar stochastic dynamics for taking action r ght; action stra ght means no change in the lane. Irrespective of the action taken, the agent always moves forward.
Environment MDP. Based the above setup, we define the environment MDP, Mear. consisting of 8 types of tasks, namely TO-T7, and 5 tasks of each type. Every location in the environment is associated with state. Each task is of length 10 and width 2, leading reward function E that is dependent on an underlying feature vector of (see Fig. The feature vector of state s, denoted by df (s), is binary vector encoding the presence or absence of an object at the state. In this work we have two types of features: features indicating the type of the current cell as stone, grass, car, ped, pol volice, and HOV, as well as features providing some look-ahead information such whether there is car or pedestrian in the immediate front cell (denoted as car-front and ped-front), Now we explain the reward function R For states in tasks of type TO-T6, the reward is given by (w, of (s)) (see Fig. 2). Essentially there are different penalties (i.e., negative rewards) for colliding with specific obstacles such as stone and car. For states in tasks of type T7, there is reward of value +1 for driving on HOV; however, if pol ce is present while driving on HOV, a reward value of 5 is obtained. Overall, this results in the reward function R being nonlinear.
state space of size
5 x 8 x 20. We consider an action-independee
5.1 Teaching Algorithms
Here, we introduce the teaching algorithms considered in our experiments. The teacher's 's near- optimal policy n" is obtained via policy iteration [63]. The teacher selects demonstrations to pro- vide to the learner using its teaching algorithm. We compare the performance of our proposed CUR teacher, which implements our strategy in Eq. (1), with the following baselines:
CUR-T: A variant of our CUR teacher that samples demonstrations based the difficulty score E alone, and sets Vi to constant.
CUR-L: A similar variant of our CUR teacher that samples demonstrations based on the difficulty score / alone, and sets E to constant.
AGN: an agnostic teacher that picks demonstrations based on random ordering [28, 32).
OMN: The omniscient teacher is state-of-the-art algorithm [28, 37], which applicable only to MaxEnt-IRL learners. OMN requires complete knowledge of the parameter 0", the learner's current parameter 0r and the learner's gradients T/-t- Based on this knowledge, the teacher picks demonstrations to directly steer the learner towards 0", i.e., by minimizing ||0" (0, n.9:))
7

---

Cen
Age
AGe
Cua-7
CoN.T
Oxe
Ctn-L
BBoox
Scor
Scrr
Expected rewordd
Exppectin reward
Tosk picked
50
100
150
100
200
100
150
200
100
200
Timo
Time
Time
(a)
(c)
(d)
Figure 3: Learning curves and curriculum visualization for MaxEnt-IRL learners (with varying initial knowl- edge) trained on the car driving environment: (a) reward convergence plot and (b) curriculum generated by the CUR teacher for the learner with initial knowledge of IO; (c) reward convergence plot and (d) curriculum
generated by the CUR teacher for the learner with initial knowledge of TC-T3.
CUR.L
Coee +op- AGH
Expected roward rowardd
2.5
17
Expected reward
17
T6
picked
T6
75
75
T4
-0.5
Task
73
23
70-
70
200
300
420
100
200
300
400
100
200 (c)
300
400
100
200 Time
300
400
TTme
(a)
(b)
(d)
Figure 4: Learning curves and curriculum visualization for CrossEnt-BC learners (with varying initial knowl- edge) trained on the car driving environment: (a) reward convergence plot and (b) curriculum generated by the CUR teacher for the learner with initial knowledge of IO; (c) reward convergence plot and (d) curriculum
generated by the CUR teacher
leamer with initial knowledge of TO-T3.
The blackbox teacher [28] is designed based on the functional form of gradients for the linear MaxEnt-IRL learner model.2 Specifically, the teacher picks a demonstration which maximizes Ea (p (s', (s' ( Rt a')l, where p denotes state visitation frequency vectors. The BBoX teacher does not require access to 0* or the learner's current parameter Oe: however, it requires access the reward function Re
BBoX:
Scor: The ScOT teacher [18] aims to find the smallest set of demonstrations required to teach an optimal reward function to the MaxEnt-IRLLearner The teacher uses cover algorithm to pre- compute the entire curriculum as a batch, prior to training. In our implementation, after having provided the entire batch, the teacher continues providing demonstrations selected random.
5.2 Learner Models
Next, we describe the MaxEnt-IRL and CrossEnt- BCC learner models. For the MaxEnt-IRL learner, we evaluate all the above-mentioned teaching algorithms that include state-of-the-ar baselines; for the CrossEnt-BC learner, evaluate CUR, CUR-T, CUR-L, and AGN algorithms.
MaxEnt-IRU IRL learner. For alignment with the prior state-of-ther -hrt--rrtt work on teaching sequential MaxEnt-IRL learners [28], we perform teaching over states in our experiments. More concretely, at time the teacher picks a state st (where Po(st) > 0) and provides all demonstrations starting from 8, the learner given by E., = {5 (0f of) s.t. sf 8t The gradient gt of the MaxEnt- IRL learner is then given by gr = pt . م Sس where (i) p E.,| Eeez,, μ and (ii) n.
the feature expectation vector of policy with starting state set to St (see Section 4.1). Based on [28], we consider the learner's feature mapping as Φ(s, a) = (F(ss and the learner uses a non- linear parametric reward function R/ (s,a) (0 ó (s, a)) + (0 1:24', & (s,a))² where d' is the
'The BBOX teacher's objective is derived under the assumptions that the reward function can be linearly parameterized as (w", , (s)) and gradients are based on the linear MaxEnt-IRR IRL learner model. Under these assumptions, the teacher's objective be equivalently written (w" gt)
8

---

dimension of ((s, a). As explained in [28], a linear reward representation cannot capture the optimal behaviour for Mear. We consider learners with varying levels of initial knowledge, i... the learner is trained on a subset of tasks before the teaching process starts. In this setting, for our curriculum strategy in Eq. (1) the difficulty score of set of demonstrations associated with a starting state E, is computed as the mean difficulty score of individual demonstrations in the set.
CrossEnt-BC learner. We consider the CrossEnt-BC learner model of Section 4.2 as our sec- ond learner model. The learner's feature mapping is given by Φ (s, a) (. (Ø (s').. A quadratic parametric form is selected for the scoring function, i.e., He (s, = (Ol.dr o (s, a)} + 1:2', (s, a))²² where d' is the dimension of (6s; a). We consider learners with varying initial knowledge and perform teaching over states similar to the MaxEnt-IRI. learner.
5.3 Experimental results
Figs. 3a, 3c and 4a, 4c show the convergence of the total expected reward for the MaxEnt-IRI. and CrossEnt-BC learners respectively, averaged over 10 runs. The CUR teacher outperforms OMN de- spite not requiring information about the learner's dynamics. For non-linear parametric reward functions, the MaxEnt-IRL learner no longer solves convex optimization problem. As result, foreing the learner to converge to a fixed parameter doesn't necessarily perform well, as seen by the poor performance of the OMN teacher in Fig. 3c. The CUR teacher is competitive with the BBOX teacher. Unlike our CUR teacher, the BBOX teacher does require exact access to the true reward function, R The CUR teacher consistently outperforms the AGN and ScOT teachers, as well as both the CUR-T and CUR-L variants.
Figs. 3b, 3d and 4b, 4d visualize the curriculum generated by the CUR teacher for the MaxEnt- IRL and CrossEnt-BC learners respectively. Here, the curriculum refers to the type of task, TO-T7, associated with the demonstrations provided by the teacher to the learner at time step f. For both types learners we see that at the beginning of training, the teacher focuses on tasks which teach skills the learner is yet to master. For example, in Fig. 4d, the teacher picks tasks T4, T6, and T7, which teaches the learner to avoid grass. pedestrians, and to navigate through police and HOV. We also notice that the CUR teacher can identify degradation in performance previously mastered tasks, e.g., task T1 in Fig. 3d, and corrects for this by picking them again later during training.
Additional results under limited observability. In the above ex- periments, we consider the learner's policy to be fully observable by the teacher at every time step. Here, study the performance of our CUR teacher under the limited observability setting, similar to [28], where the learner's policy needs be estimated by probing the learner. The probing process is formally characterized by two param- eters, (B,k), where the learner's policy is probed after every B time steps and each probing step corresponds to querying the learner's pol- icy of total of l times from each state 8 S in the MDP. The learner's 's policy, < Va, s, is then approximated based on the fraction of the queries in which the learner performed action a from state In between every B time steps that the learner is probed, the CUR teacher does not update its estimate of the learner's policy. We note that the (B = 1. k - oo) setting corresponds to full observability of the learner. Fig. 5 depicts the performance of the CUR teacher for different values of (B, k). Even under limited observability, the CUR teacher's ' performance is competitive with the full observability setting. The performance of (B- = 40, k + oo) is even slightly better at certain time steps during later stages of training compared to (B = 1, k: + oo), which is possibly due to the strategy of greedily picking demonstra- tions not being necessarily optimal. Also, for the limited observability setting it can be interesting to explore approaches that alleviate the need to query the full policy of the learner [64, 65].
Expocted reward
12
Cu 40,0 mx)
Cuica 40,2
0
50
100
150
200
Time
Figure 5: Learning curves for the MaxEnttIRIL learner under limited observability.
6 Experimental Evaluation: Learner-Centrie Setting
In this section, we evaluate our curriculum strategy in a learner-centric setting, i.e., no teacher agent is present, and the teacher's difficulty E is expressed by task-specific difficulty score (see Sec- tion 3). We evaluate our approach for training a multi-task neural policy to solve discrete optimiza- tion problems. Here, we provide an overview of the results with additional details in the Appendix.

---

Coa.l Aox
Dun-T
Now
CoR.T T
Expected soward
o
12
Expected
-12
5500
11000
4903 Time
9000
Time
X32
(a)
(b)
(h)
Figure 6: Illustration of shortest path navigation
Figure 7: Illustration of a travelling salesman naviga- tion task (left) and convergence curves (right).
lask (left) and convergence curves (right).
Experiment setup. We begin by describing the synthetic navigation-based environments consid- ered in experiments. Our first navigation environment comprises of tasks based on the shortest path problem [66]. We represent each task with grid-world (see Fig. 6a) containing goal cells (depicted by stars), and cells with muds and bombs (shown brown and red respectively). The agent aims to navigate to the closest goal cell while avoiding muds and bombs. Our second navi- gation environment comprises of tasks inspired by the travelling salesman problem (TSP) [67, 68] (see Fig. 7a). Again we represent each task with a grid-world, where the agent's 's goal is to find the shortest tour which visits all goals and returns to its initial location (see Fig. 7a). Orange arrows in Figs. 6a and 7a depict the optimal path for the agent. In our experimental setup, we begin by creating a pool of tasks and split them into training and test sets. The curriculum algorithms order the training tasks during the training phase based on their strategy to speed up the learner's ' progress. The aim of the learner is to learn multi-task neural policy that can generalize to new unseen tasks in the test set.
Curriculum algorithms. We compare the performance of four different curriculum algorithms: (i) the CUR algorithm picks tasks from the training set using Eq. (1) where the numerator is / and the denominator f is defined by task-specific difficulty score (detailed in the Appendix); (ii) the CUR-L algorithm picks tasks from the training set using Eq. (1) where the numerator is % and the denominator is set to (iii) the CUR-T algorithm picks tasks from the training set using Eq. (1) where the numerator is set to 1 and the denominator is set to q E (iv) the AGN algorithm picks tasks with uniform distribution over the training set.
Learner model. We consider neural CrossEnt-BC learner (see Section 4.2). The learner's scor- ing function Hg is parameterized by 6-layer Convolutional Neural Network (CNN). The CNN takes as input a feature mapping of the agent's current position in a task, and outputs a score for each action. The learner minimizes the cross-entropy loss between its predictions and the demonstrations.
Results. Figs. 6b and 7b, show the reward convergence curves on the test set for the different curriculum algorithms averaged over runs. The CUR algorithm leads to faster reward convergence compared to the AGN algorithm, which is the common approach for training a neural policy. CUR- is competitive with CUR this setting which highlights the importance of the learner's difficulty.
Discussion and Conclusions
We presented unified curriculum strategy, with theoretical guarantees, for the sequential MaxEnt- IRL and CrossEnt-BC learner models, based on the concept of difficulty scores. Our proposed strategy is independent of the learner's internal dynamics and is applicable in both teacher-centric and learner-centric settings. Experiments on a synthetic car driving environment and on navigation- based environments demonstrated the effectiveness of our curriculum strategy.
Our work provides theoretical underpinnings curriculum design for teaching via demonstrations, which can be beneficial in educational applications such as tutoring systems and also for self- curriculum design for imitation learners. such we do not see any negative societal impact of our work. Some of the interesting directions for future work include: obtaining convergence bounds for CrossEnt-BC and other learner models, designing curriculum algorithms for reinforcement learning agents based the concept of difficulty scores, and designing approaches to efficiently approximate the learner's policy using less queries.
10

---

References
[1] S. Schaal. Learning from demonstration. In NewrIPS, pages 1040-1046, 1997.
[2] Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469 483, 2009.
[3] Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning robotics: A survey. The International Journal of Robotics Research, 32(1 52(11):1238-1274, 2013.
[4] Maya Cakmak and Andrea L Thomaz. Eliciting good teaching from humans for machine learners. Artificial Intelligence, 217:198- 215, 2014.
[5] Daphna Buchsbaum, Alison Gopnik, Thomas L Griffiths, and Patrick Shafto. Children's imi- tation of causal action sequences is influenced by statistical and pedagogical evidence. Cogni- tion, 120(3):331-340, 2011.
[6] Patrick Shafto, Noah D Goodman, and Thomas L Griffiths. A rational account of pedagogical reasoning: Teaching by, and learning from, examples. Cognitive psychology, 71:55-89, 2014.
[7] D. A. Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Computation, 3(1):88-97, 1991.
[8] Stuart Russell. Learning agents for uncertain environments. In COLT, pages 101-103, 1998.
[9] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In ICML, page 1, 2004.
[10] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In AAAI, 2008.
[1]] Abdeslam Boularias, Jens Kober, and Jan Peters, Relative entropy inverse reinforcement learn- ing. In AISTATS, pages 182-189, 2011.
[12] Markus Wulfmeeer, Peter Ondruska, and Ingmar Posner. Maximum entropy deep inverse rein- forcement learning. arXiv preprint arXiv: 150,008877,0888, 2015.
[13] Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In ICML., pages 49-58, 2016.
[14] Dorsa Sadigh, Anca Dragan, Shankar Sastry, and Sanjit Seshia. Active preference-based learning of reward functions. RSS, 2017.
[15] Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, and Jan Peters. An algorithmic perspective on imitation learning. Foundations and Trends in Robotics, 2018.
[16] Thomas J Walsh and Sergiu Goschin. Dynamic teaching in sequential decision making envi- ronments. UAI, 2012.
[17] Maya Cakmak and Manuel Lopes. Algorithmic and human teaching of sequential decision tasks. In AAAI, volume 26, 2012.
[18] Daniel S Brown and Scott Nickum. Machine teaching for inverse reinforcement learning: Algorithms and applications. In AAAI, 2019.
[19] Manuel Lopes, Francisco Melo, and Luis Montesano. Active learning for reward estimation in inverse reinforcement learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 31-46. Springer, 2009.
[20] Kareem Amin, Nan Jiang, and Satinder Singh. Repeated inverse reinforcement learning. In NeurlPS, pages 1815-1824, 2017.
[21] Maxwell Svetlik, Matteo Leonetti, Jivko Sinapov, Rishi Shah, Nick Walker, and Peter Stone. Automatic curriculum graph generation for reinforcement learning agents. In AAAI, volume 31, 2017.
[22] Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse curriculum generation for reinforcement learning. In CORL, pages 482-495, 2017.
[23] Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In ICMI., 2018.

---

[24] Wojciech Czarnecki, Siddhant Jayakumar, Max Jaderberg, Leonard Hasenclever, Yee Whye Teh, Nicolas Heess, Simon Osindero, and Razvan Pascanu. Mix & match agent curricula for reinforcement learning. In ICMI., 2018.
[25]
Sanmit Narvekar and Peter Stone. Learning curriculum policies for reinforcement learning. In AAMAS, pages 25-33, 2019.
[26] Sainbayar Sukhbaatar, Zeming Lin, Πya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Robert Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play. In ICLR, 2018.
[27] Martin A Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. In JCML, 2018.
[28]
Parameswaran Kamalaruban, Rati Devidze, Volkan Cevher, and Adish Singla. Interactive teaching algorithms for inverse reinforcement learning. In IJCAI, 2019.
[29] Jeffrey L Elman. Learning and development in neural networks: The importance of starting small. Cognition, 48(1):71 1993.
[30] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ICML, pages 41-48, 2009.
[31] Wojciech Zaremba and Ilya Sutskever. Learning to execute. arXiv preprint arXiv:1410.4611 2014.
[32] Daphna Weinshall, Gad Cohen, and Dan Amir. Curriculum learning by transfer learning: Theory and experiments with deep networks. In ICML, 2018.
[33] Daphna Weinshall and Dan Amir. Theory of curriculum learning, with convex loss functions. arXiv preprint arXiv: mat:::8123.0777 2018.
[34] Minoru Asada, Shoichi Noda, Sukoya Tawaratsumida, and Koh Hosoda. Purposive behavior acquisition for a real robot by vision-based reinforcement learning. Machine learning, 23(2- 3):279-303, 1996.
[35] Yuxin Wu and Yuandong Tian. Training agent for first-person shooter game with actor-critic curriculum learning. In ICLR, 2016.
[36] Michael Bain. A framework for behavioural cloning. In Machine Intelligence 15, pages 103- 129, 1995.
[37] Weiyang Liu, Bo Dai, Ahmad Humayun, Charlene Tay, Chen Yu, Linda B Smith, James M Rehg, and Le Song. Iterative machine teaching. In ICML, 2017.
[38] Alex Graves, Mare G Bellemare, Jacob Menick, Rémi Munos, and Koray Kavukcuoglu. Au- tomated curriculum learning for neural networks. In /CML., pages 1311-1320, 2017.
[39] Lu Jiang, Zhengyuan Zhou, Thomas Leung. Li-Jia Li, and Li Fei-Fei. Mentomet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In ICML, pages 3304-2315 2018.
[40] Tianyi Zhou and Jeff Bilmes. Minimax curriculum learning: Machine teaching with desirable difficulties and scheduled diversity. In ICLR, 2018.
[41] Tianyi Zhou, Shengjie Wang, and Jeff Bilmes. Curriculum leaminng by optimizing learning dynamics, In AISTATS, pages 433-441, 2021.
[42] Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey. Journal of Machine Learning Research, 21:1-50, 2020.
[43] Mark K Ho, Michael Littman, James MacGlashan, Fiery Cushman, and Joseph L Austerweil. Showing versus doing: Teaching by demonstration. In NewrIPS, 2016.
[44] Mark Ho, Michael Littman, and Joseph L Austerweil Teaching by intervention: Working backwards, undoing mistakes, or conrecting mistakes? In Conference of the Cognitive Science Sociery, 2017.
[45] Mark Ho, Michael Littman, Fiery Cushman, and Joseph I. Austerweil. Effectively learning from pedagogical demonstrations. In Conference of the Cognitive Science Society, 2018
12

---

[46] Mark K Ho, Fiery Cushman, Michael Littman, and Joseph L Austerweil. People teach with rewards and punishments as communication not reinforcements. In Journal of Experimental Psychology: General, page 520-549, 2019.
[47] Mark Ho, Fiery Cushman, Michael Littman, and Joseph L Austerweil. Communication in action: Planning and interpreting communicative demonstrations. In Journal of Experimental Psychology: General, 2021.
[48] Sally A Goldman and Michael J Kearns. On the complexity of teaching. Journal of Computer and System Sciences,
1995.
[49] Xiaojin Zhu, Adish Singla, Sandra Zilles, and Anna N. Rafferty. An overview of machine teaching. CoRR, abs/1801.0599 2018.
[50] Scott Cheng- Cheng-Hsin Yang, Yue Yu, arash Givchi, Pei Wang, Wai Keen Vong, and Patrick Shafto. Optimal cooperative inference. In AISTATS, pages 376-385, 2018.
[51] Francisco S Melo, Carla Guerra, and Manuel Lopes. Interactive optimal teaching with un- known learners. In IJCAI, pages 2567-2573, 2018.
[52] Weiyang Liu, Bo Dai. Xingguo Li, Zhen Liu, James Rehg, and Le Song. Towards black-box iterative machine teaching. In ICML, 2018.
[53] Yuxin Chen, Adish Singla, Oisin Mac Aodha, Pietro Perona, and Yisong Yue. Understanding the role of adaptivity in machine teaching: The case of version space learners. In NeurIPS, 2018.
[54] Teresa Yeo, Parameswaran Kamalaruban, Adish Singla, Arpit Merchant, Thibault Asselborn, Louis Faucon, Pierre Dillenbourg, and Volkan Cevher. Iterative classroom teaching. In AAAI, 2019.
[55] Anette Hunziker, Yuxin Chen, Oisin Mac Aodha, Manuel Gomez Rodriguez, Andreas Krause, Pietro Perona, Yisong Yue, and Adish Singla. Teaching multiple concepts to forgetful learner. In NeurlPS, 2019.
[56] Farnam Mansouri, Yuxin Chen, Ara Vartanian, Jerry Zhu, and Adish Singla. Preference-based batch and sequential teaching: Towards a unified view of models. In NewrlPS, 2019.
[57] Luis Haug, Sebastian Tschiatschek, and Adish Singla. Teaching inverse reinforcement learners via features and demonstrations. In NewrlPS, 2018.
[58] Sebastian Tschiatschek, Ahana Ghosh, Luis Haug. Rati Devidze, and Adish Singla. Learner- aware teaching: Inverse reinforcement learning with preferences and constraints. In NeurIPS, 2019.
[59] Brian D Ziebart. Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy. PhD thesis, University of Washington, 2010.
[60] Brian D Ziebart, J Andrew Bagnell, and Anind K Dey. The principle of maximum causal entropy for estimating interacting processes. IEEE Transactions Information Theory, 1964);1966-1988, 2013.
[61] Andrew Y Ng and Stuart Russell. Algorithms for inverse reinforcement learning. In ICML, 2000.
[62] Sergey Levine, Zoran Popovic, and Vladlen Koltun. Feature construction for inverse reinforce- ment learning, NeurIPS, 23:1342-1350, 2010.
[63] R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press, 2018.
[64] Alexis Jacq, Matthieu Geist, Ana Paiva, and Olivier Pietquin. Learning from a learner. In ICML, 2019.
[65] Daniel S Brown, Jordan Schneider, Anca Dragan, and Scott Niekum. Value alignment verifi- cation. In ICML, 2021.
[66] E. W. Dijkstra. A note on two problems in connexion with graphs. Numer Math., 1:269-271, December 1959.
[67] Wouter Kool, Herke van Hoof, and Max Welling. Attention, learn solve routing problems!
In ICLR, 2019.
[68] Chaitanya K. Joshi, Thomas Laurent, and Xavier Bresson. On learning paradigms for the travelling salesman problem. In NeurlPS graph representation learning workshop, 2019.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] In Sections and 4, we present our curriculum strategy and provide theoretical analysis as described in the introduction. The experimental re- sults mentioned in the abstract and introduction are detailed Sections 5 and 6.
(b) Did you describe the limitations of your work? [Yes] We discuss the limitations of work in Sections 5 and 7. We point out that convergence bounds for CrossEnt- BC learners have not been obtained yet and that it would be beneficial to explore approaches to estimate the learner's policy sr using fewer queries.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] As stated in Section 7, this work presents the theoretical underpinnings of curriculum design for teaching via demonstrations. This can be beneficial for education applications and for self-curricllum design for imitation learners. As such in the present form, we do not see any direct negative societal impacts of our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We confirm that our paper conforms with the ethics review guidelines.
2. If you are including theoretical results...
(a) Did you state the full set assumptions all theoretical results? [Yes] In Section 4.1, we state our assumptions of deterministic MDP and linear reward structure. Simi- larly, we state our assumption a linear scoring function for the CrossEnt-BP learner in Section 4.2. The simplifications considered are reiterated in the proofs.
(b) Did you include complete proofs of all theoretical results? [Yes] Complete proofs of all theoretical results are included in the Appendix.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] The code and instructions are included as a URL.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Training details are presented in the Appendix and are also present in the code.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] Eror bars are included in all the result graphs, as can be seen in figures.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] The details provided in the Appendix.
4. If you are using existing assets (e-g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you'l 'e using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifi- able information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "bC_nfbIsRM", "reviewer": "Reviewer_RaMd", "summary": "This paper considers the problem of designing a curriculum of trajectories to teach an imitation learning agent. The authors propose to sequentially give trajectories that are both difficult (low probability) for the current learner and easy (high probability) for the demonstrator. Some theoretical justification is provided for why this approach works as well as some empirical results on simple domains. In general this paper seems novel, but there are many areas that deserve more clarification and it is unclear how far this idea can be taken given the strong assumptions.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I like this paper. The problem setting is interesting and the theoretical analysis is nice and some of the empirical results are compelling. However, there are three main weaknesses that I see that should be addressed:.\n\n(1) The authors claim that batch curriculum approaches are not as good as interactive settings. However, the approach is not compared with batch curriculum approaches. The authors need to compare with a batch setting to justify why to select the trajectories sequentially. For example, the SCOT algorithm (citation [18]) uses a set cover approximation to design the entire curriculum offline. While this method was not originally designed for training BC agents, it can be used as a baseline in the MaxEnt IRL agent setting. It is unclear whether the proposed methods are that much better as they require access to the learner's policy at each iteration. It seems that an offline curriculum algorithm will be much more efficient as it only requires access to the MDP and the teacher's policy and does not require repeatedly probing the learner. However, I agree that there is a potential for the sequential approach to be better as it can adapt to the learning progress. This hypothesis needs to be rigorously tested by comparing the proposed approach with the prior work on batch teaching for LfD.\n\n(2) Probing the learner's policy seems very difficult. The authors mention monte-carlo techniques but this would require repeatedly querying for actions at every state, if I understand correctly. Is there a possibility of devising a curriculum that doesn't require this many queries? See some ideas below. Also, it's unclear how much of an advantage it is to have access to the policy but not the internal parameters as in prior work [28]. What is the benefit of also having access to the internal parameters as done in prior work? I don't think this point needs to be solved, but more discussion of this limitation would strengthen the paper.\n\n(3) Figure 5 shows that you don't need the hand-crafted difficulty score since you can just use the learner's difficulty. This seems to make the entire section 6 unnecessary. I'm not convinced that equation (4) is a good surrogate for teacher difficulty. In particular it seems that having more goals makes it easier not harder since there are more ways to solve the task near optimally. \nAlso the interpretation of the curriculum in Figure 6 seems iffy. The number of goals is included in the surrogate difficulty score f^E so it seems like that is the main reason later stages have more goals, not that this is actually a more difficult concept. I would recommend that the authors compare qualitatively with CUR-L to see if you get the same or different curricula if you don't use f^E.\n\nI like the car driving example for visualizing and testing curriculum learning. \n\nClarity:\nThe authors often say that their method doesn't require access to the \"learner's internal dynamics\". This is incorrect and the wording should be changed. The internal dynamics are the way that the learner updates its model. This is assumed to be known. The unknown thing is the learner's parameters, not their internal dynamics. Just saying \"internal parameters\" would be much better and clearer.\n\n\nQuestions:\nWhat happens with deterministic policies? How do you define/deal with 0/0 when selecting trajectories.\n\nIt would be good to relate this approach to prior work by Brown et al. on \"Value Alignment Verification\"? There they also seek an epsilon-approximate value difference. But look at a testing problem rather than a teaching problem. This enables them to verify the student policy is within epsilon of the teacher without requiring access to the entire policy of the learner as assumed in this work. Can something similar be done here?\n\nAlong similar lines, could ideas from Jacq et al. \"Learning from a learner\" allow curriculum design for a MaxEnt agent without needing to query for the entire policy?\n\nDoes the proposed approach work for continuous action spaces? What would be needed to extend to this setting?\n\nWhy have two separate theorems (1 and 3)? They say basically the same thing. If the proof techniques are different enough to justify different theorems it would be nice to give a proof sketch for how they are different. Or just merge them into one theorem.\n\nAlso, why is there no analogy to Thm 2 for BC? I think this is fine to leave for future work, but it would be beneifical to add a brief description of why this setting is so much more challenging than that for Thrm 2.\n\nIn figure 3b and d, why teach what is known initially later in training? Is this implicitly accounting and correcting for catastrophic forgetting?\n\nThe reward function still appears to be linear since there is an indicator for the conjunction, right?\n\nThe Ominicient teacher is not really ever explained in section 3.\n\nHow would this work for educational applications? There I'd assume that you know the teaching policy but can never really know the learner policy. This seems to make the proposed approach ill-suited since the experiments show that you can get around not knowing the teacher's policy, but need access to the full policy of the student.\n\n---\nThank you for your extensive response. The authors have answered my questions and based on the response, I think the paper quality will improve. \n\n---\nUpdate: After a fruitful discussion with the other reviewers and the authors I am satisfied with the claims of novelty in the paper. I think these claims are nuanced and deserve better attention and I would like to see a more careful discussion of the differences between the submission and the blackbox approach proposed by Kamalaruban et al.. I think these points are easy to fix and will strengthen the paper. I would recommend that the paper be accepted.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "Gm1FApezcMn", "reviewer": "Reviewer_kSL2", "summary": "The paper investigates approaches to curriculum design for teaching via demonstrations, unifying two models in the literature. Some theoretical results are proven and experiments show the effectiveness of the approach. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presents a unified approach to curriculum design for MaxEnt-IRL and CrossEnt behavioral cloning. The primary contributions appear to be a teaching approach that does not require direct access to the learner's internal parameters, theoretical results regarding convergence rate, and modest sized empirical demonstrations of effectiveness. \n\nPositive aspects: \n- The problem of curriculum design is interesting and important, and compelling theoretical results are relatively limited at this time. \n- The proposed approach based on difficulty scores is interesting. \n- The empirical results are nice, if a bit modest in scope. \n\nWeaknesses: \n- The review of literature was somewhat limited. In particular, there are several papers by Mark Ho and Michael Littman and colleagues that seem relevant. \n- The motivation for difficulty scores was weak. In particular, it seems connected to the broader class of approaches that trade off effort of the teacher and effect for the learner that appear in probabilistic formulations. \n- The paper is rather niche. The theoretical results appear to be limited to the two approaches considered, and not general to curriculum learning. The empirical demonstrations are on small scale gridworld style problems. The exposition does not flow particularly well in that there are terms used without introduction, motivations for choices early in the paper are explained later, etc. \n- Although the paper is billed as being about curriculum learning theory and applications, it is not general enough to live up to these promises. \n\nDetailed comments: \n- \"much less work is done from the teacher’s point of view to reduce the number of demonstrations required to achieve the learning objective\" There is a literature on Teaching dimension that is relevant. Also see Optimal Cooperative Inference. \n- There is relevant work by Mark Ho, Michael Littman and others that does not appear to be reviewed. \n- \"We define difficulty scores for any demonstration\" Please define terms such as \"difficulty scores\" when they are introduced. \n- \"We then study the differential effect of the difficulty scores\" What does this mean? \n- \"We adapt our curriculum strategy to utilize domain knowledge, in the form of task-specific difficulty scores, when the teacher’s optimal policy is unknown.\" What does this mean? What constitutes domain knowledge? What are difficulty scores? Teacher's policy unknown to whom? \n- \", the teacher could approximately infer the policy πLt by probing the learner and using Monte Carlo methods\" Probing the learner? \n- \"Leads to CUR teaching algorithm\" How does it lead to it? \n- Self-curriculum is not defined. \n- I had trouble understanding the feasible set introduced on lines 142-143. The difficulty values are the same or different from difficulty scores? What type are they? \n- \"In our experiments we relax the simplification\" What does this mean? \n- Rather than using the phrase \"differential effect\" it would be clearer to just say what the effect is. Differential effect is not that helpful for the reader to whom the details have not been explained. \n- \" The result is analogous to that obtained for linear regression\" Analogous how?  \n- I do not support the choice to put the technical details on nu_t in the appendix. One could at least give the reader a flavor. \n- \"Theorem 1 motivates our curriculum strategy in Eq.\" How? Also, the ordering of the exposition is not ideal, given that Eq 1 appears much earlier. \n- In theorem 3, what does it mean for the convergence rate to be \"roughly\" something? \n- The use of a scheduling mechanism is fine in principle. But it is important to assess all algorithms on the same footing, so we can differentiate between what is due to the algorithm and what is due to the training approach. It is not ideal to put details relevant to understanding performance in the appendix. \n- \" This clearly shows the effectiveness of our proposed ranking strategy that combines the two difficulty scores\" This does not seem quite accurate. The result shows that both difficulty scores matter. I do not believe we can tell what it says about the particular manner of combining them. \n- \"Our work provides theoretical underpinnings of curriculum design for teaching via demonstrations, which can be beneficial for education applications and faster training of IRL/BC learners across application domains.\" This is a huge overstatement. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No limitations were discussed. Societal impact was not discussed. ", "review_text": "", "extra_scores": {"ethical_concerns": "None. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "6rXcxXw8q5f", "reviewer": "Reviewer_3tg2", "summary": "The authors introduce a new method for imitation learning, and more specifically, for selecting an expert demonstration that is maximally useful for the learner.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality:** \n\nThe method that is introduced here is not inherently bad but it only appears to work on small isolated grid world problems in which all states and actions can be enumerated. I believe that with automatic curriculum methods like DAGGER [1] and AGGREVATE [2] (which work in continuous state/action spaces), and astounding results such as \"Learning to drive in a day\" [3], the relevancy of this contribution is questionable at best. I guess the only way to convince me otherwise would be to include experiments in continuous state/action space but I don't think this can be part of the discussion period.\n\nThe authors also omitted a related literature section and instead list a few related works in the introduction, which does not help separate this work clearly from its predecessors.\n\nI also feel like Behavior Cloning is treated as state of the art for imitation learning and from my best understanding that is the most naive possible approach. Some more contemporary baselines are DAC [4], TCN [5], and VIRL [6] that learn trajectory embeddings of the expert which could be used for comparison, rather than the product of single-state differences, which is one of the proposed metrics in this work.\n\n**Quality:** \n\n1. In the car environment, the learned curriculum is visualized (Fig.3 and 4, b & d) and there are artifacts that need to be explained: (a) why does the learner with initial knowledge of T0 go back to learning T0 after T=150, why does it ignore T1 instead? (b) what is the \"time\" axis? Is that supposed to be environment interactions? (c) Why does the agent with knowledge of T0-T3 gradually resume these tasks? (d) ...and why does it initially ignore T4 and T5? (e) How many seeds were used to train the agents in Fig. 3/4?\n2. All that the experiments in the 6x6 grid world show is that the agent learns to maximize equation 4, i.e. create more goals and fewer bombs/mud and therefore create easier and easier environments, which is backed up by Figure 5c. What are these experiments supposed to show? The agent has not acquired any resilience to bombs and mud but rather found a way to remove them from the environment altogether.\n\n**Clarity:** \n\nThe writing is mostly clear.\n1. A dedicated \"Related Literature\" section would be good\n2. line 80 $\\Theta$ is not defined\n3. line 82 should be an equation and not line-broken\n4. Algorithm 1 can be cut because it's trivial or can be explained in 2 sentences.\n5. The entirety of sections 4(.0) and 4.1 is pointless in my eyes because it assumes comparability of the agent parameters $\\theta$ to the expert parameters $\\theta^*$ and that is not a realistic or even plausible assumption, given that an expert can be of a completely different structure (e.g. a human demonstrating to a neural network)\n6. Fig. 1 and 2 are missing an explanation of what \"HOV\" is. (I'm a native English speaker and I had to google that.)\n7. Fig. 6 could use a legend of what the different symbols mean. It's clear from the text but Fig. 6a would be easier to parse with one.\n\n**Significance:** \n\nAs outlined above, I do not believe this method is pushing the state-of-the-art.\n\n**Nitpicks and Questions:**\n\n1. The authors keep repeating the phrase \"under minor technical conditions\" 3 times in the paper and I have no idea what this is supposed to mean.\n\n**References:**\n\n- [1] Ross, Stéphane, Geoffrey Gordon, and Drew Bagnell. \"A reduction of imitation learning and structured prediction to no-regret online learning.\" Proceedings of the fourteenth international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 2011.\n- [2] Ross, Stephane, and J. Andrew Bagnell. \"Reinforcement and imitation learning via interactive no-regret learning.\" arXiv preprint arXiv:1406.5979 (2014).\n- [3] Kendall, Alex, et al. \"Learning to drive in a day.\" 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019.\n- [4] Kostrikov, Ilya, et al. \"Discriminator-actor-critic: Addressing sample inefficiency and reward bias in adversarial imitation learning.\" arXiv preprint arXiv:1809.02925 (2018).\n- [5] Sermanet, Pierre, et al. \"Time-contrastive networks: Self-supervised learning from video.\" 2018 IEEE international conference on robotics and automation (ICRA). IEEE, 2018.\n- [6] Berseth, Glen, and Christopher J. Pal. \"Visual Imitation Learning with Recurrent Siamese Networks.\" arXiv preprint arXiv:1901.07186 (2019).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did not properly explain the limitations of this work and neither the societal impact. Both were only mentioned off-hand in the conclusion.\nHere are some suggestions:\n- Instead of \"our approach could be beneficial for education applications\", you could write something more concrete like \"We believe that this method can be employed by MOOC platforms to gauge which order of courses would be most beneficial for the students...\"\n- As for limitations, I would recommend to put the contribution of this work into perspective to other contemporary works and add a few sentences a la \"we only experimented in a limited grid world setting, we only experimented with a low number of random seeds, we assume that we have unlimited access to a teacher,  etc\"", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "2pgwcChdATJ", "reviewer": "Reviewer_Ecwk", "summary": "This paper considers the problem of providing experts demonstrations\nto a learner, modeled as a Maximum Entropy IRL or a Cross Entropy\nBehavioral Cloning algorithm, in order for them to achieve a target\nperformance on some task. The key contribution of the paper is to\nprovide a demonstration ranking strategy which can be greedily\nutilized to define a curriculum. This strategy resulting in\nconvergence for both types of learners. Importantly, and in contrast\nto prior work, this approach can be done in a manner that does not\nexplicitly see the learner's internal dynamics.  Moreover, in the case\nof the maximum entropy IRL (with deterministic dynamics), the prove\nthat the convergence is logarithmic. Finally, they show the efficacy\nof this approach in various domains.\n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The considered problem of efficiently teaching common learning from\ndemonstration algorithms is certainty interesting and has a number\nof theoretical and practical applications. For example, there is a \nlarge body of literature of treating humans as maxEnt IRL learners\nand optimizing the informativeness of demonstrations, particularly\nin the context of human robot collaboration.\n\nThe key contribution of this paper, the ranking function, is fairly\nsimple to compute both directly given the learner's internal state\nas well as empirically using monte carlo methods. As such, this\nwork opens up several possible applications compared to prior work.\n\nAn interesting feature, although whether it is a positive or negative\nfeature is another question, is that the convergence guarantees are\nagnostic to the particular MDP being considered. One would naturally\nexpect that some MDPs are \"easier\" than others, but this is not\nreflected in the bounds. An MDP sensitive analysis while more\ndifficult, may give a much sharper characterization of the learning\ncurves observed in practice.\n\nMy primary concern with the paper was the simple setting of\ndeterministic dynamics for maxEnt considered (although weakened in the\nexperiments). While the convergence guarantees are fairly strong, its\nunclear to me if this should be expected to hold in more general\ndynamics or is a special feature of the deterministic setting.\n\nThat said, the empirical evaluation seems to suggest that the\ncurriculum really does make a difference compared to an agnostic\nteacher which is un-aware of the trace level difficultly.\n\n## Minor points\nUnless I'm misunderstanding, t, seems to be used in multiple ways,\nboth as the demonstration index and the time within an episode. \n\nShould the footnote on the bottom of page 3 point to section 4?\n\nThe technical details of eta would really be nice in the main draft.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This approach extends the theoretical understanding of teaching to existing algorithms and potentially enable new forms of human robot interaction. None of these are particularly concerning and I would argue this work is a net positive impact.\n\nRegarding the limitations, the primary limitation (acknowledged in the conclusion) is the sharpness of the convergence bounds for cross entropy and extending to more general dynamics. That said, I feel that it is perfectly reasonable to leave these as future work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is interested in defining an interactive curriculum for learning from demonstrations. Like in Weinshall et al., 2018, the idea is that the most useful demonstrations are the easiest ones for the teacher/oracle and the most difficult ones for the current learner (where the difficulty of a demonstration w.r.t. a policy is computed as its probability w.r.t. this policy). 
The lesson is backed by a theoretical analysis in the linear case, considering MaxEnt-IRL and CrossEnt-BC algorithms. 

Three approaches are experimentally compared: CUR (selecting the demonstration less difficult wrt the expert and most difficult wrt the current learner), CUR-L (where the difficulty wrt the expert is replaced with a heuristic measure) and CUR-T (that mostly considers the difficulty wrt the expert). The results suggest that the most important is to take into account the difficulty w.r.t. the current learner. 

The area chair did not take into account the most negative review (nb: this reviewer was eventually convinced by the good and thorough job done by the authors w.r.t. all reviewers, to explain the issues and to promise to revise their paper w.r.t. all points and suggestions raised by the reviewers). 

However, the area chair considered that the novelty of the paper w.r.t.  Kamalaruban et al.'s paper  (IJCAI 2019, cited) needed to be discussed:

* a main difference lies in  considering MaxEnt-IRL and CrossEnt-BC whereas Kamalaruban et al. considered MCE-IRL; 

* the structure of the proof (section 4.3 in Kamalaruban et al; Appendix C in the submitted paper) is similar ;

* the experimental setting (the car driving problem) is the same;

* this paper argues that it improves on Kamalaruban et al as it does not require the knowledge of the optimal policy, as it replaces the difficulty w.r.t. the expert policy with a heuristic function. The quality of this heuristic function however does not seem to be very important (point 3 in Response to Reviewer RaMd); unless I missed it, the authors did not comment further on it and did not consider any lesion of this, to study its impact.  

The significance of the contributions in the submitted paper w.r.t. Kamalaruban et al's (K, in the following) has been extensively discussed, concluding that 
* K presents an omniscient algorithm, aimed to yield the optimal $\theta^*$ policy parameter using the gradient $g_t$
* K also presents an approximation thereof, based on an estimation of $\theta_t$, though with no guarantees and specifically based on the MaxEnt-IRL / MCE-IRL freamework.


Overall, the paper is considering a hot topic and presenting interesting results; the many additional complements (explanations, discussion and results) in the answers to the reviews let us hope that the revised version will be way better than the submitted one. 
It is mandatory that in the final version, the authors clarify the novelty issue, and  compare their results with the Blackbox teaching baseline (section 5 in K).

Decision: Accept (Poster)