TaSIL: Taylor Series Imitation Learning
Authors: Daniel Pfrommer
Thomas TCK Zhang
Stephen Tu
Nikolai Matni
Venue: NeurIPS 2022 Accept (2022)
Keywords: learning theory
control theory
nonlinear systems
imitation learning

Abstract: We propose Taylor Series Imitation Learning (TaSIL), a simple augmentation to standard behavior cloning losses in the context of continuous control. TaSIL penalizes deviations in the higher-order Tayler series terms between the learned and expert policies. We show that experts satisfying a notion of incremental input-to-state stability are easy to learn, in the sense that that a small TaSIL-augmented imitation loss over expert trajectories guarantees a small imitation loss over trajectories generated by the learned policy.  We provide sample-complexity bounds for TaSIL that scale as $\tilde{\mathcal{O}}(1/n)$ in the realizable setting, for $n$ the number of expert demonstrations. Finally, we demonstrate experimentally the relationship between the robustness of the expert policy and the order of Taylor expansion required in TaSIL, and compare standard Behavior Cloning, DART, and DAgger with TaSIL-loss-augmented variants.  In all cases, we show significant improvement over baselines across a variety of MuJoCo tasks.

TaSIL: Taylor Series Imitation Learning
Daniel Pfrommer" Massachusetts Institute of Technology! Cambridge, MA dpfrom@mit.com edu
Thomas T.C.K. Zhang' University of Pennsylvania Philadelphia, PA ttz2@seas upenn. edu
Stephen Tu
Nikolai Matni
Robotics at Google New York, NY
University of Pennsylvania
Philadelphia, PA nmatn @seas. upenn edu aereaas.ueenn..
stephentufgoogle.comm
Abstract
We propose Taylor Series Imitation Learning (TaSIL), simple augmentation standard behavior cloning losses in the context of continuous control. TaSIL pe- nalizes deviations in the higher-order Taylor series terms between the learned and expert policies. We show that experts satisfying notion of incremental input-to- state stability easy learn, in the sense that a small TaglL-augmented imitation loss over expert trajectories guarantees a small imitation loss over trajectories gen- erated by the learned policy. We provide generalization bounds for TaSIL that scale (1//n) in the realizable setting, for the number of expert demonstrations. Finally, we demonstrate experimentally the relationship between the robustness of the expert policy and the order of Taylor expansion required in TaSIL, and com- pare standard Behavior Cloning, DART, and DAgger with MASL--loss-augmnnn variants. In all cases, we show significant improvement over baselines across a variety of MuJoCo tasks.
Introduction
Imitation learning (IL), wherein expert demonstrations are used to train a policy [1, 2), has been successfully applied to wide range of tasks, including self-driving cars [3,4], robotics [5], and video game playing [6]. While IL is typically more sample-efficient than reinforcement learning-based alternatives, is also known be sensitive to distribution shift: small errors the learned policy can lead to compounding errors and ultimately, system failure [3, 6]. In order to mitigate the effects of distribution shift caused by policy error, two broad approaches have been taken by the community. On-policy approaches such as DAgger [6] augment the data-set with expert-labeled or corrected trajectories generated by learned policies. contrast, off off-polliy approaches such as DART [7] and GAIL [8] augment the data-set by perturbing the expert controlled system. In both cases, the goal is to provide examples to the learned policy of how the expert recovers from errors. While effective, these methods either require interactive expert (on-policy), or access to simulator for policy rollouts during training (off-policy), which may not always be practically feasible.
In this work, we take a more direct approach towards mitigating distribution shift. Rather than providing examples of how the expert policy recovers from error, seek to endow a learned policy with the robustness properties of the expert directly. In particular, we make explicit the underlying assumption in previous work that a good expert is able recover from perturbations through
Both authors contributed equally this work.
I This work was done while affiliated with University of Pennsylvania.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

the notion of incremental input-to-state stability, well-studied control theoretic notion of robust nonlinear stability. Under this assumption, we show that if the p-th order Taylor series approximation of the learned and expert policies approximately match on expert-generatee learned policy induces closed-loop behavior similar to that of the expert closed-loop system. Here the order p determined by the robustness properties of the expert, and makes quantitative the informal observation that more robust experts are easier to learn. Fundamentally, we seek to characterize settings where offline imitation learning is Probably Approximately Correctly (PAC)-learabble [9]; that is, defining notions of expert data and designing algorithms such that low training time error on offline expert demonstrations implies low test time error along the learned policy's 's trajectories. in spite of distribution shift.
trajectories, then the
Contributions We propose and analyze Taylor Series Imitation Learning (TaSII.), which augments the standard behavior cloning imitation loss to capture errors between the higher-order terms of the Taylor series expansion of the learned and expert policies. We reduce the analysis of TaSIL to the analysis of supervised learning problem over expert data: in particular, we identify a robustness criterion for the expert such that a small FaSIL-augmented imitation loss over expert trajectories guarantees that the difference between trajectories generated by the learned and expert policies is also small. We also provide a finite-difference based approximation that is applicable to experts that cannot directly query their higher-order derivatives, expanding the practical applicability of TaSIL. We show in the realizable setting that our algorithm achieves generalization bounds scaling as Õ (n), for the number of expert demonstrations. Finally, we empirically demonstrate (i) that the relationship between the robustness of the expert policy and the order of Taylor expansion required in TaSIL predicted by our theory is observed in practice, and (ii) the benefits of using the YaSlL-augmented imitation loss by comparing the sample-efficiency of standard and TaSIL-loss-augmentee behavior cloning, DART, and DAgger on a variety of MuJoCo tasks: on hard instances where behavior cloning fails, the TaSIL-augmented variants show significant performance gains.
1.1 Related work
Imitation learning Behavior cloning is known to be sensitive to compounding errors induced by small mismatches between the learned and expert policies [3, 6]. On-policy [6] and off-policy [7, 8] approaches exist that seek to prevent this distribution shift by augmenting the data-set created by the expert. While DAgger is known enjoy Ô(T) sample-complexity in the task horizon T for loss functions that are strongly convex in the policy parameters, we are not aware of finite-data guarantees for DART GAIL. In addition these seminal papers, there is a body of work that seeks to leverage control theoretic techniques [10, 11] to ensure (robust) stability of the learned policy, More closely related to our work are the results by Ren al. [12] and Tu et al. [13]. In Ren al. [12], two-stage pipeline of imitation learning followed by policy optimization via PAC-Bayes regularization term is used to provide generalization bounds of the learned policy across randomly drawn environments. This work is mostly complementary to ours, as TaSIL could in principle be used to augment the imitation losses used the first stage of their pipeline (leaving the second stage unmodified).
In Tu et al. [13], sample-complexity bounds for II. are provided under the assumption that the learned policy can be explicitly constrained to match the incremental stability properties of the expert. While conceptually appealing, practically enforcing such stability constraints is difficult, and as such Tu et al. [13] resort to heuristics in their implementation. In contrast, we provide sample-complexity guarantees for a practically implementable algorithm, and under much milder stability assumptions.
Robust stability and learning for continuous control There is a rich body of work applying Lyapunov stability or contraction theory [14] to learning for continuous control. For example [15-18] use stability-based regularizers to trim the hypothesis space, and empirically show that this leads to more sample-efficient and robust learning methods. Lyapunov stability and contraction theory have also been used to provide finite sample-complexity guarantees for adaptive nonlinear control [19] and learning stability certificates [20].
'This bound degrades to Ö(1²) when loss function is only convex, and does not hold for the nonconvex loss functions consider.
2

---

2 Problem formulation
We consider the nonlinear discrete- discrete-time dynamical system
T++1 fzt, uz), xo = E.
where xt E Rd is the system state, Ur E R. the control input, and f IR R - Rd defines the system dynamics. We study system (1) evolving under the control input u, ( + Ar, for Rd - R a suitable control policy, and A E R an additive input perturbation (that will be used to capture policy errors). We define the corresponding perturbed closed-loop dynamics by Ft-1 = fa (ar, A+) == f(x x (x) + At), and use x (E, { {==) denote the value of the state dr at time & evolving under control input = π(x) + A starting from initial condition To = Ç. To lighten notation, we use x² (E) to denote x; (c. (O)=-)), and overload ||-|| to denote the Euclidean norm for vectors and the operator norm for matrices and tensors.
Initial conditions are assumed to be sampled randomly from a distribution D with support restricted to compact set X' We assume access to rollouts of length T from an expert T., generated by drawing initial conditions {E:]== i..dd from D. The IL task is to learn a policy which leads to a closed-loop system with similar behavior to that induced by the expert policy æ,, measured by the expected imitation gap Eg maxigig7
||a (E) x; (6)lI-
The baseline approach to II, typically referred to as behavior cloning (BC), casts the problem as an instance of supervised learning. Denote the discrepancy between an evaluation policy it and the expert policy TT a trajectory generated by rollout policy va starting at initial condition by A (E; +) ;= (x)""(E) -n- n. (27 Td (S)). BC directly solves the supervised empirical risk minimization (ERM) problem
1 be E argmiin - Σ (S:
#Ell
over a suitable policy class IT. Here, h: (RT")T - IR is a loss function which encourages the discrepancy terms - (E,; π) be small along the expert trajectories. While behavior cloning is conceptually simple, it can perform poorly in practice due to distribution shifts triggered by errors in the learned policy foc. Specifically, due to the effects of compounding errors, the closed-loop system induced by the behavior cloning policy Toe may lead to dramatically different distribution over system trajectories than that induced by the expert policy π., even when the population risk on the expert data, Egh( (AT (E; =T') ), is small.
As described in the introduction, existing approaches to mitigating distribution shift seek to augment the data with examples of the expert recovering from errors. These approaches either require interactive oracle (e.g.. DAgger) or access to a simulator for policy rollouts (e.g., DART, GAIL), and may not always be practically applicable. To address the distribution shift challenge without resorting to data-augmeetttion, we propose TaSIL,, an off-policy IL algorithm which provably leads to learned policies that are robust to distribution shift.
The rest of the paper is organized as follows: in Section 3, we focus on ensuring robustness to policy errors for single initial condition ş. We show that the imitation gap ||~ (5) - -x" (S)I] between a test policy π and the expert policy т. can be controlled by the TaSIL-augmented imitation loss evaluated on the expert trajectory {2, (E)), effectively reducing the analysis of the imitation gap to a supervised learning problem over the expert data. In Section 4, we integrate these results with tools from statistical learning theory to show that Z -88 trajectories are sufficient to achieve imitation gap of at most e with probability at least - 6: here r > 0 is a constant determined by the stability properties of the expert policy т., with more robust experts corresponding to smaller values of Finally, in Section 5, we validate our analysis empirically, and show that using the TaSIL-augmented loss function in IL algorithms leads to significant gains in performance and sample efficiency.
3 Bounding the imitation gap on a single trajectory
In this section, we fix an initial condition & and test policy v, and seek control the imita- tion gap TT(E:=) π) == max (g,
(g, (0)) xz (E, (0))]. A natural way to compare the
3

---

closed-loop behavior of test policy ι that of the expert policy म. is to view the discrep- ancy A7 =f; == ग (2. (E)) (x. (()) as an input perturbation to the expert closed-loop sys- (I,, 0) = fa (æt, (f(TT)), the imitation gap can be written as (5: = max, ||=; (c. ( (E: +)) x," (c. [O})]l suggesting that closed-loop expert sys- tems that are robust to imput perturbations, as measured by the difference between nominal and perturbed trajectories, will lead to learned policies that enjoy smaller imitation gaps.
tem. By writing 21+1
Stability conditions defined in terms of differences between nominal and perturbed trajectories have been extensively studied in robust nonlinear control theory via the notion of incremental input-to- state stability (Ó-ISS) (see c.g., Angeli [21] and references therein). Before proceeding, we recall definitions of standard comparison functions [22]: a function (xx) is class K. if it is continuous, strictly increasing, and satisfies 70) = O, and a function (I, t) is class K.C. if is continuous, 3(-,t) class K. for each fixed t, and ((x,) .) is decreasing for each fixed z.
Definition 3.1 (6-ISS system). Consider the closed-loop system evolving under policy æ and subject to perturbations De given by I++1 = fā (It, 41). The closed-looy system fä (xr, At) is incremental input-to-state stable (8-ISS) if there exists class K.C function B and class IC function y such that for all initial conditions Eı, E2 E X, perturbation sequences [Ae3120 and I E N:
[oT (A.))b) - x7 (S2; (0)==0) S B(1EE - -211, +7
max OSASI-1 lAull)
(2)
Definition 3.1 says that: (i) trajectories generated by 8-1SS systems converge towards each other if they begin from different initial conditions, and (ii) the effect of bounded perturbations {Ar} on trajectories is bounded. Our results will only require the stability conditions of Definition 3.1 to hold for a class of norm bounded perturbations. In light of this, we say that a system is 1--locally 8-ISS if equation (2) holds for all input perturbations satisfying supten || S 77.
By writing F++1 = 1y-locally 8-ISS, and that suPtEn | (E: # )| S 17, then equation (2) yields
13 (Ix, 0) = fā (aa, A7 (E; T)), we conclude that if 2++1 = fa (xt,, A.) is
(xT(E) - (E-(1)) S>
max OSAST-1
| (E:
(T(::)) (r(s:=))) S = - -
[|AT(:)))I (3) 0SIST
max
Equation (3) shows that the imitation gap x77))==((5)) is controlled by the maximum discrep- incurred on trajectories generated by the policy π. A natural way of bounding the test discrepancy || ({;=))i =) defined over trajectories generated by æ, by the training discrepancy ||A7" (5: Il.. defined over trajectories generated by т., is to write out a Taylor series expansion of the former around the latter, i.e., to write:
ancy || [E) m)
(E:
1 (E: m| + 11, (Ei ((E +0
(1 (E,)).
(4)
where 0, (5: π)) denotes the partial derivative of the discrepancy with respect to the argument of the policies x and k.. The challenge however, is that the imitation gap ('r(E: T) also appears in the Taylor series expansion (4), leading to an implicit constraint. We show that this can be overcome if the order p of the Taylor series expansion (4) is sufficiently large, as determined by the decay rate of the class IC function --) defining the robustness of the expert policy. We begin by identifying condition reminiscent of adversarially robust training objectives that ensures a small imitation gap.
Proposition 3.1. Let the expert closed-lop system fä be nolloallly 5-ISS for some 17 > O. Fix an imitation gap bound e > 0, initial condition E, and policy Then if
max 0SSST-1
sup (1,x + - -x) (E) + 8)ll S min(n).7
(5)
16l|.e
we have that the imitation gap satisfies (r((E:)) S.
Proposition 3.1 states that if a policy π is sufficiently close to the expert policy т. in tube around the expert trajectory, then the imitation gap remains small. How to ensure that inequality (5) holds using only offline data from expert trajectories is not immediately obvious. The Taylor series expansion (4) suggests that a natural approach to satisfying this condition is to match derivatives of the test policy æ to those of the expert policy n.. We show next that sufficient order for such a Taylor series
'We only take first order expansion here for illustrative purposes.

---

expansion is naturally determined by the decay rate of the class K. function (xx) towards O. Less robust experts have functions that decay to O more slowly and will lead to stricter sufficient conditions. Conversely, more robust experts have functions that decay to 0 more quickly, and will lead more relaxed sufficient conditions. We focus on two disjoint classes of class K. functions: (i) functions that decay in their argument faster than a linear function, i.e. (ax) < ((x) as x + 0f, and (ii) functions that decay in their argument no faster than a linear function, i.e. ((x) 2 a((z)) as x + 0+.
Rapidly decaying class IC functions
We show that when the class K. function (ar) decays to 0 faster than (xx) in some neighborhood of 0, then matching the zeroth-order difference max, A (E: T)ll on the expert trajectory, as is done in vanilla behavior cloning, suffices to close the imitation gap. We make the following assumption the test policy π and expert policy π..
A ssumption 3.1. There exists non-negative constant Lg such that [T - =(v)ll2 S Lx || ull2 for all x, y, and π. E {T,
Proposition 3.1 then leads to the following guarantee on the imitation gap. Theorem 3.1. Fix test policy π and initial condition & X, and let Assumption 3.1 hold. Let fal be r-locally 6-ISS for some 1) > O, and assume that the class K. function 7()) in (2) satisfies 7(x) & Oxx!+T) () for some r > O. Choose constants p., a > such that
2L (x/μ) - (x) for all S x S a.
(6)
Provided that the imitation error on the expert trajectory incurred by safisfies:
max 1 ||A (5: (I= Sa 0SKST-
max 2La || (E: ([]= + ( (E: S n.
(7)
0S15T-1
then for all 1 1sts the instantaneous imitation gap is bounded as || (E) - 27 (E) S
mmxx 0CkSt-1
/1 || (E:T))I**r,
(8)
Theorem 3.1 shows that if policy π is sufficiently good approximation of the expert policy
on an expert trajectory {a," (5), then the imitation gap |x,"" (5) -x)= can be upper bounded in terms of the discrepancy term maxos* 1 (E: n ml evaluated on the experl Irajectory. To
help illustrate the effect of the decay parameter r on the choices of Jı and a, we make condition (6) [0, 1]. Then one can choose μ 2 and a (() This makes clear that a larger i.e. a more robust expert, leads to less restrictive conditions (7) on the policy т and a tighter upper bound on the imitation gap (8) (assuming maxogA<i-1 || A (E: T] < 1). In particular, for such systems, vanilla behavior cloning is sufficient to ensure bounded imitation gap. This also makes clear how the result breaks down when r = 0, i.e., when the decay rate is nearly linear, as the neighborhood can become arbitrarily small, such that it may be impossible to learn a policy π that satisfies the bound (7) with a practical number of samples n. The interplay of the bounds (7) in Theorem 3.1 (and the subsequent theorems of its like) and the sample-complexity of imitation learning will be discussed in further detail in Section 4.
more explicit by assuming that 7(x) I Cx for all
Slowly decaying class IC functions
When the class IC function ((x) decays to 0 slowly, for reasons discussed above, controlling the zeroth-order difference (5: .T may not be sufficient to bound the imitation gap. In particular, we consider class IC functions satisfying (xx) S (x///r for some r 2 1. Setting p = [r]. we now show that matching up to the p-th total derivative of T, is sufficient to control the imitation gap. Analogously to Assumption 3.1, we make the following regularity assumption on the test policy π and expert policy ...
Assumption 3.2. For given non-negative p E N, assume that the test policy π and expert policy n are p-times continuously differentiable, and there exists constant Larz 2 0 such that
T
( - (J () <
||
roll P1
(p
+ 1)!
for all x, To and t E (,,TTa), Taylor polynomial of के evaluated at zo. and & denotes the tensor product.
TT), where (Jz, $) (x) Ji
1 (
( (sr - zo i is the p-th order
5

---

With this assumption in hand, provide the following guarantee the imitation gap.
Theorem 3.2. Let fa be r-localty 8-1SS for some 1) > O, and assume that the class K. function 7(-) in (2) satisfies ((x) & (0(2lr))fol for some 2 1. Fix test policy π and initial condition E X, and let Assumption 3.2 hold for p E I4 satisfying p +
O. Choose μ, a > such that
Lapa (p 1)!
1
+ (a/N)) SY (x), for all 0 S S
(10)
Provided the jth total derivatives, j = o.....P. of the imitation error on the expert trajectory incurred by x satisfy:
2 110147 (4:+)1 1/r sa,
mAX max 0SIST-10SISP
(11)
2Lar
2 + [| (E: s..
max max 0S157-1051 10SiSP
(12)
(p + 1)!
then for all S T the instantaneous imitation gap is bounded by
( (I0IAT 1/7.
(1) - =7(6
max max 1 Ockgr-IUSSSI
(13)
Theorem 3.2 shows that if the p-th order Taylor series of the policy x approximately matches that of the expert policy T, when evaluated on expert trajectory {x. (5)), then the imitation gap || (E) - x7 (E) can be upper bounded in terms of the derivatives of the discrepancy term, i.c., by maxos&çt-1 maxusisp |ag -. (f:ar) T)|l- evaluated on the expert trajectory. To help illustrate the effect of the choice of the order on the constants and a, we make condition (10) more explicit by assuming that (x) S Cr for all x E [0. 1]. Then one can choose μ 21/rC and = (La) C-CCC"//--1--_-1 -1(++--r) This expression highlights tradeoff: by picking larger order p. the right hand side a of bound (11) increases, but at the expense of having to match higher-order derivatives. This also highlights that both the order p and closeness required by Equation (11) get increasingly restrictive as r increases, matching our intuition that less robust experts lead to harder imitation learning problems.
Using estimated derivatives We show in Appendix C that the results of Theorems 3.1 and 3.2 extend gracefully to when only approximate derivatives He (z) be obtained, e.g., through diffe-difference methods. In particular, if (|02 (x) a n, (a) S e, then suffices appropriately tighten the constraints (11) and (12) by (st//r) and O()), respectively. Please refer to Appendix C for more details.
4 Algorithms and generalization bounds for TaSIL
The analysis of Section 3 focused on a single test policy æ and initial condition g. Theorems 3.1 and 3.2 motivate defining the p-TaSIL loss function:
(E: = p
max
(14)
p+1
1= 0StST
The corresponding policy daslLp is the solution to the empirical risk minimization (ERM) problem:
1 VI
TTS E aargmin TEN
t (EG
(15)
which explicitly seeks to learn a policy E II, for II suitable policy class, that matches the p-th order Taylor series expansion of the expert policy. 3 In this section, we analyze the generalization and sample-complexity properties of the p- p-TaSII. ERM problem (15).
Although we focus on the supremum loss maxosesT-1 | (5: (G:m)] in our analysis, we note that any surrogate loss that upper bounds the supremum loss, e.g.. Ei= ||aA (E: m)]l can be used.

---

Our analysis in this section focuses on the realizable setting: we assume that for every dataset of expert trajectories { {(=," (E:)) Ji=i- Ji=l, there exists a policy π II that achieves (near) zero empirical risk. This is true if, for example, R, E II. In this setting, we demonstrate that we can attain generalization bounds that decay as O(n where (-) hides poly-log dependencies on n. These rates are referred to as fast rates in statistical learning, since they decay faster than the n n-1/2 rate prescribed by the central limit theorem. We present analysis for the non-realizable setting in Appendix B: this analysis is standard and yields generalization bounds scaling as O(n
Let C [0, 1 be a set of functions mapping some domain X' to [0, 1]. Let be a distribution with support restricted to X' and denote the mean of 9 with respect to 2 ~ 'D by Ex [g). Similarly, fixing data points ,1....,Iy, In E X. we denote the empirical mean of g by [n[g] := n Σ g(x)). We focus our analysis on the following class of parametric Lipschitz function classes.
Definition 4.1 (Lipschitz parametric function class). parametric function class c [0. 1|x is called (Ba, Lg. 9) 9 if g = (ge(-) A θ} with θ C R2, and it satisfies the following boundedness and uniform Lipschitz conditions:
(g) (0,(z)]
sup ||0|| S Be, sup sup
S Lg.
(16)
x 02 02E0,011
100 021l
We assume without loss of generality that Lg 2 1.
The description (16) is very general, and as we show next, is compatible with feed-forward neural networks with differentiable activation functions. We then have the following generalization bound, which adapts [23, Corollary 3.7] to Lipschitz parametric function classes using the machinery of local Rademacher complexities [23]. Alternatively, the result can also be derived from [24, Theorem 3].
Theorem 4.1. Let G C [0. 1 be (Bo, La, - Lipichhit parametric function class. There exists universal positive constant K < 10 such that the following holds. Given ó E (0. 1), with probability at least 8 over the i...d. draws I1, , Tn ~ DD for all 9 E G. the following bound holds:
q (glog( Bg Lgn) + log(1/8)
Bylg < [g KK
(17)
We now use Theorem 4.1 to analyze the generalization properties of the p-TSSIL ERM problem (15). In what follows, assume that the expert-closed loop system is stable in the sense of Lyapunov, ie., that there exists Bx > 0 such that suPtEN, (E)Il I Bx, and consider the following parametric class of p + continuously differentiable policies:
To.p == {z 0) 0 E R2, |10|| < Bo: (0, 0) = 0 V0, a is p + continuously differentiable).
(18)
Define the constants
sup B₁ o 111.ix110
ml 0]] ly== =
sup
|| (z),,)))))
| Px.10
for j = O, ... P. and note that they are guaranteed to be finite under our regularity assumptions. Finally, define the loss function class:
6 o Io == (4p (::)) defined in (14) π E I.
(19)
(6T. o lls is (Bg, B p Le,p. ( Lipschitt parametric function class for Bt.p _ E)= B, and Lt,p Bx Ey= Ly. We now combine this with Theorem 4.1 to bound the population risk achieved by the p+l
From a repeated application of Taylor's theorem, we show in Lemma B.2 that 1
solution to the TaSII. ERM problem (15).
Corollary 4.1. Let the policy class Is. be defined as in (18), and assume that T. E II. Let the function class ET. o Ie be defined as in (19), and constants Bt.,, Le be defined as above. Let T be any empirical risk minimizer (15). Then with probability at least over the initial conditions { Lid. D
Ee I (5: [ S Bt) q log BB - Legn) + log(1/6)
(20)
TThis without loss of generality for [o, B. -boudded functions by considering the normalized function class B- (B-'g 9 g E 9) c 10
7

---

We note that since these generalization bounds solely concern the supervised learning problem of matching the expert on the expert trajectories, the constants do not depend on the stability properties of the trajectories generated by the learned policy TTaSIL.p- To convert the generalization bound (20) to probabilistic bound on the imitation gap (5; p) we first apply Markov's 's inequality to bound the probability that the conditions of Theorem 3.1 3.2 hold by the expected TaSIL loss (20), and then apply Corollary 4. together with Markov's inequality.
Theorem 4.2 (Rapidly decaying class K functions). Assume that nI E 1lg.0 and let the assumptions of Theorem 3.1 hold for all æ E Io Let Equation (6) hold with constants μu, a O, and assume without loss of generality that o μ S 1, Lan 2 1/2. Let TTSLLL be an empirical risk minimizer of Ltd.
fo over the policy class o initial conditions {Er} ~ and assume that
DD Fix a failure probability E (0, 1),
n 2 O(1) max Be, ofr log So Bybeo)
BB,0 fi log
Kig BogBobr
}
where Ki == q( (L//a))(//)) 1/(+7) Kn := ql n. Then with probability at least 1 6, the imitation gap evaluated on ~ D (drawn independently from (E.)==)) satisfies
1tr
Bro09 Beoglog BB Le,on) Le,0n)
(r (TSILL,) S O(1) μ
Theorem 4.3 (Slowly decaying class K. functions), Assume that n. E le. and let the assumptions of Theorem 3.2 hold for all 1p Let Equation (10) hold with constants a.a >> O. Let TTSSI.. iid. D".
be an empirical risk minimizer of E over the policy class Hl for initial conditions {E.} Fix a failure probability d (0, and assume
KyJ log Kaj BBB BBBTIBxLi ) B,
Kn, BaB BBBBBBLLjj
B Ka.j log
2 O(1) max max JSP
8
8
8
where Ka,j (H) 2 and Kn.
4++ (p+1] G1)0+ny7 + 1) 23. Then with probability at least 1 8,
Lap
J!
the imitation gap evaluated on & ~ D (drawn independently from {5:)i-) satisfies
1/r
Byg log BB BxL,m) JSP j18
PE: S 0(1) j max
n
In the rapidly decaying setting, corresponding to more robust experts, Theorem 4.2 states that e- - /5 expert trajectories sufficient ensure that the imitation gap (r: TasI.L.) with probability at least to smaller cample-complexity bounds. In contrast, to achieve the same guarantees the imitation gap in the slowly decaying setting, Theorem 4.3 states n Z E =-/88 expert trajectories are sufficient, where we recall that less robust experts have larger values of 1. These theorems quantitatively show how the robustness of an underlying expert affects the sample-complextty of II., with more robust experts enjoying better dependence on than less robust experts. We note that analogous dependencies on expert stability are reflected in the burn-in requirements, i.c., the number of expert trajectories required to ensure with high probability no catastrophic distribution shift occurs, of each theorem.
Recall that more robust experts have larger values of r > 0, leading
5 Experiments
We compare three standard imitation learning algorithms, Behavior Cloning, DAgger, and DART, to TaSIL--agmented loss versions. In TaSIL- algorithms, we replace the standard imitation loss function
C (t&.)imii Σn
(E::))
with the p-TaSII- loss
4i (L ((E.)) ==
1 IT (0
8

---

sample efficient than T&STL order
TaSIL order
Mean Policy Discrepancy
0.6
TaSIL order 1
TaSIL order
TaSIL order 2
Imitation Gap
3
TaSIL order
0.4
2-
0.2
0.5
1.0
1.5
2.0
2.5
3.0
0.5
1.0
1,5
2.0
2.5
3.0
V
Figure 1: Left: The average Euclidean norm difference between the expert and learned policies trajectories rolled out under the learned policy. Right: The maximum deviation between expert and learned policy trajectory starting from identical initial conditions. statistics are averaged across test trajectories and we plot the mean and standard deviation for random seeds.
where {Ay 1zo are positive tunable regularization parameters. We use the Euclidean nomm of the vectorized error the derivative tensors as more optimizer-amenaal surrogate the operator norm.
We experimentally demonstrate (i) the effect the expert stability properties and order of the TaSIL loss, and (ii) that TaSII--loss-based imitation learning algorithms their standard counterparts. All experiments' are carried out using Jax [25] GPU acceleration and automatic differentiation capabilities compute the higher-order derivatives, and the Flax [26] neural network and Optax [27] optimization toolkits.
Stability Experiments To illustrate the effect of the expert closed-loop system stability sample- complexity, we consider a simple 5-ISS stable dynamical system with a tunable input-to-state gain function. For state and input Te, ut E 10 the dynamics are:
#++1 TH4 + (1 7(1((())) + vell) (I((2) + u.). [1(())) uell
The perturbation function h R.0 R is set randomly initialized MLP with two hidden layers of width 32 and GELU [28] activations such that the expert т, (I) = =h((x) yield closed loop system f (ar, A) which is 6-ISS stable with the specified class I. function 7 (see Appendix D).
We use 0.95 for all experiments presented here.
We investigate the performance of p- p-TSSIL loss functions for 8-ISS system with different class K. stability. We sweep X. functions y(x) = Cx" for v E [0.05, 3). C == and p-TaSIL loss functions for p E {0, 2, (additional details be found in Appendix D). The results of this sweep are shown in Figure Higher-order p- TaSIL losses significantly reduce both the imitation gap and the mean policy discrepancy test trajectories. Notably, the first and second order TaSIL loss maintain their improved performance for slower decaying class IC functions. Theorem 3.1 and Theorem 3.2 yield lower bounds of 1, v 2-1, and 3-1 for closing the imitation gap using the O-TaSIL, I-TaSIL, and 2-TaSIL losses respectively. Figure demonstrates significant performance degradation in policy discrepancy and decaying imitation gap starting around these threshold values.
MuJoCo Experiments We evaluate the ability the TaSIL loss to improve performance on standard imitation learning tasks by modifying Behavior Cloning, DAgger [6], and DART [7) to use &TSIIL, loss and testing them in simulation on different OpenAl Gym MuJoCo tasks [29]. The MuJoCo environments we use and their corresponding (state, input) dimensions are: Walker2d-v3 (17, 6), HalfCheetah-v3 (17, 6), Humamoid-V v3 (376, 17). and Ant-v3 (111, 8).
For all enviromments we use pretrained expert policies obtained using Soft Actor Critic reinforcement learning by the Stable-Band Baselines3 [30] project. The experts consist of Multi-Layer Perceptrons with two hidden layers of 256 units each and ReLUU activations. For all environments, learned policies have 2 hidden layers with 512 units each and GELU activations in addition to Batch Normalization. The final policy output for both the expert and learned policy are rescaled to the valid action space
The code used for these experiments be found https: //:g:://ttttt comm/unstable-zree

---

Swimmeer
Walker2d
HallCheetthh
Humanoid
RC
0.25
BC.uit 1.VL
Normalized Reward
0.00
1.00
DART
DART
BARI with 1-tiett
0.00
1.00
DAgger
0.75
0.50
15
45
15
45
13
30
30
Training Trajectories
Figure 2: Cumulative
rewards as a function of trajectory budget for policies trained using different algorithms with and without 1-TaSIL loss.
after applying tanh nonlinearity. We used trajectories of length T 300 for all experiments. We refer to Appendix E for additional experiment details.
In Figure 2 we report the mean expert-normalized rewards across seeds for all algorithms and environments as a function of the trajectory budget provided. Algorithms with 1-TaSIL loss showed significant improvement in sample-complexity across all challenging environments. The expert for the Swimmer environment very robust due to the simplicity of the task, and so as predicted by Theorem 3.1 and 4.2, all algorithms (with the exception of the vanilla DAgger algorithm due to initially selecting poor rollout trajectories) are able to achieve near expert performance across all trajectory budgets. We also able to nearly match exceed the performance of standard on-policy methods DAgger and DART with our off-policy 1-TaSIL loss Behavior Cloning in all environments. Additional experimental results be found in the appendix. These include representative videos of the behavior achieved by the expert, BC, and TaSlL--agmmettd BC policies in the supplementary material, where again, striking improvement is observed, especially in low-data regimes for harder environments, well as systematic study of Linit-difffrrenc--asse approximations of 1-TaSIL which achieve comparable performance to aeoobia--basee implementations.
Conclusion
We presented Taylor Series Imitation Learning (TaSIL), a simple augmentation to behavior cloning that penalizes deviations the higher-order Tayler series terms between the learned and expert policies. We showed that 6-ISS experts are easier to learn, both in terms of the loss-function that needs to be optimized and sample-complexity guarantees. Finally, we showed the benefit of using TaSIL-augmented losses in BC, DAgger, and DART across variety of MuJoCo tasks. This work opens up many exciting future directions, including extending TaSIL to pixel-based IL. and to (offline/inverse) reinforcement leamning settings.
Acknowledgements
We thank Vikas Sindhwani, Sumeet Singh, Jean-Jacques Slotine, Jake Varley, and Fengjun Yang for helpful feedback. Nikolai Matni is supported by NSF awards CPS-2038873, CAREER award and Google Research Scholar award.
10

---

References
[1] Ahmed Hussein, Mohamed M. Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1-35, 2017.
[2] Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J. Andrew Bagnell, Pieter Abbeel, and Jan Peters. An algorithmic perspective on imitation learning. Foundations and Trends@ in Robotics, 7(1 (11-2):1-179, 2018.
[3] Dean A. Pomerleau. Alvinn: An autonomous land vehicle in neural network. In Advances in Neural Information Processing Systems, volume 1, 1988.
[4] Felipe Codevilla, Matthias Miiller, Antonio López, Vladlen Koltun, and Alexey Dosovitskiy. End-to-end driving via conditional imitation learning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 4693-4700, 2018.
[5] Stefan Schaal. Is imitation learning the route to humanoid robots? Trends in cognitive sciences, 3(6):233-242, 1999.
[6] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. reduction of imitation learning and structured prediction no-regret online learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15, pages 627-635. PMLR, 2011.
[7] Michael Laskey, Jonathan Lee, Roy Fox, Anca Dragan, and Ken Goldberg, Dart: Noise injection for robust imitation learning. In Proceedings of the 1st Annual Conference on Robot Learning. volume 78, pages 143-156. PMLR, 2017.
[8] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, volume 29, 2016.
[9] Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory algorithms. Cambridge university press, 2014.
[10] Michael Hertneck, Johannes Köhler, Sebastian Trimpe, and Frank Allgöwer. Learning approximate model predictive controller with guarantees. IEEE Control Systems Letters, 2(3): 543 148,548,20 2018.
[11] He Yin, Peter Seiler, Ming Jin, and Murat Arcak. Imitation learning with stability and safety guarantees. IEEE Control Systems Letters, 6:409-414, 2022.
[12] Allen Ren, Sushant Veer, and Anirudha Majumdar. Generalization guarantees for imitation learning. In Proceedings of the 2020 Conference on Robot Learning, volume 155, pages 1426-1442. PMLR, 2021.
[13] Stephen Tu, Alexander Robey, Tingnan Zhang, and Nikolai Matni. On the sample complexity of stability constrained imitation learning. arXiv preprint arXiv:2102.09161 2021.
[14] Winfried Lohmiller and Jean-Jacques E. Slotine. On contraction analysis for non-linear systems. Automatica, 34(6):683-666, 1998.
[15) Sumeet Singh, Spencer M. Richards, Vikas Sindhwani, Jean-Jacques E, Slotine, and Marco Pavone. Learning stabilizable nonlinear dynamics with contraction- hased regularization. The International Journal of Robotics Research, 40: 123-1150, 2020.
[16] Andre Lemme, Klaus Neumann, R. Felix Reinhart, and Jochen J. Steil. Neural learning of vector fields for encoding stable dynamical systems. Neurocomputing, 141:3-14, 2014.
[17] Harish Ravichandar, Iman Salchi, and Ashwin Dani. Learning partially contracting dynamical systems from demonstrations. In Proceedings of the 1st Annual Conference on Robot Learning, volume 78, pages 369-378. PMLR, 2017.
[18] Vikas Sindhwani, Stephen Tu, and Mohi Khansari. Learning contracting vector fields for stable imitation learning. arXiv preprint arXiv: 04878, 2018.

---

[19] Nicholas M. Boffi, Stephen Tu, and Jean-Jacques E. Slotine. Regret bounds for adaptive nonlinear control. In Proceedings of the 3rd Conference on Learning for Dynamics and Control, volume 144, pages 471-483, PMLR, 2021.
[20] Nicholas M. Boffi, Stephen Tu, Nikolai Matni, Jean-Jacques E. Slotine, and Vikas Sindhwani. Learning stability certificates from data. In Proceedings of the 2020 Conference on Robot Learning, volume 155, pages 1341-1350. PMLR, 2021.
[21] David Angeli. A Iyapunov approach to incremental stability properties. IEEE Transactions on Automatic Control, 4703)4410-21,, 2002.
2002.
[22]
Hassan K. Khalil. Nonlinear Systems. Pearson Education. Prentice Hall, 2002.
[23] Peter L. Bartlett, Olivier Bousquet, and Shahar Mendelson. Local rademacher complexities. The Annals of Statistics, 33(4):1497-1553 2005.
[24] David Haussler. Decision theoretic generalizations of the pac model for neural net and other learning applications. Information and Computation, 100(1):78-150) 1992.
[25] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wandeeman-Milnn and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
http://gitan com/ hamm/google/jaa jax.
[26] Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc van Zee. Flax: A neural network library and ecosystem for JAX, 2020. URL http://github. com/ coon/gogle//fax
[27] Matteo Hessel, David Budden, Fabio Viola, Mihacla Rosca, Eren Sezener, and Tom Hennigan. Optax: composable gradient transformation and optimisation, jax!, 2020. URL http: // //github. . wwt.coo///eaaee www.ddeemmnd//pptotooooooooooeee
[28] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 2016.
[29] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016
[30] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Emestus, and Noah Dormann. Stable-baselines:: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268): 2021.
[31] Nathan Srebro, Karthik Sridharan, and Ambuj Tewari. Smoothness, low noise and fast rates. In Advances in Neural Information Processing Systems, volume 23, 2010.
[32] Martin J. Wainwright. High-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cambridge University Press, 2019.
[33] Peter L. Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463 2002.
[34] Olivier Bousquet. Concentration inequalities and empirical processes theory applied to the analysis of learning algorithms. PhD thesis, École Polytechnique: Department of Applied Mathematics Paris, France, 2002.
[35] Roman Vershynin. High--Dienssioal Probability: An Introduction with Applications in Data Science. Cambridge University Press, 2018.
[36] Max Simehowitz, Horia Mania, Stephen Tu, Michael I. Jordan, and Benjamin Recht. Learning without mixing: Towards a sharp analysis of linear system identification. Proceedings of the s1 Conference On Learning Theory, volume 75, pages 439-473 PMLR, 2018
12

---

[37] Franco Woolfe, Edo Liberty, Vladimir Rokhlin, and Mark Tygert. A fast randomized algorithm for the approximation of matrices. Applied and Computational Harmonic Analysis, 25(3): 335-366, 2008.
[38] Yann A LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. Efficient backprop. In Neural networks: Tricks of the trade, pages 9-48. Springer, 2012.
13


Reviews:
[{"review_id": "zs1sahNql9L", "reviewer": "Reviewer_jRuX", "summary": "The paper proposes a variant of behavioural cloning that learns higher-order derivatives of the expert policy. The authors provide a sample-complexity bound and show experimentally that learning higher-order derivatives leads to better-performing learned policies.", "questions": "+ The setting seems odd. Can the authors give some examples of where this setting of having Jacobians along with policies may occur? This is the key issue limiting the significance of the paper.\n+ Do the results on sample complexity count a trajectory plus derivatives as a single trajectory? This seems unfair, since e.g. with finite differences it would take at least two trajectory evaluations to get a trajectory plus derivatives.\n+ Typically the IL problem is formulated with a stochastic policy. Is a strictly deterministic policy required for the ${\\cal O}(n^{-1})$ rates? How would the notion of $p$ -order derivatives translate to a stochastic policy?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n+ Clarity. The paper is well-written and clear. The main contribution is spelled out clearly and the theory and experiments are effective at bolstering the claims made.\n+ Theory. It's nice to see some finite-sample bounds for imitation learning, and the theory seems quite robust at proving the required results, at least given the assumptions in the paper. Generally we only see asymptotic results.\n+ Originality. As far as I am aware, the setting and idea is quite novel. The idea of matching derivatives reminds me of PID controllers, which have seen a huge amount of success practically. \n\nWeaknesses\n+ Odd Setting. The assumptions needed for the TASIL approach to work seem quite odd. Typically in imitation learning, we assume access to a fixed set of expert trajectories, and potentially the ability to roll out new trajectories (online) or not (offline). From a practical point of view, we can achieve this by recording human demonstrators solving the task themselves. In the TASIL setting, we would need to record the Jacobians of the human policy along with the policy itself. This doesn't necessarily seem possible: you could ask the demonstrators how they would change their action given a small change in the observations, but it's not clear to me that demonstrators would be very good at answering this query. \n\n  Of course, if the expert policy is given as a differentiable program, it would be possible to tractably compute the Jacobian with autodiff(though this would incur an extra factor of $\\min(d, m)$ in the complexity). But in that case we already have the expert policy, so what's the point in doing imitation learning? Finally, numerical differences are proposed as a way of getting the Jacobian, but that would assume query access to the policy with arbitrary observations. This is again different from the typical IL set-up, where we are able to sample rollouts but are not able to evaluate the policy with arbitrary inputs.\n\n  The odd setting limits the significance of the work quite substantially.\n\n+ Unclear Evaluation. The authors show convincingly that having higher-order derivatives of the policy helps quite a bit in learning the correct policy. However, this doesn't seem surprising. To take an extreme case, if we are trying to fit an analytic function and we know *all* the derivatives, we can fit the entire function exactly from a single point. Generally speaking, the extra information is going to help. The authors need to present a fair test; for instance, if we are envisioning a situation where we are querying experts to get our expert policy and we require $d$ more queries per human to get the Jacobian, how does it compare to use those $d$ queries on the Jacobian vs getting $d$ more complete rollouts?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RXprFOFBOz", "reviewer": "Reviewer_tDpY", "summary": "This paper propose an augmentation to standard imitation learning losses by penalizing the deviation in higher order Taylor series terms between learned and expert policy. The paper analyzes when the experts are easier to learn with the notion of incremental input-to-state stability. A main result is if the p-th order Taylor series of the policy matches that of the expert policy when evaluated on trajectories from $\\pi_E$, then the imitation gap is small. This naturally leads to the TaSIL algorithm. For realizable case, a sample complexity bound is provided. Experiments on various continuous control tasks also demonstrate the effectiveness of the algorithm.", "questions": "Please see \"Strengths And Weaknesses\".", "limitations": "No potential negative societal impact.", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- The paper provides an interesting perspective to study the well-known distribution shift problem of standard behavior cloning. \n- The theoretical analysis leads to a novel augmentation to existing imitation learning algorithm.\n- The math derivation is solid and the presentation quality is good.\n\nComments and Questions:\n- the discrepancy in the algorithm $\\Delta_t^{\\pi} := \\pi(\\phi_t^{\\pi} (\\xi)) - \\pi_* (\\phi_t^{\\pi} (\\xi))$ has an explicit dependence on $\\pi_*$. Do we need the knowledge of the expert policy during optimization (to evaluate the objective). In BC, we only need samples from $\\pi_*$.\n- In Mujoco environments, will 2-TaSIL further improve the performance?\n- What are the limitations of your algorithm? For example, besides sample efficiency, what are the computational efficiency compared to previous methods? Is the computation of those higher-order taylor series terms expensive?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IAItMwRvf-s", "reviewer": "Reviewer_8nEZ", "summary": "The authors target the distribution shift problem in imitation learning, especially behavior cloning (BC). They propose TaSIL, which augments BC by penalizing deviations in the higher-order Tayler series terms between the learned and expert policies. Theoretical analysis shows that smaller TaSIL-augmented loss over expert trajectories promises a small imitation loss over generated trajectories. Extensive experiment results using OpenAI gym environment support the effectiveness of the proposed solution. ", "questions": "What will the performance comparison be like when compared with other approaches that deal with the distribution shift problem requiring environment interactions such as GAIL. ", "limitations": "The authors have stated potential limitations and future directions of the TaSIL work. It is interesting to learn how this TaSIL idea will be applied in offline IL and pixel-based IL. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. Proposed an interesting TaSIL that augments BC to capture learned and expert policy errors from a higher-order perspective. Theoretical justification also is given telling a small TaSIL loss likely promises a small expert and learner performance difference. \n2. The proposed solution is justified to be sample efficient both theoretically and empirically. \n3. Extensive experiment results validate major claims in the paper. \nWeakness: \n1. The notation used in this paper is different from major works such as in BC, DAGGER and GAIL, making it a little bit difficult to follow. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-X_TBManyXz", "reviewer": "Reviewer_dYK3", "summary": "The paper considers an imitation learning problem for which the goal is to reduce the trajectory discrepancy (i.e., imitation gap) between the expert policy and the learned policy. In order to study this, the paper proposes to consider such gap in the context of $\\delta$-ISS (incremental input-to-state stability) system — a special transition dynamics system such that: a) the trajectories generated converge to the same fixed points regardless of the initial state distribution; and b) the trajectory difference caused by different policies can be bounded. One nice property of such system is the trajectory difference (with the same initial conditions) can be effectively bounded by a class K function, i.e., the $\\gamma$ function. The paper then leverages this property to bound the imitation gap over the trajectories generated by the expert policy. Specifically, the paper investigates two types of $\\gamma$ function. The first type concerns the rapidly decaying functions, faster than linear. For the system with this type of function, the paper shows that BC suffices to recover the expert policy with small imitation gap. The second type of $\\gamma$ function refers to slowly decaying functions. For this system, the paper shows that matching up to $p$-th order of policy derivative (aka Taylor Series Imitation Learning, TaSIL) is sufficient to bound the imitation gap. The paper also evaluates the TaSIL idea by augmenting the BC loss with extra loss from the $p$-th order matching (via finite differences). ", "questions": "Q1: The on-policy and off-policy terms are specifically reserved for reinforcement learning training settings. It would be confusing to say GAIL is off-policy approaches. Would it be better to explicitly explain the off-policy in the paper? Or to change on-policy/off-policy to interactive and simulation-based? \n\n\nQ2: In Fig 1, the mean policy discrepancy is quite small for order 1 and 2 (as opposed to order 0) when $\\nv$ is close to 0. But why is the imitation gap for order 1 and 2 still very large? \n\n\nQ3: For the Mujoco experiments, it’s unclear what the $\\gamma$ function is and how it would affect the final performance of TaSIL. Would it be possible to explicitly present the dynamics and check what types of $\\gamma$ function each environment corresponds to? \n", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths** \n\n[originality] Studying the imitation learning from the system perspective seems novel and interesting. The idea of leveraging system property to effectively reduce the imitation gap is also inspiring. One nice insights this paper sheds light on is when the BC suffices to recover the expert policy — for a system with rapidly decaying $\\gamma$ functions. \n\n[clarity & quality] The paper is relatively easy to follow (the notations are quite different from the RL area though). \n\n**Weakness**\n\n[significance] The assumption that a system is $\\delta$-ISS might be quite strong and far from being realistic, especially for the imitation learning setting where typically no system information is available. Although the paper presents detailed discussion and theoretical analysis of the imitation gap for such system, it still remains largely unclear how to decide if a system is $\\delta$-ISS, or what additional information is needed to identify such system. According to Theorem 3.1 and 3.2, knowing the $\\gamma$ type for $\\delta$-ISS system is crucial as it determines whether to use BC or TaSIL. Furthermore, from the empirical results in Fig 2, it looks that the performance boost via this TaSIL augmentation is insignificant. Also, as the number of trajectories increases, the performance gap between BC and BC-augmented is diminishing. This might put it in question that whether it is really necessary to consider the $\\delta$-ISS and how this consideration would really make a difference in practice. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept