TaSIL: Taylor Series Imitation Learning

Abstract: We propose Taylor Series Imitation Learning (TaSIL), a simple augmentation to standard behavior cloning losses in the context of continuous control. TaSIL penalizes deviations in the higher-order Tayler series terms between the learned and expert policies. We show that experts satisfying a notion of incremental input-to-state stability are easy to learn, in the sense that that a small TaSIL-augmented imitation loss over expert trajectories guarantees a small imitation loss over trajectories generated by the learned policy.  We provide sample-complexity bounds for TaSIL that scale as $\tilde{\mathcal{O}}(1/n)$ in the realizable setting, for $n$ the number of expert demonstrations. Finally, we demonstrate experimentally the relationship between the robustness of the expert policy and the order of Taylor expansion required in TaSIL, and compare standard Behavior Cloning, DART, and DAgger with TaSIL-loss-augmented variants.  In all cases, we show significant improvement over baselines across a variety of MuJoCo tasks.

Reviews and author responses:

 A friendly reminder that the reviewer/author discussion period ends tomorrow.  If there are any additional points regarding our response that you would like clarified, please don't hesitate to reach out with further questions.

 Thank you for your detailed reply to my questions and the questions of the other reviewers. 

In my review I wasn't able to think of situations where it would be realistic to have oracle access to the Jacobian of the expert. I can see that distilling a complex policy from e.g. MPC could be such a use-case. 

I will therefore raise my score from 4 to 5. 



 - **Regarding the odd setting:** As we note in the general response there are many non-human experts which may be interesting to imitate, including larger neural networks and expensive non-linear model predictive controllers. Prior work in these areas includes:

Imitating MPC:
“Efficient Representation and Approximation of Model Predictive Control Laws via Deep Learning,”  Karg et al.
“Approximating Explicit Model Predictive Control Using Constrained Neural Networks,” Chen et al.

Imitating large neural nets:
“Distilling the Knowledge in a Neural Network,” Hinton et al.
“Knowledge distillation: A good teacher is patient and consistent,” Beyer et al.

Since TaSIL has finite sample guarantees for tracking the expert trajectory, our work provides a method for transferring the safety properties of these controllers to neural networks. For many of these experts, they may be cheaply queryable (so derivatives can be potentially numerically approximated via finite differencing–we explore this experimentally in Appendix C), but the cost of collecting additional trajectories can be quite high and involve running physical experiments (i.e., consider a self-driving vehicle or quadrotor with MPC).

It was noted that querying from arbitrary points to numerically estimate the derivatives is very different from a traditional IL setup. However, this query model (of being able to ask the expert to label arbitrary states) is assumed in e.g., DAgger. Furthermore, interactive algorithms such as DART, DAgger, and GAIL assume the ability to roll out new trajectories under new policies, which we do not. Ultimately the takeaway from such comparisons is that if one has an interactive environment, then one should use a method tailored to that. If one can query the expert faster compared to rolling out trajectories, one should consider using TaSIL, especially given the strong theoretical guarantees on performance.

- **Regarding unclear evaluation and extra information:** Extra information will of course always improve performance. The principal finding of our paper is not just that TaSIL improves performance, but that capturing higher order behavior is more important in more unstable settings, and that BC is only sufficient if the system is already extremely robust. This can be seen in Figure 1, where 1-TaSIL and 2-TaSIL have both lower imitation gap and mean policy difference for the same system, but crucially 1-TaSIL and 2-TaSIL both do not lose their performance edge in the less robust regime, whereas 0-TaSIL (BC) already starts to fail. Similarly in the Mujoco setting (Figure 2), although we cannot precisely characterize the $\gamma$ function for the expert, TaSIL disproportionately helps with more difficult environments (i.e., Humanoid, HalfCheetah).

- **Regarding sample complexity comparisons:** In Appendix C, we run experiments where we consider TaSIL with X additional queries to the expert per state and a finite-differencing loss. Naturally for systems with very large state spaces (Humanoid), or ones where 1-TaSIL helps but they are relatively easy to learn anyways (i.e., Ant, Walker), it would be better to just collect more trajectories since large state spaces/very stable systems benefit less from matching directional derivatives. However, for HalfCheetah, just 1 additional query per state to compute derivatives via finite-differencing yields > 75% of expert reward with just 15 trajectories (i.e., 30 trajectories worth of queries to the expert) while vanilla BC fails to reach that level even after 50 trajectories. We expect that in this domain of low-dimensional complex systems, TaSIL will be competitive with BC even if querying the expert is non-negligibly expensive. 

Furthermore, a more clever adversarial approach using Prop 3.1 could allow for more intelligently picking the most important places to query the expert rather than attempting to estimate the entire derivative. This would potentially allow the finite-sample guarantees of TaSIL to be transferred to an interactive or more query-efficient setting. We leave this to future work.

- **Regarding stochasticity** We originally formulated TaSIL in a block MDP setting but did not include this to improve clarity. One can treat the stochasticity of a policy as an augmentation of the state where the noise components over time are drawn as part of sampling the initial condition. One would then match the derivatives of the original state components and ignore the partials of the stochastic component as that evolves independently of the policy. However the state derivatives must match for identical noise instances and the closed loop system must similarly be dISS stable for each noise realization. This could also be applied to policies which output a distribution from which the action is sampled–it is simply a matter of reparameterizing the policy in terms of (state, noise) as an input (or something that bijectively maps onto this space).


 - **Regarding Weakness**: The performance gap between BC and 1-TaSIL decreases in the Mujoco experiments because with sufficiently many trajectories, even BC can attain good performance by virtue of the training data largely covering the relevant state-action space. With this in mind, we emphasize that the performance gains of 1-TaSIL over vanilla BC is very significant in the low-to-intermediate sample regime, especially for the *more complex* tasks such as Half-Cheetah or Humanoid. In fact the performance of BC with 1-TaSIL is notably better than vanilla DART and DAgger. The results match our assertion that less robust expert closed-loop systems may require TaSIL augmentation to successfully imitate–especially with the time-independent sample complexity of TaSIL. We note that the less significant improvements for other tasks are equally important to include, as they illustrate the complementary assertion that vanilla BC can suffice to imitate more robust expert closed-loop systems, see: Theorem 3.1.

- Q1: We agree that on-policy and off-policy, though standard terms in the RL literature, may be potentially confusing or uninformative in our paper mostly dedicated to the pure offline IL/BC setting. We currently use on-policy and off-policy as discussed in the DART paper by Laskey et. al.–i.e., “on-policy” refers to when more rollouts are performed on a learnt policy and “off-policy” refers to when only the expert policy (or modified expert policy) is used. The author’s descriptive alternatives are appreciated and we will take them into account during revision to reduce the inherent confusion. 

- Q2: The dISS of the system suggests an relationship of the form $\textrm{gap} = \textrm{constant}\cdot(\textrm{policy discrepancy})^\nu$ between the policy discrepancy and imitation gap (this follows from application of the dISS property). Therefore for small $\nu$, regardless of how small the discrepancy is between learned and expert controllers, the final state gap will be magnified. As mentioned in our general response, in the extreme $\nu = 0$ (step function), the final imitation gap will be constant regardless of the TaSIL variant used, which explains why we see the imitation gaps between 0, 1, and 2 TaSIL approaching each other for small $\nu$ (we note that $\nu$ = 0 is not contained in our plots, the smallest $\nu$ tested was 0.05).

- Q3: For the Mujoco environments, unfortunately the “true” $\gamma$ function depends not just on the environment but also on the expert policy, which are RL-trained neural networks. This makes the $\gamma$ function very difficult to characterize; in fact, the effect of input perturbation on state is almost certainly both perturbation direction and state dependent–a $\gamma$ function in this case would be the worst-case bound over all states and local perturbation directions realizable via expert trajectories. We direct the reviewer to our general response on the reasonableness of our dISS assumptions, and how we propose to choose $p$ in practice.


 We thank the reviewer for their kind remarks. We invite the reviewer to read our general response for a summary of what we believe to be the central foci of our paper, as well as the main subtleties to take away. Regarding comparisons to other algorithms, we experimentally demonstrate (see Figure 2) not only superior performance of BC + TaSIL to interactive methods like DART and DAgger across a variety of Mujoco environments, but also show that using a TaSIL loss can also boost DART and DAgger themselves. Although we did not test GAIL specifically, we expect that similar improvements could be brought by using a TaSIL-augmented loss.

Unfortunately our notation does differ from the standard RL/IL works. As dISS is a stability notion originating from dynamical systems / control theory, the aesthetics of our theoretical analysis naturally lean toward those communities. We hope the intuition behind our theory can still be readily extracted, despite the notational barrier.


 - Q1: We bound the discrepancy the reviewer refers to (between the expert and learned policies under the learned policy) in terms of the discrepancy on the expert-policy trajectories–this is a consequence of our key Proposition 3.1. Therefore we only need to collect a dataset of trajectories under the expert (in addition to the expert policy output + higher order information on those trajectories) but do not need to actually evaluate the expert policy during training. What we are proposing is essentially a modification of the Behavior Cloning imitation loss that additionally clones higher-order derivatives.

- Q2: We do not evaluate the 2-TaSIL due to the prohibitive complexity of the Hessians involved for Mujoco environments. Humanoid-v3 has a state dimension of 376 and an input dimension of 17, meaning the Hessian has 2.4 million parameters (before counting symmetries). We note that this is how we envision TaSIL to be used in practice, where one augments their existing IL method with $p$-TaSIL, with $p$ chosen as a hyperparameter depending on one’s data collection pipelines or compute resources. As we note in our general response, the case $p=1$ should be sufficient for most experts. We test 2-TaSIL experimentally on our synthetic system primarily to demonstrate that adding higher order terms improves performance on experts that have very poor robustness, in line with our theoretical predictions.

- Q3: The limitations of our algorithm are inherently balanced between the interactivity properties of the expert and the computational cost of derivatives, and thus cannot be characterized uniformly. For example, DAgger requires learned policy trajectories to be rolled-out for each expert initial condition every optimization iteration, and also that the expert to be queried along each of these trajectories. As such DAgger (or DART) does not require higher-order expert information, but is also a poor choice when either the expert is non-interactive or rolling out learned trajectories is expensive. On the other hand, if the expert trajectories are annotated with derivative information offline, then TaSIL is a fully offline method like BC, and is as computationally feasible as computing the Jacobians of the learner policy. When the learner is a moderately-sized neural network, this is reasonable. We also discuss approximating derivatives via finite-differencing in our paper (Appendix C), where we assume a query-able expert like in DAgger, but do not require policy roll-outs. 
Regardless of the computational trade-offs, we emphasize that the analysis of TaSIL is the first to demonstrate strong guarantees on distribution shift (i.e., no dependency on time-horizon $T$), under a broad stability notion that brings to light the implicit assumptions of methods like DAgger, DART and GAIL, as discussed in the general response.

 We thank the reviewers for their constructive and insightful feedback. While all of the reviewers agree that the paper is well written and provides **novel finite-data guarantees for IL**, certain common questions were raised.  We address these here first, and then provide detailed responses to individual reviewers below. 

**Why the dISS assumption, and how reasonable is it in practice?**

IL methods such as DAgger, DART, and GAIL implicitly assume that the expert is able to recover from errors due to noise injection or learned policy errors. We propose using *Incremental Input-to-State-Stability (dISS)*, a control theoretic notion of robust nonlinear stability, to quantify this type of expert robustness.  We only require robustness to perturbations, captured by the $\gamma$ function in Definition 3.1, and not convergence of trajectories from different initial conditions, captured by the $\beta$ function.  This $\gamma$ function captures how much perturbations to the system (i.e., errors between the learned and expert policy) induce deviations from a nominal (expert) trajectory, and is not restrictive at all, in the sense that it can interpolate from perfect robustness (i.e., $\gamma(x)=0 \forall x$), to extreme fragility (e.g., $\gamma(x) \approx \mathbf{1}_{> 0}(x)$).  Our results show that if the expert policy has a reasonable amount of robustness to disturbances, e.g., a polynomial $\gamma$ function, then sample-efficient IL is possible via TaSIL.  We again emphasize that such an assumption of expert robustness is implicit in standard IL algorithms, and one of our contributions is introducing a formalism for quantifying its effects on the sample-efficiency of IL.

**How does one verify the dISS assumption?**:

Rather than verifying dISS, we propose operationalizing the theoretical insights of TaSIL by treating the order $p$ as a hyperparameter: in particular, one could imagine starting with BC and examining whether the desired performance is achieved given the current data budget. If not, a first order term could be added, etc. Furthermore, any expert that induces locally exponentially stable error dynamics in a control-affine system (i.e., any good expert applied to a mechanical system should meet this assumption) is locally-dISS with a linear gamma function, which makes 1-TaSIL a good default choice for robotics settings.

**What is the practical setting/oracle model for which TaSIL makes sense?**:

Here are two settings where TaSIL is immediately applicable:
- **Differentiable experts.**  Such experts include nonlinear model-predictive-controllers (MPC) (Jacobians obtainable by autodiff or KKT conditions) and large neural networks (derivatives obtainable via autodiff).  One may wish to perform IL for such experts, despite access to function calls and derivatives, when the expert is too computationally expensive/slow to run in real time (e.g., a nonlinear MPC relying on expensive optimization primitives), or the expert function is too large to fit on-board a robot (e.g., a large neural net with too many parameters). In these settings, IL can learn a compact explicit representation of the policy that meets real-time and computational requirements. Indeed, there are entire literatures centered around IL of MPC controllers and distillation of large deep networks into smaller networks, motivated by real-time and compute constraints.  We provide specific references in our detailed response to Reviewer 4. Further, since TaSIL has finite-sample bounds, it is well suited for imitating experts that provide safety guarantees but cannot be run in real time, e.g., nonlinear MPC.

- **Queryable experts.** In settings where there is a *queryable* expert (one can query $\pi(x)$ for any state $x$), then TaSIL is reasonable since derivatives along the observed trajectories can be easily approximated via finite-differencing.  We provide a detailed analysis of this setting in Appendix C, including how to efficiently approximate the Jacobian using randomized linear algebra. In Figure 3 (Appendix C.3), we experimentally show that only a few finite-difference directions suffice to reap the benefits of Jacobian cloning. We note that this queryable expert model is assumed, e.g., in the oracle model of DAgger, where experts (interactively) label how to recover from failures. We further add that using finite-differencing is particularly appealing in settings where it is much cheaper to query an expert at a particular state configuration, compared to rolling out an entire trajectory. For example, for robotics or autonomous vehicles, collecting a trajectory may be significantly more expensive than collecting a labeled pair $(x, \pi_\star(x))$, since the former requires actually rolling out the system, whereas the latter may only involve querying a function. In these settings, TaSIL with extra queries may still be more efficient than BC with extra trajectories, e.g., see the Half-Cheetah results in Figure 3, Appendix C.3.


 This paper propose an augmentation to standard imitation learning losses by penalizing the deviation in higher order Taylor series terms between learned and expert policy. The paper analyzes when the experts are easier to learn with the notion of incremental input-to-state stability. A main result is if the p-th order Taylor series of the policy matches that of the expert policy when evaluated on trajectories from $\pi_E$, then the imitation gap is small. This naturally leads to the TaSIL algorithm. For realizable case, a sample complexity bound is provided. Experiments on various continuous control tasks also demonstrate the effectiveness of the algorithm. Strength:
- The paper provides an interesting perspective to study the well-known distribution shift problem of standard behavior cloning. 
- The theoretical analysis leads to a novel augmentation to existing imitation learning algorithm.
- The math derivation is solid and the presentation quality is good.

Comments and Questions:
- the discrepancy in the algorithm $\Delta_t^{\pi} := \pi(\phi_t^{\pi} (\xi)) - \pi_* (\phi_t^{\pi} (\xi))$ has an explicit dependence on $\pi_*$. Do we need the knowledge of the expert policy during optimization (to evaluate the objective). In BC, we only need samples from $\pi_*$.
- In Mujoco environments, will 2-TaSIL further improve the performance?
- What are the limitations of your algorithm? For example, besides sample efficiency, what are the computational efficiency compared to previous methods? Is the computation of those higher-order taylor series terms expensive? Please see "Strengths And Weaknesses". No potential negative societal impact.

 The authors target the distribution shift problem in imitation learning, especially behavior cloning (BC). They propose TaSIL, which augments BC by penalizing deviations in the higher-order Tayler series terms between the learned and expert policies. Theoretical analysis shows that smaller TaSIL-augmented loss over expert trajectories promises a small imitation loss over generated trajectories. Extensive experiment results using OpenAI gym environment support the effectiveness of the proposed solution.  Strength:
1. Proposed an interesting TaSIL that augments BC to capture learned and expert policy errors from a higher-order perspective. Theoretical justification also is given telling a small TaSIL loss likely promises a small expert and learner performance difference. 
2. The proposed solution is justified to be sample efficient both theoretically and empirically. 
3. Extensive experiment results validate major claims in the paper. 
Weakness: 
1. The notation used in this paper is different from major works such as in BC, DAGGER and GAIL, making it a little bit difficult to follow.  What will the performance comparison be like when compared with other approaches that deal with the distribution shift problem requiring environment interactions such as GAIL.  The authors have stated potential limitations and future directions of the TaSIL work. It is interesting to learn how this TaSIL idea will be applied in offline IL and pixel-based IL. 

 The paper considers an imitation learning problem for which the goal is to reduce the trajectory discrepancy (i.e., imitation gap) between the expert policy and the learned policy. In order to study this, the paper proposes to consider such gap in the context of $\delta$-ISS (incremental input-to-state stability) system — a special transition dynamics system such that: a) the trajectories generated converge to the same fixed points regardless of the initial state distribution; and b) the trajectory difference caused by different policies can be bounded. One nice property of such system is the trajectory difference (with the same initial conditions) can be effectively bounded by a class K function, i.e., the $\gamma$ function. The paper then leverages this property to bound the imitation gap over the trajectories generated by the expert policy. Specifically, the paper investigates two types of $\gamma$ function. The first type concerns the rapidly decaying functions, faster than linear. For the system with this type of function, the paper shows that BC suffices to recover the expert policy with small imitation gap. The second type of $\gamma$ function refers to slowly decaying functions. For this system, the paper shows that matching up to $p$-th order of policy derivative (aka Taylor Series Imitation Learning, TaSIL) is sufficient to bound the imitation gap. The paper also evaluates the TaSIL idea by augmenting the BC loss with extra loss from the $p$-th order matching (via finite differences).  **Strengths** 

[originality] Studying the imitation learning from the system perspective seems novel and interesting. The idea of leveraging system property to effectively reduce the imitation gap is also inspiring. One nice insights this paper sheds light on is when the BC suffices to recover the expert policy — for a system with rapidly decaying $\gamma$ functions. 

[clarity & quality] The paper is relatively easy to follow (the notations are quite different from the RL area though). 

**Weakness**

[significance] The assumption that a system is $\delta$-ISS might be quite strong and far from being realistic, especially for the imitation learning setting where typically no system information is available. Although the paper presents detailed discussion and theoretical analysis of the imitation gap for such system, it still remains largely unclear how to decide if a system is $\delta$-ISS, or what additional information is needed to identify such system. According to Theorem 3.1 and 3.2, knowing the $\gamma$ type for $\delta$-ISS system is crucial as it determines whether to use BC or TaSIL. Furthermore, from the empirical results in Fig 2, it looks that the performance boost via this TaSIL augmentation is insignificant. Also, as the number of trajectories increases, the performance gap between BC and BC-augmented is diminishing. This might put it in question that whether it is really necessary to consider the $\delta$-ISS and how this consideration would really make a difference in practice. 
 Q1: The on-policy and off-policy terms are specifically reserved for reinforcement learning training settings. It would be confusing to say GAIL is off-policy approaches. Would it be better to explicitly explain the off-policy in the paper? Or to change on-policy/off-policy to interactive and simulation-based? 


Q2: In Fig 1, the mean policy discrepancy is quite small for order 1 and 2 (as opposed to order 0) when $\nv$ is close to 0. But why is the imitation gap for order 1 and 2 still very large? 


Q3: For the Mujoco experiments, it’s unclear what the $\gamma$ function is and how it would affect the final performance of TaSIL. Would it be possible to explicitly present the dynamics and check what types of $\gamma$ function each environment corresponds to? 
 Yes

 The paper proposes a variant of behavioural cloning that learns higher-order derivatives of the expert policy. The authors provide a sample-complexity bound and show experimentally that learning higher-order derivatives leads to better-performing learned policies. Strengths
+ Clarity. The paper is well-written and clear. The main contribution is spelled out clearly and the theory and experiments are effective at bolstering the claims made.
+ Theory. It's nice to see some finite-sample bounds for imitation learning, and the theory seems quite robust at proving the required results, at least given the assumptions in the paper. Generally we only see asymptotic results.
+ Originality. As far as I am aware, the setting and idea is quite novel. The idea of matching derivatives reminds me of PID controllers, which have seen a huge amount of success practically. 

Weaknesses
+ Odd Setting. The assumptions needed for the TASIL approach to work seem quite odd. Typically in imitation learning, we assume access to a fixed set of expert trajectories, and potentially the ability to roll out new trajectories (online) or not (offline). From a practical point of view, we can achieve this by recording human demonstrators solving the task themselves. In the TASIL setting, we would need to record the Jacobians of the human policy along with the policy itself. This doesn't necessarily seem possible: you could ask the demonstrators how they would change their action given a small change in the observations, but it's not clear to me that demonstrators would be very good at answering this query. 

  Of course, if the expert policy is given as a differentiable program, it would be possible to tractably compute the Jacobian with autodiff(though this would incur an extra factor of $\min(d, m)$ in the complexity). But in that case we already have the expert policy, so what's the point in doing imitation learning? Finally, numerical differences are proposed as a way of getting the Jacobian, but that would assume query access to the policy with arbitrary observations. This is again different from the typical IL set-up, where we are able to sample rollouts but are not able to evaluate the policy with arbitrary inputs.

  The odd setting limits the significance of the work quite substantially.

+ Unclear Evaluation. The authors show convincingly that having higher-order derivatives of the policy helps quite a bit in learning the correct policy. However, this doesn't seem surprising. To take an extreme case, if we are trying to fit an analytic function and we know *all* the derivatives, we can fit the entire function exactly from a single point. Generally speaking, the extra information is going to help. The authors need to present a fair test; for instance, if we are envisioning a situation where we are querying experts to get our expert policy and we require $d$ more queries per human to get the Jacobian, how does it compare to use those $d$ queries on the Jacobian vs getting $d$ more complete rollouts? + The setting seems odd. Can the authors give some examples of where this setting of having Jacobians along with policies may occur? This is the key issue limiting the significance of the paper.
+ Do the results on sample complexity count a trajectory plus derivatives as a single trajectory? This seems unfair, since e.g. with finite differences it would take at least two trajectory evaluations to get a trajectory plus derivatives.
+ Typically the IL problem is formulated with a stochastic policy. Is a strictly deterministic policy required for the ${\cal O}(n^{-1})$ rates? How would the notion of $p$ -order derivatives translate to a stochastic policy? Yes

Meta-review: Guided by theoretical analysis, this paper proposes a new objective for behavioral cloning based on penalizing deviations in higher-order Taylor series terms of policies and demonstrates the benefits in MuJoCo experiments. The method is limited by its need for the Jacobians of the policy to be imitated (demonstrated human trajectories in standard imitation learning do not provide such information, instead algorithmically-produced policies are needed). However, the authors provide some examples that seem sufficiently motivating. Another emphasized reviewer concern is the feasibility of identifying the necessary conditions for the theoretical analysis in actual tasks/demonstration policies, but I tend to agree with the authors that this is an orthogonal question to operationalizing the assumption to develop the proposed methods. Overall, I think the authors have addressed many of the lesser concerns of the reviewers and agree with the majority positive opinion of the reviewers and recommend the paper be accepted.

Decision: Accept