Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper propose an augmentation to standard imitation learning losses by penalizing the deviation in higher order Taylor series terms between learned and expert policy. The paper analyzes when the experts are easier to learn with the notion of incremental input-to-state stability. A main result is if the p-th order Taylor series of the policy matches that of the expert policy when evaluated on trajectories from $\pi_E$, then the imitation gap is small. This naturally leads to the TaSIL algorithm. For realizable case, a sample complexity bound is provided. Experiments on various continuous control tasks also demonstrate the effectiveness of the algorithm.
strengths_and_weaknesses: Strength:
- The paper provides an interesting perspective to study the well-known distribution shift problem of standard behavior cloning. 
- The theoretical analysis leads to a novel augmentation to existing imitation learning algorithm.
- The math derivation is solid and the presentation quality is good.

Comments and Questions:
- the discrepancy in the algorithm $\Delta_t^{\pi} := \pi(\phi_t^{\pi} (\xi)) - \pi_* (\phi_t^{\pi} (\xi))$ has an explicit dependence on $\pi_*$. Do we need the knowledge of the expert policy during optimization (to evaluate the objective). In BC, we only need samples from $\pi_*$.
- In Mujoco environments, will 2-TaSIL further improve the performance?
- What are the limitations of your algorithm? For example, besides sample efficiency, what are the computational efficiency compared to previous methods? Is the computation of those higher-order taylor series terms expensive?
questions: Please see "Strengths And Weaknesses".
limitations: No potential negative societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors target the distribution shift problem in imitation learning, especially behavior cloning (BC). They propose TaSIL, which augments BC by penalizing deviations in the higher-order Tayler series terms between the learned and expert policies. Theoretical analysis shows that smaller TaSIL-augmented loss over expert trajectories promises a small imitation loss over generated trajectories. Extensive experiment results using OpenAI gym environment support the effectiveness of the proposed solution. 
strengths_and_weaknesses: Strength:
1. Proposed an interesting TaSIL that augments BC to capture learned and expert policy errors from a higher-order perspective. Theoretical justification also is given telling a small TaSIL loss likely promises a small expert and learner performance difference. 
2. The proposed solution is justified to be sample efficient both theoretically and empirically. 
3. Extensive experiment results validate major claims in the paper. 
Weakness: 
1. The notation used in this paper is different from major works such as in BC, DAGGER and GAIL, making it a little bit difficult to follow. 
questions: What will the performance comparison be like when compared with other approaches that deal with the distribution shift problem requiring environment interactions such as GAIL. 
limitations: The authors have stated potential limitations and future directions of the TaSIL work. It is interesting to learn how this TaSIL idea will be applied in offline IL and pixel-based IL.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper considers an imitation learning problem for which the goal is to reduce the trajectory discrepancy (i.e., imitation gap) between the expert policy and the learned policy. In order to study this, the paper proposes to consider such gap in the context of $\delta$-ISS (incremental input-to-state stability) system — a special transition dynamics system such that: a) the trajectories generated converge to the same fixed points regardless of the initial state distribution; and b) the trajectory difference caused by different policies can be bounded. One nice property of such system is the trajectory difference (with the same initial conditions) can be effectively bounded by a class K function, i.e., the $\gamma$ function. The paper then leverages this property to bound the imitation gap over the trajectories generated by the expert policy. Specifically, the paper investigates two types of $\gamma$ function. The first type concerns the rapidly decaying functions, faster than linear. For the system with this type of function, the paper shows that BC suffices to recover the expert policy with small imitation gap. The second type of $\gamma$ function refers to slowly decaying functions. For this system, the paper shows that matching up to $p$-th order of policy derivative (aka Taylor Series Imitation Learning, TaSIL) is sufficient to bound the imitation gap. The paper also evaluates the TaSIL idea by augmenting the BC loss with extra loss from the $p$-th order matching (via finite differences). 
strengths_and_weaknesses: **Strengths** 

[originality] Studying the imitation learning from the system perspective seems novel and interesting. The idea of leveraging system property to effectively reduce the imitation gap is also inspiring. One nice insights this paper sheds light on is when the BC suffices to recover the expert policy — for a system with rapidly decaying $\gamma$ functions. 

[clarity & quality] The paper is relatively easy to follow (the notations are quite different from the RL area though). 

**Weakness**

[significance] The assumption that a system is $\delta$-ISS might be quite strong and far from being realistic, especially for the imitation learning setting where typically no system information is available. Although the paper presents detailed discussion and theoretical analysis of the imitation gap for such system, it still remains largely unclear how to decide if a system is $\delta$-ISS, or what additional information is needed to identify such system. According to Theorem 3.1 and 3.2, knowing the $\gamma$ type for $\delta$-ISS system is crucial as it determines whether to use BC or TaSIL. Furthermore, from the empirical results in Fig 2, it looks that the performance boost via this TaSIL augmentation is insignificant. Also, as the number of trajectories increases, the performance gap between BC and BC-augmented is diminishing. This might put it in question that whether it is really necessary to consider the $\delta$-ISS and how this consideration would really make a difference in practice. 

questions: Q1: The on-policy and off-policy terms are specifically reserved for reinforcement learning training settings. It would be confusing to say GAIL is off-policy approaches. Would it be better to explicitly explain the off-policy in the paper? Or to change on-policy/off-policy to interactive and simulation-based? 


Q2: In Fig 1, the mean policy discrepancy is quite small for order 1 and 2 (as opposed to order 0) when $\nv$ is close to 0. But why is the imitation gap for order 1 and 2 still very large? 


Q3: For the Mujoco experiments, it’s unclear what the $\gamma$ function is and how it would affect the final performance of TaSIL. Would it be possible to explicitly present the dynamics and check what types of $\gamma$ function each environment corresponds to? 

limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a variant of behavioural cloning that learns higher-order derivatives of the expert policy. The authors provide a sample-complexity bound and show experimentally that learning higher-order derivatives leads to better-performing learned policies.
strengths_and_weaknesses: Strengths
+ Clarity. The paper is well-written and clear. The main contribution is spelled out clearly and the theory and experiments are effective at bolstering the claims made.
+ Theory. It's nice to see some finite-sample bounds for imitation learning, and the theory seems quite robust at proving the required results, at least given the assumptions in the paper. Generally we only see asymptotic results.
+ Originality. As far as I am aware, the setting and idea is quite novel. The idea of matching derivatives reminds me of PID controllers, which have seen a huge amount of success practically. 

Weaknesses
+ Odd Setting. The assumptions needed for the TASIL approach to work seem quite odd. Typically in imitation learning, we assume access to a fixed set of expert trajectories, and potentially the ability to roll out new trajectories (online) or not (offline). From a practical point of view, we can achieve this by recording human demonstrators solving the task themselves. In the TASIL setting, we would need to record the Jacobians of the human policy along with the policy itself. This doesn't necessarily seem possible: you could ask the demonstrators how they would change their action given a small change in the observations, but it's not clear to me that demonstrators would be very good at answering this query. 

  Of course, if the expert policy is given as a differentiable program, it would be possible to tractably compute the Jacobian with autodiff(though this would incur an extra factor of $\min(d, m)$ in the complexity). But in that case we already have the expert policy, so what's the point in doing imitation learning? Finally, numerical differences are proposed as a way of getting the Jacobian, but that would assume query access to the policy with arbitrary observations. This is again different from the typical IL set-up, where we are able to sample rollouts but are not able to evaluate the policy with arbitrary inputs.

  The odd setting limits the significance of the work quite substantially.

+ Unclear Evaluation. The authors show convincingly that having higher-order derivatives of the policy helps quite a bit in learning the correct policy. However, this doesn't seem surprising. To take an extreme case, if we are trying to fit an analytic function and we know *all* the derivatives, we can fit the entire function exactly from a single point. Generally speaking, the extra information is going to help. The authors need to present a fair test; for instance, if we are envisioning a situation where we are querying experts to get our expert policy and we require $d$ more queries per human to get the Jacobian, how does it compare to use those $d$ queries on the Jacobian vs getting $d$ more complete rollouts?
questions: + The setting seems odd. Can the authors give some examples of where this setting of having Jacobians along with policies may occur? This is the key issue limiting the significance of the paper.
+ Do the results on sample complexity count a trajectory plus derivatives as a single trajectory? This seems unfair, since e.g. with finite differences it would take at least two trajectory evaluations to get a trajectory plus derivatives.
+ Typically the IL problem is formulated with a stochastic policy. Is a strictly deterministic policy required for the ${\cal O}(n^{-1})$ rates? How would the notion of $p$ -order derivatives translate to a stochastic policy?
limitations: Yes
review_title: null

Meta-review: metareview: Guided by theoretical analysis, this paper proposes a new objective for behavioral cloning based on penalizing deviations in higher-order Taylor series terms of policies and demonstrates the benefits in MuJoCo experiments. The method is limited by its need for the Jacobians of the policy to be imitated (demonstrated human trajectories in standard imitation learning do not provide such information, instead algorithmically-produced policies are needed). However, the authors provide some examples that seem sufficiently motivating. Another emphasized reviewer concern is the feasibility of identifying the necessary conditions for the theoretical analysis in actual tasks/demonstration policies, but I tend to agree with the authors that this is an orthogonal question to operationalizing the assumption to develop the proposed methods. Overall, I think the authors have addressed many of the lesser concerns of the reviewers and agree with the majority positive opinion of the reviewers and recommend the paper be accepted.
confidence: Less certain
Decision: Accept