CCIL: Continuity-Based Data Augmentation for Corrective Imitation Learning
Authors: Liyiming Ke
Yunchu Zhang
Abhay Deshpande
Siddhartha Srinivasa
Abhishek Gupta
Venue: ICLR 2024 poster (2024)
Keywords: imitation learning
data augmentation
robotics
model-based method
Primary area: reinforcement learning
TL;DR: We generate corrective labels for imitation learning by leveraging the presence of local continuity in the dynamics function.

Abstract: We present a new technique to enhance the robustness of imitation learning methods by generating corrective data to account for compounding error and disturbances. While existing methods rely on interactive expert labeling, additional offline datasets, or domain-specific invariances, our approach requires minimal additional assumptions beyond expert data. The key insight is to leverage local continuity in the environment dynamics. Our method first constructs a dynamics model from the expert demonstration, enforcing local Lipschitz continuity while skipping the discontinuous regions. In the locally continuous regions, this model allows us to generate corrective labels within the neighborhood of the demonstrations but beyond the actual set of states and actions in the dataset. Training on this augmented data enhances the agent's ability to recover from perturbations and deal with compounding error. We demonstrate the effectiveness of our generated labels through experiments in a variety of robotics domains that have distinct forms of continuity and discontinuity, including classic control, drone flying, high-dimensional navigation, locomotion, and tabletop manipulation.

CCIL: CONTINUITY-BASED DATA AUGMENTATION FOR CORRECTIVE IMITATION LEARNING
Liyiming Ket, Yunchu Zhang", Abhay Deshpande, Siddhartha Srinivasa, Abhishek Gupta Paul G. Allen School of Computer Science and Engineering, University of Washington (kayke, yunchuz, abhayd, s ddh, abhgupta @cs. washingtoon. edu
ABSTRACT
We present new technique enhance the robustness of imitation learning methods by generating corrective data to account for compounding errors and disturbances. While existing methods request interactive experts, additional offline datasets, or domain-speciff invariances, our approach requires minimal additional assumptions beyond access to expert data. Our key insight is to leverage local continuity in the environment dynamics generate corrective labels. Our method constructs dy- namics model from expert demonstrations, emphasizing local Lipschitz continuity in the learned model. In regions exhibiting local continuity, our algorithm generates corrective labels within the neighborhood of the demonstrations, extending beyond the actual set of states and actions in the dataset. Training on the augmented data improves the agent's ss resilience against perturbations and its capability to address compounding errors. To validate the efficacy of our generated labels, we con- duct experiments across diverse robotics domains in simulation, encompassing classic control problems, drone flying, navigation with nigh-dimensional sensor observations, legged locomotion, and tabletop manipulation.
INTRODUCTION
The application of imitation learning in real-world robotics necessitates extensive data, and the success of simple yet practical methods. such as behavior cloning (Pomerleau) 1988). relies on datasets with comprehensive coverage (Bojarski et al.) 2016: Florence et al.) 2022). However, obtaining such datasets can be prohibitively expensive, particularly in domains like robots that lacks readily available plethora of expert demonstrations. Furthermore, robotic policies can behave unpredictably and dangerously when encountering states not covered in the expert dataset, due to various factors such sensor noise, stochastic environments, external disturbances, or compounding errors leading to covariate shift (Ross et al., 2011 de Haan et al. 2019). For widespread deployment of imitation learning in weal-world robotic applications, policies need ensure their robustness when encountering unfamiliar states.
Many strategies address this challenge revolve around augmenting the training dataset. Classic techniques for data augmentation require either an interactive expert (Ross et al. 2011] Laskey et al. 2017) knowledge about the systems' invariances (Bojarski al.] 2016 F Florence et al. 201
Zhou et al. 2023). This information is necessary to inform the learning agent how to recover from out-of-distriuttion states. However, demanding such information can be burdensome (Ke al. 2021b), expensive (Zhou et al. 2023). or practically infeasible across diverse domains. As result, behavior cloning, which assumes only access to expert demonstrations, has remained the de-facto
solution in many domains (Silver et al.) 2008 Rahmatizadeh et al. 2018 Ke et al. 202 Florence
et al.
2022 Chen et al. 2022: Zhou et al. 2022).
For general applicability, this work focuses on imitation learning (IL. that rely solely expert demonstrations. We propose a method to enhance robustness IL by generating corrective labels for data augmentation. We recognize under-exploited feature of dynamic systems: local continuity. Despite the complex transitions and representations in system dynamics, they need to adhere to the laws of physics and exhibit some level of local continuity, where small changes actions or states result in small changes in transitions. While realistic systems may contain discontinuities certain state space portions, the subset exhibiting local continuity proves to be valuable asset.

---

Expert Demo
Distributional Shift
Corrective Labels
MOTIVATION
Demonstrations
Generate Corrective Labels.
Learned Dynamics
PROPOSAL
Local Continuity in Dynamics
KEY INSIGHTS
+
DisturbedAction
BackTrack
APPLICATIONS
Our proposed framework, CCIL. To enhance the robustness of imitation learning, we propose augment the dataset with synthetic corrective labels. We leverage the local continuity regularized dynamics function and generate corrective labels near the expert data support. We provide theoretical guarantees on the quality of the generated labels. We present empirical evaluations CCIL over 14 robotic tasks 4 domains showcase CCIL improving imitation learning agents' robustness disturbances.
Figure 1:
dynamics, learn
Armed with this insight, our goal is to synthesize corrective labels that guide agent encountering unfamiliar states back to the distribution of expert states, Leveraging the presence of local continuity makes the synthesis of these corrective labels more tractable. A learned dynamics model with local Lipschitz continuity can navigate an agent from unfamiliar out-of-distributic states to in-distribution expert trajectories, even extending beyond the expert data support. The local Lipschitz continuity allows the model to have bounded error in regions outside the expert data support, providing the ability to generate appropriately corrective labels.
We propose mechanism for learning dynamics models with local continuity from expert data and generating corrective labels. Our practical algorithm, CCIL, leverages local Continuity in dynamics to generate Corrective labels for Imitation Learning. We validate CCIL empirically on variety of robotic domains in simulation. In summary, our contributions are:
Problem Formulation; We present a formal definition of corrective labels to enhance robustness
for imitation learning. (Sec. 4.1.
Practical Algorithm: We introduce CCIL, a framework for learning dynamics functions, and leveraging local continuity generate corrective labels. Our method is lean assumptions, primarily relying availability of expert demonstrations and the presence of local continuity in the dynamics.
Theoretical Guarantees: We showcase how local continuity in dynamic systems would allow extending learned model's capabilities beyond the expert data. We present practical methods to enforce desired local smoothness while fitting dynamics function to the data and accommodating discontinuity (Sec. 4.2) We provide theoretical bound on quality the model this area and the generated labels (Sec.
Extensive Empirical Validation: We conduct experiments over 4 distinct robotic domains across 14 tasks in simulation, ranging from classic control, drone flying, high- digeedimensional car navigation, legged locomotion and tabletop manipulation. We showcase our proposal's ability to enhance the performance and robustness of imitation learning agents (Sec. 5
RELATED WORK
Imitation Learning (IL) and Data Augmentation. Given only the expert demonstrations, Behavior Cloning (BC) remains a strong empirical baseline for imitation learning (Pomerleau) 1988). formu- lates II. as supervised learning problem and has plethora of data augmentation methods. However,
2

---

(Venkatraman et al. 2013 Bojarski et al. 2016 Florence et al. 2019 Spencer et al. 2021 Zhou et al. 2023), which
previous augmentation methods mostly leverage expert some form of invariance
a domain -peecttcc property. Ke et al. (2021b) explores nosse-based data augmentation but lacks principle for choosing appropriate noise parameters. Park & Wong (022) learns dynamics model for data augmentation, similar our approach. But it learns an inverse dynamics model and lacked theoretical insights. In contrast, our proposal leverages local continuity in the dynamics, is agnostic to domain knowledge, and provides theoretical guarantees on the quality of generated data.
Mitigating Covariate Shift in Imitation Learning. Compounding errors push the agent astray from expert demonstrations. Prior works addressing the covariate shift often request additional information. Methods like DAGGER (Ross et al., 201T), LOLS (Chang et al. 2015), DART (Laskey et al. 2017) and AggrevateD Sun et al. (2117) use interactive experts, while GAIL (Ho & Ermon 2016). SQIL
(Reddy et al.
2019) and AIRI (Fu et all 2017) sample more transitions in the environment to minimize the divergence of states distribution between the learner and expert (Ke et al., 202 a
Swamy et al.
2021p. Offline Reinforcement Learning methods like MOREL (Kidambi et al. 2020
are optimized to mitigate covariate shift but demand access to ground truth reward function. Reichlin
et al. (2022) trains recovery policy to move the agent back to data manifold by assuming that the
dynamics is known. MILO (Chang et al. (02T)) learns a dynamics function to mitigate covariate
shift but requires access to abundant offline data to learn a high-fidelity dynamics function. In contrast, our proposal is designed for imitation learning without requiring additional data or feedback, complementing existing IL methods. We include MILO and MOREL as basclines in experiments.
Local Lipschitz Continuity in Dynamics. Classical control methods often assume local Lipschitz continuity in the dynamics to guarantee the existence and uniqueness of solutions differential equations. For example, the widely adopted C² assumption in optimal control theory (Bonnard et al. 2007) and the popular control framework iLQR (Li & Todorov 2004). This assumption is particularly useful in the context of nonlinear systems and is prevalent in modern robot applications (Seto et al. 1994, Kahveci 2007, Sarangapani 2018). However, most of these methods leveraging dynamics continuity are in the optimal control setting, requiring pre-specified dynamics model and cost function. In contrast, this work focuses on robustifying imitation learning agents by learning a locally continuous dynamics model from data, without requiring human-specified model or cost function.
Learning Dynamics using Neural Networks. Fitting a dynamics function from data is an active
area of research (Hafner et al., 2019 Wang et al., 2022 Kaufmann al. 2023). For continuous states and transitions, Asadh et al. (2018) proved that enforcing Lipschitz continuity in training dynamics model could lower the multi-step prediction errors. Ensuring local continuity in the trained dynamics, however, can be challenging. Previous works enforced Lipschitz continuity in training
neural networks (Bartlett et al.) 2017, Arjovsky al. 2017 MMyato al. 2018), but did not apply it to dynamics modeling or generating corrective labels. [hh et al. (2199) learned smooth dynamics functions via enforcing global Lipschitz bounds and demonstrates on the problem of drone landing. Pfrommer et al. (2021) learned smooth model for fictional contacts for manipulation. While this work is not focusing on learning from visual inputs, works such as Khetarpal al. (220); Zhang et al. 2020); Zhu et al. (2023) are actively building techniques for learning dynamics models from high-dimensional inputs that could be leveraged in conjunction with the insights from our proposal.
Concurrent Works In the final stages of preparing our manuscript, we became aware of concurrent work (Block et al. 2024 that shares similar motivations with our study - to stabilize imitation learning near the expert demonstrations. Instead of data augmentation, their proposal directly generates stabilizing controller. We hope both works can serve as a foundation for future research into robustifying imitation learning.
3 PRELIMINARIES
We consider an episodic finite-horizon Markov Decision Process (MDP), .M- = (. where is the state space, A is the action space, f is the ground truth dynamics function and Po is the initial state distribution. A policy maps a state to distribution of actions E I1 - The dynamics function f can be written as a m apping from a state $, and an action a, at time f to the change to a new state st+1. This is often represented in its residual form: 8141 8 + f(s,, a.). Following the imitation learning setting, the true dynamics function ff is unknown, the reward is unknown, and we only have transitions drawn from the system dynamics.
3

---

We are given set of demonstrations D collection of transition triplets: D =
where 8j, 5 1 E S, 3 E A. We can learn a policy from these traces via supervised learning (behavior cloning) by maximizing the likelihood of the expert actions being produced at the training states: arg max; Rsg (5 ~D- log( (aa) x)))- In practice, we can optimize this objective by minimizing the mean-squared error regression using standard stochastic optimization.
4 GENERATING CORRECTIVE LABELS FOR ROBUST IMITATION LEARNING LEVERAGING CONTINUITY IN DYNAMICS
Our objective is to enhance the robustness of imitation learning methods by generating corrective labels that bring an agent from out-of-distriuution states back to in-distribution states. We first define the desired high quality corrective labels to make imitation learning more robust in Sec. I then discuss how these labels can be generated with known ground truth dynamics of the system (App. A). Without the ground truth dynamics, we discuss how this can be done using learned dynamics model. Importantly, to generate high-quality corrective labels using learned dynamics functions, the dynamics need to exhibit local continuity. We present method to train locally supschizz-bunndee dynamics model in Sec. 4.2 and then show how to use such a learned model to generate corrective labels (Sec. 4.3 with high confidence, beyond the support of the expert training data. Finally, in Section 4.4 we instantiate these insights into a practical algorithm CCIL that improves the robustness of imitation learning methods with function approximation.
We
4.1 CORRECTIVE LABELS FORMULATION
To robustify imitation learning, we aim to provide corrections to disturbances or compounding errors by bringing the agent back the "known" expert data distribution, where the policy is likely to be successful. We generate catte-action-saaee state triplets (s9 ,a,'"' )²) that are corrective: executing action a in state g can bring the agent back state s* support of the expert data distribution.
[Corrective Labels]. (s a s*) is corrective label triplet with starting state s corrective action label a and target state (in the expert data) 8* E if, with respect to dynamics function /.
I|[s9 + f(s (a.) - cc.
(1)
This definition is trivially satisfied by the given expert demonstration (s). a5. $7 Dr. We aim to search for a larger set of corrective labels for outtof-distribution states. If the policy has bounded error on not just the distribution of expert states but on distribution with larger support, Support(d"') Sd is robust against larger set of states that it might encounter during execution. However, without knowledge of the true system dynamics f, we have only an approximation f of the dynamics function derived from finite samples.
Definition 4.1. [High Quality Corrective Labels under Approximate Dynamic Models]. (sC s²)s*) is a corrective label if [|[s@ + f(sE, [(sS,a))] ] s*Il I ce. w.r.. an approximate dynam- ics i for a target state s* E 'D*. Such a label is "high-quality" if the approximate dynamics function also has bounded error w.r.t. the ground truth dynamics function evaluated at the given state-action pair [s a8) f(s0, a S coo-
The high-quality corrective labels represent the learned dynamics model's 's best guess at bringing the agent back into the support of expert data. However, an approximate dynamics model is only reliable in a certain region of the state space, i.e., where the predictions of closely align with the true dynamics of the system, f. When the dynamics model is trained without constraints, such as in maximum likelihood framework, its performance might significantly degrade when extrapolating beyond the distribution of the training data. To address this challenge, will first describe how to learn locally continuous dynamics models from data. Subsequently, we will outline how to utilize these models for high-quality corrective label generation beyond the expert data.
4.2 LEARNING LOCALLY CONTINUOUS DYNAMICS MODELS FROM DATA
Our core insight for learning dynamics models and using them beyond the training data is to leverage the local continuity in dynamic systems. When the dynamics are locally Lipschitz bounded, small
4

---

a
a
8
OR
BackTrack Labels
DisturbedAction Labels
Figure 2: Generating corrective labels from demonstration. Given expert demonstration state 8 and action a" that arrives at Sezt- BackTrack Labels: search for alternative starting state 8 that can arrive at expert state s if it executes expert action a*. DisturbedAAttonn Labels: search for an alternative starting state 8 that would arrive at next if it executes slightly disturbed expert action a = + A, where is sampled noise.
changes in state and/or action yield small changes in the transitions. A dynamics function that exhibit locally continuity would allow us to extrapolate beyond the training data to the states and actions that in proximity to the expert demonstration region in which we can trust the learned model.
We follow the model-based reinforcement learning framework to learn dynamics models from data (Wang et al. 2019). Essentially, we learn a residual dynamics model f(" a;) + 8 s via mean-squared error regression (MSE). We can use any function approximator (e.g., neural network) and write down the learning loss:
arg min Eag P3 [MS]]. MSF = (If( (5)
+11l
A model trained with the MSE loss alone is not guaranteed to have good extrapolation beyond the training data. Critically, modify the above learning objective ensure the learned dynamics model to contain regions that are locally continuous. Previously [Shi et al. (2019) used spectral normalization to dynamies functions that are globally Lipschitz bounded. However, most robotics problems involve dynamics models that are hybrids of local continuity and discontinuity: for instance, making and breaking contacts. face discontinuity, we present a few practical methods to fit dynamics model that (1) enforces much local continuity as permitted by the data and (2) discards the discontinuous regions when subsequently generating corrective labels. Due to space limit, we defer the reader to App. C.I for complete list of candidate loss functions considered. Here, show an example of sampling-bases penalty of violation of local Lipschitz constraint.
Local Lipschitz Continuity via Sampling-hased Penalty. Following Gulrajani et al. (017)) relax the global Lipschitz constraint by penalizing any violation of the local Lipschitz constraint.
MSE + ((IIf(s) 1)]
arg min E3j,0g.3541 f
(3)
(() f(s) + +,, f00-f((3.00) az)
A
We use samples estimate the local Lipschtiz continuity f'() aa))) by perturbing the data points with small sampled Gaussian noises A, ~ N.. Alternatively one can compute the Jacobian matrix to estimate the local Lipschitz continuity. The penalty term weighted by enforces the local Lipschitz constraint at the specific expert datapoint. Doing so ensures that the approximate model is mostly while being predictive of the transitions the expert data. This approximate dynamics model can then be used to generate corrective labels.
Lipschitz-bouut
4.3 GENERATING HIGHI-QUALITY CORRECTIVE LABELS
We employ learned dynamics models to generate corrective labels, leveraging local continuity to ensure the generated labels have bounded error in proximity the expert data 's support. When the local dynamics function is bounded by a Lipschitz constant w.r.t. the state-action space, we can (1) perturb the dynamics function by introducing slight variations in either state action and (2)
5

---

quantify the prediction enror from the dynamics model based on the Lipschitz constant. We outline two techniques, BackTrack label and DisturbedAction label, to generate corrective labels (Fig. 2
Technique 1: BrckTrack Label. Assuming local Lipschitz continuity w.r.t states, given expert state-action pair st dj. we propose to find a different state oh_1 that can arrive at st using action di. To do so, we optimize for a state sy such that:
of f( =
(5)
The quality of the generated label (s9,a,, 8,) is bounded and we present the proof in Appendix B.1 Theorem 4.2. When the dynamics model has a training error of e on the specific data point under the assumption that the dynamics models f and are locally Lipschitz continuous w.r. state with Lipschitz conslant Ki and K2 respectively, if ot is in the neighborhood of local continuity, then
|s (04)) - [58 (1t)) Se+(Ki K2) [2 sit1ll-
(6)
Technique 2: DisturbedAAction. Assuming local Lipschitz continuity w.r.t,, state-action, given an expert tuple (st, of, s1+1), we ask: Is there an alternative action a² that slightly differs from the demonstrated action ai, i.e., a = at + A. that can bring an imaginary state 8 to the expert state i We randomly sample small action noise A and solve for an imaginary previous state sf:
f(s + s;+ = 0.
For every data point, we can obtain a set of labels (sf aj + A) by randomly sampling the noise vector A. We show that the quality of the labels is bounded (proof in App. B.2
Theorem 4.3. Given 8441 1 fos at A) - oy and that the dynamics function is locally Lipschitz continuous w.r.l. states and actions, with Lipschitz constants KA and Ks respectively, then
|| + A i(09, ag A)ll Ka||A|I (1 Ks)[[e||-
(8)
Solving the Root-Finding Equation in Generating Labels Both our techniques need solve root-finding equations (Bq. 5 and 7) This suggests an optimization problem: arg min,e ||s@ f(s9, aه) -*" r0|| given f and a s. There are multiple ways to solve this optimization problem (e.g..
gradient descent or Newton's method). For simplicity, we choose fast-to-compute and conceptually simple solver, Backward Fuler, widely adopted in modern simulators (Tooorov et al. 2012). It lets us use the gradient of the next state to recover the earlier state. To generate data given a s", Backward Euler solves a surrogate equation iteratively: 8e + s - fss",a))
Rejection Sample. We can reject the generated labels if the root solver returns solution that result in large error bound, i.e., ||s sf > e, where is hyper-parameed Rejecting labels that are outside a chosen region lets us directly control the size of the resulting error bound.
4.4 CCIL: CONTINUITY-BASED CORRECTIVE LABELS FOR IMITATION LEARNING
We instantiate practical version of our proposal, CCIL Coontinuty--baeed data augmentation to generate Corrective labels for Imitation Learning), as shown in Alg with three steps: (1) fit an approximate dynamics function that exhibit local continuity, (2) generate corrective labels using the learned dynamics, and (3) use rejection sampling to select labels with the desired error bounds. We highlight the difference combine it with the expert data and train policies using standard behavior cloning on the augmented dataset. We evaluate the augmented agent empirically test whether training with synthetic corrective labels would help imitation learning.
our two techniques in color. To use the generated data D we simply
5
EXPERIMENTS
We evaluate CCIL over 4 simulated robotic domains of 14 tasks to answer the following questions: Q1. Can empirically verify the theoretical contributions the quality of the generated labels;

---

Algorithm 1 CCIL: Continuity-based data augmentation for Corrective labels for Imitation Learning
1: Input: D* = (s) $741)
10: Function LearnDynamics f
2: Initialize: DM Ø
11: Optimize a chosen objective from Sec.
3: LearnDynamics f
12: Function
4: for 1..n do
13:
a
5: (sf, a.) t GenLabels (8) 08, ((1)
14:
F arg m ||s/ + f(s aa, stll
6: if | *ill < then
15: Function
7: D D U (s, 167,aa )
16:
az + A, A ~ NO
8: end if
17: oy arg min, |lo@ + ffs s41ll
argarin
9:
end for
Q2. How well does CCIL handle discontinuities in environmental dynamics;
Q3. How would training with CCIIL-generated labels affect imitation learning agents' performance.
We compare CCIL to standard Behavior Cloning (BC) (Pomerleat 1988), NoisyBC (Ke et al. 202 MILO (Cang et al. 202T) and MOREL (Kidambi al. 2020) using the same model architecture across all methods, over 10 random seeds. Note that MOREL requests additional access to reward function. Appendix, D contains details reproduce the experiments.
5.1 ANALYSIS ON CLASSIC CONTROL OF PENDULUM AND DISCONTINUOUS PENDULUM
Angle
-2269
OCTL
-2536
-2744
Demo Trajctury
-3000
Generated States
-3004
Generated Action
Steps
500
Target
3500
Continunus
Discontinazus
(a) The Discontinuous Pendulum
(b) Example generated labels
(c) Performance Comparison
Figure 3: Evaluation the Pendulum and Discontinuous Pendulum Task.
We consider the classic control task, the pendulum, and also consider a variant, The Discontinuous Pendulum, by bouncing the ball back a few chosen angles, as shown in Fig. 3a
Verifying the quality of the generated labels (Q1). We visualize a subset of generated labels in Fig. 3b Note that the generated states (red) are outside the expert support (blue) and that the generated actions are torque control signals, which are challenging for invariance-batt methods to generate. To quantify the quality of the generated labels, we use the fact that ground truth dynamics can be analytically computed for this domain and measure how closely our labels can bring the agent the expert. We observed an average L.2 norm distance of 0.02367 which validates our derived theoretical bound of 0.065: K1|8| + (1 + K2) = 12 * 0.0001 + 13 0.005 (Equation 6
data augmentation
The Impact Local Lipschitz Continuity Assumption (Q2) To highlight how local discontinuity in the dynamics affects CCIL, we compare CCIL. performance in the continuous and discontinuous pendulum in Fig. 3e CCIL improved behavior cloning performance even when discontinuity is present, albeit with smaller margin for the discontinuous Pendulum. In ablation study, also tried generating labels using a naive dynamics model (without explicitly assuming Lipschitz continuity) which performed slightly better than vanilla behavior cloning but worse than CCIL, highlighting the importance of enforcing local Lipschitz continuity in training dynamics functions for generating corrective labels.
CCIL improved behavior cloning agent (Q3). For both the continuous and discontinuous Pendulum, CCIL outperformed behavior cloning given the same amount of data (Fig. 3c

---

5.2
DRIVING WITH LIDAR: DIGE-DIMENSIONAL STATE INPUT
Method
Succ. Rate
Avg. Score
Expert
100.0%
1.00
Morel
0.0%
0.001 + 0.001
MILO
0.0%
0.21 0.003
BC
31.9%
0.58 1: 0.25
NoiseBC
39.3%
0.62 + 0.28
CCIL
56.4%
0.75 + 0.25
Figure Fltenth
Figure 5:
LiDar POV
Table 1: Performance on Racing
We compare all agents the Fitenth racing simulator (Fig. D. which employs high-dimensional, LiDAR sensor as input (Fig. 5) We evaluate each agent over 100 trajectories. At the beginning of each trajectory, the car is placed random location on the track. It needs to use the LiDAR input to decide on speed and steering. earning scores for driving faster or failing by crashing.
CCIL can improve the performance of behavior cloning for hign-dimensional state inputs (Q2, Q3). Table. shows that CCIL demonstrated an empirical advantage over all other agents, achieving higher success rate and better score while having fewer crashes. Noticeably, LiDar inputs can contain highly-complicater discontinuities and impose challenges to model-basee methods (Morel and MILO), while CCIL could reliably generate corrective labels with confidence.
5.3 DRONE NAVIGATION: HIGH- BIGH-FREEUENCY CONTROL TASK AND SENSITIVE TO NOISES
Method
Hover
Circle
Fly FlyThrough
Expert
-1104
-10
-4351
BC
-1,0888
-9.5677
-1.0688
NoisyBC
-1.13E8
-9.88E7
-1.07E8
Morel
-1.25E8
-1.24E8
-1,25E8
MILO -1.26E8
-1.25E8
-1,25E8
CCIL
-0.9618
-8.03E7
-0.78E8
Figure 6: Drone Flying Tasks
Table 2: Performance on Drone
Drone navigation is high-frequency control task and can be very sensitive noise, making an appropriate testbed for the robustness of imitation learning. We use an open-source quadcopter simulator, gym-py cym--ybullet-dron
(Panerati et 202T) and consider three proposed tasks: hover, circle, fly-through-gate, shown in Fig. 6 Following Shi et al. (209), we inject observation and action noises during evaluation to highlight the robustness of the learner agent.
CCIL improved performance for imitation learning agent and its robustness to noises (Q3).
CCIL outperformed baselines by a large margin.
5.4 LOCOMOTION AND MANIPULATION: DIVERSE SCENES WITH VARYING DISCONTINUITY
Manipulation and locomotion tasks commonly involve complex forms of contacts, raising consider- able challenges for learning dynamics models and for our proposal. We evaluate the applicability of CCIL in such domains, considering tasks from MuJoCo locomotion suite: Hopper, Walker2D, Ant, HalfCheetah and 4 tasks from the Meta World manipulation suites: CoffeePull, ButtonPress, CoffeePush, DrawerClose. During evaluation, we add a small amount of randomly sampled Gaussian noise the sensor (observation state) and the actuator (action) to simulate real-world conditions of robotics controllers and to test the robustness of the agents.
CCIL outperforms behavior cloning or at least achieves comparable performance even when varying form of discontinuity is present (Q2, Q3). On out of tasks considered, CCIL outper- forms all baselines. Across all tasks, CCIL at least achieves comparable results to vanilla behavior cloning, shown in Table. B indicating that it's a effective alternative to behavior cloning without necessitating substantial extra assumptions.
8

---

Table Evaluation results for Mujoco and Metaworld tasks with noise disturbances. We the expert scores a noise-free setting for reference. In the face of varying discontinuity from contacts, CCLL remains the leading agent on 4 out of 8 tasks (Hopper. Walker. HalfCheetah, CoffeePull). Comparing CCIL with BC: across all tasks, CCIL. can outperform vanilla behavior cloning at least achieve comparable performance.
Majoco
Metaworld
Hopper
Walker
Ant
Hallchectal:
CalfcePull
ButtoaPress
CoffeePush
DrowerClose
Expest
$234,30
4592.30
3879:70
12135.00
4409)
3895 82
4488.29
4529 34
BC
1983 + 672 66
1922.55
141009
2965 20
202.1
1798.98 + 791
3552,
+233,44
3693,02 104,99
1288: 19+ 746,
3247
468.13
Moeel
152, 19:534.12
20.27
3.59
1000.77
15:21
-2.24 = 0.02
18.76+00.00
14.85=17.08
18.664 0.02
1222.23dt
1241.47
MILO
S66 96: 100.32
526
12:772
1006
13-160433
151.06::11.00
230,49:
110.44
966
106-466:15..79
230.62 19:37
4621.11-:9966
NosseBC
1563.55 56 1012.01
2893:21
+ 107639
3776.6
442.13
2944.24
1:91,599
31722.86
4: 78878888 91
3663.4463.. 10
2551.11 857.79
4226.71= 18.90
CCIL
2631.25 300.86
1538.48
573.23
2338.35
474.17
8893.81
1 472.70
4168..6
1: 192.98
3775.22191.24
2484.191
976.03
4145.452 76.23
5.5 SUMMARY
We conducted extensive experiments to address three research queries. For Q1, we verified the proposed theoretical bound on the quality of the generated labels on the Pendulum task. For Q2, tested that CCIL is robust to environmental discontinuities, improving behavior cloning in both con- tinuous (Pendulum, Drone) and discontinuous scenarios (Driving, MuloCo). For Q3, CCIL emerged as the best agent on 10 out of 14 tasks. In the rest of tasks, it achieved comparable performance behavior cloning, demonstrating its cost-effectiveness and efficiency as an alternative approach without requiring significant additional assumptions.
6 CONCLUSION
We address the problem of encouraging imitation learning agents to adhere closely to demonstration data support. We introduce corrective labels and propose CCIL, a novel framework that leverages local continuity in environmental dynamics to generate corrective labels. The key insight is to use the expert data train a dynamics model that exhibits local continuity while skipping the discontinuous regions. This model, in turn, can augment imitation learning with corrective labels. While the assumption is that system dynamics exhibiting some local continuity, we've validated the effectiveness of our method on various simulated robotics environments, including drones, driving, locomotion and manipulation.
Limitation: Optimality of Corrective Labels While corrective labels constrain the learner distri- bution within the expert distribution, similar to stabilizing controller, they may not guarantee an optimal solution. There's 's possibility that following corrective labels confines the agent to a limited subset of expert-visited states, potentially leading to task failure.
However, we view our approach as a catalyst for promising future research:
Quantifying Local Continuity: Further exploration into quantifying and capturing local continuity in the dynamics of robotic systems, especially through data-driven approach. Optimizing Dynamics Model Training: The quality of the learned dynamics model directly affects the efficacy of generated labels, necessitating research into learning dynamics models while preserving continuity.
Handling Complex Discontinuities: The application of CCIL in domains with intricate discontinuities. such as pixel- bisel images, remains an open area for investigation. Solutions may involve compact state representations and smoothness in latent spaces.
Real-world Robotics Application: Extending our findings real-world robotics presents an exciting frontier for future research, offering the potential for practical implementation and impact.
ACKNOWLEDGMENTS
We thank Sanjiban Choudhury for detailed feedback on the manuscript; Jack Umenberger for enlightening discussions; Khimya Khetarpal, Sidharth Talia and folks at the WEIRDLab at UW for edit suggestions. This work was (partially) funded by the National Science Foundation NRI (#2132848) and CHS (#200701 I), the Office of Naval Research 17-10-4---1-26117 #2022-016-01 UW), and Amazon.
and

---

REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 214-22% PMLR, 2017.
Kavosh Asadi, Dipendra Misra, and Michael I.. Littman. Lipschitz continuity in model-based reinforcement learning. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML. 2018, Stockholmsmüsssan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 264-273. PMLR, 2018, URL http::/proceediing
m
Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky, Spectrally-normalln margin bounds for neural networks. NeurIPS, 30, 2017.
Adam Block, Ali Jadbabaie, Daniel Pfrommer, Max Simchowitz, and Russ Tedrake. Provable guarantees for generative behavior cloning: Bridging low-level stability and high-level behavior. Advances in Neural Information Processing Systems, 36, 2024.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp. Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars. 0rX1v:1604.07316 2016.
Bernard Bonnard, Jean-Bu Baptiste Caillau, and Emmanuel Trélat. Second order optimality conditions in the smooth case and applications in optimal control. ESAIM: Control, Optimisation and Calculus of Variations, 13(2):207-2366 2007.
Jonathan D Chang, Masatoshi Uehara, Dhruv Sreenivas, Rahul Kidambi, and Wen Sun. Mitigating covariate shift in imitation learning via offline data without great coverage. arXiv preprint arXiv:2106.03207 2021.
Kai-Wei Chang, Akshay Krishnamurthy, Alekh Agarwal, Hal Daumé III, and John Langford. Learning to search better than your teacher. In International Conference on Machine Learning, 2015.
Zoey Qiuyu Chen, Karl Van Wyk, Yu-Wei Chao, Wei Yang, Arsalan Mousavian, Abhishek Gupta, and Dieter Fox. Learning robust real-world dexterous grasping policies via implicit shape augmentation. arXiv preprint 10. 13638, 2022.
Pim de Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion imitation learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancou- ver, BC, Canada, pp. 11693 1 1704, 2019. URL URL/htpp::://rrceeee..... wtw.:eeeeddddd..e...........p..p cc/
Abstract.
Pete Florence, Corey Lynch, Andy Zeng, Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. In Conference on Robot Learning, pp. 158-168. PMLR, 2022.
Peter Florence, Lucas Manuelli, and Russ Tedrake. Self-supervised correspondence in visuomotor policy learning. IEEE Robotics and Automation Letters, 5(2):492 4992224997700 2019.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforce- ment learning. arXiv preprint arXiv: 1248, 2017.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint 191iv:1912.01600 2019.
10

---

Marti A. Hearst, Susan T Dumais, Edgar Osuna, John Platt, and Bernhard Scholkopf. Support vector machines. IEEE Intelligent Systems and their applications, 13(4):18-22 28, 1998.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. NeurlPS, 29, 2016.
Nazli E Kahveci. Robust Adaptive Control For Unmanned Aerial Vehicles. PhD thesis, University of Southern California, 2007.
Elia Kaufmann, Leonard Bauersfeld, Antonio Loquercio, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza. Champion-level drone racing using deep reinforcement learning. Nature, 620 (7976):982-987, 2023.
Liyiming Ke, Sanjiban Choudhury, Matt Barnss, Wen Sun, Gilwoo Lee, and Siddhartha Srinivasa. Imitation learning f-divergence minimization. In Algorithmic Foundations of Robotics XIV: Proceedings of the Fourteenth Workshop on the Algorithmic Foundations of Robotics 14, pp. 313-329. Springer, 2021a.
Liyiming Ke, Jingqiang Wang, Tapomayukh Bhattacharjee, Byron Boots, and Siddhartha Srinivasa. Grasping with chopsticks: Combating covariate shift in model-free imitation learning for fine manipulation. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 6185-6191. IEEE, 2021b.
Khimya Khetarpal, Zafarali Ahmed, Gheorghe Comanici, David Abel, and Doina Precup. What can do here? theory affordances reinforcement learning. International Conference on Machine Learning, pp. 5243-5253, PMLR, 2020.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. Advances in neural information processing systems, 33: 21810-21823, 2020.
Michael Laskey, Jonathan Lee, Roy Fox, Anca Dragan, and Ken Goldberg, Dart: Noise injection for robust imitation learning. In Conference on robot learning. pp. 143-156. PMLR, 2017.
Weiwei Li and Emanuel Todorov. Iterative linear quadratic regulator design for nonlinear biological movement systems. ICINCO (I), pp. 222-229. Citeseer, 2004.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for 2018.
generative adversarial networks. arXiv preprint
Felipe Dennis de Resende Oliveira, Eduardo Luiz Ortiz Batista, and Rui Seara. On the compression of neural networks using 0-norm regularization and weight pruning. arXiv preprinl arXiv:2109.05077 2021.
Jacopo Panerati, Hehui Zheng, SiQi Zhou, James Xu, Amanda Prorok, and Angela P Schoellig. Learning to fly- gym environment with pybullet physics for reinforcement learning of multi- agent quadcopter control. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7512-7519. IEEE, 2021.
Jung Yeon Park and Lawson Wong. Robust imitation of few demonstrations with backwards model. Advances in Neural Information Processing Systems, 35:19759-19772 2022.
Samuel Pfrommer, Mathew Halm, and Michael Posa. Contactnets: Learning discontinuous contact dynamics with smooth, implicit representations. In Conference Robot Learning, pp. 2279-2291. PMLR, 2021.
Dean A Pomericau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems, 1, 1988.
Rouhollah Rahmatizadeh, Pooya Abolghasemi, Ladislau Bölöni, and Sergey Levine. Vision-based multi-task manipulation for inexpensive robots using end-to-end learning from demonstration. In 2018 IEEE international conference on robotics and automation. IEEE, 2018
Nathan D Ratliff, Andrew Bagnell, and Martin A Zinkevich. Maximum margin planning. In Proceedings of the 23rd international conference Machine learning, pp. 729-736. 2006.

---

Siddharth Reddy, Anca D Dragan, and Sergey Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards. arXiv preprint 10XXvv1905.111000 2019.
Alfredo Reichlin, Giovanni Luca Marchetti, Hang Yin, Ali Ghadirzadch, and Danica Kragic. Back to the manifold: Recovering from out-of-distribution states. In International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference artificial intelligence and statistics, pp. 227-633, JMLR Workshop and Conference Proceedings, 2011.
Jagannathan Sarangapani. Neural network control of nonlinear discrete-time systems. CRC press, 2018.
Danbing Seto, Anuradha Annaswamy, and John Baillicul. Adaptive control of nonlinear systems with triangular structure. IEEE Transactions Automatic Control, 39(7):1411-1428, 1994.
Guanya Shi, Xichen Shi, Michael O'Connell, Rose Yu, Kamyar Azizzadeneshelii Animashree Anandkumar, Yisong Yue, and Soon-Jo Chung. Neural lander: Stable drone landing control using learned dynamics. In 2019 International Conference on Robotics and Automation (ICRA), pp. 9784-9790. IEEE, 2019.
David Silver, James Bagnell, and Anthony Stentz. High performance outdoor navigation from overhead data using imitation learning. Robotics: Science and Systems IV, Zurich, Switzerland, 1, 2008.
Jonathan C. Spencer, Sanjiban Choudhury, Arun Venkatraman, Brian D. Ziebart, and J. Andrew Bag- nell. Feedback imitation learning: The three regimes covariate shift. CoRR, abs/2102.02872. 2021. BRRLattp.. wttpaaaxvvvvv...... org. abs /2102/1122 02872)
/ttps://arxiv.org)
Wen Sun, Arun Venkatraman, Geoffrey Gordon, Byron Boots, and J Andrew Bagnell. Deeply aggrevated: Differentiable imitation learning for sequential prediction. In International conference on machine learning, pp. 3309-3301333 3318. PMLR, 2017.
Gokul Swamy, Sanjiban Choudhury, J Andrew Bagnell, and Steven Wu. Of moments and matching: A game-theoretic framework for closing the imitation gap. In International Conference on Machine Learning, pp. 10022- 10032. PMLR, 2021.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-base control. In 2012 JEEE/RSJ international conference on intelligent robots and systems, 2012.
Arun Venkatraman, Martial Hebert, and Andrew Bagnell. Improving multi-step prediction of learned time series models. In Twent-Nintt AAAI Conference on Artificial Intelligence, 2015.
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model- reinforcement learning. arXiv preprint arXiv: 00050007.......... 2019.
Tongzhou Wang, Simon S Du, Antonio Torralba, Phillip Isola, Amy Zhang, and Yuandong Tian. De- noised mdps: Learning world models better than the world itself. arXiv preprint arXiv:2206.15 154777 2022.
Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint arXiv:2006.10742 2020.
Allan Zhou, Moo Jin Kim, Lirui Wang, Pete Florence, and Chelsea Finn. Nerf in the palm of your hand: Corrective augmentation for robotics via novel-view synthesis. arXiv:2301.088 088556 2023.
Gaoyue Zhou, Liyiming Ke, Siddhartha Srinivasa, Abhinav Gupta, Aravind Rajeswaran, and Vikash Kumar. Real world offline reinforcement learning with realistic data source. arXiv preprint trXiv:2210.064777 2022.
Chuning Zhu, Max Simchowitz, Siri Gadipudi, and Abhishek Gupta. Repo: Resilient model- -bsed reinforcement learning by regularizing posterior predictability. wrxXiv2339.000822 2023.
12

---

A GENERATING CORRECTIVE LABELS WITH KNOWN DYNAMICS FUNCTION
With a known dynamics function, we show an example algorithm that one can use to generate N corrective labels (as defined in See 11 The algorithm first trains behavior cloning policy, samples test time roll-out trajectories from the learned policy, and then derives labels using a root-finding solver.
Algorithm 2 Generating Corrective Labels using Dynamics Function
1: Input Expert Data D* (5) st41).
2: Input Dynamies function [(8 18 as',
3: Input Parameter N.
4: Initialize DD + - Ø
5: ft arg min; - -ap,0 pDD log(ft(;} s())
6: for i E 1..N do
7: =i Po, 29 ~ [(s) n (()))) - s' -
8: end for
9: for E E. do
of + arg min I1ss + ffs f(s) stil
D9 < 4D u Usf ag)
12: end for
13: return D
B PROOFS
B.1 PROOF OF THEOREM
Notation. Let f be the ground truth 1-step residual dynamics model, and let ff be the learned approximation of f. Given a demonstration 4141). we have generated a corrective label
(8 11+1).
Assumptions:
1. The estimation error of the learned dynamics model at the training data is bounded.
[s a(+1) f( 07+1
S
2. f is locally K₁ KiiLishchitz in neighborhood Ô around (51+1,4141) i.e. for oy Ū: [(0 ((sf.at+1) at+1) (((+1) < Ki 1| -
3. f is locally K₂ Li-Lhhhhi neighborhood U around (5t+1, a(41). i.e., for of E U:
|| ( - f8//+1 a < K2 || 1
4. G is within the region of local continuity around 8741 for both f and f: of Usi,, Usips
Proof:
(1s),at1)) - f(s9,att1)) ait1)
= =1|1(oi, ((+1) 5( ((41) (sit1 f(siz a(+1) + - (of,
< [|]((s], ((+1) f( |(+1))l ((siin ((
f(s.
S (K1 + K2) 100 ا
Remark Our assumption (1) about the error of the learned dynamics model is not a global constraint but simply requires the model have small prediction error at the specific data point. Our
13

---

assumptions (2) and (3) about the size of the local Lipschitz continuity is not requirement on the global Lipschitz continuity in the dynamics, but simply requires space near the expert support to exhibit local continuity. Our proof leverages simple triangle inequality and is valid only near expert data support.
B.2 PROOF OF THEOREM
Notation. Let f be the ground truth 1-step residual dynamics model, and let f be the learned approximation of f. Given demonstration (st oj s(+1), we have generated a corrective label
(s9 of A, $
Assumptions
1. f is locally Kg-Lipschitz in state in a neighborhood Us around (s7, az). i... for of Us:
[f a f(ss, at)ll - Ks || -
st|l-
2. f is locally Kx-Lipschitz in action in neighborhood UA around ag). i.e., for aj + E UA:
[sf a.) - [(+ oz + A) - Ka |||..
3. Using rejection sampling, we can enforce -s $ S Crej.
4. Given that the generated labels come from root solver:
i1+ f(s aj + A) oe = Eapt where Eopt + ot [( ai) f(,), af + A) of tapt
a,) f(s a + +A) of s + Copt
5. The corrective label is within the neighborhood of local Lipschitz continuity of f: I Ug and of + A UA
Proof
[( at + + - f(0) of + A)
||1(08, ot + + (588 + f(s) at) (() [s) f(8 az A)
[[f(s] at + + f()),,))) + f - [(s] f(s aF A)
I KA ||||- + Kg -56 s; + || - s; + ||copell
3 KA ||||| + (1 + Ks) Erej + ||Eopel|
When the root solver yields solution with |/copel| have S KA KAll + (1 Ks) Crej
C DETAILS FOR CCIL
Our proposed framework for generating corrective labels, CCIL, takes three steps:
1. Learn a dynamics model: fit a dynamics model that is locally Lipschitz continuous.
2. Generate labels: solve root-finding equation in Sec. 4.3 to generate labels.
3. Augment the dataset and train a policy: We use behavior cloning for simplicity to train policy.
C.I LEARNING A DYNAMICS FUNCTION WHILE ENFORCING LOCAL LIPSCHITZ CONTINUITY
There are many function approximators to learn a model. For example, using Gaussian process can produce smooth dynamics model but might have limited scalability when dealing with large amount
14

---

of data. In this paper demonstrate examples of using neural network to learn the dynamics model. There are multiple ways to enforce Lipschitz continuity on the learned dynamics function. with varying levels of strength that trade off theoretical guarantees and learning ability. In Sec. 4.2
we discuss a simple penalty to enforce local Lipschitz continuity, here we provide a few alternative methods, including the one used by [hh et al. (2019) and the one inspired by /Gulrajani al. (2017).
Global Lipschitz Continuity via Spectral Normalization. Using spectral norm with coefficient provides the strongest guarantee that the dynamic model is global L-Lipschitz. Concretely, spectral normalization Miyato et al. (2018) normalizes the weights of the neural network following each gradient update. Shi et (2019) used this method to train a dynamies function for drone navigation.
W 1a
arg min E f 03.00
MSE
while W + W/ max
1)
(9)
However, spectral normalization enforces global Lipschitz bound. may hinder the model's ability to learn the true dynamics.
Local Lipschitz Continuity via Sampling- Penalty. Following Guirajani et al. (011 7), we can relax the global Lipschitz continuity constraint by penalizing any violation of the local Lipschitz constraint.
arg min Egg 133.03.0011000 I MSE A EAYON max (f's) a.:
(10)
To estimate the local Lispehitz continuity, we can use a sampling- based method, EA, max (f'(s) As,a)) I, O), which is indicative of whether the local continuity constraint is violated for given state-action pair. The sampling- penalty perturbs the data points by some sampled noise and enforces the Lipschitz constraint between the perturbed data and the original using penalty term in the loss function.
Doing ensures that the approximate model is mostly L Lipschitz- Lipschitz-bounded while being predictive of the transitions in the expert data. This approximate dynamics model can then be used to generate corrective labels.
Local Lipschitz Continuity via Slack-variable. We can allow for a small amount of discontinuity in the learned dynamics model in the same way that slack variables are modeled in optimization problems, e.g., SVM (Hearst et al., 1998) or max-margin planning with slack variables (Ratiifi al. 2006). With these insights in mind, we can reformulate the dynamics model learning objec- live as learning models that maximize likelihood (MSE) while minimizing the Lipschitz constant (Lipschitz).
arg min Bog.ng B MSE X;- Lipschitz(s3,a)) ||A; \l1o
where || - Allo ~ 1 - exp (- -1) and Lipschitz a.) EA, ~ max (f'5) As, I,,)
(1I)
To account for small amounts of discontinuity, we introduce a state-dependent variable X, to allow violation of the Lipschitz continuity constraint. We minimize the number of non-zero entries in the slack variables (as noted by the LO norm, || - Xllo) to ensure the model is otherwise as smooth as possible besides small amounts of local discontinuity. Building use practical, differentiable approximation for the I.0 norm method, since the 1.0 norm itself non-differentiable. Equipped with these locally continuous learned dynamics models, can then generate corrective labels for robustifying imitation learning.
Oliveira et al. (202T), we can
Local Lipschitz Continuity via Weighted Loss. Since we are optimizing continuous function approximators, if the data contain subsets of discontinuity, such regions are likely to induce high training loss. We can create slack variable for each data point, λy. to down- down-weight the those regions. We then reformulate the dynamics model learning objective as learning models that maximize weighted likelihood (MSE). is the Sigmoid function:
15

---

|
arg min
(Az)) MSE
Co(A))
while W - W/ max
1)
(12)
f
Intuitively, we expect that spectral normalization tends to work better for simpler environments where the ground truth dynamics global Lipschitz with some reasonable L, whereas the soft constraint should be better suited for data regimes with more complicated dynamics and discontinuity. remains research question how best train dynamics function for each domain and representation with different physical properties.
C.2 GENERATING CORRECTIVE LABELS
In Sec. 4.3 we discuss two techniques to generate corrective labels. Depending on the structure of the application domain, one can choose to generate labels either by backtrack or by disturbed actions. Both techniques require solving root-finding equations (Eq. and Eq. 7). To solve them, we can transform the objective to an optimization problem and apply gradient descent.
Eq. specifies the root finding problem for backtrack labels.
f(5 07) 5 =0
Given si of and the learned dynamics function f. we need to solve for sf that satisfies the equation. We can instead optimize for
arg min 118t 1(0 1h11l
(13)
Similarly, we can transform Eq.
to become
arg min |s + f(08 + sttil
(14)
af With access to the trained model f and its gradient d24 one can use any optimizer. For simplicity, we use the Backward Euler solver that apply an iterative update 2a + of s. af where s is a step size. auy
We repeat the update until the objective is within threshold ||s + f(s at + A) si-1ll S Copt-
C.3 USING THE GENERATED LABELS
There are multiple ways use the generated corrective labels. We can augment the dataset with the generated labels and treat them as if they are expert demonstrations. For example, for all experiments conducted in this paper, we train behavior cloning agents using the augmented dataset. Optionally, one can favor the original expert demonstrations by assigning higher weights to their training loss. We omit this step for simplicity in this paper.
query trained imitation learning policy with the generated labels and measure the difference between our generated action and the policy output. This difference can be used as an alternative metric to evaluate the robustness of imitation learning agents when encountering subset of out-of -iit-----itriiutioon states. However, for a query state, our proposal does not necessarily recover all possible corrective actions. We defer exploring alternative ways of leveraging the generated labels to future work.
Alternatively,
D EXPERIMENTAL DETAILS
We provide details to reproduce experiments, including environment specification, expert data, parameter tuning for our proposal and details about the baselines. We will also open-source the code and the configuration for each experiment, the proposal is published.
16

---

D.1 ENVIRONMENT AND TASK DESIGN
We conduct experiments on different domains and 8 robots, including the pendulum, a drone, car, four robots for locomotion and one robot arm for manipulation. We consider 14 tasks: the pendulum, a modified pendulum swing task with discontinuity, three drone navigation tasks (fly- through, circle, hover). one LiDar racing task on Fltenth, four MuJoCo tasks (Hopper, HalfCheetah. Ant. Walker2D) and 4 MetaWorld tasks (coffee-pull, button-press- press- topdoww,, coffee-push. drawer- close). The Fitenth, MuloCo and Metaworld environments are from open source implementations, and the drone environment is from a slightly modified open source implementation, discussed below. will also describe how we set up the Pendulum environment and how we modify to test our method with discontinuity.
Pendulum Formulation. The pendulum environment asks policy to swing a pendulum up to the vertical position by applying torque. The properties of the system are controlled by the constants g. the gravitational acceleration, and 1, the length of the pendulum. all experiments, we take = 9.81 and 1.
pendulum's 's state characterized by 8, the current angle, and 6, the current angular velocity. To avoid any issues regarding angle representation,
do not directly store in the state representation; A policy can control the system by
instead, we parameterize the state as 8 ssi 0 cos applying torque the pendulum, which we represent as scalar a, which is clamped to the range []..
The continuous-ting dynamics function given by:
0 cos 0
ds di - ((s,a)=
0 sin 0
sin 0
This continuous dynamics model is then discretized to timestep of 0.02 seconds using RK4. Addi- tionally, although not required by the algorithms we study, we create the following reward function. where 9 is the normalized pendulum angle in the range [0, 2TT) metricize policy performance:
0 r(s,a) = 1100
1
0 11: a
Expert Formulation for Pendulum. We formulate the expert policy using a combination of LQR and energy shaping control, where LQR is applied when the pendulum is near the top and energy- shaping is applied everywhere else. Note that the LQR gains were calculated by linearizing the dynamics around n, along with the cost function c(s, a) -r(s, a). So, the expert policy has the form:
20.11 - T) 7.008
0 - 0.1
Te (s)
á 1100 9.81 cos 0 9 otherwise.
Discontinuous Pendulum. We create wall in the Pendulum environment create local disconti- nuity. When the ball hits the wall, we revert the sign of its velocity, creating discontinuity in the dynamics.
Drone. The drone environment used in our experiments is from a slightly modified fork of the original gym-py cern--ybulle--drmee project. There are two notable modifications, Firstly, we added the circle task, where the agent aims to move in a circular trajectory. Second, we removed the floor, which better simulates a completely airborne drone and reduces training complexities caused by crashing into the floor.
D.2 DEMONSTRATION DATA
To feed expert data to train imitation learning agents, we design expert policies for the pendulum. For all other environments, the expert data from the D4RL dataset Fu al. (2020). For drone
17

---

Environment
Trajectories
Pendulum
50
Discontinuous Pendulum
500
Fltenth Racing
1
Drone hover
Drone circle
30
Drone fly-through
MuloCo Ant
10
MuJoCo Walker2D
MuJoCo Hopper
25
MuJoCo HalfCheetah
MetaWorld all tasks
50
Table 4: Number of expert demonstration trajectories used in our experiments. We limit the amount of expert data to avoid making the task trivially solvable by naive behavior cloning.
environments, we first generate bunch of via points alongside the target trajectories and then use low-level PID) controller to hit the via points one by one.
We note that it is possible to solve most tasks with naive behavior cloning if we feed them with sufficient number demonstrations. We thus limit the number of demonstrations we use for all tasks, as shown in Table. 4
D.3 PARAMETER TUNING.
Our proposal first trains a dynamics model. In this process, the most important L, the enforeed local Lipschitz smoothness per NN layer. Additional hyper-parameterr depend the exact protocol used to train the dynamics function and might include: À (weight of the Lipschitz penalty) and (size of perturbation for estimating local Lipschitz continuity).
is
We first fit a dynamics model without enforcing any Lipschitz smoothness, to obtain an average prediction error for reference. To enforce local Lipschitz continuity, we then adopt the sampling- based penalty and train a series of dynamics models by sweeping parameters, L [2, 5, 10), soft dynamics [0.3, 0.5] and [o. [0.0001, 0.0003, 0.0005].
To choose a dynamics model for generating labels, we will follow Theorem. and Theorem. We note that the local Lipschitz bound of a neural network the product of the Lipschitz bound of each layer. Given that we are using two-layer NN to train our dynamics model, 7. = T.2. We denote the empirical prediction error on the validation set for each trained dynamics model as c. We choose the best dynamics model that has the smallest error bound. For Theorem, 4.2 we optimize for e + 2L ||C']| where a constant that we pick to be the average 81+1 s across the training data. optimize for 0.001 L (1 + L)e.
For Theorem. 4.3
To generate corrective labels, we pick the dynamics model, the size of the perturbation (Sigma) and the rejection threshold (Epsilon). Empirically, Sigma 0.00001 and Epsilon 0.01 worked for all our environments. It is also possible to fine-tune Epsilon, the rejection threshold, for each task and each trained dynamics model to optimize the error bound. In experiments, omit this step for simplicity.
After generating corrective labels, two-layer MLPs (64,64) plus ReLu activation to train Behavior Cloning agent with both original and augmented data.
D.4 BASELINES
We evaluate the following algorithms to gain a thorough understanding of how our proposal compares to relevant baselines.
EXPERT: For reference, we plot the theoretical upper bound of our performance, which is the score achieved by expert during data collection.
18

---

BC: naive behavior cloning agent that minimizes the KL divergence given dataset.
NOISEBC: modification to naive behavior cloning that injects a small disturbance noise to the imput state and reuses the action label, as described in Ke et al. (2221b).
MOREL: Morel Kidambi et al. (2020) trains an ensemble of dynamics functions and uses the learned model for model- reinforcement learning. It uses the variance between the model output as proxy estimation of uncertainty to stay within high confidence region. We use the author implementation.
MILO: MILO Chang et al. (2021) learns dynamic function from given offline dataset in an attempt to mitigate covariate shift. We use the author's implementation in our experiment. MILO traditionally requires a large batch of offline dataset to train a dynamics function, we use only the expert demo dataset to train dynamics function.
CCIL:: our proposal to generate high-quality corrective labels.
19


Reviews:
[{"review_id": "n4nllYlCOT", "reviewer": "Reviewer_cJ4u", "summary": "This work presents a method for augmenting imitation learning data by learning locally lipschitz-continuous dynamics models and then generating additional labels by perturbing the action to find noisy states as well as tracing states that would lead to the current state with the current action according to the learned dynamics model. Experiments on a diverse set of simulated tasks demonstrate the effectiveness of the proposed method.", "questions": "See weakness for major concerns.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Offline data augmentation is an important area of research that could lead to more robust  policies. This paper proposes an intuitive solution by generating additional data around existing data points by querying a locally smooth dynamics model.\nThe proposed solution categorizes two types of data augmentation: one by perturbing action labels and finding states that would land in the next state given this noise label, and the other by tracing states that would land in the current state given the current action.\nThis work presents thorough evaluation of the proposed method by experimenting with diverse task settings ranging from controlling a drone to manipulation tasks that have discontinuous dynamics.", "weaknesses": "The theoretical and algorithmic contribution is novel and exciting but the empirical results are not as impressive.\n\nIt would be great if the authors could test augmenting the training data with ground truth dynamics models (isn’t it deterministic -> computable given low-dimensional state representations?) to showcase the full potential of data-augmentation-based methods and situate the performance of the proposed method: i.e. help the audience understand if the performance gain/no-gain attribute to additional data or quality of the dynamics model.\n\nThis work could also benefit from additional experiments with varying number of demonstrations in a particular domain to show how much data is needed to learn a good dynamics model and at the same time could still benefit from additional augmentation data. \n\nThis work only conducted experiments in simulation, where dynamics models are deterministic and different from real applications. The authors should comment more on what challenges there would be to apply the proposed method in the real world and if one can benefit more or less from this paradigm of data augmentation.\n\n\n----- Edit ------\nThe authors presented additional results during rebuttal that address some of my concerns about the evaluation. However, I do think a real-world experiment is practical and valuable for the true impact of this paper, given the proposed method is fully offline.\n\nI am happy to raise my evaluation to weakly accept.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "52ZHSOG8Tv", "reviewer": "Reviewer_mpW3", "summary": "This study is dedicated to enhancing the robustness of imitation learning through the generation of corrective data, compensating for compounding errors and disturbances. Numerous experiments have been executed on an array of tasks, ranging from drone navigation and locomotion to robot manipulation, to validate the effectiveness of the proposed approach.", "questions": "The reason why noise BC underperforms compared to vanilla BC is not clear. If we reduce the noise added to the BC, noiseBC's performance should align more closely with that of vanilla BC. Nonetheless, in multiple tasks, noiseBC exhibits significantly poorer results. This could potentially be attributed to the fact that the added noise has not been not carefully chosen and thus, an excessive amount of noise has been injected into the system?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work offers a detailed theoretical analysis, providing evidence that the quality of the generated label is bounded under specific assumptions related to the dynamics.\n\n2. Various tasks ranging from drone navigation to locomotion and robot manipulation have been extensively experimented and analyzed in the study.", "weaknesses": "1. The proposed method is only compared to vanilla BC and noisy BC. The proposed method declares it constructs a dynamics model for policy learning and has used implementation with a model-based RL framework; therefore, it would be more robust to also include a comparison with other model-based RL methods. As model-based RL also constructs a dynamic model first before planning the most effective actions.\n\n2. There is a lack of clarity in important implementation details. The process of generating corrective labels is discussed in Section 4 but the paper does not make it clear how these labels are employed in later stages. The additional corrective labels could be used to train the imitation learning agent, presumably a neural network? However, if this is the case, further details on the network's implementation could be discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x5oWlK7xtf", "reviewer": "Reviewer_vWGV", "summary": "This paper proposes the data augmentation method for behavioral cloning (BC) utilizing the local Lipschitz constraint. To train the forward dynamics from expert data, the proposed method (CCIL) minimizes mean-squared error with the regularization that is computed from the local Lipschitz constraint. Then, two techniques are proposed to generate transition triplets that can be used as expert data.  Once the dataset is augmented, naive BC is applied to find a policy. CCIL is evaluated on various tasks and outperforms BC and NoiseBC.", "questions": "1. The proposed method is formulated in a discrete-time state transition model, whereas the corresponding true system operates in continuous-time. Therefore, the proposed method implicitly applies a time discretization. In this case, the time interval is critical, and I think the Lipschitz constant depends on the time interval. How did the authors determine an appropriate Lipschitz constant? Or, are there any assumptions on the time discretization in the proposed method? \n2. I do not fully understand the major differences between the techniques of the proposed method and Data as Demonstrator (DaD) proposed by Venkatraman et al. (2015)? I think the core idea is similar; therefore, it is worth discussing the advantages of the proposed method.  \n3. Two augmentation techniques are proposed, but I am unsure whether either would be equally useful. Is it possible to conduct an additional ablation study where one of the techniques is removed? \n4. In the paragraph above Definition 4, the authors introduce $\\mathrm{Support}(d^\\pi)$, but it is not defined. Is $d^\\pi$ a stationary distribution induced by $\\pi$? \n5. The first paragraph on page 5: $\\hat{f}(s_t, a_t) \\to s_{t+1}^* - s_t^*$ should be $\\hat{f}(s_t^*, a_t^*) \\to s_{t+1}^* - s_t^*$. \n6. Is $\\bar{\\lambda}$ in Equation (3) is an average of $\\{ \\lambda_j \\}_j$? \n7. Please define $f'$ in Equation (3) explicitly.\n8. Regarding the technique 1 (Backtrack label), what does \"xlabel\" mean?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Although the proposed idea is simple, the experimental results show that CCIL is very powerful even if the environmental dynamics is not globally continuous. \n2. The authors evaluated CCIL on various tasks, and it suggests that the proposed method is appealing to practitioners. \n3. The manuscript is written well and easy to follow and understand.", "weaknesses": "1. My major concern is that the proposed method has to solve relatively complicated optimization problems. For example, Equation (3) contains two complicated terms: Lipschitz constraint and L0 norm. How to deal with the max operator in the Lipschitz constraint term is unclear. \n2. The proposed method assumes a deterministic transition function. I am curious when the proposed method is applied to stochastic systems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3vEvL6zKzI", "reviewer": "Reviewer_iEcJ", "summary": "The paper proposes to enhance the robustness of imitation learning methods by generating corrective data to account for compounding error and disturbances. Their work is based upon utilizing the local continuity in the environment dynamics. The paper augments the original expert's dataset with generated corrective labels within the neighborhood of the demonstrations but beyond the actual set of states and actions in the dataset. The authors' argue that this augmentation helps the agent to recover from perturbations and deal\nwith compounding error.", "questions": "Please check the Weaknesses part.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Their problem is well-defined, and methods are explained clearly.\n- Related work covers basic prior work.\n- The data augmentation part where the authors utilize the local continuity of dynamics model helped them achieve better performance than the basic Behavioral Cloning Algorithm.", "weaknesses": "- While the writing was clear and easy to understand, the paper lacked substantial content. I didn't find any need to pause and think while reading and I skimmed through the paper rather quickly .\n- Performance comparisons of their work are only done with basic Behavior Cloning and NoiseBC algorithms that are basic Imitation Learning (IL) Algorithms. Comparison with state-of the-art IL methods are missing.\n-  I would recommend the authors to include experiments to compare the sample efficiency with other state of the art algorithms in terms of trajectories needed as that is also an important metric in IL paradigm.\n- There are many Offline IL algorithms proposed recently in literature that have the same settings where they don't make any new interaction with the environment or the expert. Comparisons with them would be interesting to see.\n- I would recommend the authors to also report results on Humanoid environment from Mujoco.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)