Stabilized Likelihood-based Imitation Learning via Denoising Continuous Normalizing Flow
Authors: Xin Zhang
Yanhua Li
Ziming Zhang
Christopher Brinton
Zhenming Liu
Zhi-Li Zhang
Hui Lu
Zhihong Tian
Venue: ICLR 2022 Submitted (2022)

Abstract: State-of-the-art imitation learning (IL) approaches, e.g, GAIL, apply adversarial training to minimize the discrepancy between expert and learner behaviors, which is prone to unstable training and mode collapse. In this work, we propose SLIL – Stabilized Likelihood-based Imitation Learning – a novel IL approach that directly maximizes the likelihood of observing the expert demonstrations. SLIL is a two-stage optimization framework, where in stage one the expert state distribution is estimated via a new method for denoising continuous normalizing flow, and in stage two the learner policy is trained to match both the expert’s policy and state distribution. Experimental evaluation of SLIL compared with several baselines in ten different physics-based control tasks reveals superior results in terms of learner policy performance, training stability, and mode distribution preservation.

Under review as a conference paper at ICLR 2022
STABILIZED LIKELIHOOD-BASEH IMITATION LEARN- ING VIA DENOISING CONTINUOUS NORMALIZING
FLOW
Anonymous authors Paper under double-blind review
ABSTRACT
State-of-the-an imitation learning (IL) approaches, e.g, GAIL, apply adversarial training to minimize the discrepancy between expert and learner behaviors, which is prone to unstable training and mode collapse. this work, we propose SLIII- Stabilized iikelihood-based Imitation Learning novel II. approach that directly maximizes the likelihood of observing the expert demonstrations. SLIL is a two- stage optimization framework, where in stage one the expert state distribution is estimated via a new method for denoising continuous normalizing flow, and stage two the learner policy is trained to match both the expert's policy and state distribution. From the best of our knowledge, none of existing works solve the unstable training and mode collapse problem of GAIL. Experimental evaluation of control tasks
SLIL compared with several baselines in ten different reveals superior results in terms of learner policy performance, training stability, and mode distribution preservation.
INTRODUCTION
Imitation learning (IL) (Abbeel & Ng 2004: Ho & Ermon 2016) aims to learn sequential decision- making policies directly from expert demonstrations, without access to reward signals from the environment. State-of-thee-rt (SOTA) IL approaches have primarily followed one of two paradigms: behavior cloning (BC) (Pomerleau 1991 and generative adversarial imitation learning (GAIL) (Ho & Ermon 2016). While both BC and GAIL. have been studied extensively, each has crucial limitations. On the one hand, BC approaches employ supervised learning. which requires large amount of expert demonstrations to avoid compounding errors due to covariate shifts (Ross et al. 2011 Ross
Bagnell] 2010). On the other hand, GAIL approaches (Jena & Sycaraa 2020 Fei et al. 2020) Li et al.
2017
Hausman et al.
2017
Fu et al. 2017 Ke et al. 2019)] Ghasemiporr et al. 2019 [Zhang et al,
2020
Arjovsky et al.
2017 connect IL with generative adversarial networks (GAN) (Goodfellow
et al.
2014). but adversarial training processes are intrinsically unstable (Jena & Sycaral 2020
Arjovsky et al. 2017) and prone mode collapse (especially when learning from multi-mode expert
demonstrations) (Ghasemipour et al. 2019 Fei et al. 2020 Ke et al., 2019). To see this, consider
Fig. [ which shows an example of different IL algorithms on the Reacher task (Todorov al.) 2012)
Hausman et al. 2017) with four targets (Fig. La While the expert tends to visit all targets equal
(Fig. lb the policy learned by GAIL mode collapsed, primarily visiting the green one (Fig. lc
SLIL: Overview and Contributions. In this work, we are motivated to design an IL methodology with a stabilized training process and mode distribution preservation from expert demonstrations, We propose SLIII- Stabilized likelihooddbasee Imitation Learning a novel II. approach that directly maximizes the likelihood of observing the expert demonstrations. SI.II. stable two- stage optimization framework, with the 1st stage focusing on accurately estimating the expert state distribution, and the 2nd stage training the learner policy to match the expert's 's policy and state distribution. Fig. shows quick view of our results: SLIL leads leamer policy preserving the mode visitation distribution of the expert demonstrations. Our key contributions are as follows:
We develop the first likelihood-basee IL methodology that tackles the training instability issue of GAIL. SLIL's two-stage optimization framework is based on a tight lower bound on the joint policy and state distribution training objective from the likelihood-basee II. formulation (Sec

---

Under review as a conference paper at ICLR 2022
(a) Reacher4 with four mode targets.
(b) Mode coverage of expert policy.
(c) Mode coverage of SLIL policy.
(d) Mode collapse of DRIL policy.
EMD
SLIL
0.33
DRIL
0.55
GAIL
0.41
0.48
Soft-SLIL
0.67
(e) Mode collapse of GAIL policy.
(f) Mode collapse of PWIL policy-
(g) Mode collapse
(h) Learner policy EMD result.
of Soft-SLIL policy.
Figure 1: Example results obtained by SLIL (Ours) and baselines on mode coverage. (a): A Reacher task, with four targets in different colors. (b)-(g show the mode coverage (i.e, state distribution) with expert policy (b), SLIL policy (c), DRIL policy (d), GAIL policy (e), PWIL policy (f), and SLIL implemented with SoftFlow (Soft-SLIL, (g). (h) show the earth mover's distance (EMD) (Ling & Okada 2007) between expert and learner policy state distributions. All the distributions visualized
using kernel density estimation (KDE) (Sheather & Jones 1991.. None of the compared approaches solve the mode collapse problem.
We propose the denoising continuous normalizing flow (DCNF) algorithm accurately estimate the expert state distribution in the 1a stage of SLIL to preserve expert modes, DCNF maps the expert state to the Gaussian noise distribution with a neural ODE, and trains the ODE using perturbed expert states (Sec 4.2).
Our evaluation on ten different physics-based control tasks reveals that SLII obtains superior results compared with SOTA baselines in terms of learner policy performances, training stability, and mode distribution preservation (Sec
RELATED WORK
Behavior Cloning (BC) (Pomerleau) 1991 Bohg et al., 2020) approaches for IL learn the expert policy via maximizing expert trajectory likelihood in the demonstration data, i.e, maximizing expert action likelihood. Though effective with abundant demonstrations, BC suffers from the covariate shift problem with limited data (Ross et al. 201T Ross & Bagnell| 2010). BBranteyy et al. (0199) aims address this via first pre-training an ensemble of BC policies, and then using reinforcement learning (RL) (Sutoon& & Barto 2018) to train learner policy whose cost is proportional to ensemble policies' prediction variance. In SLIL, we instead strive solve the covariate shift problem via expert state-action distribution matching, learning the expert state distribution explicitly.
Generative Adversarial Imitation Learning (GAIL) (Ho Ermon 2016) employs the GAN framework (Goodfellow et al. 2014) to minimize the discrepancy between expert and learner state- action distributions. It jointly trains a generator (i.e, learner policy) to imitate the expert behaviors, and discriminator (i.e, reward signal) to distinguish the state-action pair distributions between the expert and the learner. In GAIL, the discrepancy between the behavior distributions of the expert and the learner is measured by JS divergence. Using the variational lower bound of an f-divergence measure, several studies (Ke et al. 2019; Ghasemipour et al. 2019 Fu et al. 2017: Nowozin et al. 2016 Arumugam al.] 2019 Zhang et al. 2020) have extended GAIL from JS to pre-defined f- divergence, e.g, KI. (Fu et al.) 2017). Reverse KI. Ke et al., 2019), Total Variation (Ross et al.) 2011). However, all these approaches adversarially train the generator and the discriminator
minimax
game to reach an equilibrium, which can lead to training instability and mode collapse (Schroecker
al.
2019 Ghasemipour al. 2019
al. 2020
Ke al. 2019)
For reproducibility, the code for experiments is available at pettpa://o www. wwwpbox..com.....oooo..oooo.
h:cps://ww.drrpbbxxx
2

---

Under review as a conference paper at ICLR 2022
Several recent works try avoid adversarial training in IL Dadashi al. 2020 Liu et al. 2020
Kim et al.] 2020b; Rhinehhar et al. 2018 Schroecker et al. 2019). To do so, Primal Wasserstein IL
(PWIL) Dadashi et al. 2020) considers the primal form of Wasserstein distance to match learner's s
and expert's state-action distributions. Neural Density Imitation (NDI) (Kim et al. 20206) estimates expert's occupancy measure using which as a reward for maximum occupancy entropy reinforcement
learning. Ennery-Baaeed IL (EBIL) (Liu et al. 2020) stems from Max-Entropy IRL Ziebart et al.)
2008) and estimates surrogate reward function with score matching from expert demonstrations. Imitative Models (IM) (Rhinehart et al. 2018) learns a flow model that assigns high likelihoods to expert-like trajectories for test time goal-directed directed planning. GPRIL (Schroecker al. 2019) applies masked autoregressive flows (Papamakarios et al. 2017) to learn predecessor state-action distribution in each training iteration, which adds complexity to learner policy training and requires training stabilization. Unlike these methods, we maximize expert state-action likelihood and strive to not only avoid adversarial training, but solve the mode collapse and unstable training problems as well.
3 LIL PROBLEM FORMULATION
Notations. We denote S as of states, A as set of actions, P SS AxS - [0, 1] as the transition probability distribution, r A H RR as the reward function, Po S + R as the distribution of the initial state 50, and y E [0, 1] as the discount factor. An agent makes decisions following a policy
FF
Upper level
S A + [D, 1J. which specifies probability distribution of choosing an action a E A at a state s E S. With so ~ Po. then over time f. ar I x(a (t)) and 8t+1 ~ P(s az) according to the policy æ. We will denote the expert policy as TE. and the learner policy as æ. We denote (s a) as the probability of observing state-action pair (s,a) when executing the learner policy and denote PÆ(s, a) to represent Prg (8, a) for brevity. Moreover, for
Aecun x
Lower level
D0
Env
Figure 2: LIL framework.
function (ss,, a) of interest, we use an expectation over a policy to
Dg
denote an expectation with respect to the trajectories it generates, i.e, En [h(s, a)] _ E "'h(s: a:)].
x
Likelihood- Imitation Learning (LIL). To avoid the instability from adversarial training (e.g, in GAIL), we introduce a likelihood- based imitation learning paradigm to learn the policy by directly maximizing the likelihood of the state-action pairs from the expert demonstration data Dz. The LII. objective can be formally modeled as a bilevel optimization problem (Sinha et al. 2017:
Stagets Dr
Execate 1
Stagelt
Evaluato
PE
Env
max E (e (lo (s),a)), s.t. Px = arg max log P(s, Figure 3: SLIL framework.
with Pa a) as the probability of observing an expert state-action pair (s. a) E Dg when executing the learner policy и. Notice that instead of maximizing trajectory likelihood as in BC, LIL maximizes the likelihood of observed state-action pairs. The variable function (= a) can be learned from the lower-level optimization to maximize the log likelihoehood of observing state-action pairs (s, a) E D, collected by interacting with the environment using the learner policy (i.e, the upper-level variable function). Moreover, since (r a) = π (a) 8) P>(s), with (a (s) as the learner policy and Pa (s) as the state distribution when executing म, the LIL objective can be rewritten as
max (E (x,a)eD, [log [[]ss)] + EE [log (s)), S.L.
P = arg max ESED, [log P(s)].
(2)
Challenges. The LIL solution framework alternates between policy learning (i.e, the upper-level optimization to update T) and state distribution estimation (i.e, the lower-level optimization to learn P=), as shown in Fig. 2 In each iteration, the learner policy π is used to interact with the environment to collect D,,. A normalizing flow is trained to estimate the state distribution P. of policy æ using Dr (i.e, lower-level optimization). Then, the policy is updated based on the state distribution Px using expert data DE (i.e, upper-level optimization). However, two key challenges prevent a straightforward application of LIL. C#1 Training stabiliry: It is challenging to properly grow π and Pe jointly to speed up and stabilize the training process in the bilevel optimization framework of Eq. B. C#2 Manifold hypothesis: Normalizing flows are only valid when the data distribution and
3

---

Under review as a conference paper at ICLR 2022
the target noise distribution have the same dimensions, which conflicts with the observation that real-world data usually concentrate on low dimensional manifolds in a high dimensional space (a.k.a the ambient space) (Papamakarios al., 2017; Kim et al. 2020al Song & Ermon 2019 Ho et al. 2020; Belkin & Niyogi 2003). Given an expert state distribution supported on a low dimensional manifold, the LII framework will fail to estimate the ground-truth state distribution P, and fail to learn an expert-like learner policy efficiently.
To address these challenges. we propose the Stabilized Likelihood-baset Imitation Learning (SLIL) framework shown in Fig. 3 SLIL relaxes the LIL problem to a two-stage optimization problem for improved training stability, and learns expert state distribution with denoising continuous normalizing flow (DCNF) to overcome the manifold hypothesis challenge. We formalize this methodology next.
4
OUR SLIL APPROACH
4.1 SLIL WITH EXPERT STATE DISTRIBUTION
As we discussed earlier, the training instability of the LII framework in 2 comes from the joint training of policy (in the upper-level objective) and the corresponding state distribution Ps (in the lower-level objective). In Theorem below, we show that the LIL problem (as a bilevel optimization problem) be relaxed to a two-stage optimization problem, which estimates the expert state distribution PE (in Stage #I), and trains the learner policy π based on PE (in Stage #2).
Theorem 4.1. The optimal objective of the likelihood- based imitation learning (LIL) problem in Eq. is lower bounded tightly by the optimal objective of the following two-stage optimization problem (see our proof in Appx. A
max (s,,) [logllloollll T (a]s)] + E,DD, [log Pr(()],
5.1.
Pe arg max ExEDE [log P(s)]-
Expert action matching.
Based on Theorem 4. we can relax the LII. problem
g
by maximizing the two-stage optimization problem
in Eq. B.
Env
The solution framework of SLIL. To solve the two- stage objective in (. we model the learner policy deep neural network parameterized by 8, and the expert state distribution P& using our DCNF pa- rameterized by w. The training process involves two stages, described in Fig. 4 (Stage #1) expert state distribution matching, where we train an expert state distribution PEW based on DCNF to tackle the man- ifold hypothesis challenge (see details in Sec. 4.2): (Stage #2) policy learning, where learner policy Tg with the objective in Eq.3I. Note that updating the policy πa with PF guidance involves action sampling. Thus, follow an RL approach and employ proximal policy gradient (PPO) (Schulman et al. 2017) for back-propagatte
Sa (DCNF)
Sax (DCNF)
Expert state matching,
PE.())
(s)
(a) Stage#1: Expert state
(b) Staget2: Policy learning.
distribution matching.
Figure 4: Illustration of our SLIL framework.
the learned expert state distribution PE to train the
with the Adam optimizer (Kingma & Ba 2014)
for updating the policy ie. Comparing to the bilevel optimization framework of LIL in Fig. the
stabilized two-stage optimization framework for SLIL is shown in Fig. 3
The objective function in Eq. (3) has two components: expert policy matching (i.e, BC), and expert state distribution matching (with PEw-). In each training iteration, the learning speed between the two components matter, and have impacts on the training stability. Similar observations are made in (Lee et al. 2018). To further improve the training stability, we introduce gating function 1-(-) with parameter X, which governs whether to run the second component of matching expert distribution
in particular iteration, i.e, 1x(1) = 1 iff (i mod A) 0 for any training iteration i. For example, = 2 means that the second component is included in all training iterations with even sequence numbers. A lower À leads to more iterations to match the expert distribution, which can mitigate the compounding errors from BC. On the other hand, a higher encourages supervision from expert policy matching (i.e, through BC). Therefore, with the gating function, the objective in Eq be rewritten as
B(e,))DD (log na, (a|s)] + 1x(i) (log Pru- (8)], Vi.
(4)
4

---

Under review as a conference paper at ICLR 2022
4.2 DCNF FOR LEARNING EXPERT STATE DISTRIBUTION
There have been many solutions proposed to estimate the data distribution from a real world dataset {x} X. The discrete normalizing flow (DNF) and continuous normal- izing flow (CNF) methods both have advantages as ex- plicit models for expressing data sample likelihood via the change of variable formula (Rezende Mohamed 2015).
Data Distribution
Laleni Space
Target Distribution
Discrete versions of normalizing flows (NF) (Rezende & Dinh et al. 2016 Kim et al.] 2020aa
Mohamed 2015;
Dinh et al.
2014) feature sequence of n bijective map-
pings, i.e, Jw In o fn-10.... fi and require computing the Jacobian of fa for back proppapoppaatiin. These features restrict DNF's modeling capability as fa needs to be a once-differentiable bijection, and makes it computation- ally costly as the Jacobian matrix requires large computa- tion power (Chen et al., 2018 Grathwohl al. 2018). In contrast, CNF (Chen et al. 2018 Grathwohl et al., 20188)
O
views the transformation between Gaussian noise and data samples
Figure Illustration of FFJORD trained on 2D manifold (top), on ID manifold (middle), and our DCNF trained on ID manifold (bottom).
ordinary differential equation (ODE) de-
termined by Jun which does not have model restrictions (Grathwohl et al. 2018). and expresses data sample density via the instantancous change of variable formula (Chen ct al. 2018). Therefore, CNF, e.g, FFJORD (Grathwohl et al. 2018), is more promising than DNF in terms of es- timating the expert state distribution PEW. where the ODE system function fu learns a bijection between random noise z Z and expert state (s(s), i.e, Sw 2 + S. The ODE and the expert distribution density follow the instantaneous change of variable formula, i.e.
6
Sale
Sann
Jua
+CC
C2
Figure 6: Illustration of DCNF.
2tt) = f=(=((), with z(40) x(11) = t
0fm = E;loog (s) = logp(=((o)) - S" Tr 0z(1))
Manifold hypothesis challenge. The manifold hypothesis, i.e, real-world data tend to concentrate on a low dimensional manifold in a high dimensional space (Roweis Saul 2000 Belkin & Niyogi 2003). has been observed on many datasets. As a result, since CNF is only valid when the data distribution and the target noise distribution have the same dimensions, the CNF will fail to estimate the ground-truth state distribution PE(s) if the expert states lie in a low dimensional manifold.
Fig. shows examples to illustrate the manifold hypothesis challenge. The goal is to transform the data distribution displayed on the left column to the target Gaussian distribution the right column using CNF. The transformed latent variable scatter plots (from the data distribution) based on placed in the middle column. is clear that when the true data populates 2D manifold over a 2D space, its transformed latent variables match target Gaussian distribution well (the 1st row). On the other hand, when the true data resides in a ID manifold of the 2D space, its transformed latent variable does not match the target Gaussian distribution (the 2nd row). This example illustrates how the manifold hypothesis deteriorates CNF's performance; when the data distribution lies in a low- dimensinnion manifold of the ambient space, no homeomorphism can be easily created (Dupont al. 2019 Kim et al. 2020a).
FFJORD (Grathwohl et al.) 2018) and DCNF (described below)
DCNE Our experimental results in Fig. 5 (the 3rd row) and SOTA works (Kim et al.) 2020a Liu
et al. 2019 Song & Ermon 2019 Ho al. 2020) have each observed that perturbing data with random Gaussian noise N (0, o (with 2 O as the noise level) can tackle the manifold hypothesis challenge, since the perturbed data will expand the data (i.e, expert states) from a low-dimension manifold to high dimensional ambient space. It can also increase model generalizability given limited amounts of expert data. Denote a perturbed expert state $ ~ DE as 8, with = s ae 0. approaches Therefore. we decrease the noise level a in each iteration when training the ODE system function f.. of the expert state distribution Pg. Fig. 6 and Alg. give our denoising continuous normalizing flow
where (0, I). When the noise level is very small, i.e,
5

---

Under review as a conference paper at ICLR 2022
(DCNF) algorithm. We first pre-define arithmetic sequence of decreasing noise levels (o,) ) whose common difference of successive numbers is I where L is the number training iterations. For each training iteration,
sample B states $j from expert demonstrations Dr, and B random noises Ej (0, of with j 1, ..... B (line 2&3). The sampled noises Ej applied to the sampled states 8 to obtain perturbed states 8, (line 4). The perturbed states 3j are input into the expert state distribution PE to be evaluated, whose value will be used to compute the gradient with Adam (Kingma & Ba 2014) to update w (line 5). After training, a learned expert state distribution PR (governed by the learned fw- function) is obtained and used for learner policy training.
Algorithm Denoising Continuous Normalizing Flow (DCNF)
Require:
Initial parameters wi for the ODE system function fw of PEw: expert demonstrations DE containing state-action pairs; predefined initial noise level 11, total number of training iteration L and batch size B.
Ensure: The ODE system function fo- of the expert state distribution PEw".
1: for each epoch = 1, 2... L do
2: Sample B states 8, E DE where j ,....,, B.
3: Sample B random noises Ej (0, where j 1,,,, and 0, =0 121-111
4 1).
4: Apply noise on sampled states 8j = 5j + E
5: Update wi wi++ by ascending with the gradients: Aw; = -E== Va; log Pres (8)).
6: end for
5 EXPERIMENTAL EVALUATION
To evaluate our proposed SLIL methodology, we conduct experiments on ten physics-based control tasks, including CartPole (Barto et al. 1983). Reacher (with and 4 targets), Hopper, Walker, HalfCheetah (with 1 and 2 running directions), Ant, and Humanoid all simulated with MuloCo [ooorov et al. 2012). From these experiments, we show that: i) learner policy from our SLII avoids mode collapse, by accurately preserving the expert mode distribution; ii) our DCNF can properly address the manifold hypothesis challenge in estimating the expert state distribution; iii) learner policies from SLIL have comparable or better performance than IL baselines; and vi) SLIL has more stability to hyper-parameter changes than IL baselines using alternative training (e.g.. GAIL).
Implementation Settings and IL Baselines. We obtain expert policies of all tasks by running TRPO (Schulman et al. 2013a) with their ground-trute reward functions defined in the OpenAI Gym the expert policies generate expert demonstrations. We use Reacher with two and four target modes, and HalfCheetah with two target modes [Todorov et al. 2012) respectively (i.e, Reacher2, Reacher4 and HalfCheetah2)) analyze the mode coverage of our SLIL vs IL baselines. Each expert policy obtained by TRPO has a particular distribution of reaching different target modes. We relegate more implementation details to Appx. Below are the four IL. baselines we use to compare with SLIL:
(Brockman al. 2016). Then,
Behavior Cloning (BC) (Pomerleaul 1991) Expert demonstrations as a set of state-action pairs are split into 70% training data and 30% validation data. The learner policy is trained with supervised learning where actions are viewed as labels and states are viewed as input features.
Generative Adversarial Imitation Learning (GAIL)) (Ho & Ermon] (0T6): GAIL. is an II. method that consists of generator as a policy network mimicking the expert behaviors, and a discriminator reward signal distinguishing between learner and expert behaviors.
Generative PRedecessor models for Imitation Learning (GPRIL) (Schroecker et al. 2019 GPRIL performs the state-action distribution matching by jointly training the learner policy and the corresponding multi-step predecessor state-action distribution. In each iteration, the predecessor state-action distribution is estimated using masked autoregressive flows (Papamakarios al.) 2017). SlsugreementtReguuariiece. Regularised Imitation Learning (DRIL) (Brantley et al., (019)): DRIL. pre-trains Disagreement- an ensemble of BC policies with expert demonstration data, and uses RL train a learner policy whose cost function is proportional to the sum of the variance of ensemble policies' predictions.
5.1
MODE COVERAGE
Fig. shows the results of Reacher4 with four mode targets in different colors in Fig. I and Fig. shows the results of HalfCheetah2 with two mode directions running forward and backward.

---

20 Under review as a conference paper at ICLR 2022
Expert -0- BC a GAIL + DRIL SLIL
Scaled Return
Segled Return
0.3
Backward
Forward
1,02
1.02
0.98
0.98
Density
0.2
0.94
0.94
0.5
0.7
0.9
0.4
0.6
0.8
EMD
EMD
0.1
(a) Reacher2 w. 2 modes.
(b) Reacher4 w. modes,
Scaled Return
Scaled Return
1
0.9
0.9
-5
0
5
0.4
0.85
1.05
1.15
1.2
1.6
EMD
EMD
(c) HalfChoctab w. I mode.
(d) HalfCheetah2 w. modes.
Figure 7: Results of SLIL (Ours)
Figure 8: EMD vs scaled return in tasks with multiple
and baselines mode coverage in modes. The x-axis is the EMD (Ling Okada 2007
HalfCheetah2. All the distributions are visualized using KDE (Sheather & Jones 1991).
between expert and learner policy state distribution. The y-axis is the expected return (i.e. total reward), scaled so that the expert achieves and a random policy achieves 0.
Results of Reacher2 are in Appx. C as its observations are similar to the Reacher4 task. Fig. shows the mode coverage (i.e, the state distribution) of the expert policy, where the expert tends cover all four targets (as four modes) evenly. The learner policy obtained by SLIL preserves the expert mode coverage very well shown in Fig. Ic On the other hand, the learner policy from GAIL is prone to mode collapse. It only focuses on the green target out of the four targets. Consistent with results in SOTA works (Fei al. 2020 Arjovsky et al.) 2017 Jena & Sycara 2020), this mode collapse is due to the adversarial training process used in GAIL. DRIL also fails to cover the green target mode as its defined reward function likely encourages mode seeking behavior. SLII. successfully preserves the mode coverage from the expert, because it uses our DCNF to accurately estimate the expert state distribution, and the stabilized two-stage training to update the learner policy.
In the HalfCheetah2 task shown in Fig. 7 the x-axis represents the rumning velocity, and the plots show the velocity distributions of expert and IL policies. The black curve demonstrates two modes (i.e, running forward and backward) in expert demonstrations. SLIL (blue curve)
DCNF (Ours)
CNF
SoftFlow
0
Test Log Likelihood
Test Log Likelihood
-10
-15
able to preserve all modes, while DRIL and BC are collapsed to running forward and GAIL fails to reveal any mode. We further calcu- lated the earth mover's distance (EMD) (Ling & Okada 2007) between expert and learned poli- cies' state distributions as the x-axis in Fig. g in the Reacher and HalfCheetah tasks with mul- tiple modes. A lower EMD value indicates better learner policy at recovering expert demon- stration modes, and the results echo the above observations quantitatively. We omit GPRIL as no meaningful results are obtained.
-15
-20
10
10
20
Number of Trajectories
Number of Trajectories
(a) Hopper
(b) Walker
Figure 9: Test state log likelihood using PEW
learned from our DCNF and CNF.
# of Traj.
Hopper
Walker
CNF
SoftFlo
DCNF SoftFlow DCNF
4
3.73
1.88
0.63
3.22
2.04
1.44
2.69
1.42
0.52
3.72
1.53
0.80
18
2.09
0.94
0.49
3.73
1.10
0.80
25
1.95
0.68
0.49
2.85
1.02
0.64
5.2 EXPERT STATE ESTIMATION
Table 1: The EMD
between the Gaussian and the latent noise recovered from
(Ling & Okada 2007)
To show how our proposed denoising mecha- nism in DCNF addresses the manifold hypothe- sis challenge in estimating the expert state distri-
PE learned from DCNF, SoftFlow and CNF,
bution, we compare our DCNF to SoftFlow and CNF both in terms of i) expert state matching and ii) latent space (i.e, Gaussian noise) recovery. The results below obtained using Hopper and Walker tasks. We make similar observations for other tasks, so their results are omitted for brevity.
Expert state matching. To quantify the quality of the expert state matching, we expert policies to generate 50 trajectories (each consisting of 1,000 state-action pairs) as a test set and evaluate the log-likelihood of states in the test set using PF learned by DCNF, SoftFlow (Kim al. 2020ab
7

---

Under review as a conference paper at ICLR 2022
SLIL (Ours) DRIL
GAIL
+ BC * GPRIL
Random Policy Expert
Scaled Return
Scaled Return
0.8
Scaled Return
0.8
0.9
0.6
0.6
0.8
0.4
0.4
0.7
0.2
0.2
5
13
5
10
20
10
20
Number of Trajectories in Dataset
Number of Trajectories Dataset
Number of Trjectoriis in Dataset
(a) Reacher.
(b) Hopper.
(c) Walker.
Scaled Return
0.8
Scaled Return
Scaled Return
0.9
0.6
0.4
0.8
0.2
0.7
13
10
15
20
80
120
160
200
240
Number of Trajectories in Dataset
Number of Trajectories in Dataset
Number of Trajectories in Dataset
(d) HalfCheetah.
(c) Ant.
(f) Humanoid.
Figure 10: Performance of learner policies in tasks with one mode. The y-axis is the expected return
(i.e, total reward), scaled so that the expert achieves and random policy achieves 0.
and CNF (Grthhwhhl et al. 2018). A higher test state log-likelihood score indicates a more accurate model in estimating the expert state distribution. The results are shown in Fig. 9 Clearly, in both tasks with different numbers of expert demonstrations, DCNF outperforms SoftFlow and CNF which validates the necessity of our denoising mechanism in tackling the manifold hypothesis challenge for expert state distribution estimation.
ti) Latent space (i.e., Gaussian noise) recovery. We use the test set to recover latent noises and com- pute its EMD (Ling & (ko7) to the Gaussian noise distribution. A lower EMD value indicates better normalizing flow model, with its learned data transformation closer Tab. M shows the EMD results in Hopper and Walker with different number of expert demonstrations. demonstrates that DCNF is more likely recover latent space closer to Gaussian.
homeomorphism.
5.3 PERFORMANCE OF THE LEARNER POLICY FROM SLIL
Fig. g shows the EMD (Ling & Okada 2007) vs scaled return results in Reacher and HalfCheetah with different numbers of demonstration modes. In all the tasks. SLIL has both lower EMD and higher scaled return compared with baselines. This is because DCNF is able to recover expert state distribution PE with multiple modes well, and thus provides useful feedback for the learner policy to recover expert behaviors. Comparing between Reacher with 2 and targets, SLIL shows larger return margin when the target modes are more distant from each other the Reacher2 task.
Fig. 10 shows the performances of the learner policies from our SLIL and IL baselines under different numbers of expert trajectories when they only contain one mode. In all tasks, the learner policies from SLIL have comparable performances with GAIL, which is because SLIL directly maximizes the log-likelihood of expert data without using adversarial training. SLIIL outperforms DRIL particularly with limited number of expert trajectories because the use of DCNFFlearred Pe has good generalization ability and provides useful feedback to learner policy я to reach a better policy. Moreover, in both easy tasks (Reacher) and complex tasks (Hopper and Walker), SLIL consistently outperforms BC with different numbers of expert demonstrations. which is because SLIL uses the state distribution matching on top of the BC objective to overcome the covariate shift problem. The performance of the learner policy from BC increases when using more expert demonstrations, as more training data mitigate the overfitting problem and compounding errors. However, the learner policies from GPRIL have the lowest performances in all tasks. This is primarily because the policy is jointly Iearned with its multi-step predecessor state-action distribution. With random initial parameters for these two functions, is hard to progressively improve them jointly.
5.4 TRAINING STABILITY OF SLIL
8

---

Under review as a conference paper at ICLR 2022
2. = -
== = 225
SLIL (Ours)
GAIL
0.8
Scaled Return
0.75
Scaled Return
0.5
0.4
0.25
0.2
3
5
1o-5
76-5
1o-4
1o-3
76-3
Number Environment Interactions( x 10²)
Learning Rate
Figure 11: SLIL (Ours) performance with differ- ent gating parameters À in Walker.
Figure 12: Policy performances with different
learning rates using GAIL and SLIL in Walker.
When training the learner policy, the conver- gence speed, performance fluctuation, and ro- bustness to the learning rate are all important. stable II algorithm leads to a high conver- gence speed, low performance fluctuation in training, and works in a wide-range of learning rate choices. Now, we investigate the training stability of our SLIL from the perspectives of i) the gating parameter X, ii) the learning rate, and iii) the number of environment interactions. Be- low, we show our results from the Walker task with 18 expert trajectories; similar observations were made for other tasks.
SLIL (Ours)
GAIL
0.5
Staled
0.25
0
2
10
Number of Enviromment Interactions( 10¹)
Figure 13: Policy performance over environment interaction numbers (SLIL vs GAIL) in Walker.
Impact of the gating parameter A. Fig. shows the training curves of SLIL with different gating parameters A. It is clear that when 2, the SLIL training process stable, with small performance fluctuation, and fast convergence rate. On the other hand. lower gating parameter (i.e, À 1) leads to an unstable training curve, with high fluctuations; and a higher gating parameter e.g.. À = 3 and leads slower convergence rate. This indicates that both behavior cloning and expert state matching matter in learning expert policy, and balancing the weights of these two components is crucial. If the weight to the expert state matching is too much (small A), leads a high training fluctuation. On the other hand, when the weight of the expert state matching is too small (large A), it slows down the training speed.
Impact of the learning rate. We further study SLIL's robustness in choosing different learning rates. Fig. 12 shows the performance of SLIL and GAIL given different learning rates. shows that SLIL works in wide range of learning rates, while GAIL tends to fail/crash when learning rates are le-3 and higher. GAIL. with adversarial training is less robust the change of learning rate due to vanishing gradient and the complex interactions between the discriminator and the generator (Jena & Sycara 2020 Arjovssy et al. 2017). However, our SLIL implemented in a two-stage optimization Tramework, which is more robust to hyper-parameter changes.
Impact of the number of environment interactions. Fig. 13 shows the performance change with the number of environment interactions seeff-supervision steps). This figure shows that SLIL is able to attain a higher return than GAIL given a small number of environment interactions (at around 3-6x10* steps). An explanation is that the pre-trained PE good guiding the learner policy me to explore on those states frequently visited by the experts. Moreover, differing from the adversarial training in GAIL with two generator and discriminator trained alternatively, SLIL employs the two-stage training process to training PE and Ty separately, thus leading to a fast convergence rate.
6 CONCLUSION
In this work, we proposed SLIL- stable likelihood- based imitation learning which trains a learner policy by directly maximizing the likelihood of expert demonstrations. SLIL is stable two-stage optimization framework, where in stage one accurately estimate the expert state distribution using a novel denoising continuous normalizing flow method, and in stage two we train the learner policy match both expert's policy and state distribution. Comparing our SLIL with baselines in ten different physics-base based control tasks, we present superior evaluation results in terms of learner policy performance, training stability, and mode distribution preservation.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Pieter Abbeel and Andrew Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference Machine learning, pp- 1. ACM, 2004.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan. arXiv preprint
arXiv:1701.07875 2017.
Dilip Arumugam, Debadcepta Dey, Alekh Agarwal, Asli Celikyilmaz, Elnaz Nouri, Eric Horvitz, and Bill Dolan. Reparameterized variational divergence minimization for stable imitation. 2019.
Andrew G Barto, Richard S Sutton, and Charles W Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE transactions on systems, man, and cybernetics, (5): 834-846, 1983.
Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. Neural computation, 15(6):1373-139, 2003.
Jeannette Bohg, Marco Pavone, and Dorsa Sadigh. Principles of robot autonomy http::///ww.ssstooooooeccc/////a http://p////ettee/eettrrr 10111213. 20111223.....imttttttoleeem..s.p (On1://ipooonnnmiinnn.
[On ine]] 2020.
Kianté Brandly, Wen Sun, and Mikael Henaff. Disagreement-regulartee imitation learning. In International Conference Learning Representations, 2019.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540. 2016.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. arXiv preprint arXiv: 1806 2018.
Robert Dadashi, Léonard Hussenot, Matthiuu Geist, and Olivier Pietquin. Primal wasserstein imitation learning. arXiv preprint arXiv:2006.04674. 2020.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516, 2014.
Laurent Dinh, Jascha Sohk-Dickst,in, and Samy Bengio. Density estimation using real nvp. arXiv preprint wxXiv:1605.0880) 2016.
Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural odes. arXiv preprint 1rX1v:1904.01681, 2019.
Cong Fei, Bin Wang, Yuzheng Zhuang. Zongzhang Zhang, Jianye Hao, Hongbo Zhang, Xuewu Ji. and Wulong Liu. Triple-gail: a multi-modal imitation learning framework with generative adversarial nets. arXiv preprini arXiv:2005. 10622, 2020.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforce- ment learning. arXiv preprint 1r0iv: / 2017.
Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence minimization perspective on imitation learning methods. arXiv preprint arXiv: 1.1022250 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems. pp. 2672-2688 2680, 2014.
Will Grathwohl, Ricky TQ Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. arXiv preprint arXiv:1810.01367. 2018.
David Ha, Andrew Dai, and Quoe V Le. Hypemetworks.. arXiv preprint
2016.
Karol Hausman, Yevgen Chebotar, Stefan Schaal, Gaurav Sukhatme, and Joseph Lim. Multi-modal imitation learning from unstructured demonstrations using generative adversarial nets. arXiv preprint arXiv: 1705. 10479, 2017.
10

---

Under review as a conference paper at ICLR 2022
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, pp. 1663,,53, 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006. 12399, 2020.
Rohit Jena and Katia Sycara. Loss-annealed gail for sample efficient and stable imitation learning. arXiv preprint arXiv:2001.07798, 2020.
Liyiming Ke, Matt Barnes, Wen Sun, Gilwoo Lee. Sanjiban Choudhury, and Siddhartha Srinivasa. Imitation learning as f-divergence minimization. arXiv preprint arXiv: /2888, 2019.
Hyeongju Kim, Hyeonseung Lee, Woo Hyun Kang, Joun Yeop Lee, and Nam Soo Kim. Softflow: Probabilistic framework for normalizing flow on manifolds. Advances in Neural Information Processing Systems, 33, 2020a.
Kuno Kim, Akshat Jindal, Yang Song. Jiaming Song, Yanan Sui, and Stefano Ermon. Imitation with neural density models. arXiv preprint arXiv:2010.09808, 2020b.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Dongsoo Lee, Parichay Kapoor, and Byeongwook Kim. Deeptwist: Learning model compression via occasional weight distortion. arXiv preprint arXiv:1810.12823, 2018.
Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual demonstrations. In Advances in Neural Information Processing Systems, pp. 3812-3822, 2017.
Haibin Ling and Kazunori Okada. An efficient earth mover's distance algorithm for robust histogram comparison. IEEE transactions on pattern analysis and machine intelligence, 2945):840-853, 2007.
Minghuan Liu, Tairan He, Minkai Xu, and Weinan Zhang. Energy-basse imitation learning. arXiv preprint vrXiv:2004.09395, 2020.
Xuanqing Liu, Si Si, Qin Cao, Sanjiv Kumar, and Cho-Jui Hsieh. Neural sde: Stabilizing neural ode networks with stochastic noise. arXiv preprint 12Xx:19906.02355, 2019.
Radford M Neal. Annealed importance sampling. Statistics and computing, (1(2):125-1) 139, 2001.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Advances in neural information processing systems, pp. 271-279, 2016.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. arXiv preprint vrXiv:100S.07057. 2017.
Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Computation, 3(1):88-97, 1991.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference on Machine Learning, pp. 1530-1538. PMLR, 2015.
Nicholas Rhinehart, Rowan McAllister, and Sergey Levine. Deep imitative models for flexible inference, planning, and control. arXiv preprint
2018
Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 661-668, 2010.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-635, 201
Sam T Roweis and Lawrence K Saul. Nonlinear dimensionality reduction by locally linear embedding. science, 290(5500) 2000.
11

---

Under review as a conference paper at ICLR 2022
Yannick Schroecker, Mel Vecerik, and Jonathan Scholz. Generative predecessor models for sample- efficient imitation learning. arXiv preprint arXiv: 2019.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, 2015a.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint 15Xiv 02438, 2015b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.063777 2017.
Simon J Sheather and Michael C Jones. A reliable data-based bandwidth selection method for kernel density estimation. Journal of the Royal Statistical Society: Series B (Methodological), 53(3): 683-690, 1991.
Ankur Sinha, Pekka Malo, Kalyanmoy Deb. review on bilevel optimization: from classical to evolutionary approaches and applications. IEEE Transactions Evolutionary Computation, 22 (2):276-295, 2017.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. arXiv preprint arXiv:1907.05500 2019.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033 IEEE, 2012.
Xin Zhang, Yanhua Li, Ziming Zhang, and Zhi-Li Zhang. f-gail: Learning f-divergence for generative adversarial imitation learning. Advances in Neural Information Processing Systems, 33, 2020.
Brian D Ziebart, Andrew L Maas, Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In AAAI, volume pp. 1433-1438. Chicago, IL. USA, 2008.
12


Reviews:
[{"review_id": "hCQKz-LLxEr", "reviewer": "Reviewer_SsHp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong Points\n* The empirical results are good: SLIL achieves good results, no matter whether few demonstrations or many demonstrations are provided.\n* The presentation is good, however, I have some suggestions below.\n\nWeak Points\n* Technical correctness (1): The main claim (Theorem 4.1) that SLIL tightly bounds the likelihood objective (LIL) is not supported. The SLIL objective differs by substituting the term $E_{p_\\text{E}(s)}\\big[ \\log p_{\\pi}(s) \\big]$ with $E_{p_{\\pi}(s)}\\big[ \\log p_{\\text{E}}(s) \\big]$, that is, switching the expert' and agent's state distribution. The proof has several issues: First of all, it starts with circular reasoning, by assuming that the optimal policy of the SLIL objective, $\\pi^\\star$,  matches the expert's state distribution. It then derives the bound (which is only valid for $\\pi^\\star$) that uses the equality $\\max_x  f(x) + \\max_x g(x) \\ge \\max_x \\big[ f(x) + g(x) \\big]$ which is not at all tight in general.\n* Technical correctness (2): The SLIL objective is further modified by introducing a hyper-parameter that disables the RL objective ever $\\lambda$ iterations, which effectively changes the objective being optimized without any theoretical justification.\n\n* Novelty (1): SLIL is a hybrid between BC and Energy-Based Imitation Learning (EBIL) [2], a reference certainly missing. Like SLIL, State-only EBIL starts by estimating the expert's state distribution and uses it as a fixed reward function for imitation learning. As the SLIL loss is basically an interpolation between two prior imitation learning losses the contribution seems minor, in particular given that the objective of SLIL is not well motivated (Technical correctness (1)). It is also worth noting that AIRL uses noise contrastive estimation to estimate the expert policy and uses this density estimate directly as reward for MaxEnt-RL, which is similar to the 2nd term of the SLIL objective. The differences are thus that AIRL ues MaxEnt-RL instead of RL, $r=\\log p_E(a|s)$ instead of $r=\\log p_E(s)$, and NCE with an EBM for density estimation rather than maximum likelihood with normalizing flows. \n\n* Novelty (2): The \"Denoising continuous normalizing flow\" seems to be a minor variant of Softflow [Kim et al. 2020]. Furthermore, this minor contribution is not evaluated at all (neither Softflow, nor any other methods for handling the manifold hypothesis challenge were compared). \n\n* Evaluation:  The improved stability compared to GAIL was not demonstrated. Please add learning curves for the data of Fig. 10 to the Appendix (choosing a suitable number of demonstrations (e..g. 5) for each experiment).\n\nAdditional Comments\n* It is not clear why the hyper-parameter lambda was introduced in favor of a fixed coefficient that trades off the two objectives, which would at least maintain a fixed objective function. \n* Much of 4.2 seems to be related work. I think it would make more sense to put it before the \"Method\" section to make the contributions more clear. In exchange, the problem formulation of LIL could be put under method, since it is introduced to \"derive\" the SLIL objective.\n* Algorithmbox 1 is almost useless, because it mainly shows that we have an iterative training loop, but doesn't really show what happens at each iteration.\n* End of page 5. The definition of $P_{E,\\sigma}(\\tilde{s}|s)$ looks like  $P_{E,\\sigma}(\\tilde{s},s)$ to me.\n\nReferences:\n[1] Liu, R., Gao, J., Zhang, J., Meng, D., & Lin, Z. (2021). Investigating bi-level optimization for learning and vision from a unified perspective: A survey and beyond. arXiv preprint arXiv:2101.11517.\n\n[2] Liu, M., He, T., Xu, M., & Zhang, W. (2020). Energy-based imitation learning. arXiv preprint arXiv:2004.09395.", "summary_of_the_paper": "The paper present a novel method for imitation learning aiming to stabilize training and mode coverage compared to existing methods. The proposed method (SLIL) extends behavioral cloning by an additional objective of maximizing a state-only reward that is defined by a density estimate (trained once beforehand) of the expert's state distribution.\nFor density estimation SLIL uses a continuous normalizing flow where the expert states are corrupted by Gaussian noise, where the noise-level is decreased during training. The noise is introduced to tackle the \"Manifold Hypothesis Challenge\", that is, the inability of normalizing flows to accurate approximate distributions that lie on a lower-dimensional manifold. \n\nThe main contributions are:\n1. A novel imitation learning method that combines the BC loss with a RL loss for reaching expert states\n2. A slightly modified approach to tackle the manifold hypothesis challenge (Kim et al. [2020] used Gaussian noise where the noise-level was sampled from a uniform distribution, independently for each data set, rather than using the same noise-level for all data and annealing it during optimization)", "summary_of_the_review": "The contributions seem rather incremental and the claims are not well-supported.\nI still gave a relatively high recommendation, because the results are fine and the overall ideas of 1) tackling covariance shift of BC by using an RL objective to match the expert state distribution, and 2) estimating the expert state distribution once instead of iteratively based on NCE with adapted noise to improve stability, are reasonable.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bbxs6c9q_VN", "reviewer": "Reviewer_JhT1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "$\\textbf{Strengths}$\n- The approach is very well motivated, aiming to mitigate issues of both BC (distributional shift) and GAIL-like approaches (mode collapse and instability of minimax optimization)\n- The algorithm performs very well on all benchmark environments, which are themselves distinct enough. \n- The ablations illustrate the robustness of the method to hyperparameter variations. \n\n$\\textbf{Weaknesses}$\n- In the description of the proposed approach to generative modeling, coined denoising continuous normalizing flows, there are a few points I do not understand, and where their may be some incorrect statements:\n  - It is claimed $P_{E,\\sigma}(\\tilde{s}|s) = P_E(s)\\mathcal{N}(\\tilde{s}| s, \\sigma^2I)$. I think the RHS is actually equal to the joint distribution  $P_{E,\\sigma}(\\tilde{s}, s)$ instead?\n  - The statement that follows about the $0$-noise limit is also incorrect I thinkj, their should be some sort of marginalization in order to have an accurate proof?\n  - In the algorithm, I think the noise scale in the Gaussians should be $\\sigma_i^2$ instead of $\\sigma_j^2$?\n\n", "summary_of_the_paper": "The paper proposes an approach to the problem of state-action based imitation learning, in which an agent aims to solve a specific task given state-action expert trajectories. Their objective SLIL combines (or interpolates between) generative modeling and behavioral cloning in order to avoid mode collapse of the agent policy (common issue of adversarial approaches to IL like GAIL) and to mitigate distributional shift which BC typically suffers from. \n\nContributions:\n- An extension of continuous normalizing flows, coined denoising continuous normalising flows (DCNF), which is claimed to alleviate the manifold hypothesis, and is trained via maximum likelihood estimation.\n- A strong imitation learning algorithm consisting in estimating the expert's state distribution, and learning a policy that recovers the expert policy and that has an occupancy distribution close to the distribution induced by generative model learned via the DCNF.\n", "summary_of_the_review": "I believe the paper proposes an interesting approach to imitation learning, however the technical issues described in the weaknesses section makes me currently tend towards a weak reject. If the authors were to successfully clarify these, I would consider raising my score. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "V9TTx5anU1", "reviewer": "Reviewer_nLuT", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Main strength\n--\nThis paper's main strength is that it proposes a straightforward method that both conceptually and empirically satisfies important desiderata of imitation learning methods (1) address the cascading error problem of BC-based methods (2) address the instability (mode collapse) of adversarially-trained IL methods (3) avoid dependence on an interactive demonstrator (i.e. DAGGER).\n\nMain weakness\n--\nThe paper's main weakness is some imprecision in a few places\n\n### Imprecision\nThere are some issues (a,b,c) with the proof. (a) The proof refers to component $\\enclose{circle}{2}$ in Eq 1, yet Eq. 1 has no denoted components. Is this reference correct? If so, then Eq 1. should denote the second component with an underbrace. If not, should it instead be Eq. 2 or Eq. 3? (b) The proof states \"with sufficient expert data $\\mathcal D_E$, $\\enclose{circle}{2}$ can be expressed with $\\sum_{s \\in \\mathcal D_\\pi} \\frac{P_E(s)}{P_\\pi(s)} \\log P_\\pi(s)$. I don't see how this is true, because $\\enclose{circle}{2}$ in Eq (3) is $\\sum_{s \\in \\mathcal D_\\pi} \\log P_E(s)$, i.e. the state distribution is the state distribution of the policy. Again, is the reference to $\\enclose{circle}{2}$ correct? These ambiguities / errors prevent me from being able to completely follow the proof. (c) It's not obvious from the definition of $g(\\pi, P_\\pi)$ that $\\sum_{s \\in \\mathcal D_{\\pi^*}} \\log P_{\\pi^*}(s) \\geq \\max_\\pi \\sum_{s \\in \\mathcal D_\\pi} \\log P_E(s)$ for two reasons (c.i) because the sums are not normalized by the the dataset size, it would be easy to invalidate the inequality by changing the dataset sizes (c.ii) What if $\\pi$ was such that $\\mathcal D_\\pi$ only contains $s$ at the mode of $P_{\\pi^*}(s)$? Wouldn't the RHS be at least as large as the LHS, in that case?\n\nS4.1: $P_E$ is undefined (it's mentioned that it's the expert's state distribution, but a formal definition is needed). Is it more correctly denoted $P_{\\pi_E}$, to \"typecheck\" with the subscripting of $P$ with a policy (i.e. $P_\\pi$) to denote the state visitation distribution induced by policy $\\pi$?\n\n### Other issues\n- The motivation of the gating parameter ($\\lambda$) seems to come from dataset size imbalance (between $\\mathcal D_E$ and $\\mathcal D_{\\pi_{\\theta_i}}$). Why can't this dataset balance be addressed by simply normalizing the losses by the size of the datasets? Can the authors comment? If this is insufficient, the paper needs to discuss why.\n- Missing related work: like the submitted paper, [A] also performs likelihood-based imitation learning with a normalizing flow for expert distribution matching, is not adversarially trained, and uses perturbation of the expert trajectories with gaussian noise. Discussion of the differences is important to contextualize the current paper's contributions, e.g. [A] used the distribution for planning rather than to learn a final policy\n- The description of the experiment corresponding to Figs 11, 12, and 13 is lacking. It's not clear what environment(s) the experiment are run on.\n- The introduction needs more justification for why DCNF is used in favor of other likelihood-based deep generative modeling approaches. Is continuity necessary? Is denoising necessary? Section 3 talks about the latter (C#2), perhaps incorporate some of this motivation into the introduction.\n\n[A] Rhinehart N, McAllister R, Levine S. Deep imitative models for flexible inference, planning, and control. arXiv preprint arXiv:1810.06544. 2018 Oct 15.\n", "summary_of_the_paper": "This paper proposes a straightforward non-adversarially-trained imitation learning method. The goal is to avoid mode collapse and training instability that can occur in some adversarially-trained imitaiton learning methods, while still addressing the distribution shift problem that basic Behavior Cloning is subject to. \n\nThe paper:\n- Motivates a two-stage learning procedure by starting with a likelihood-based state-action distribution matching objective that would require bi-level optimization\n- Presents a lower-bound of the objective in which optimization can occur serially of two components: MLE of the expert's vistation distribution, and then policy optimization over two terms, one of which is the LL of visited states under the the modelled visitation distribution, and the other is the behavior cloning loss.\n- Presents a proof that it is a lower bound. \n- Proposes a concrete approach to performing this optimization problem by using a denoising conditional normalizing flow for modeling the expert's visitation distribution, where the CNF seems to be motivated because of its ability to model densities of complex distributions, and the denoising is motivated to enable stable training in situations in which the data lie in a lower-dimensional manifold. \n- Performs experiments on 10 mujoco environments that illustrate that the proposed approach achieves better mode coverage than some comparable methods, that the denoising is empirically effective for generalization, that the algorithm is more robust than GAIL to variation in learning rate, among other hyperparmaeter variations.", "summary_of_the_review": "Despite some important issues with method presentation and justification, I think the paper presents a solid method and solid evidence of its utility. The paper can definitely be improved, but as it stands now, I think it's a good contribution.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FqhEb_SshP", "reviewer": "Reviewer_kBN4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Regarding the strength of this work, the contribution of this work over given baselines is clear, and I'm satisfied with the detailed explanations and experiments, e.g., comparison with LIL, the justification of using DCNF rather than CNF, how DCNF is trained algorithmically, quantitative tests using EMD and test log likelihoods, etc. \n\nRegarding the weakness of this work, I think some recent relevant works are missed, e.g., Dadashi et al., \"Primal Wasserstein Imitation Learning\", Kim et al., \"Imitation with Neural Density Models\", and some references therein. Although these papers didn't aim to solve the mode collapsing directly, their algorithms are much more sample-efficient than GAIL in terms of the number of expert trajectories and the number of environment interactions (i.e., GAIL is no more a SOTA algorithm as stated in this work), and both are relevant to this work in the sense that expert support estimation is used. Primal Wasserstein Imitation Learning (PWIL) shows their algorithm works efficiently even when we have a small number of expert trajectories (1 or 11 trajectories), whereas this work presented empirical results assuming more than 4 expert trajectories (more than 80 expert trajectories for Humanoid). In \"Imitation with Neural Density Models\", energy-based model is used to estimate expert support estimation, which are based on maximmum likelihood estimation similar to this work. ", "summary_of_the_paper": "This work proposed Stabilized Likelihood-based Imitation Learning (SLIL) which iteratively estimates the expert state distribution by using Denoising Continuous Normalizing Flow (DCNF) and maximizes the policy learning objective in Eq. (3) that matches expert policy and state distribution. Compared to other baselines such as BC, DRIL, GAIL, SLIL was shown to find out better expert state distribution (in terms of seeking multiple modes, Earth Mover's Distance (EMD) while achieving the empirical return close to that of the expert (which is widely used in the imitation learning literature). The strength of using DCNF instead of using Continuous Normalizing Flow (CNF) is well desribed and supported by the experiments evaluating the test log likelihood in Figure 9. Authors also analyze SLIL's hyperparameter sensitivity that supports the claim on the SLIL's stability. \n", "summary_of_the_review": "Overall, writing is clear, and the empirical results are shown to outperform GAIL and some other baselines. However, some SOTA algorithms (which I believe is relevant to this work) are missed, and it is unclear that the mode collapsing issue also happens for those relevant works or not. Although I agree that there is an algorithmic novelty for this work, I believe more elaborated results should be given with the aforementioned references. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes the use of a state distribution estimation objective with a classic behavioral cloning objective for imitation learning. 
The submission also proposes the use of a continuous normalizing flow training technique coined "denoising normalizing flow" to learn the state distribution. The authors experimentally validate their method on several MuJoCo continuous control benchmarks.
The theorem 4.1 does validate the fact that this proposed objective is can be maximized by the target policy.
However, the technical contributions (proposal of new objective and the denoising normalizing flow method) are marginal compared to previous work (e.g., SoftFlow or Energy-Based Imitation Learning). 
The empirical validation is lacking more extensive comparison with PWIL or NDI, which are more recent methods attempting to address the challenges described in the submission. 
I'm recommending this paper for rejecting for this conference.
Decision: Reject