Imitation with Neural Density Models
Authors: Kuno Kim
Akshat Jindal
Yang Song
Jiaming Song
Yanan Sui
Stefano Ermon
Venue: NeurIPS 2021 Poster (2021)
Keywords: Imitation Learning
Reinforcement Learning
Density Estimation
Density Model
Maximum Entropy RL
Mujoco

Abstract: We propose a new framework for Imitation Learning (IL) via density estimation of the expert's occupancy measure followed by Maximum Occupancy Entropy Reinforcement Learning (RL) using the density as a reward. Our approach maximizes a non-adversarial model-free RL objective that provably lower bounds reverse Kullback–Leibler divergence between occupancy measures of the expert and imitator. We present a practical IL algorithm, Neural Density Imitation (NDI), which obtains state-of-the-art demonstration efficiency on benchmark control tasks. 

Imitation with Neural Density Models
Kuno Kim', Akshat Jindal', Yang Song', Jiaming Song', Yanan Sui?, Stefano Ermon' 'Department Computer Science, Stanford University NELN, School of Aerospace Engineering, Tsinghua University
Contact: khkimêcs. stanford. edu
Abstract
We propose a new framework for Imitation Learning (IL) via density estimation the expert's s occupancy measure followed by Maximum Occupancy Entropy Rein- forcement Learning (RL) using the density as a reward. Our approach maximizes non-adversarial model-free RL objective that provably lower bounds reverse Kull- back-Leibler divergence between occupancy measures of the expert and imitator. We present practical IL algorithm, Neural Density Imitation (NDI), which obtains state-of-the-ee demonstration efficiency on benchmark control tasks.
Introduction
Imitation Learning (IL) algorithms aim to learn optimal behavior by mimicking expert demonstrations. Perhaps the simplest HI. method is Behavioral Cloning (BC) (Pomerleau 199T) which ignores the dynamics of the underlying Markov Decision Process (MDP) that generated the demonstrations, and treats II. as a supervised learning problem of predicting optimal actions given states. Prior work showed that the learned policy incurs a small BC loss, the worst case performance gap between the expert and imitator grows quadratically with the number decision steps (Ross & Bagnell 2010 Ross et al. 2 The crux of their argument is that policies that are "close" as measured by BC loss can induce disastrously different distributions over states when deployed in the environment. One family of solutions to mitigating such compounding errors is Interactive II. (Guo er al.) 2014) Ross ct al. 2011b 3013). which involves running the imitator's 's policy and collecting corrective actions from an interactive expert. However, interactive expert queries are expensive and seldom available.
Another family of approaches (Fu al. 2017 Ho & Ermon, 2016: Ke et al., 2020 Kim & Park 2018 Kostrikov et al. (2200 Wang et al. 2017) that have gained much traction Is to directly minimize statistical distance between state-action distributions induced by policies of the expert and imitator, ic the occupancy measures Pag and Pr.. As Pro is an implicit distribution induced by the policy and environment distribution matching with PTD typically requires likelihood-fres methods involving sampling. Sampling from Pn. entails running the imitator policy in the environment, which was not required by BC. While distribution matching IL requires additional access to environment simulator, it has been shown to drastically improve demonstration efficiency, i.e the number of demonstrations needed to succeed at IL (Ho & Ermon 2016). A wide suite of distribution matching IL algorithms adversarial methods to match Pr. and Parpa which requires alternating between reward (discriminator) and policy (generator) updates (Fu et al. 2017 Ho & Ermon 2016; Ke et al.)
2020: Kim et al. 2019: Kostrikov et al., 2020). A key brawback to such Adversarial Imitation Learning (AIL) methods that they inherit the instability of alternating min-max optimization Miyato et al. 2018; Salimans et al. 2016) which is generally not guaranteed to converge (Jin et al. 2019). Furthermore, this istability is exacerbated in the IL setting where generator updates involve high-variance policy optimization and leads to sub-optimal demonstration efficiency. To alleviate this instability, (Brantley et al. 2020 Reddy et al. 2017 Wang et al. 2019) have proposed to do RL with fixed heuristic rewards. Wang et al. 2019), for example, uses heuristic reward that estimates the
assume only samples can be taken from the environment dynamics and its density is unknown
35th Conference Neural Information Processing Systems (NeurIPS 2021).

---

support of Pxx which discourages the imitator from visiting out-of-support states. While having the merit of simplicity, these approaches have no guarantee of recovering the true expert policy.
In this work, we propose a new framework for IL via obtaining a density estimate qs of the expert's occupancy measure Prs, followed by Maximum Occupancy Entropy Reinforcement Learning (Max- OccEntRL) [Ilam et al. 2019 Lee et al. 2019). In the MaxOccEntRL step, the density estimate ds used as a fixed reward for RL and the occupancy entropy H((pz) ) is simultaneously maximized, leading to the objective max, Ep., [log (s( a) + H(Pm)). Intuitively, our approach encourages the imitator to visit high density state-action pairs under PTE. while maximally exploring the state-action space. There are two main challenges to this approach. First, we require accurate density estimation of Prg. which is particularly challenging when the state-action space is high dimensional and the number of expert demonstrations are limited. Second, in contrast to Maximum Entropy RI. (Max-
EntRL), Max McXOOLLE requires maximizing the entropy of implicit density ara- We address the former challenge leveraging advances in density estimation (Du & Mordatch] 2018 Germain
et al. 2015 Song et al. 2019). For the latter challenge, we derive a non-adversarial model-free RL objective that provably maximizes a lower bound to occupancy entropy. As byproduct, we also obtain a model-free RL. objective that lower bounds reverse Kullback-Lieber (KL. divergence between Pr. and pag. The contribution of our work is introducing novel family of distribution matching IL algorithms, named Neural Density Imitation (NDI), that (I) optimizes principled lower bound to the additive inverse of reverse KI., thereby avoiding adversarial optimization and (2). advances state-of tf-t---rrt demonstration efficiency in IL.
2 Imitation Learning via density estimation
We model agent's decision making process as discounted infinite-horizon Markov Decision Process (MDP) .M- = (S, A, P, Po. r, 7.7.. Here S,A are state-action spaces, P : S x A + ((S) is transition dynamics where S?(S) is the set of probability measures on S, Po : S + R is an initial state distribution, r S x A- -> R is reward function, and y E [0, 1) is a discount factor. A parameterized policy T : S + (2A) distills the agent's s decision making rule and {st, ae } is the stochastic process realized by sampling an initial state from ~ Po(s) then running To in the environment, i.e at Tha (-[s,), st+1 P(-|s,, oz). We denote by p9, 29 the joint distribution of states {st, $t+1,..., )t+k), where setting Po.e recovers the marginal of 8t. The (unnormalized) occupancy measure of xy is defined as Pro (s) Ei= 1o--o"PPe (')),r(s) (als). Intuitively, Pro (s. a) quantifies the frequency of visiting the atatenaction pair (s, a) when running Ta for long time, with more emphasis on earlier states.
We denote policy performance as J(TT.. F) = Exo EE= ("F(st. at)l = E(s.a>>p., [(s a)] where F is a (potentially) augmented reward function and E denotes the generalized expectation oper- ator extended to non-normalized densities p X' + T and functions X' - y so that [p[f(z)] = Σ f(=)(()). The choice of depends on the RL framework. In standard RL, we simply have F = r, while in Maximum Entropy RL (MaxEmtLL) (Haarnoja et al. 2017). we have F(s, a) = r(s.a) log (alss). We denote the entropy of Pao (s, a) as H(Pne).
1p-,,- Po, (s, a)] and overload notation to denote the 7-discounted causal entropy of policy To as H((T)) = Ex, 1E -y log (a st = Ep., |- log xe (a) s)]. Note that we use general- ized notion of entropy where the domain is extended to non non-nnomaalized densitics. We can then define Lee et al., 2019) objective as J 7 r) (Pr,)). Note the key difference between MaxOecEntR. and Max entropy regularization is on the occupancy measure instead of the policy, i. seeks state diversity instead of action diversity. We will later show in section 2.2 that lower bound on this objective reduces to complete model-free RL objective with an augmented reward
the Maximum Occupancy Entropy RL (MaxOccEnRL) (Islam et al. Lee
Let TE. no denote an expert and imitator policy, respectively. Given only demonstrations D {(s, ) { learn a policy ne which matches the expert, ie ne T Tg. Formally, IL can be recast as a distribution
Te of state-action pairs sampled from the expert, Imitation Learning (IL) aims to
matching problem (Ho & Ermon 2016
Ke et al. 2020) between occupancy measures Pr. and Pagi
maximizea - d(pra PTx)
(1)
where d(p. q) is a generalized statistical distance defined on the extended domain of (potentially) non normalzad probability densities f(x), q(x) with the same normalization factor Z > 0, ie
S. (())\\Z =Ix (Z=)/Z = 1. For Pa and Pag. we have Z Nlx. As we are only able to take

---

samples from the transition kernel and its density is unknown, Pro, is an implicit distributior Thus, optimizing Eq. [ typically requires likelihood-free approaches leveraging samples from Pr.. i.e running Th in the environment. Current state-of the-ahttt art IL approaches use likelihood- adversarial methods to approximately optimize Eq. for various choices of d such as reverse Kullback-Liebler (KL) divergence (Fu al. 2017 Kostrikov al., 2020) and Bensen-Shannon (JS) divergence (Ho & Ermon 2016). However, adversarial methods are known to suffer from optimization instability which is exacerbated in the IL setting where one step in the alternating optimization involves RL.
We instead derive a non-adversarial objective for IL. In this work. we choose d to be (generalized) reverse-Ki KI. divergence and leave derivations for alternate choices of to future work.
-DKI(P50llPte)
= Epn, [lg Pr Pr. a) - log Par (s, a)]
= [ F Pxx) + H((pm))
(2)
We see that maximizing negative reverse-KL. with respect to Ty is equivalent to Maximum Occupancy Entropy RL (MaxOc<EntRL) with log Pr as the fixed reward. Intuitively, this objective drives ла to visit states that are most likely under Prg while maximally spreading out probability mass so that if two state-action pairs are equally likely, the policy visits both. There are two main challenges associated with this approach which we address in the following sections.
1. log Pig is unknown and must be estimated from the demonstrations D. Density estimation remains challenging problem, especially when there a limited number of samples and the data is high dimensional (Liu al. 2007). y. Note that simply extracting the conditional x (als) from an estimate of the joint Px s. a an alternate way to do BC and does not resolve the compounding error problem (Ross al.
201
2. H(ppre) is hard maximize PTo is an implicit density. This challenge is similar to the difficulty of entropy regularizing generators (Belghazi et al. 2018: Dieng et al. 2019: Mohamed & Lakshminarayanan 2016) for Generative Adversarial Networks (GANs) (Goodfellow et al.[2014] and most existing approaches (Dieng et al. 2019
Lee et al. 2019) use adversarial optimization.
2.1 Estimating the expert occupancy measure
We seek learn a parameterized density model (s a) of Pry from samples. We consider two canonical families of density models: Autoregressive models and Energy-based models (EBMs).
Autoregressive Models (Germain et al. 2013; Papamakarios et al. 2017;: An autoregressive model
(x(x) for x (s, a) learns a factorized distribution of the form: lo (x) T, (Zg(aes)) For instance, each factor 9o. could be a mapping from Ze Gaussian density over z;. When given a
prior over the true dependency structure of {a: ). this can be incorporated by refactoring the model. Autoregressive models typically trained via Maximum Likelihood Estimation (MLE).
Energy-based Models (EBM) (Du & Mordatch 2018 Song et al., 2019): Let Ess : S A be an energy function. An energy based model IS a parameterized Boltzman distribution of the form (s, a) Z(a) e -E((s,)) where Z(0) = SSxA e dada) denotes the partition function. Energy-based models are desirable for high dimensional density estimation due to their expressivity, but are typically difficult to train due to the intractability of computing the partition function. However, our II. objective in Eq. [ conveniently only requires non-normalized density estimate as policy optimality is invariant to constant shifts in the reward. Thus, we opted to perform non non-normalized density estimation with EBMs using score matching which allows us to directly learn & without having to estimate (Ø))
2.2 Maximum Occupancy Entropy Reinforcement Learning
In general maximizing the entropy of implicit distributions is challenging due to the fact that there no analytic form for the density function. Prior works have proposed using adversarial methods involving noise injection (Dieng et al. 2019) and fictitious play (Brown) 1951]Les Lee et al., 2019). We instead propose to maximize novel lower bound to the additive inverse of an occupancy divergence which we prove is equivalent to maximizing a non-adversarial model-free RI. objective. We first make clear the assumptions on the MDPs considered henceforth.
ppoobability models that have potentially intractable density functions, but can be sampled from to estimate expectations and gradients of expectations with respect to model parameters (Huszár 2017).
3

---

Assumption All considered MDPs have deterministic dynamics governed by a transition function
P S x + 8. Farthermore, P is injective with respect to a A, ie Vs, 0, a' it holds that a# =
We note that Assumption I holds for most continuous robotics and physics environments as they are deterministic and inverse dynamics functions p-1 S S A have been successfully used in benchmark RL environments such as Mujoco (Todorov] 2014; Todorov et al. 2012) and Atari (Pathak et al.] 2017). Next we introduce crucial ingredient in deriving our occupancy entropy lower bound, which is a tractable lower bound Mutual Information (MI) first proposed by Nguyen. Wainright, and Jordan (Nguyen et al. 2010), also known as the f-GAN KL (Nowozin
et al.
2016) and MINE-f (Belghazi et al. 2018). For random variables X, Y distributed according to Po.y (2 Po_ (4), Po,(0 (v) where & = (Bay, Uz By), and any critic function f X x y + E, it holds
that
T ((XY 2 Iwwa (X; (X; Y where,
kww (X; 010) == Evezy (fxx, v)] - -E-'Eop [ro, [e
This bound is tight when f is chosen to be the optimal critic/ (", v) = log now ready to state lower bound to the occupancy entropy.
Peay (a.w) poz, (4)ppy + 1. We are
Theorem
Let MDP M satisfy assumption (App. A. For any critic x:SxS-EE it holds that
H(PT)) H (pro)
(4)
where
H. (PTx) = ((0))) + (1 + ())) + Σ (8
(5)
1=0
See Appendix A. for the proof and discussion of the bound tightness. Here onwards, we refer to H (px.) from Theorem as the State-Action Entropy Lower Bound (SAELBO). The SAELBO mainly decomposes into policy entropy H((Ta) and Mutual Information (MI) between consecutive When Assumption
1 does not hold, may still obtain SAELBO with only the policy entropy term, i. H (p=.) H(mT)) < H((m)) but this bound has more slack and is limited to discrete state-spaces. (see Appendix [A] for details) Since occupancy entropy maximization is also a desirable exploration strategy in sparse environments (Hazan et al. 2019 Lee al. 2019), another interpretation of the SAELBO is surrogate objective for state-action level exploration. Furthermore, we posit that maximizing the SAELBO is more effective for state-action level exploration, i.e occupancy entropy maximization, than solely maximizing policy entropy. This because, in discrete state-spaces, the SAELBO is tighter lower bound to occupancy entropy than policy entropy, i. H((T)) < H (px.) ) S H(Pre). and in continuous state-spaces, where Assumption hoolds the SAELBO still a lower bound while policy entropy alone is neither a lower nor upper bound to occupancy entropy. Please see Appendix C.J for experiments that show how SAELBO maximization can improve state-action level exploration over just policy entropy maximization. Next, we show that the gradient of the SAELBO equivalent to the gradient of model-free RL objective.
states (
Theorem 2
Let qu (als) and (ge()) be probability densities such that Vs, E S x A satisfy qa (als) (a(a)) and q((s) (s), Then for all S x S R.
VOHt (pro) (oJ ra +rz)
(6)
where
Tx(8 at) = ((1 7) log qz (at1st)
(7)
ry ry(s (1, 8441) = (st, 8041) 2 S (fessee))) + ((0,54+11)
(8)
See Appendix A.2 for the proof. Theorem shows that maximizing the SAELBO is equivalent to maximizing a discounted model-free RL objective with the reward Fn +Ty, where Ta contributes to maximizing H(())) and contributes to maximizing Eeo rf entails estimating expectations with respect to qt. 9t+1- This can be accomplished by rolling out multiple trajectories with the current policy and collecting the states from time-step t, f + Alternatively if we assume that the policy is changing slowly, we can simply take samples of states from time-step t, t + from the replay buffer. Combining the results of Theorem T we end the section with a lower bound on the original distribution matching objective from Eq. aad show that maximizing this lower bound is again, equivalent to maximizing model-free RL objective.
Note that evaluating

---

Algorithm 1: Neural Density Imitation (NDI)
Require: Demonstrations D Tg. Reward weights Ar. Ay, Fixed critic f
2 Phase I. Density estimation:
3 Learn go(s, a) from D using MADE or EBMs
4 Phase 2. MaxOCcEntRL:
5 for ht = 1,2,... do Collect (st, ar, ar, S1+1, F) na and add to replay buffer B, where
F log 9s 7T1T + A
Tx (1,, ar) -1 + (og)o((r(ss) ( at, S(+1) = =f s(+1) - B.
[ef 1.8() + e( [,00
and the critic is computed by
||
f(+++1; 8t) log
18+1 [
|80+1 St
Update using Soft Actor-Critic (SAC)
(Haarnoja et al.
2018
end
Corollary Let MDP M satisfy assumption (App.
4
For any critic f: 8 - IR, it holds that
-DKI. (etelllPg) 2 [ F
log pmg) ) + H! (pro)
(9)
Furthermore, les Tw, Vy be defined as Theorem P Then,
(e( log ( + H (Pxo)) VeJ F = log PTE +Tx +
(10)
In the following section we derive practical distribution matching II. algorithm combining all the ingredients from this section.
3 Neural Density Imitation (NDI)
From previous section's results, we propose Neural Density Imitation (NDI) that works in two phases:
Phase 1: Density estimation: We leverage Autoregressive models and EBMs for density estimation
of the expert's 's occupancy measure PTh from samples. As in (Fu al. 2017 Ho & Ermon 2016). we take the state-action pairs in the demonstration set D {(s, a): samples from Pr.. and fit qs on D. For Autoregressive models, Density Estimation (MADE) (Germain et al. 2015) where the entire collection of conditional density models {go; is parameterized by a single masked autoencoder network. Specifically, we use a gaussian mixture variant (Papamakarios al. 2017) of MADE where each of the conditionals 4s, map inputs wei to the mean and covariance of gaussian mixture distribution over I;, The MADE is trained via Maximum Likelihood Estimation. With EBMs, we perform non-normalized log density estimation and thus directly parameterize the energy function &g. with neural networks since log 4s Eg + log (()) We use Sliced Score Matching (Song et al. 2019) to train the EBM.
TE to approximate Masked Autoencoders for
Phase 2: MaxOccEntRL After we've acquired a log density estimate log qa from the previous phase, we perform RI with entropy regularization on the occupancy measure. Inspired by Corollary we propose the following RL objective
max J (To. F log 96 + AxT' + Xyrs) #
(11)
where Xx, Ay > 0 are weights introduced to control the influence of the occupancy entropy regular- ization. In practice, Eq. [1 can be maximized using any RL algorithm by simply setting the reward function to be F from Eq. In this work, we use Soft Actor-Critic (SAC) (Haarnoja et al. 2018). Note that SAC already includes a policy entropy bonus, critic/ f. we fix be normalized RBF kernel for simplicity,
do not separately include one. For
e || 2
f(s st) log E 8041
+1
(12)
5

---

Table 1: Comparison between different families of distribution matching IL algorithms
IL method
Learned Models
Relation between -iveegencce
Objective Type
and optimized objective
Support
policy ne. support estimator f
Neither Upper nor Lower Bound
max
Adversarial
policy na, discriminator D
Tight Upper Bound
min max
NDI (ours)
policy ne. critic f. density 4
Loose Lower Bound
max max
but future works could explore learning the critic to match the optimal critic. While simple, our
choice of emulates two important properties of the optimal critic (x, log p(z) + 1: (I).
p(xly]
follows the same "form" of log-density ratio plus constant (2). consecutively sampled states from the joint, i.e St, St+1 p0,a++ have high value under our f since they are likely to be close to each other under smooth dynamies. while samples from the marginals 8z, 8t+1 qr. 9t+1 are likely to have lower value under / since they can be arbitrarily different states. To estimate the expectations with respect to qu, 92+1 in Eq. [ we simply take samples of previously visited states at time t, + from the replay buffer.
4 Trade-offs between Distribution Matching IL algorithms
Adversarial Imitation Learning (AIL) methods find a policy that maximizes an upperbound the additive inverse of an f fddivergence between the expert and imitator occupancies (Ghasemipour et al. 2019 Ke et al. 2020 For example, if the f-divergence is reverse KL, then for any D:8xA==
max DKL. (Pro llpa)) )
max log (E (E ((oa))) Ex [D(
Prip (s.a) where the bound tight at (ss, a) log
+ C for any constant C. AIL alternates between,
peg (5,a)
min log [B 1e ((s,)))
Ex,, (D),
D
max - E., (((,aa) Tw
The discriminator update step in AIL minimizes the upper bound with respect to D, tightening the estimate of reverse KL, and the policy update step maximizes the tightened bound. We thus see that by using an upper bound, AlL innevitably ends up with alternating min-max optimization where policy and discriminator updates act in opposing directions. The key issue with such adversarial optimization lies not in coordinate descent itself, but in its application to a max objective which is widely known to gives rise to optimization instability (Salimans et al. 2016).
The key insight of NDI is instead derive objective that lower bounds the additive inverse of reverse KI. Recall from Eq. 9 that NDI maximizes the lower bound with the SAELBO H' (pm,):
max XB
DKI. (Pro (l
max J F log PTT + H (prw)
Unlike the AIL upper bound, this lower bound is not tight. With critic f updates, NDI alternates
Y 7' (1,111011)))10))
max [ F logPrg) Pmg) + (1 (1+7)H(TTT) + Σ 1
The critic update step in NDI maximizes the lower bound with respect to f. tightening the estimate of reverse KI., and the policy update step maximizes the tightened bound. In other words, for AlI., the policy ne and discriminator the policy no and critic f push the lower bound in the same direction. Unlike All, NDI does not perform alternating min-max but instead alternating max-max!
seek to push the upper bound in opposing directions while in NDI

---

While NDI enjoys non-adversarial optimization, it comes at the cost of having to use non-tight lower bound to the occupancy divergence. On the otherhand, AlL optimizes tight upper bound at the cost of unstable alternating min-max optimization. Support matching IL algorithms also avoid min-max but their objective is neither an upper nor lower bound to the occupancy divergence. Table summarizes the trade-off's between different families of algorithms for distribution matching II..
Related Works
Prior literature on Imitation learning (IL) in the absence of an interactive expert revolyes around
Behavioral Cloning (BC) (Pomerleau) 1991 Wu al. 2019), distribution matching IL (Ghasemipour
et al., 2019 Ho & Ermon] 2016 Ke et al. 2020: Kim et al.
2019:
Kostrikov et al. 2020; Song
er al. 2018). and Inverse Reinforcement Learning (Brown et al. 2019 Fu et 2017 Uchibe 2018).
Many approaches in the latter category minimize statistical divergences using adversarial methods
to solve a min-max optimization problem, alternating between reward (discriminator) and policy (generator) updates. ValueDICE, more recently proposed adversarial IL approach, formulates reverse KL divergence into completely off- oof-col objective thereby greatly reducing the number of environment interactions. A key issue with such Adversarial Imitation Learning (AIL) approaches is optimization instability Q et al. 2019, Miyato et al. 2018). Recent works have sought to avoid adversarial optimization by instead performing RL with heuristic reward function that estimates the support of the expert occupancy measure. Random Expert Distillation (RED) (Wang et al. 2019) and waaggreemenn-regulatie
IL (Brantley et al. 2020) are two representative approaches this family. A key limitation of these approaches is that support estimation insufficient to recover the expert policy and thus they require an additional behavioral cloning step. Unlike AIL, we maximize a non-adversarial RL objective and unlike heuristic reward approaches, our objective provably lower bounds reverse KL between occupancy measures of the expert and imitator. Density estimation with deep neural networks is an active research area, and much progress has been made towards modeling high-dimensional structured data like images and audio. Most successful approaches parameterize a normalized probability model and estimate with maximum likelihood, e.g., autoregressive models (Germain et al. 2015, Uria et al., [2033] 2016) van den Oord et al., (016) and normalizing flow models (Dinh et al.J 2014 2) Kingma & Dhariwal (2018). Some other methods explore estimating
non-normalized probability models with MCMC (Du & Mordatch 2019 Yu et al. 2020) or training
with alternative statistical divergences such as score matching (Hyvärinen] 2005 |Song & Ermon
2019 Song et al. 2019) and noise contrastive estimation (Gao et al. [2019] Gutmann & Hyvärinen
2010). Related to MaxOccEntRRL, recent works (Hazan et al. 2019 Islam et al.) 2019 [Lee et al.
2019) on exploration in RL have investigated sate-marginal occupancy entropy maximization. To
do so, (Hazan al. 2019) requires access to a robust planning oracle, while (Lee et al,, 2019) uses fictitious play, an alternative adversarial algorithm that is guaranteed to converge. Unlike these works, our approach maximizes the SAELBO which requires planning oracle nor min-mas optimization, and is trivial to implement with existing RL algorithms.
Experiments
Following prior work, we experiments on benchmark Mujoco (Brockman et al.. 2016 Todorov et al. 2012) tasks: Hopper (11, 3), HalfCheetah (17, 6). Walker (17, 6), Ant (111, 8), and Humanoid (376. 17), where the (observation, action) dimensions are noted parentheses.
Environment:
We train expert policies using SAC (Haarnoja et al. 2018). All of our results are averaged across five random seeds where for each seed we randomly sample a trajectory from an expert, perform density estimation, and then MaxOceEntRL. Performance for each seed is averaged across 50 trajectories. For each seed we save the best imitator as measured by our augmented reward F from Eq. [ and report its performance with respect the ground truth reward. We don't perform sparse subsampling on the data as in (Ho & Ermon 2016) since real world demonstration data typically aren't subsampled to such extent and using full trajectories was sufficient compare performance.
Pipeline:
We experiment with two variants of our method, NDI+MADE and NDI+EBM, where the only difference lies the density model. Across all experiments, our density model qo is a two-layer MLP with 256 hidden units. For hyperparameters related to the MaxOccEmRLL step.
Architecture:
d = 0.2 is fixed and for 'y see Section 6.3 For full details architecture see Appendix B
Baselines: We compare our method against the following baselines: (1). Behavioral Cloning
(BC) (Pomerleau) 199T): learns a policy via direct supervised learning on D. (2). Random Expert
Distillation (RED) (Wang et al. 2019): estimates the support of the expert policy using predictor
7

---

Table 2: Task Performance when provided with demonstration. NDI (orange rows) outperforms all baselines on all tasks. See Appendix C.2 for results with varying demonstrations.
HOPPER
HALF-CHEETAH
WALKER
ANT
HUMANOID
RANDOM
14
282 1 80
1 1
-70 = 111
123 1 35
BC
1432 + 382
2674 + 633
1691 + 1008
1425 + 812
353 + 171
RED
448 + 516
383 819
309 + 193
910 175
242 + 67
GAIL
3261 + 533
3017 531
3957 + 253
2299 + 519
204 + 67
VALUEDICE
2749 +
3456 + 401
3342 1514
1016 313
364 + 50
NDI+MADE
3288 + 94
41
4518 - 127
555 311
6088 - 689
NDI+EBM
3458 + 210
4511 + 569
5061 + 135
4293 + 431
5305 + 555
EXPERT
3567 + 4
4142 1: 132
5006
472
4362 1 827
5417 1 2286
2018). followed by RL using this heuristic reward. (3). Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon 2016): on-policy adversarial IL method which
and target network (Burda et
alternates reward and policy updates. (4). ValueDICE Kostrikov et al. 2020): current state-of-the-ar
adversarial IL. method that works off- off-policy. See Appendix B for baseline implementation details.
6.1 Task Performance
Table P compares the ground truth reward acquired by agents trained with various IL algorithms when one demonstration is provided by the expert. (See Appendix C.2 for performance comparisons with varying demonstrations) NDI+EBM achieves expert level performance
all mujoco bench- marks when provided one demonstration and oulperforms all baselines on all mujoco benchmarks. NDI+MADE achieves expert level performance on 4/5 tasks but fails on Ant. We found spurious modes in the density learned by MADE for Ant, and the RL algorithm maxima. We found that baselines are commonly unable to solve Humanoid with one demonstration (the most difficult task considered). RED is unable to perform well on all tasks without pretraining with BC as done in (Wang et al.] 2019). For fair comparisons with methods that do not use pretraining, we also do not use pretraining for RED. See Appendix C.4 for results with a BC pretraining step added to all algorithms. GAIL and ValueDICE perform comparably with each other, both outperforming behavioral cloning. We note that these results are somewhat unsurprising given that ValueDICE (Kostrikov et al. 2020) did not claim to improve demonstration efficiency over GAIL (Ho & Ermon 2016). but rather focused on reducing the number of environment interactions. Both methods notably under-perform the expert on Ant-v3 and Humanoid-v3 which have the largest state-action spaces. Although minimizing the number of environment interactions was not a targeted goal of this work, we found that NDI roughly requires order of magnitude less environment interactions than GAIL. Please see Appendix for full environment sample complexity comparisons.
converging to these local
6.2 Density Evaluation
In this section, we examine the learned density model qa for NDI+EBM and show that it highly correlates with the true mujoco rewards which are linear functions of forward velocity. We randomly sample test states and multiple test actions aa per test state, both from uniform distribution with boundaries the minimum/maxiimmm state-action values in the demonstration set. We then visualize the log marginal log 2e (8) = log Ea, (8(8, ag) projected on to two state dimensions: one corresponding to the forward velocity of the robot and the other a random selection, e.g the knee joint angle. Each point in Figure D corresponds to a projection of sampled test state and the colors scale with the value of log (s). For all environments besides Humanoid, we found that the density estimate positively correlates with velocity even on uniformly drawn state-actions which were not contained in the demonstrations. We found similar correlations for Humanoid on states in the demonstration set. Intuitively, a good density estimate should indeed have such correlations, since the true expert occupancy measure should positively correlate with forward velocity due to the expert attempting to consistently maintain high velocity.
6.3 Ablation studies
As intuited in Section 2 maximizing the SAELBO can be more effective for occupancy entropy maximization, than solely maximizing policy entropy. (see Appendix C.T for experiments that support this) This is because in discrete state-spaces the SAELBO H. (p*o) is tighter lower bound
8

---

Hoppet-73
boothCbe
Walker2d-13
Ant-v3
Hnmanosd-v3
Forward Volocity
Soug (
0c
0.0
60
00
-08
-14
Randem Observation Damension
Figure 1: Learned density visualization. We randomly sample test states s and multiple test actions as per test state, both from a uniform distribution, then visualize the log marginal log (s(() = log Ex, go(8, ax) projected onto two state dimensions: one corresponding forward velocity and the other random selection. Much like true reward function in Mujoco environments, we found that the log marginal positively correlates with forward velocity on 4/5 tasks.
Table 3: Effect of varying MI reward weight X on (1). Task performance of NDI-EBM (top row) and (2). Imitation performance of NDI-EBM (bottom row) measured as the average KL divergence between π, TE states sampled by running in the true environment, i. Egwn [DKL (T( (TT
normalized by the average DKL between the random and expert poli- cies. (T [T)) can be computed analytically since म, T are conditional gaussians, Density model qø is trained with one demonstration. Setting ' too large hurts task performance while setting
too small suboptimal for matching the expert occupancy. middle point of Xy 0.005 achieves
a balance between the two metrics.
HOPPER
HALF-CHEETAH
WALKER
ANT
HUMANOID
REWARD
3576 154
5658 + 698
5231 122
4214
5809 + 591
Xy
0
KL
0.13 + 0.09
0.35 + 0.12
0.31 = 0.08
0.58 + 0.09
0.55 1 0.21
REWARD KL
3506 188
5697 + 805
5171 157
4158 523
5752 632
Xy
= 0.0001
0.15 + 0.05
0.32 = 0.15
0.25 0.04
0.51 0.05
0.41 + 0.18
REWARD
3458 + 210
4511 + 569
5061 + 135
4293 + 431
5305 + 555
\ 0.005
KL
0.11 + 0.02
0.17 = 0.09
0.22 0.14
0.32 - 0.12
0.12 + 0.14
REWARD
1057 + 29
103 + 59
2710 + 501
-1021 + 21
142 + 50
Xy 0.1
KL
0.78 + 0.13
1.41 = 0.51
0.41 = 0.11
2.41 + 1.41
0.89 + 0.21
EXPERT
REWARD
3567
4142 + 132
5006 + 472
4362
827
5417 + 2286
to occupancy entropy H)))))) ) than policy entropy H((T)) ie H(TT)) < H (pm.) < H(Pz)). and in continuous state-spaces, where Assumption [atuupptoonhhhlls holds, the SAELBO is still lower bound while policy entropy alone is neither a lower nor upper bound to occupancy entropy. As an artifact, we found that SAELBO maximization (A, >> 0) leads to better occupancy distribution matching than sole policy entropy maximization (Aş 0). Table shows the effect of the varying X on task (reward) and imitation performance (KL), ie similarities between π. Te measured as Ex-n [DKL (T (-|s)) T (-]s Setting D too large (2 0.1) hurts both task and imitation performance as the MI reward ry dominates the RL objective. Setting too small (S 0.0001), i.c only maximizing policy entropy H((T)). turns out to benefit task performance, sometimes enabling the imitator to outperform the expert by concentrating most of it's trajectory probability mass to the mode of the expert's 's trajectory distribution. However, the boosted task performance comes at the cost of suboptimal imitation performance, e.g imitator cheetah running faster than the expert. We found that a middle point of X = 0.005 simultaneously achieves expert level task performance and good imitation performance. In summary, these results show that SAELBO H maximization (Af > 0) improves distribution matching between π, Tg policy entropy H(To) maximization (), 0), but distribution matching may not be ideal for task performance maximization, e.g in apprenticeship learning settings. See Appendix C.I for extended ablation studies.
7 Discussion and Outlook
This work's 's main contribution is a new principled framework for IL and an algorithm that obtains state-of-the-an demonstration efficiency. One future direction is to apply NDI to harder visual IL tasks for which All. is known perform poorly. While the focus of this work is to improve on demonstration efficiency, another important IL performance metric is environment sample complexity. Future works could explore combining off- -folcolicy RI. or model-based RI. with NDI to improve on this end. Finally, there is a rich space of questions to answer regarding the effectiveness of the SAELBO reward ry. We

---

posit that, for example, in video game environments ry may be crucial for success since state-action entropy maximization has been shown be far more effective than policy entropy maximization (Burda et al, 2018). Furthermore, one could improve on the tightness of SAELBO by incorporating negative samples (an Den Oord et al., 2018) and learning the critic function / so that it is close to the optimal critic.
Acknowledgements
This research was supported in part by NSF(1651565, 1522054, 1733686), ONR (N00014191214)) AFOSR (FA95501910024), ARO (W911NT-21-1-01225) TRI, and Sloan Fellowship.
References
Belghazi, M. I., Baratin, A., Rajeswar, S., Ozair, S., Bengio, Y., Courville, A., and Hjelm, R. D.
Mine: Mutual information neural estimation. arXiv preprint urXiv:1801.04000 2018.
Brantley, K., Sun, W., and Henaff, M. Dissagremeent-regulariizdd imitation learning. 2020.
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang. J., and Wojciech, Z. Openai gym. arXiv preprint arXiv:1606.01540) 2016.
Brown, D., Goo, W., and Niekum, S. Better-haan-demonstattoo imitation learning via automatically- ranked demonstrations. arXiv preprint arXivv: 2019.
Brown, G. Iterative solution of games by fictitious play. Activity Analysis of Production and Allocation, 1951.
Burda, Y., Edwards, H., Storkey, A., and Klimov, O. Exploration by random network distillation. arXiv preprint arXiv: 1810.12944, 2018.
Dieng, A., Ruiz, F., Blei, D. M., and Titsias, M. K. Prescribed generative adversarial networks. arXiv preprint arXiv: 043000, 2019.
Dinh, I.., Krueger, D., and Bengio, Y. NICE: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516. 2014.
Dinh, L., Sohl-Dickstein, J., and Bengio, S. Density estimation using real NVP. arXiv preprint arXiv: May 2016.
Du, Y. and Mordatch, I. Implicit generation and generalization in energy-based models. arXiv preprint arXiv:1903.0800 08689,, 2018.
Du Y. and Mordatch, I. Implicit generation and generalization in energy-based models. arXiv preprint erxiv:190: 08688, 2019.
Fu. J., Luo, K., and Levine, S. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprini arXiv::110. 12248 2017.
Gao, R., Nijkamp, E., Kingma, D. P., Xu, Z., Dai, A. M., and Wu, N. Flow contrastive estimation of energy- based models. arXiv preprint arXiv: 1912/99220099 2019.
Germain, M., Gregor, K., Murray, I., and Larochelle, H. Made: Masked autoencoder for distribution estimation. arXiv preprint 1rxiv:1502.0350000 2015.
Ghasemipour, S., Zemel, R., and Gu, S. A divergence minimization perspective imitation learning methods. arXiv preprint arXiv: 1rXiv:191..02256. 2019.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Advances in neural information processing systems, pp. 2672-2680, 2014.
Guo, X., Singh, S., Lee, H., Lewis, R. L., and Wang. X. Deep learning for real-time atari game play using offline monte-carlo tree search planning. Advances in Neural Information Processing Systems 27 (NIPS 2014), 2014.
10

---

Gutmann, M. and Hyvärinen, A. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings the Thirleenth International Conference on Artificial Intelligence and Statistics, pp. 297-304, 2010.
Haarnoja, T., Tang, H., Abbeel, P., and Levine, S. Reinforcement learning with deep energy-based 2017.
policies. arXiv preprint arXiv:
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft Actor-Critic: Off- maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv: /e..1/0011000000 January 2018.
Hazan, E., Kakade, S., Singh, K., and Soest, A. V. Provably efficient maximum entropy exploration. arXiv preprint arXiv: arXiv:1812.0000 1812.02690, 2019.
Hill, A., Raffin, A., Ernestus, M., Gleave, A., Kanervisto, A., Traore, R., Dhariwal, P., Hesse, C., Klimov, O.,, Nichol, A.. Plappert, M., Radford. A., Schulman, J., Sidor, S., and Wu, Y. Stable baselines. https: http:///ggttu.t..// com/hil1 a/ stable- baselines 2018.
Ho, J. and Ermon, S. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, pp. 4565-4573, 2016.
Huszár, F. Variational inference using implicit distributions. arXiv preprint arXiv: 2017.
Hyvärinen, A. Estimation of Non-Normalized statistical models by score matching. Journal of machine learning research: JMLR, 6(Apr):695-709 2005. ISSN 1532-4435, 1533-7928.
Islam, R., Seraj, R., Bacon, P.-L., and Precup. D. Entropy regularization with discounted future state distribution in policy gradient methods. arXiv preprint arXiv: 2019.
Jin, C., Netrapalli, P., and Jordan, M. I. What is local optimality in noncomvex-nonconeee minimax optimization? arXiv preprint arXiv: 10t.:::/9220000 006600 2019.
Ke, L., Barnes, M., Sun, W., Lee, G., Choudhury, S., and Srinivasa, S. Imitation learning as f-divergence minimization. arXiv preprint 1905. 1908. 2020.
Kim, K., Gu, Y.. Song, J., Zhao, S., and Ermon, S. Domain adaptive imitation learning. arXiv preprint arXiv:1910.001 2019.
Kim, K.-E. and Park, H. S. Imitation learning via kernel mean embedding. AAAI, 2018.
Kingma, D. and Ba, J. Adam: method for stochastic optimization. arXiv preprint 1412::441.69900 December 2014.
Kingma, D. P. and Dhariwal, P. Glow: Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.
Kostrikov, I., Nachum, O., and Tompson, Imitation learning via off-policy distribution matching. 2020.
Lee, L.,, Eysenbach, B., Parisotto, E., Xing, E.,, Levine, S., and Salakhutdinov, R. Efficient exploration via state marginal matching. arXiv preprint arXiv: 1906.05274 2019.
Liu, H., Lafferty, J., and Wasserman, L. Sparse nonparametric density estimation in high dimensions using the rodeo. volume Proceedings of Machine Learning Research, pp. 283-290, San Juan, Puerto Rico, 21-24 Mar 2007. PMLR
Miyato, T., Kataoka, T., Koyama, M., and Yoshida, Y. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957 2018.
Mohamed, S. and Lakshminarayannn, B. Learning in implicit generative models. arXiv preprint arXiv:1610.03483. 2016.
Nguyen, X., Wainwright, M. J., and Jordan, M. Estimating divergence functionals and the likelihood ratio by convex risk minimization. arXiv preprint arXiv:0809.085: 2010.
Nowozin, S., Cseke, B., and Tomioka, R. f-GAN: Training generative neural samplers using
variational divergence minimization. arXiv preprint arXiv:1606.00700 00709, June 2016.

---

Papamakarios, G.. Pavlakou, T., and Murray, I. Masked autoregressive flow for density estimation. arXiv preprini arXiv: 1000107057.7777 2017.
Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T. Curiosity-driven exploration by self-supervised prediction. arXiv preprint T 2017.
Pomerleau, D. A. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 3(1):88-97 97, 1991. ISSN 0899-7667.
Reddy, S., Dragan, A. D., and Levine, S. Sqil: Imitation learning via reinforcement learning with sparse rewards. arXiv preprint arXiv:1905.11 10088, 2017.
Ross, S. and Bagnell, D. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pp. 661-668, 2010.
Ross, S., Gordon, G., and Bagnell, D. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-635, 201
Ross, S., Gordon, G., and Bagnell, D. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-635, 201 lb.
Ross, S., Melik- Barkhudarov, N., Shankar, K. S., Wendel, A., Dey, D., Bagnell, A. J., and Hebert, M. Learning monocular reactive uav control in cluttered natural environments. International Conference on Robotics and Automation (ICRA), 2013.
Salimans, T., Goodfellow, I., 7aremba, W. Cheung, V., Radford, A., and Chen, X. Improved techniques for training gans. arXivv1606.034498 2016.
Song, J., Ren, H., Sadigh, D., and Ermon, S. Multi-agen generative adversarial imitation learning. 2018.
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. arXiv preprint arXiv:/907.0505 0,500000 2019.
Song, Y., Garg, S., Shi, and Ermon, S. Sliced score matching: A scalable approach to density and score estimation. arXiv preprint 1xXiv:1905.07088 2019.
Todorov, E. Convex and aallytically-ivertbbbe dynamics with contacts and constraints: Theory and implementation in mujoco. IEEÉ International Conference on Robotics and Automation (ICRA), 2014.
Todorov, E., Erez, T., and Tassa, Y. MuloCo: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, October 2012. doi: 10.1109/IROS 201 22002...66 2.6386109.
Uchibe, E. Model-free deep inverse reinforcement learning by logistic regression. Neural Processing Letters, 47:891-905. 2018
Uria, B., Murray, I., and Larochelle, H. RNADE: The real-valued neural autoregressive density- estimator, In Burges, C. C., Bottou, L., Welling, M., Ghahramani, Z., and Weinberger, K. Q. (eds.), Advances in Neural Information Processing Systems 26, pp. Inc., 2013.
Curran Associates,
Uria, B., Côté, M.-A., Gregor, K.,, Murray, I., and Larochelle, H. Neural autoregressive distribution estimation. The Journal of Machine Learning Research, 17(1):714-7222) 2016.
van den Oord, A., Kalchbrenner, N., and Kavukcuoglu, K. Pixel recurrent neural networks. arXiv preprint ar Xix January 2016.
Van Den Oord, A., Li, Y., and Vinyals, O. Representation learning with contrastive predictive coding. arXiv preprint 1807 2018
Wang, R., Ciliberto, C., Amadori, P., and Demiris, Y. Random expert distillation: Imitation learning via expert policy support estimation. 2019.
12

---

Wang, Z., Merel, J., Reed, S., Wayne, G., Freitas, N. d., and Heess, N. Robust imitation of diverse behaviors. arXiv preprint arXiv:
2017.
Wu, A., Piergiovanni, A., and Ryoo, M. S. Model-based behavioral cloning with future image similarity learning. arXiv preprint arXi::9110.3317., 2019.
Yu, L.,, Song. Y., Song, J., and Ermon, S. Training deep energy-based models with f-divergence minimization. arXiv preprint arXiv:2003.034p0 2020.
13


Reviews:
[{"review_id": "uEd1rVyMrlZ", "reviewer": "Reviewer_wy8U", "summary": "This paper presents an imitation learning approach based on density estimation of the state-action pairs. The underlying idea is to take expert demonstrations and find a distribution over the inherent state occupancy and then follow-up with a reinforcement learning step. As a result, the learner visits the high density state action pairs while also exploring new states during training. The paper makes a number of interesting contributions, among a new cost function for distribution matching IL and a proof showing that it is equivalent to maximizing a non-adversarial model-free RL objective. In contrast to other methods, like AIL, no adversarial optimization is needed; which typically would have resulted in instabilities. Instead the new objective pushes the lower bound in a single direction. The paper also features a list of tradeoffs between different distribution matching IL methods. Experiments on Mujoco tasks show state of the art performance.   ", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I generally enjoyed this paper. It starts with a good background on modern IL methods and their relationships and goes on to pinpoint the challenges to be addressed. Obviously, the idea of distribution matching IL algorithms is not new. However, this paper goes about it in a rigorous and well-motivated way. On one hand, it shows how to maximize an implicit density and how to avoid the instabilities that afflict other methods. Generally, a bunch of small but non-trivial tricks are combined to make this algorithm work. The density estimation step itself is performed via existing methods, e.g., EBMs. While not necessarily completely original, the strength of the paper is in a deep analysis and careful and rigorous derivation of an improved distribution matching IL algorithm.\n\nThe quality of writing is very high. Overall, it's always clear what the authors are trying to achieve. However, at times parsing the paper can become difficult due to the mathematical notation and dense mixtures of equations and text. I would recommend making another pass for readability and accessibility. In particular, some long sentences can be cut down into pieces. As for the quality of the presented ideas, I think this is a good paper with important theoretical and practical contributions to imitation learning. As mentioned before, it provides a deep dive into the intricacies of modern distribution matching IL methods. I am slightly confused however why the authors chose to show two variants, NDI+MADE / NDI+EBM? Is the underlying idea to show that the method is extensible?\n\nThe presented results in the mujoco task outperform the other SotA methods and clearly show the advantage of the approach. One important piece of information that I couldn't find is \"how many RL iterations have been performed for the experiments?\". Maybe the authors can comment on that.  Overall, this is a good paper with interesting insights, a technically sound derivation and great results in simulation.  \n\nBTW, the paper mentions code but I could not find any code for verification in the supplemental material.\n\nEdit after rebuttal: Thanks to the authors for the comprehensive and satisfying answers. I am maintaining my score. This is a good paper and I argue for acceptance. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors talk about limitations and provide a table of tradeoffs when compared to other methods.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "f2DzEhJT9Kq", "reviewer": "Reviewer_VQpP", "summary": "Online imitation learning algorithm based on neural density estimation is proposed. The objective of this paper is to maximize $-D_{KL}$ in Eq. (2), and the authors consider maximizing the lower bound of Eq. (2) in Corollary 1 as a surrogate learning objective. The aforementioned neural density model is used to estimate the expert state-action visitation distribution (or occupancy measure) which works as a component of a reward function (the log of the neural density model) during the RL phase, together with the entropy regularization ($r_\\pi$) and a critic-based reward ($r_f$). While Adversarial Imitation Learning requires the alternate optimization for policy and reward estimators, the proposed idea does not require the reward learning phase (similar to RED, Disagreement regularized IL and SQIL), which may stabilize the learning procedure. While the proposed idea is straightforward to be understood, the empirical results show that NDI outperforms its baselines especially when the number of expert demonstrations is small. The ablation studies on learned density and varying MI reward weight are also interesting to follow. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the submission is clearly written, and theoretical challenges and how to address them are well-described. A wide range of baselines are considered and compared with the proposed idea, and the experiment supports the authors’ claim. I couldn’t find any major flaw in this work overall, and I had minor questions below:\n- While I agree that using an unnormalized density estimate is okay in theory, I wonder if it doesn’t affect the algorithmic stability since the reward may be extremely large or small.\n- I read through Islan et al., and Lee et al. but couldn’t find the term “Maximum Occupancy Entropy RL”. Is this named by the author?\n- One question I had while evaluating this submission was if the discounted sum over causal entropies and the entropy of occupancy measure are different. I concluded they are different based on the theoretical result in GAIL’s Lemma 3.2, and the difference is related to the entropy of the state-visitation occupancy measure. I would like authors to more emphasize the difference between the two.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No societal issue provided. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "XZkb8M77LF", "reviewer": "Reviewer_8iqv", "summary": "The authors propose a new imitation learning algorithm that attempts to (indirectly) minimize the KL between the imitation learner distribution and the expert distribution. The main novelty consists in the fact that the objective includes the entropy over the state-action space (as opposed to the average entropy of the action distributions given the state). In order to deal with this entropy term, the authors use a learned approximation to the state-action occupancy distribution.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is original. The quality of the writing is high. The paper is mostly clear although some sections are quite dense. The proposed imitation learning method achieves the level of significance expected of a NeurIPS submission.\n\nMy main concerns are as follows:\n1. The proposed method requires access to a kernel function in the state space (for computing the critic). Since we have a kernel anyway, why not use a non-parametric kernel density estimator for the state-action distribution of the expert, rather than the solutions proposed on Section 2.1 and based on neural networks? It seems strange to use the kernel for the critic but not for estimating densities. \n2. What benefits does the proposed method have over MMD-GAIL [1]?.\n3. You assume that the environment is deterministic. That is OK, but can you explicitly describe in the paper what breaks when the algorithm is run on a stochastic MDP?\n4. One difficulty with maximising differential entropy in the state-action space is that it is not invariant to transformation of the input. Do you use raw observations provided by MuJoCo, or do you transform them somehow?\n5. Because of the complexity of the proposed approach and the amount of engineering required to get good density estimates, this work is only really reproducible if the source code is released. I don't think there are enough details in the paper text itself to fully replicate the results.\n\nMinor point:\nIn table 2, when you say \"one demonstration\" do you mean one episode or one timestep of an episode?\n\n[1] https://www-users.cs.umn.edu/~hspark/mmd.pdf", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "In the checklist, the authors say that the question of whether there are negative societal consequences of the work does not apply to this paper.  I think that the ethical questions for this paper are the same as for any other work that tries to push SOTA on imitation learning. One could for example cite the ease of construction of bots that replace humans. A short discussion of this would be nice.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "IEs8wzeJBa", "reviewer": "Reviewer_dc5s", "summary": "The paper proposes an non-adversarial imitation learning algorithm based on density estimation of expert demonstrations. The proposed method derives a lower bound on the reverse KL divergence ($KL(\\pi | \\pi_{E})$), which is translated into a composite reward function consisting of 3 addictive terms. Empirical results show improved performance in Mujoco benchmarks.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is well presented. The derivation for the proposed method is easy to follow and appears correct. The method is well motivated by establishing a rigorous lower bound for the reverse KL divergence.\n\n**On non-adversarial reward**: It is difficult to assess whether the non-adversarial reward proposed in the paper improves algorithm stability, when stability is not clearly defined. In addition,  the proposed reward function appears **not fixed**, even when it’s non-adversarial. Consequently, the proposed reward function may pose similar training difficulty as the adversarial methods, since a common explanation for training difficulty stems from learning value functions under an evolving reward. The experiments seem to suggest that $\\lambda_f$  needs to be small (i.e. reduce how much the reward function changes) to obtain good performances, while the best policy performance is obtained with $\\lambda_f = 0$. More generally, could the authors further discuss how much the dynamic terms actually contribute to the reward function (including the auto-tuned value for $\\lambda_\\pi$), and how they affect policy learning?\n\n**Efficiency wrt demonstrations**: This claim is difficult to support with the current experiments. Based on the existing conventions, 1 full trajectory contains 1000 state-action pairs, while GAIL and some previous methods use 4 sub-sampled trajectories of 50 state-actions each (200 state-actions in total, except for the Humanoid task). It is therefore unclear that the proposed method is more sample efficient wrt demonstrations. Could authors further discuss the differences between the two settings and their implications for the proposed method?\n\n**Practical algorithm**: The reward in the form of log(x) is typically challenging to optimize, since the reward has a very large range. In addition, the proposed method has three terms with possibly different ranges. Could the authors clarify (if any) modification in the actual implementation for “normalizing” the reward?\n\nTo my knowledge, SAC is already an off-policy algorithm. The limitation “Future works could explore combining off-policy RL or model-based RL with NDI to improve on (environment sample complexity)” is thus confusing to me. Please clarify this point.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As discussed above, the primary limitation of the paper is that the experiments are limited in their ability to fully demonstrate the advantages and properties of the proposed method. Broadening the evaluation (e.g. different environments/tasks) would more clearly showcase the advantages of the proposed method.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new framework for imitation learning via obtaining a density estimation of the expert’s occupancy measure, followed by Maximum Occupancy Entropy Reinforcement Learning. The intuitive idea is to encourage the imitator to visit high density state-action pairs under the expert’s occupancy measure while maximally exploring the state-action space. 

In their initial reviews, the reviewers find the paper well-written. The proposed method is based on in-depth analysis of the problem and it has strong performance in comparison with a wide range of baselines. On the other hand, they also think that the claim on data efficiency not is well supported and lack some necessary discussions (evolving reward and stability of learning, use of non-parametric kernel density, advantages over MMG-GAIL, etc)

During discussions, two reviewers stood by their votes for accept. Two other reviewers felt that their concerns were not satisfactorily addressed by authors’ responses. One of them even lowered his/her score.

Decision: Accept (Poster)