Imitation with Neural Density Models
Authors: Kuno Kim
Akshat Jindal
Yang Song
Jiaming Song
Yanan Sui
Stefano Ermon
Venue:  (2021)
Keywords: Imitation Learning
Reinforcement Learning
Density Estimation
Density Model
Maximum Entropy RL
Mujoco

Abstract: We propose a new framework for Imitation Learning (IL) via density estimation of the expert's occupancy measure followed by Maximum Occupancy Entropy Reinforcement Learning (RL) using the density as a reward. Our approach maximizes a non-adversarial model-free RL objective that provably lower bounds reverse Kullback–Leibler divergence between occupancy measures of the expert and imitator. We present a practical IL algorithm, Neural Density Imitation (NDI), which obtains state-of-the-art demonstration efficiency on benchmark control tasks. 

Under review as a conference paper at ICLR 2021
IMITATION WITH NEURAL DENSITY MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
We propose a new framework for Imitation Learning (IL) via density estimation of the expert's 's occupancy measure followed by Maximum Occupancy Entropy Rein- forcement Learning (RL) using the density as a reward. Our approach maximizes a non-adversarial model-free RL objective that provably lower bounds reverse Kull- back-Leibler divergence between occupancy measures of the expert and imitator. We present a practical IL algorithm, Neural Density Imitation (NDI), which obtains state-of-the-arr demonstration efficiency on benchmark control tasks.
INTRODUCTION
Imitation Learning (IL) algorithms aim to learn optimal behavior by mimicking expert demonstrations. Perhaps the simplest IL. method is Behavioral Cloning (BC) (Pomerleau 1991) which ignores the dynamics of the underlying Markov Decision Process (MDP) that generated the demonstrations, and treats IL as a supervised learning problem of predicting optimal actions given states. Prior work showed that if the learned policy incurs a small BC loss, the worst case performance gap between the expert and imitator grows quadratically with the number of decision steps (Ross & Bagnell 2010
Ross al. 2011a). The crux of their argument is that policies that are "close" as measured by BC loss can induce disastrously different distributions over states when deployed in the environment. One family of solutions mitigating such compounding errors is Interactive IL (Ross al. 2011b 2013
Guo et al. 2014), which involves running the imitator's 's policy and collecting corrective actions from an interactive expert. However, interactive expert queries can be expensive and are seldom available.
Another family of approaches (Ho & Ermon 2016 Fu et al.
2017 Ke al. 2020 Kostrikov et al.,
2020: Kim & Park 2018 Wang et al.] (017) that have gained much traction is to directly minimize a statistical distance between state-action distributions induced by policies of the expert and imitator, ie the occupancy measures Pag and Pr., As Pr, is an implicit distribution induced by the policy and environment distribution matching with Pro typically requires likelihood methods involving sampling. Sampling from Pao entails running the imitator policy in the environment, which was not required by BC. While distribution matching IL requires additional access to environment simulator, it has been shown to drastically improve demonstration efficiency, i.c the number of demonstrations needed to succeed at IL. (Ho & Ermon 2016). A wide suite of distribution matching IL algorithms use adversarial methods to match Pr, and parg which requires alternating between reward (discriminator) and policy (generator) updates (Ho & Ermon 2016 Fu et al. 2017: Ke et al.
2020: Kostrikov al. 2020 Kim et al., 2019). A key drawback
such Adversarial Imitation Learning (AIL) methods is that they inherit the instability of alternating min-max optimization (Salimans et al. 2016, MMyato et al. 2018) which is generally not guaranteed to converge (Tin et al. 2019p. Furthermore, this instability exacerbated in the IL setting where generator updates involve high-variance policy optimization and leads to sub-optimal demonstration efficiency. To alleviate this instability, (Wang et al. 2019 Brantley et al., 2020; Reddy et al. 2017) have proposed to do RL with fixed heuristic rewards. Wang et al. (019), for example, uses a heuristic reward that estimates the support of PxR which discourages the imitator from visiting out-of-suppor states. While having the merit of simplicity, these approaches have no guarantee of recovering the true expert policy.
In this work, we propose a new framework for IL via obtaining a density estimate 9s of the expert's occupancy measure Pxg followed by Maximum Occupancy Entropy Reinforcement Learning (Max- OccEntRi.) (Lee et al. 2019 Islam et all, 2019p. In the MaxOccEntRI, step, the density estimate 4 is used as fixed reward for RL and the occupancy entropy H.(p.)) ) is simultancously maximized, leading to the objective max E.p., [log qo a) + H(PT,)) Intuitively, our approach encourages the imitator visit high density state-action pairs under P=ε while maximally exploring the state-action
assume only samples can be taken from the environment dynamics and its density is unknown

---

Under review as a conference paper at ICLR 2021
space. There are two main challenges to this approach. First, we require accurate density estimation of Pag. which is particularly challenging when the state-action space is high dimensional and the number of expert demonstrations are limited. Second, in contrast to Maximum Entropy RL (MaxEntRL), MaxOecEntRL requires maximizing the entropy of an implicit density Pag. We address the former challenge leveraging advances in density estimation (Germain et al., 2013|Du & Mordatch, 2018 Song et al. 2019). For the latter challenge, that provably maximizes a lower bound to occupancy entropy. As a byproduct, we also obtain a model-free RL objective that lower bounds reverse Dellbacck-ieber (KL) divergence between Pro and Pag. The contribution of our work is introducing novel family of distribution matching II algorithms, named Neural Density Imitation (NDI), that (1) optimizes a principled lower bound to the additive inverse of reverse KL, thereby avoiding adversarial optimization and (2). advances state-of-the-arr demonstration efficiency in II..
derive non-adversarial model-free RL objective
2 IMITATION LEARNING VIA DENSITY ESTIMATION
We model an agent's s decision making process as a discounted infinite-horizon Markov Decision Process (MDP) .M= = (S. AS P, Po. T.). Here S are state-action spaces, P S x A - ((S) is a transition dynamics where (2(S) is the set of probability measures on S. Po : S + R is an initial state distribution, r 8 x A -> IR is a reward function, and 7 E [0, 1) is a discount factor. parameterized policy Te : S + ((A)) distills the agent's s decision making rule and {st, at 3
is the stochastic process realized by sampling an initial state from 8n ~ (s(s)) then running To in the environment, i.c a, To (-]s,), Se+1 P(-|s,, oz). We denote by P k the joint distribution of states {st, $t+1. S1*k), where sctting Pe.c recovers the marginal of s,. The (unnormalized) occupancy measure of xo is defined as PTo (T. (s. = Eg=o""po,t ('po, (als). Intuitively, PT. (s, a) quantifies the frequency of visiting the state-action pair (s, a) when running Ta for long time, with more emphasis on earlier states.
We denote policy performance as FTTo E 7"Foee at)] = py [Fss, a)] where F is a (potentially) augmented reward function and E. denotes the generalized expectation oper-
ator extended to non-normalized densities p X' + TR+ and functions X' - y so that Ep]f(z)] = Σx fz)) The choice of depends on the RL framework. In standard RL. we simply have r = r, while in Maximum Entropy RL (MaxEmRLL) (Haarnoja et al. (017)) we have F(s, a) = r(s,a) log (a(ss). We denote the entropy of Paoo (s, a) as H(Pr))
I- Pro (8, a)] and overload notation to denote the 7-discounted causal entropy of policy Ty as H(TT)) = Ex, 1E/= -7" log (a(a,) [s]]] = Ep.,, log To [)]. Note that we use general- ized notion of entropy where the domain is extended to non-normalized densities. We can then define the Maximum Occupancy Entropy RL (MaxOccEmRL) (Lee et al, 2019 Islam et al., 2019) objective as J(TT, f = r) + H(P+)). Note the key difference between MaxOceEntRR and Max entropy regularization is on the occupancy measure instead of the policy, i.c seeks state diversity instead of action diversity. We will later show in section 2.2 that lower bound this objective reduces to complete model-free RL objective with an augmented reward F.
Let Te. Ta denote an expert and imitator policy, respectively. Given only demonstrations 'D {(s, a)x) [ ~ Te of state-action pairs sampled from the expert, Imitation Learning (IL) aims to learn policy Te which matches the expert, i.e Te = .g. Formally, IL can be recast distribution Ke et al. 2020) between occupancy measures Pr, and pagi maximizeo
matching problem (Ho & Ermon 2016
P(p)
where d(p. ②) is a generalized statistical distance defined on the extended domain of (potentially) non-normalized probability densities p(x),, ((x) with the same normalization factor 2 > 0, i. f. f& / = 1. For P.r and PTR. we have Z = r As we are only able to take samples from the transition kernel and its density is unknown, Pa. is an implicit distribution F Thus. optimizing Eq. [ tyyiicall requires likelihood frre approaches leveraging samples from Pr.. i.e IL approaches use likelihood-free adversarial methods to approximately optimize Eq. for various choices of d such as reverse Kullback-Liebler (KL) divergence (Fu et al.) [017] Kostrikov et al. 2020) and Sensen-Shannod (JS) divergence (Ho) & Ermon] (016). However, adversarial methods are known to suffer from optimization instability which is exacerbated in the IL setting where one step in the alternating optimization involves RL.
running T in the environment. Current state-of-the-an
"probability models that have potentially intractable density functions, but can be sampled from to estimate expectations and gradients of expectations with respect to model parameters (uszáá 2017
2

---

Under review as a conference paper at ICLR 2021
We instead derive non-adversarial objective for IL. In this work. we choose d to be (generalized) reverse-KL. divergence and leave derivations for alternate choices of future work.
- (Po, (P Pp... [lo PTg (8, a) log Pay (s, aj] [ F log Prg) ) H(Pm.)
We see that maximizing negative reverse-KL KL with respect to To is equivalent to Maximum Occupancy Entropy RL (MaxOceEntRL) with log Prg as the fixed reward. Intuitively, this objective drives me to visit states that are most likely under Paı. while maximally spreading out probability mass so that if two state-action pairs are equally likely, the policy visits both. There are two main challenges associated with this approach which we address in the following sections.
1. log Prg is unknown and must be estimated from the demonstrations D. Density estimation remains a challenging problem, especially when there are a limited number of samples and the data is high dimensional (Liu et al. 2007). Note that simply extracting the conditional (als) from an estimate of the joint ( 88, a an alternate way to do BC and does not resolve the compounding error problem Roos et al. 2011a).
2. H(prr) is hard maximize as Pere an implicit density. This challenge is similar the difficulty of entropy regularizing generators (Mohamed & Lakshminarayanan (016 Belghazi et al. (2018) Dieng et al. 2019) for Generative Adversarial Networks (GANs) (Goodiellw et al. 2014). 4) and
most existing approaches (Dieng et al. 2019 Lee
2019 use adversarial optimization.
2.1 ESTIMATING THE EXPERT OCCUPANCY MEASURE
We seek to learn a parameterized density model (s, a) of Pxg from samples. We consider two canonical families of density models: Autoregressive models and Energy-based models (EBMs).
Autoregressive Models (Germain et al. 2015: Papamakarios et al. 2017;: An autoregressive model I. (z.)). For
(z) for x (s,a) learns a factorized distribution of the form: g instance, each factor ls. could be a mapping from Zei Gaussian density over z;z When given prior over the true dependency structure of { this can be incorporated by refactoring the model. Autoregressive models typically trained via Maximum Likelihood Estimation (MLE).
Energy- Models (EBM) (Du & Mordatch 2018 Song et al. 2019); Let Eg S
be an energy function. An energy based model IS a parameteried Boltzman distribution of the form (s, a) Z e where ( JSXA e (e.0) dsda denotes the partition function. Energy- based models are desirable for high dimensional density estimation due to their expressivity. but are typically difficult to train due to the intractability of computing the partition function. However, our IL objective in Eq. [ conveniently only requires a non-normalized density estimate as policy optimality is invariant to constant shifts in the reward. Thus, we opted to perform non normoazall density estimation with EBMs using score matching which allows us to directly learn Eg without having to estimate (()).
2.2 MAXIMUM OCCUPANCY ENTROPY REINFORCEMENT LEARNING
In general maximizing the entropy of implicit distributions is challenging due the fact that there is no analytic form for the density function. Prior works have proposed using adversarial methods involving noise injection (Dieng al. 2019 and fictitious play (Brown 1951 Lee et al. 2019). We instead propose to maximize novel lower bound to the additive inverse of an occupancy divergence which we prove is equivalent to maximizing non-adversarial model-free RL objective. We first make clear the assumptions on the MDPs considered henceforth.
Assumption All considered MDPs have deterministic dynamics governed by transition function : S A + S. Furthermore, P is injective with respect to a A, i. Vs, a, a' it holds that a a' + P(s,a) + a(x,a')
We note that Assumption I hholds for most continuous robotics and physics environments as they deterministic and inverse dynamics functions p-1 S S A have been successfully used in benchmark RL environments such as Mujoco (Todorov et al. 2012: Todorov 2014) and Atari (Pathak et al.] 2017). Next we introduce a crucial ingredient in deriving our occupancy entropy lower bound, which a tractable lower bound Mutual Information (MI) first proposed
by Nguyen, Wainright, and Jordan (Nguyen et al. 2010), also known as the f-GAN KL (Nowozin al. 2016) and MINE- (Belghazi al. 2018). For random variables X, Y distributed according to
3

---

Under review as a conference paper at ICLR 2021
Pr.. (z, po_ (z), Po, (v) where 0 = (0ay. 2z. 0y, and any critic function f: x R, it holds
that T(X; 2 Www 0 where,
kww (X; 0) = Epe.y (z. y)] Epo Epo, [e((x.)0]]
(3)
Pazy (a.1) poz (x)poy (v) + We are
This bound tight when f is chosen to be the optimal critic f" v) = log now ready to state lower bound to the occupancy entropy.
Theorem
Let MDP M scaisfy assumption 7 (App. For any critic J S S - R, it holds that
H(PFu) 2 Ht (Pro)
(4)
where
H (Px.) H(ss)) (1 + ()H((T))) +7 T''
(5)
See Appendix A. for the proof and a discussion of the bound tightness. Here onwards, we refer to H' (PT,) from Theorem 1 as the State-Action Entropy Lower Bound (SAELBO). The SAELBO mainly decomposes into policy entropy H(Te) and Mutual Information (MI) between consecutive states [K ( When Assumption does not hold, may still obtain SAELBO with only the policy entropy term, i. (px.))
H(TT))) I H((p))), but this bound has more slack and limited to discrete state-spaces. (see Appendix [A] for details) Since occupancy entropy maximization is also a desirable exploration strategy in sparse environments (Hazan et al. 2019 Lee al. 2019), another interpretation of the SAELBO is surrogate objective for state-action level exploration. Furthermore, we posit that maximizing the SAELBO is more effective for state-action level exploration, i.c occupancy entropy maximization, than solely maximizing policy entropy. This because, in discrete state-spaces, the SAELBO is tighter lower bound to occupancy entropy than policy entropy, i.e H(m)) < H. (Px.) S H(Ps))) and in continuous state-spaces, where Assumption holds, the SAELBO still a lower bound while policy entropy alone is neither a lower nor upper bound to occupancy entropy. Please see Appendix C. experiments that show how SAELBO maximization can improve state-action level exploration over just policy entropy maximization. Next, we show that the gradient of the SAELBO is equivalent to the gradient of model-free RL objective.
Theorem Let Or (als) and (ge()) be probability densities such that Vs, E S A satisfy 4. (a(s)) (o(as)) and (i(s) = (s). Then for all f x S IR,
VoMt (pxa) = (To Ta +rl)
(6)
where
ax)
-01 =
log ax (apls:)
(7)
Ty (8,, = 7f(s,) 8(+1) Barq le (Rrssee))) + ((nn54))
(8)
See Appendix A.2 for the proof. Theorem 2 shows that maximizing the SAELBO is equivalent to maximizing discounted model-free RL objective with the reward r. + ry. where r, contributes to maximizing (TTT)) and ry contributes to maximizing Ee= 7¹ 2 ( Note that evaluating ry entails estimating expectations with respect to 4t. 4t+1- This can be accomplished by rolling out multiple trajectories with the current policy and collecting the states from time-step t, + Alternatiely, if we assume that the policy is changing slowly, we can simply take samples of states from time-stpp t +1 +1 from the replay buffer. Combining the results of Theorem 2 we end the section with lower bound on the original distribution matching objective from Eq. I and show that maximizing this lower bound is again, equivalent to maximizing model-free RI. objective.
Corollary Let MDP M satisfy assumption
(ppp. A For any critic f S:SxS R, it holds that
-DKL (pmo (prollpss)
( log PMR) ) + H. (pao)
(9)
Furthermore, let Ta,Ty f be defined in Theorem Then,
Vo(J(Te,
png) + H (Pxp)) VoJ = log PAR + + +ry)
(10)
In the following section we derive practical distribution matching IL algorithm combining all the ingredients from this section.
4

---

Under review as a conference paper at ICLR 2021
Algorithm 1: Neural Density Imitation (NDI)
Require:
Demonstrations D TE, Reward weights Xy. Xy, Fixed critic f
Phase 1. Density estimation:
Learn (0(s, a) from D using MADE or EBMs
Phase 2. MaxOccEndRL:
for k = 1,2,... do
Collect (8t, de, 8441, P) ~ xo and add to replay buffer B, where llo qs + ATT + AgTT.
(x()),0)) - log no (apls:)
(8(8,, Ct, 81-1) = (85(8 82+1) - 2 E B (e ( + e (f(c)1191)) and the critic is computed by
e
116-1-8113
f(5+11 81) log
B8,
B,_1 [e ||0041
Update Ty using Soft Actor-Critic (SAC) (Haarnoja et al. 2018):
end
NEURAL DENSITY IMITATION (NDI)
From previous section's results, we propose Neural Density Imitation (NDI) that works in phases:
Phase 1: Density estimation: We leverage Autoregressive models and EBMs for density estimation
of the expert's occupancy measure Pry from samples. As in (Ho & Ermon 2016 Fu et al. (017). we take the state-action pairs in the demonstration set D {(s, a);) samples from PTw and fit qe on D. For Autoregressive models, Density Estimation (MADE) (Germain et al. 2013) where the entire collection of conditional density models {9,,} is parameterized by single masked autoencoder network. Specifically, we use gaussian mixture variant (Papamakarios al. 2017) of MADE where each of the conditionals qo, map inputs æei to the mean and covariance of a gaussian mixture distribution over I;, The MADE model is trained via Maximum Likelihood Estimation. With EBMs, perform normaazad log density estimation and thus directly parameterize the energy function & with neural networks since log q E, + log (()). We use Sliced Score Matching (Song et al. 2019) to train the EBM.
TE to approximate Masked Autoencoders for
Phase 2: MaxOL After we've acquired a log density estimate log d6 from the previous phase, we perform RL with entropy regularization on the occupancy measure. Inspired by Corollary propose the following RL objective
max (m F log ds + >xTx + Xyry)
(11)
0
where Ax, by O are weights introduced to control the influence of the occupancy entropy regular- ization. In practice, Eq. M can be maximized using any RL algorithm by simply setting the reward function to be F from Eq. 11 In this work, we use Soft Actor Critic (SAC) (Haarnoja et al. 2018). Note that SAC already includes a policy entropy bonus, critic/ f. we fix it to be a normalized RBF kernel for simplicity,
do not separately include one. For
e 1111+111-800 -
f(s 8t) log
+1
(12)
Egg. [e- 1e441 1
but future works could explore learning the critic to match the optimal critic. While simple, our
choice of emulates two important properties of the optimal critic (2 log p(z) + 1: (1). it follows the same "form" of log-density ratio plus constant (2). consecutively sampled states from the joint, i.e St, St+1 ~ Po.1:+11 have high value under our f since they are likely to be close to each other under smooth dynamics, while samples from the marginals 8t, 8t+1 (t+1 are likely have lower value under ff since they can be arbitrarily different states. To estimate the expectations with respect to qe 4++1 in Eq. g we simply take samples of previously visited states at time ł, t + from the replay buffer.
plaly)
5

---

Under review as a conference paper at ICLR 2021
Table 1: Comparison between different families of distribution matching II. algorithms
IL method
Learned Models
Relation between -divergence
Objective Type
and optimized objective
Support
policy ne. support estimator f
Neither Upper nor Lower Bound
max
Adversarial
policy no, discriminator
Tight Upper Bound
min max
NDI (ours)
policy ne, critic f, density 4
Loose Lower Bound
max max
4 TRADE-OFFS BETWEEN DISTRIBUTION MATCHING IL ALGORITHMM
Adversarial Imitation Learning (AIL) methods find a policy that maximizes an upperbound to the additive inverse of an f-divergence between the expert and imitator occupancies (Chasemipour et al.
2019. Ke et al. 2020 For example, if the f-divergence is reverse KI., then for any D
max - DKL (lp=g) S max log (Bng ((sa))) - Ex, DD(s Te
where the bound tight at ((s, a)
log Amg (3,a) Pag (s.a] + C for any constant C. AIL alternates between,
min log [L D(oa))) (Ds, a)),
D
max - Ex, (((,
The discriminator update step in AIL minimizes the upper bound with respect to D, tightening the estimate of reverse KL, and the policy update step maximizes the tightened bound. We thus see that by using an upper bound, AlL innevitably ends up with alternating min-max optimization where policy and discriminator updates act in opposing directions. The key issue with such adversarial optimization lies not in coordinate descent itself, but in its application to min-max objective which is widely known to gives rise to optimization instability (Salimans al. 2016
The key insight of NDI is to instead derive an objective that lower bounds the additive inverse of reverse KL. Recall from Eq. that NDI maximizes the lower bound with the SAELBO H' (pmp):
max- m6
DKL (Pzul IPTT))
max J (TTg F log PTR) + H (PTw)
me
Unlike the AlL upper bound, this lower bound is not tight. With critic ʃ updates, NDI alternates
max
max J(mO, log Pop) + (1 HH) +y Σ (w t=0
The critic update step in NDI maximizes the lower bound with respect to f. tightening the estimate of reverse KL, and the policy update step maximizes the tightened bound. In other words, for AIL, the policy we and discriminator
seek to push the upper bound in opposing directions while in NDI the policy Ty and critic f push the lower bound in the same direction. Unlike AIL, NDI does not perform alternating min-max but instead alternating max-max!
While NDI enjoys non-adversarial optimization, it comes at the cost of having to use a non-tight lower bound to the occupancy divergence. On the otherhand, AlL optimizes tight upper bound at the cost of unstable alternating min-mi max optimization. Support matching IL algorithms also avoid min- max but their objective is neither an upper nor lower bound to the occupancy divergence. Table summarizes the trade-offs between different families of algorithms for distribution matching II..
5 RELATED WORKS
Prior literature on Imitation learning (IL) in the absence of an interactive expert revolves around Behavioral Cloning (BC) (Pomerleau 1991 Wu al. 2019) distribution matching II. (Ho & Ermon

---

Under review as a conference paper at ICLR 2021
2016 Song et al., 2018: Ke et al. 2020 Ghasemipour et al. 2019; Kostrikov et al., 2020; Kim et al.
2019), and Inverse Reinforcement Learning (Fu et al. 2017; Uchibe 2018 Brown et al. 2019).
Many approaches in the latter category minimize statistical divergences using adversarial methods to solve min-max optimization problem, alternating between reward (discriminator) and policy (generator) updates. ValueDICE, a more recently proposed adversarial IL approach, formulates reverse KL divergence into completely off- off-pooicy objective thereby greatly reducing the number of environment interactions. A key issue with such Adversarial Imitation Learning (AIL) approaches is optimization instability (Miyato et al. 2018 Jin et al.) 2019). Recent works have sought to avoid adversarial optimization by instead performing RL with heuristic reward function that estimates the and
support of the expert occupancy measure. Random Expert Distillation (RFD) (Wang et al., 2019)
IL (Brantley et al.] 2020) are two representative approaches this family. A key limitation of these approaches is that support estimation insufficient to recover the expert policy and thus they require an additional behavioral cloning step. Unlike AIL, we maximize a non-adversarial RL objective and unlike heuristic reward approaches, our objective provably lower bounds reverse KI. between occupancy measures of the expert and imitator. Density estimation with deep neural networks is an active research area, and much progress has been made towards modeling high- nigh-dimensional structured data like images and audio. Most successful approaches parameterize a normalized probability model and estimate it with maximum likelihood. c.g., autoregressive
models (Uria al. 2013
2016
Germain al., 2015) van den Oord al.. 2016) and normalizing 2016; Kingma & Dhariwal, 2018). Some other methods explore
vann
flow models @Din et al.
2014
estimating lin-normalized probability models with MCMC (Du & Mordatch] 2019 Yu et al. 2020)
or training with alternative statistical divergences such as score matching (Hyvärmen] (2005
Song
iet al. 2019) Song & Ermon 2019) and noise contrastive estimation (Gutmann & Hyvärinen 2010
Gao al. (2019). Related to MaxOccEntRLL recent works (Lee al. [019) Hazan et al.) 2019 Islam et al. (2019) on exploration in RL have investigated state- marrinall occupancy entropy maximization.
To do so, (Hazan et al. 2019) requires access to a robust planning oracle, while (Lee et al. 2019) uses fictitious play, an alternative adversarial algorithm that is guaranteed to converge. Unlike these works, our approach maximizes the SAELBO which requires no planning oracle nor min-max optimization, and is trivial implement with existing RL algorithms.
6 EXPERIMENTS
Following prior work, we run experiments on benchmark Mujoco Todorov et al. 2012 Brockman et all, 2016) tasks: Hopper (11, HalfCheetah (17, 6), Walker (17, 6). Ant (III, 8), and Humanoid (376, 17), where the (observation, action) dimensions are noted parentheses.
Environment:
Pipeline: We train expert policies using SAC (Haarnoja et al. 2018). All of our results are averaged across five random seeds where for each seed we randomly sample trajectory from an expert, perform density estimation, and then MaxOccEntRL. Performance for each seed is averaged across 50 trajectories. For each seed we save the best imitator as measured by our augmented reward F from Eq. [ and report its performance with respect to the ground truth reward. We don't 't perform sparse subsampling on the data as in (Ho & Ermon 2016) since real world demonstration data typically aren't subsampled to such extent and using full trajectories was sufficient to compare performance.
We experiment with two variants of our method, NDI+MADE and NDI+EBM, where the only difference lies in the density model. Across all experiments, our density model do is a two-layer MLP with 256 hidden units. For hyperparameters related to the MaxOccEntRL step. A = 0.2 is fixed and for xy see Section 6.3
Architecture:
For full details on architecture see Appendix B
Baselines: We compare our method against the following baselines: (1). Behavioral Cloning (BC) (Pomerleau 199T): learns policy via direct supervised learning on D. (2). Random Expert Distillation (RED) (Wang et al. 2019): estimates the support of the expert policy using a predictor and target network (Burda et al. 2018 followed by RL using this heuristic reward. (3). Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon 2016): on-policy adversarial IL method which
alternates reward and policy updates. (4). ValueDICE Kostrikov et al. 20200: current -tate-of-the-art adversarial IL method that works off-policy. See Appendix B for baseline implementation details.
6.1
TASK PERFORMANCE
Table compares the ground truth reward acquired by agents trained with various IL algorithms when one demonstration is provided by the expert. (See Appendix C.2 for performance comparisons with varying demonstrations) NDI+EBM achieves expert level performance on all mujoco bench-
7

---

Under review as a conference paper at ICLR 2021
Table 2: Task Performance when provided with one demonstration. NDI (orange rows) outperforms all baselines on all tasks. See Appendix C.2 for results with varying demonstrations.
HOPPER
HALF-CHEETAH
WALKER
ANT
HUMANOID
RANDOM
14
-282 = 80
1
-70 = 111
123 35
BC
1432 + 382
2674 1: 633
1691 1: 1008
1425 1: 812
353 1 171
RED
448 1 516
383 1: 819
309 193
910 + 175
242 1: 67
GAIL
3261 + 533
3017 531
3957 = 253
2299 - 519
204 - 67
VALUEDICE
+ 2749
3456 401
3342 1514
1016 313
364 50
NDI+MADE
3288 + 94
4119
4518 127
555 311
6088 689
NDI+EBM
3458
210
4511 + 569
5061 135
4293 431
5305 555
EXPERT
3567 = 4
4142 + 132
5006 + 472
4362 = 827
5417 + 2286
Hognervv3
HallCheetnh-v:3
Walker2de33
Ant-x3
Hummoid-v3
2.8
24
Forward Velocity
log gie (s)
0.
0.0
60
04
-144
54
Randem Observation Damension
Figure 1: Learned density visualization. We randomly sample test states & and multiple test actions as per test state, both from a uniform distribution, then visualize the log marginal log (s) = log Ee, (s,s, ag) projected onto two state dimensions: one corresponding to forward velocity and the other a random selection. Much like true reward function in Mujoco environments, we found that the log marginal positively correlates with forward velocity on 4/5 tasks.
marks when provided one demonstration and outperforms all baselines on all mujoco benchmarks. NDI+MADE achieves expert level performance on 4/5 tasks but fails on Ant. We found spurious modes in the density learned by MADE for Ant, and the RI. algorithm was converging to these local maxima. We found that baselines are commonly unable to solve Humanoid with one demonstration (the most difficult task considered). RED is unable perform well all tasks without pretraining with BC as done in (Wang et al. 2019). For fair comparisons with methods that do not use pretraining, we also do not use pretraining for RED. See Appendix C.4 for results with BC pretraining step added to all algorithms. GAIL and ValueDICE perform comparably with each other, both outperforming behavioral cloning. We note that these results are somewhat unsurprising given that VICE (Kostrikov et al. 2020) did not claim to improve demonstration efficiency over GAIL (Ho & Ermon 2016). but rather focused on reducing the number of environment interactions. Both methods notably under-perform the expert on Ant-v3 and Humanoid-v3 which have the largest state-action spaces. Although minimizing the number of environment interactions was not a targeted goal of this work, we found that NDI roughly requires an order of magnitude less environment interactions than GAIL. Please Appendix C.5 for full environment sample complexity comparisons.
6.2 DENSITY EVALUATION
In this section, we examine the learned density model qo for NDI+EBM and show that it highly correlates with the true mujoco rewards which are linear functions of forward velocity. We randomly sample test states s and multiple test actions a, per test state, both from a uniform distribution with boundaries the minimun/maximum state-action values in the demonstration set. We then visualize the log marginal log 4e (s) = log E., (s,s, ag) projected on to two state dimensions: one corresponding to the forward velocity of the robot and the other random selection, e.g the knee joint angle. Each point in Figure D corresponds to a projection of a sampled test state and the colors scale with the value of log (s). For all environments besides Humanoid, we found that the density estimate positively correlates with velocity even on uniformly drawn state-actions which were not contained in the demonstrations. We found similar correlations for Humanoid on states in the demonstration set. Intuitively, a good density estimate should indeed have such correlations, since the true expert occupancy measure should positively correlate with forward velocity due to the expert attempting to consistently maintain high velocity.
8

---

Under review as a conference paper at ICLR 2021
Table 3: Effect of varying MI reward weight , on (I). Task performance of NDI-EBM (top row) and (2). Imitation performance of NDI-EBM (bottom row) measured as the average KL divergence between T states sampled by running in the true environment, i.c Rews PKL (<[||=)][TT (+))(())))] normalized by the average DKL between the random and expert poli- can be computed analytically since æ, T are conditional gaussians. Density model qú is trained with one demonstration. Setting A too large hurts task performance while setting
cies. DRI, ( (TE [T]]
too small suboptimal for matching the expert occupancy. A middle point of Xy 0.005 achieves
balance between the two metrics.
HOPPER
HALF-CHEETAH
WALKER
ANT
HUMANOID
REWARD
3576 1 154
5658 1 698
5231 + 122
4214 +: 444
5809 1 591 0.55 + 0.21
0
KL
0.13 = 0.09
0.35 = 0.12
0.31 0.08
0.58 + 0.09
REWARD
3506 + 188
5697 + 805
5171 + 157
4158 + 523
5752 + 632
,y 0.0001
KL
0.15 + 0.05
0.32 = 0.15
0.25 0.04
0.51 + 0.05
0.41 + 0.18
REWARD
3458 210
4511 + 569
5061 135
4293 431
5305 + 555
A 0.005
KL
0.11 + 0.02
0.17 + 0.09
0.22 0.14
0.32 + 0.12
0.12 1 0.14
REWARD
1057 + 29
103 + 59
2710 501
1022 +
142 50
xy 0.1
KL
0.78 0.13
1.41 + 0.51
0.41 0.11
2.41 + 1.41
0.89 1 0.21
EXPERT
REWARD
3567 + 4
4142 + 132
5006 + 472
4362 +: 827
5417 + 2286
6.3 ABLATION STUDIES
As intuited in Section 2.2 maximizing the SAELBO can be more effective for occupancy entropy maximization, than solely maximizing policy entropy. (see Appendix C.T for experiments that support this) This is because in discrete state-spaces the SAELBO H. (p=p) is tighter lower bound to occupancy entropy H((P.) than policy entropy H(mT), ). i. H((T)) ) <. H. (p < H(px)). and in continuous state-spaces, where Assumption Capationnoollee holds, the SAELBO is still lower bound while policy entropy alone is neither a lower nor upper bound to occupancy entropy. As an artifact, we found that SAELBO maximization (A >> 0) leads to better occupancy distribution matching than sole policy entropy maximization (A, = 0). Table 3 shows the effect of the varying X on task (reward) and imitation performance (KL), i.c similarities between π. T measured as Ex-a (T( T (-[s))). Setting X too large (2 0.1) hurts both task and imitation performance as the MI reward ry dominates the RL objective. Setting it too small (S 0.0001), i.e only maximizing policy entropy H(TT), turns out to benefit task performance, sometimes enabling the imitator to outperform the expert by concentrating most of it's trajectory probability mass the mode of the expert's 's trajectory distribution. However, the boosted task performance comes at the cost of suboptimal imitation performance, e.g imitator cheetah running faster than the expert. We found that middle point of X 0.005 simultaneously achieves expert level task performance and good imitation performance. In summary, these results show that SAELBO H maximization (Ay > 0) improves distribution matching between π. Te over policy entropy H(T)) maximization (Åş = 0), but distribution matching may not be ideal for task performance maximization, e.g in apprenticeship learning settings. See Appendix C.I for extended ablation studies.
DISCUSSION AND OUTLOOK
This work's main contribution is new principled framework for IL. and an algorithm that obtains state-of-te the-art demonstration efficiency. One future direction is to apply NDI to harder visual IL tasks for which AlL. is known perform poorly. While the focus of this work is to improve on demonstration efficiency, another important IL performance metric is environment sample complexity. Future works could explore combining off- aff-policy RI. or model-based -based RI. with NDI to improve on this end. Finally, there is rich space of questions to answer regarding the effectiveness of the SAELBO reward ry. We posit that, for example, in video game environments r may be crucial for success since state-action entropy maximization has been shown to be far more effective than policy entropy maximization (Burda et al., 2018). Furthermore, one could improve on the tightness of SAELBO by incorporating negative samples (an Den Oord et al., 2018) and learning the critic function f so that it is close to the optimal critic.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mine: Mutual information neural estimation. arXiv preprint arXiv:1801.0006., 2018.
Kiante Brantley, Wen Sun, and Mikael Henaff. Disagreementtregullaiiiee imitation learning. 2020.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Zaremba Wojeiech. Openai gym, arXiv preprint urXiv:1606.01540 2016.
Daniel Brown, Wonjoon Goo, and Scott Niekum. Better-than-demmonttator imitation learning via automatically-rankedd demonstrations. arXiv preprint 197.0377, 2019 2019.
George Brown. Iterative solution of games by fictitious play. Activity Analysis of Production and Allocation, 1951.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: 18/0. /2894, 2018.
Adji Dieng, Francisco Ruiz, David M. Blei, and Michalis K. Titsias. Prescribed generative adversarial networks. arXiv preprint arXiv: 2019.
Dinh, D Krueger, and Y Bengio. NICE: Non-linear independent components estimation. arXiv preprint arXivv1410.8516, 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. arXiv preprint arXiv: 10050000,, May 2016.
Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-hased models. arXiv preprint arXiv:1903.08689) 2018
Yilun Du and Igor Mordatch. Implicit generation and generalization in energy- models. arXiv preprint arXiv:1903.000 08688, 2019.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforce-
ment learning. arXiv preprint arXiv: 171.11110,, 2017.
Ruiqi Gao, Erik Nijkamp, Diederik P Kingma, Zhen Xu, Andrew M Dai, and Ying Nian Wu. Flow 2019.
contrastive estimation of energy-based models. arXiv preprint
Mathieu Germain, Karol Gregor, lain Murray, and Hugo Larochelle. Made: Masked autoencoder for distribution estimation. arXiv preprint arXiv:/502.00 I502:0330033099 2015.
Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence minimization perspective on imitation learning methods. arXiv preprint vrXiv:191..02254 2019.
Ian Goodfellow, Jean Abudie-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems. pp. 2672 2280, 2014.
Xiaoxiao Guo, Satinder Singh, Honglak Lee, Richard L. Lewis, and Xiaoshi Wang. Deep learning for real-time atari game play using offline monte-carlo tree search planning, Advances in Neural Information Processing Systems 27 (NIPS 2014), 2014.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference Artificial Intelligence and Statistics, pp. 297-304, 2010.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. arXiv preprint arXi:: 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic Off- Policy maximum entropy deep reinforcement learning with stochastic actor. arXiv preprint 1rXiv:1801.011 January 2018
10

---

Under review as a conference paper at ICLR 2021
Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. arXiv preprini arXiv: 02690, 2019.
Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines. https://githint com/
hi aill-a/stablln stable-basllleeeee 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, pp. 4565-4573, 2016.
Ferenc Huszár. Variational inference using implicit distributions. arXiv preprint IrXiv:1702.00S: 702.088235 2017.
Aapo Hyvärinen. Estimation Non-Normalized statistical models by score matching. Journal of machine learning research: JMLR, 6(Apr):695-709, 2005. ISSN 1532-4435, 1533-7928.
Riashat Islam, Raihan Seraj, Pierre-Luc Bacon, and Doina Precup. Entropy regularization with discounted future state distribution in policy gradient methods. arXiv preprint arXiv: 2019.
Chi Jin, Praneeth Netrapalli, and Michael Jordan. What is local optimality in noncomvex-nonnconcae minimax optimization? arXiv preprint 1922v:1902.006 2019.
Liyiming Ke, Matt Barnes, Wen Sun, Gilwoo Lee, Sanjiban Choudhury, and Siddhartha Srinivasa. Imitation learning as f-divergence minimization. arXiv preprint 190.::190..228000 2020.
Kee-Bung Kim and Hyun Soo Park. Imitation learning via kernel mean embedding. AAAI, 2018.
Kuno Kim, Yihong Gu, Jiaming Song, Shengjia Zhao, and Stefano Ermon. Domain adaptive imitation learning, arXiv preprint arXiv: 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988 December 2014.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.
llya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off-policy distribution matching. 2020.
Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdinov, Efficient exploration via state marginal matching. arXiv preprint 17X4541906.05274, 2019.
Han Liu, John Lafferty, and Larry Wasserman. Sparse nonparametric density estimation in high dimensions using the rodeo. volume 2 of Proceedings of Machine Learning Research, pp. 283-290, San Juan, Puerto Rico, 21-24 Mar 2007. PMLR.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957 2018.
Shakir Mohamed and Balaji Lakshminarayanan. Learning in implicit generative models. arXiv preprint 16Xiv:1610.0344 2016.
XuanLong Nguyen, Martin J. Wainwright, and Michael I. Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. arXiv preprint arXiv:0809.0 0853, 2010.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-GAN: Training generative neural samplers using variational divergence minimization. arXiv preprint 1606. 00709, June 2016.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. arXiv preprint DrXiv:170S.07057 2017.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. arXiv preprint arXiv:/708 0766,0 2017.

---

Under review as a conference paper at ICLR 2021
Dean A Pomerleau.
Efficient training of artificial neural networks for autonomous navigation. Neural computation, 3(1)888977, 1991. ISSN 0899-7667.
Siddharth Reddy, Anca D. Dragan, and Sergey Levine. Sqil: Imitation learning via reinforcement
learning with sparse rewards. arXiv preprini arXiv: /905. 108, 2017.
Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 661-668, 2010.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. reduction of imitation learning and structured prediction to no-regret online learning, In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 637-635, 201 la.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourleenth international conference on artificial intelligence and statistics, pp. 627-635, 201 lb.
Stephane Ross, Narek Melik-Barkhudarde Kumar Shaurya Shankar, Andreas Wendel, Dehadeepta Dey, Andrew J. Bagnell, and Martial Hebert. Learning monocular reactive uav control in cluttered natural environments. International Conference on Robotics and Automation (ICRA), 2013.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans, arXivv16066 03498, 2016.
Jiaming Song, Hongyu Ren, Dorsa Sadigh, and Stefano Ermon. Multi-agent generative adversarial imitation learning. 2018.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. arXiv preprint 1xXXv:1907.05000 2019.
Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach 2019.
to density and score estimation. arXiv preprint
E Todorov, T Erez, and Y Tassa. MuJoCo: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, October 2012. doi: 10.1 10.1100/RRO... 1..666666...
Emanuel Todorov. Convex and analyriclll--iveriible dynamics with contacts and constraints: Theory and implementation in mujoco. IEFE International Conference on Robotics and Automation (ICRA), 2014.
Eiji Uchibe. Model-free free deep inverse reinforcement learning by logistic regression. Neural Processing Letters, 47:891-905, 2018
Benigno Uria, lain Murray, and Hugo Larochelle. RNADE: density- In C C Burges, L Bottou, M Welling, Z Ghahramani, and K Q Weinberger (eds.), Advances in Neural Information Processing Systems 26, pp. 2175-2183. Curran Associates, Inc., 2013.
The real-valued neural autoregressive
Benigno Uria, Marc-Alexandre Côté, Karol Gregor, lain Murray, and Hugo Larochelle, Neural autoregressive distribution estimation. The Journal of Machine Learning Research, 17(1):7184 7220, 2016.
Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. arXiv preprint arXiv: 1601.06752 January 2016.
Aaron Van Den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 18700:1807 2018.
Ruohan Wang, Carlo Ciliberto, Pierluigi Amadori, and Yiannis Demiris. Random expert distillation: Imitation learning via expert policy support estimation. 2019.
Ziyu Wang, Josh Merel, Scott Reed, Greg Wayne. Nando de Freitas, and Nicolas Heess. Robust 02040,, 2017.
imitation of diverse behaviors. arXiv preprint
Alan Wu, AJ Piergiovanni, and Michael S. Ryoo. Model-based behavioral cloning with future image 2019.
similarity learning. arXiv preprint
Lantao Yu, Yang Song, Jiaming Song. and Stefano Ermon. Training deep energy-based models with f-divergence minimization. arXiv preprint arXiv:2003.03463 2020.
12


Reviews:
[{"review_id": "y8bUqIm2JMN", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "=====POST-REBUTTAL COMMENTS======== \n\nI thank the authors for the response and the efforts in the updated draft. Most of my concerns were clarified and I still think the paper should be accepted. However, I agree with Reviewer 4 that  additional experiments would be good to better tease out the reasons for this method working.\n\n\n\n##########################################################################\n\nSummary:\n \n\nThe paper proposes an interesting way to do imitation learning without using an adversarial framework. The proposed approach involves density estimation to learn a surrogate reward function that can be optimized via RL. The approach is motivated by recently proposed maximum occupancy entropy RL and extends this to the imitation learning setting. The authors derive an efficient policy optimization method that outperforms existing approaches for imitation learning.\n\n##########################################################################\n\nReasons for score: \n \n\nThis paper provides strong theory and strong empirical results validating the theory. GAIL-like methods are notorious for their instability so having non-adversarial IL methods is a significant improvement.\n \n\n##########################################################################\n\nPros: \n \n\n1. Non-adversarial approach to IL that seems well suited policy optimization with any RL algorithm.\n\n2. Significant improvement over state-of-the-art IL approaches. Also works with only one demonstration.\n\n3. Nice theoretical results showing the objective lower bounds reverse KL between expert and imitator.\n\n##########################################################################\n\nCons: \n \n\n1. Only results on mujoco tasks with state information. Most of these tasks can be solved reasonably well with just a bias toward longer episodes. It would be nice to show that the authors method qualitatively imitates a variety of behaviors rather than just being able to go really fast without falling down. Imitating something like the hopper back flip would be a nice addition (https://openai.com/blog/deep-reinforcement-learning-from-human-preferences/). Some other kind of open ended task would also be interesting where learning from demonstrations actually makes sense.\n\n2. Omits other papers that also perform efficient reward learning, then RL for non-adversarial imitation learning: \ne.g. \nUchibe. \"Model-free deep inverse reinforcement learning by logistic regression.\" Neural Processing Letters, 2018.\nBrown et al. \"Better-than-Demonstrator Imitation Learning via Automatically-Ranked Demonstrations.\" CoRL, 2019. \n\n3. Quite a few knobs that need to be tuned in terms of hyperparameters. Perhaps I missed it, but it would be nice to have better intuition for how to set these and how imitation behavior changes based on them.\n\n\n##########################################################################\n\nQuestions/Clarifications:\n\nI'm confused about why the authors think this method will work well with hard exploration video games. I'm not sure I buy the claim that optimizing the authors bound will cause an agent to explore new levels.\n\nAfter equation (9) the authors say this will cause the policy to seek out areas with low randomness and update the policy to be random there, but the alternative also seems equally likely given the objective, e.g., seek out states where there is lots of randomness and try and make the policy less random there.\n\nThe notation for the autoregressive section is confusing what does x = (x_1, ..., x_dim(S) + dim(A) ) =  (s,a) mean? What if S and A are inf dimensional? Also, autoregressive models are often used for time series data, but here is looks like q is conditioned on (s,a) pairs from some lexicographic ordering which does not seem to make sense since there may be no correlation between actions at far apart states.", "extra_scores": {}, "extra_text": {}}, {"review_id": "sMB2Y1AVLrQ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces an approach for imitation learning based on density estimation. The approach uses the previously introduced idea of minimizing some divergence between policy and expert occupancy measures, state-action distributions induced by these policies. The authors propose to first estimate expert occupancy measure either using an autoregressive density model or an energy based model. Then authors use the Donsker-Varadhan KL representation to compute a log ratio between $p(s_{t+1}|s_t)$ and $p(s_t)$ where $s_{t}$ is a state at time $t$. Then, the expert occupancy and the KL representation are used as RL reward for imitation learning. \n\nIn overall, I think that this approach is interesting and this direction is under-explored in the context of imitation learning. The paper is well written and easy to follow. However, there are some issues that require additional clarification (see my comments below). I think that experimental evaluation is adequate but limited, the paper requires an additional ablation study in order to justify certain design decisions. Moreover, I believe that since the method is using SAC as its underlying RL algorithm its necessary to perform additional comparisons with other methods in sample efficient RL ([1], [2] or [3]).\n\nIn conclusion, I think that at the moment this is a borderline paper. The approach is definitely interesting but right now the experimental evaluation is not solid enough to recommend this paper for acceptance. I highly recommend the authors to revise the paper and I'm looking forward to authors' response.\n\nComments:\n1) How many additional interactions from the environments does this approach require? Some other papers ([1], [2], [3]) plot training performance vs number of additional environment interactions.\n2) policy entropy term is included into rewards in algorithm 1 (line 6). Is it the same entropy term as in SAC? If yes, what is the benefit of including it twice?\n3) algorithm 1 would benefit from explicitly stating how f is computed in practice;\n4) what is the benefit of using the representation (7) to compute state density only? Could the same representation be used to compute $KL(\\rho_\\pi(s, a)|\\rho_{expert}(s,a))$? I think that the paper could significantly benefit from additional experiments that demonstrate that the proposed formulation is better than simply using (7) for state-action distributions.\n5) Table 2 demonstrates that smaller values of $\\lambda_f$ lead to better performance (with an exception of Ant but the difference is within the standard deviation). The figure table 2 can benefit from including results for $\\lambda_f=0$.\n\n\nReferences:\n\n[1] Sample Efficient Imitation Learning for Continuous Control, Sasaki et al., ICLR 2019: https://openreview.net/forum?id=BkN5UoAqF7\n\n[2] Discriminator-Actor-Critic, Kostrikov et al., ICLR 2019: https://openreview.net/forum?id=Hk4fpoA5Km\n\n[3] Sample-Efficient Imitation Learning via Generative Adversarial Nets, Blonde et al,. 2018: https://arxiv.org/abs/1809.02064", "extra_scores": {}, "extra_text": {}}, {"review_id": "eV7fPEP1du2", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis work proposes a novel density matching method for learning from demonstration, which achieves state-of-the-art demonstration efficiency. Prior density matching methods utilize the adversarial methods suffers from the instability of optimization. To overcome this issue, this work proposes to separate the imitation process into expert density estimation phase and density matching phase, where a model-free formulation is derived and provably served as the lower bound of reverse KL divergence between $\\pi_\\theta$ and expert policy $\\pi_E$.\n\nThis work overcomes the instability issue of min-max optimization\n through transferring the objective to the lower bound leading to \nthe model-free objective is attractive and novel as far as I am concerned. \n\nThe experimental results are mostly complete and convincing. \nThe improvement in the final performance over state-of-the-art is demonstrated. Although the sample-efficiency is not the focus of this \nwork, it would be helpful to see the averaged learning curves as in the results\nin related works, which can give a straight forward intuition on the\nrobustness of the proposed methods. \n\nPlease specify the number of random seeds and, the number of evaluation trajectories [and what is the mechanism to choose those trajectories, such as the last 10 episodes?] of \nthe reported results in Table 1 and other tables. \n\nQuestions: \n1. Does the improvement come from the better density estimation of the expert's policy? It would be helpful to separate the two phases and conduct an ablation study to show how does each phase affects the performance. \nIt would be nice to see the results of replacing density estimation with\noracle [density estimated by a sufficient amount of trajectries]. I didn't see too much difference in terms of final performance between 1 and 25 expert trajectories. Is there a large gap in the sample complexity?\n\n2. Could the author explain the reason why one expert trajectory can provide a good density estimation of expert policy? Typically, we cannot expect one trajectory to cover the state-action space. It is less attractive\nif the learned policy can reach expert-level performance but stick to\nthe original modality of the expert policy. \n\n\nAppendix: \nLemma 1:\n$-\\lambda \\sum_{x} \\hat{p}(x) \\log \\hat{p}(x)+(1-\\lambda) \\sum_{x} \\hat{q}(x) \\log \\hat{q}(x) $ $\\rightarrow$ $-\\left[\\lambda \\sum_{x} \\hat{p}(x) \\log \\hat{p}(x)+(1-\\lambda) \\sum_{x} \\hat{q}(x) \\log \\hat{q}(x)\\right]$\n\nRegarding theorem 1\nDo we need to normalize $p_{\\theta}(s)$ with $1/(1-\\gamma)$ before applying concave property? \n$\\left(\\mathcal{H}\\left(s_{0}\\right)+\\sum_{t=1}^{\\infty} \\gamma^{t} \\mathcal{H}\\left(s_{t}\\right)\\right)$ \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "PuSQHsiiQ5r", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Review\nKey points are written in bold font.\n\n* Section 1:\n  * You should maybe include a reference to \"A Divergence Minimization Perspective on Imitation Learning\" (Ghasemipour et al. 2019) since that work studies properties of different divergences used for match the joint state-action distributions (similar to Ke et al. which you cited) as well state-marginal matching using adversarial methods.\n* Section 2:\n  * For max-ent RL, there is an intuitive connection between RL and probability distributions over the space of trajectories. For example -- when we have deterministic transition dynamics -- max-ent RL's optimal policy is the policy that \"samples\" trajectories according to the (unnormalized) density $\\hat{p}(\\tau) = \\sum_t r(s_t, a_t)$. I understand the motivation of the addition of this form of entropy term instead, but is there anything that can be said about what the optimal policy for this objective looks like (in a similar vein to the max-ent RL solution I described above)?\n  * __Section 2.2, Theorem 1: Assumption 1 should definitely be included in the main text. This is quite a strong assumption (even if it holds in many physical/robotics problems), and the reader should not have to look into the appendix to realize this. Furthermore, I strongly believe that you should include a discussion in the main text of why you needed these assumptions (what goes wrong when these assumptions are not satisfied?). This is important so that 1) you provide intuition to the reader of what can go wrong if this is not satisfied, 2) future work can try to address the limitations.__ For example, if the transition dynamics was not deterministic, the intuition your provide when saying \"$I\\_{NWJ}$ encourages the agent to seek out states where...\", is not necessarily correct. In a non-deterministic transition setting, the mutual information term could seek out states where the dynamics are more determinstic, but I don't see why that would be a good thing necessarily.\n  * Section 2.2: \"... Our bound will encourage the agent to continuously seek out...\": I don't quite agree with this. You're also increasing entropy, which reduces the chances of getting to those states. Also, the initial low variance random policy might/probably won't get anywhere interesting anyways.\n* Section 3:\n  * You are using SAC as your RL algorithm. SAC itself is introducing entropy, so maybe comment on how that is playing with your entropy term. Is there any concerns one should be aware of?\n  * __Top of page 6. You are saying that your method is not adversarial *<<even if you optimize $f$>>*. I believe, despite appendix A.3, this statement is incorrect. Consider the AIRL (Fu et al.) algorithm. According to Ghasemipour et al., 2019, the AIRL objective minimizes $KL(\\rho^\\pi(s,a)||\\rho^{exp}(s,a))$.__\n$-KL(\\rho^\\pi(s,a)||\\rho^{exp}(s,a)) = E_{\\rho^\\pi(s,a)} [\\log \\frac{\\rho^{exp}(s,a)}{\\rho^\\pi(s,a)}] = \\text{RL with the loss }\\quad \\log \\frac{\\rho^{exp}(s,a)}{\\rho^\\pi(s,a)}$\n__In works such as AIRL and GAIL there are two things to consider. 1) lower bounds similar to your $I\\_{NWJ}$ lower bound are written to estimate quantities of interest (Jensen-Shannon for GAIL, and reverse KL for AIRL). These lower bounds are maximized to obtain tighter estimates of the quantity of interest. 2) Once a good estimate is obtained, an RL step is performed using the estimate. And this process is repeated.\nIn your work, you are trying to avoid directly estimating $\\log \\frac{\\rho^\\pi(s,a)}{\\rho^{exp}(s,a)}$, and have decomposed the objective. If we try to write this $KL$ objective in a form that looks closer to your equations, we get:__\n$-KL(\\rho^\\pi(s,a)||\\rho^{exp}(s,a))\n= E_{\\rho^\\pi(s,a)}[\\log \\frac{\\rho^{exp}(s,a)}{\\rho^\\pi(s,a)}] = E_{\\rho^\\pi(s,a)}[\\log \\rho^{exp}(s,a) - \\log \\rho^\\pi(s,a)]$\n$= E_{\\rho^\\pi(s,a)}[\\log \\rho^{exp}(s,a)] - E_{\\rho^\\pi(s,a)}[\\log \\rho^\\pi(s,a)] \\text{(The J and H in your equation 2)}$\n$= E_{\\rho^\\pi(s,a)}[\\log \\rho^{exp}(s,a)] - E_{\\rho^\\pi(s,a)}[\\log \\rho^\\pi(a|s)] - E_{\\rho^\\pi(s,a)}[\\log \\rho^\\pi(s)]$\n__In your work, you are estimating $\\log \\rho^{exp}(s,a)$ directly through generative models, $\\log \\rho^\\pi(a|s)$ is available from the policy, and in your derivations, it seems that you have been able to replace  $- E\\_{\\rho^\\pi(s,a)}[\\log \\rho^\\pi(s)]$ with the mutual information terms (due to Assumption 1 I believe). But this remaining mutual information terms is still being estimated using similar processes/lower bounds as prior work.__\n__Hence, it is my view that your method is as \"adversarial\" as prior work. The reason you are suggesting stability of your optimization process (which I don't think you have quantified) is maybe because you are not actually learning $f(s\\_{t+1}, s\\_t)$ which is the property that makes those prior works \"adversarial\".__\n* Section 5:\n  * __Pipeline: Why did you checkpoint based on your augmented rewards? How were the checkpoints made for the other methods?__\n  * Architecture: What is $\\lambda_\\pi$?\n  * In GAIL are you using techniques from the GAN literatures (such as gradient penalty or spectral normalization) to regularize the discriminators? And are you using the original TRPO optimization or a newer technique such as PPO?\n  * Do you have intuition for the poor performance or Ant NDI+MADE? For the 25 demos version in the appendix this value is also low.\n  * __Section 5.1: Although this does not seem to be a central claim of contribution in your work, I am wondering if your claims of sample-efficiency are well-founded. You are using SAC, which is an off-policy algorithm, whereas GAIL uses on-policy RL (e.g. TRPO). To enable a proper judgement of sample-efficiency, please respond to the following questions: 1) What implementation of GAIL did you use? Does it use TRPO. If so, this might be a contributing factor since there are better on-policy algs now such as PPO. 2) For your method, in your SAC replay buffer are you only storing on-policy samples, or maintaining a large replay buffer or past interactions? 3) When you're estimating the mutual information, you are not using on policy samples for the marginal q distributions, correct? 4) What is the ratio of model update steps to environment steps in the various algorithms?__\n  * With regards to the previous point, a more fair comparison might be to run GAIL with SAC and maintaining a full replay buffer, as done in Ghasemipour et al., 2019.\n  * __Table 2: $\\lambda\\_f$ is tuning the knob for the stregth of the mutual information maximization term. Why should this be improving the KL?__\n  * __You have a good set of experiments, but ( and I might be wrong) my biggest gripe with your experiments is that it doesn't seem that you have any experiments that would demonstrate a unique advantage of your method. Please clarify the unique advantages of your methodology, and how that is being shown in the experiments.__\n* Section 6:\n  * I don't think AIL works particularly poorly on visual domains (e.g. \"InfoGAIL: Interpretable Imitation Learning from\nVisual Demonstrations\" or \"Third Person Imitation Learning\"). Do you have references for this statement?\n  * I don't quite understand this statement: \"We posit that...\". Please clarify.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a difficult borderline decision, with the reviewers evenly split in their final recommendation.  Overall, the authors provided good responses to the reviewer questions: this was much appreciated.  The reviewers requested additional ablations and explanations, which the authors provided.

A prevailing concern is that the experimental evaluation, restricted to a few standard MuJoCo environments, does not really demonstrated a distinctive advantage for the proposed approach.  In fact, one of the new ablations added raises concerns about the significance of the paper's main technical contributions: the \lambda_f=0 row added to Table 3 shows very strong reward results, which apparently obviates a key aspect of the proposed approach. 

This work is interesting, and would like to see it published, but the current state of the evaluation does not support the significance of the main contribution.  I think the authors need to expand their empirical evaluation, as suggested, to better highlight the effectiveness of the proposed approach over the \lambda_f=0 baseline.  In the end, I think the authors would be better served by broadening the evaluation, isolating scenarios where the key proposal shows significant (rather than marginal) benefits, and publishing a more compelling version.
Decision: Reject