Imitation Learning via Off-Policy Distribution Matching
Authors: Ilya Kostrikov
Ofir Nachum
Jonathan Tompson
Venue:  (2020)
Keywords: reinforcement learning
deep learning
imitation learning
adversarial learning

Abstract: When performing imitation learning from expert demonstrations, distribution matching is a popular approach, in which one alternates between estimating distribution ratios and then using these ratios as rewards in a standard reinforcement learning (RL) algorithm. Traditionally, estimation of the distribution ratio requires on-policy data, which has caused previous work to either be exorbitantly data- inefficient or alter the original objective in a manner that can drastically change its optimum. In this work, we show how the original distribution ratio estimation objective may be transformed in a principled manner to yield a completely off-policy objective. In addition to the data-efficiency that this provides, we are able to show that this objective also renders the use of a separate RL optimization unnecessary. Rather, an imitation policy may be learned directly from this objective without the use of explicit rewards. We call the resulting algorithm ValueDICE and evaluate it on a suite of popular imitation learning benchmarks, finding that it can achieve state-of-the-art sample efficiency and performance.

Published as a conference paper at ICLR 2020
IMITATION LEARNING VIA OFF-POLICY DISTRIBUTION MATCHING
Ilya Kostrikov; Ofir Nachum, Jonathan Tompson
Google Research
(kostrikov, ofirnachum, tompson)eee goognle.ooooooooooooomoomoo
ABSTRACT
When performing imitation learning from expert demonstrations, distribution matching is a popular approach, in which one alternates between estimating dis- tribution ratios and then using these ratios rewards in standard reinforcement learning (RL) algorithm. Traditionally, estimation of the distribution ratio requires on-policy data, which has caused previous work either be exorbitantly data- inefficient or alter the original objective in a manner that can drastically change its optimum. In this work, show how the original distribution ratio estimation ob- jective may be transformed in a principled manner to yield a completely off- off-policy objective. In addition the data-efficiency that this provides, that this objective also renders the use of a separate RL optimization unnecessary. Rather, imitation policy may be learned directly from this objective without the use of explicit rewards. We call the resulting algorithm ValueDICE and evaluate it on suite of popular imitation learning benchmarks, finding that it can achieve state-of-the therthhrrttttte art sample efficiency and performance]
are able to show
INTRODUCTION
Reinforcement learning (RL) is typically framed as learning a behavior policy based on reward feedback from trial-and-error experience. Accordingly, many successful demonstrations of RL often rely on carefully handcrafted rewards with various bonuses and penalties designed to encourage intended behavior (Nachum al. 20T9a Andrychowicz al. 2018). In contrast, many real-world behaviors are easier to demonstrate rather than devise explicit rewards. This realization is at the heart of imitation learning (Ho Ermon 2016 Ng et al.) Pomerleau 1989). in which one aims to learn behavior policy from set of expert demonstrations logged experience data of near-optimal policy interacting with the environment - without explicit knowledge of rewards.
Distribution matching via adversarial learning, or Adversarial Imitation Learning (AIL), has recently become a popular approach for imitation learning (Ho & Ermon 2016 Fu et al.) 2017 Ke et al, 2019 Kostrikov et al. 2019). These methods interpret the states and actions provided in the expert demonstrations as a finite sample from a target distribution. Imitation learning can then be framed as learning a behavior policy which minimizes a divergence between this target distribution and the state-action distribution induced by the behavior policy interacting with the environment. As derived by Ho & Ermon 2016)), this divergence minimization may be achieved by iteratively performing two alternating steps, reminiscent of GAN algorithms (Goodfellow et al. 2014). First, one estimates the density ratio of states and actions between the target distribution and the behavior policy. Then, these density ratios are used as rewards for standard RL algorithm, and the behavior policy is updated to maximize these cumulative rewards (data distribution ratios).
The main limitation of current distribution matching approaches is that estimating distribution den- sity ratios (the first step of every iteration) typically requires samples from the behavior policy dis- tribution. This means that every iteration every update to the behavior policy requires new interactions with the environment, precluding the use these algorithms in settings where interac- tions with the environment expensive and limited. Several papers attempt relax this on-policy
Also at NYU.
Coode to reproduce results is availahle htpp::///ittii....o.ggog.lllle research [ooglle-rreeeerrhchtteee/eeeeee

---

Published as a conference paper at ICLR 2020
requirement and resolve the sample inefficiency problem by designing off-policy imitation learning algorithms, which may take advantage past logged data, usually in the form of a replay buffer (Kostrikov al. 2019 Sasaki et al. 2019). However, these methods do by altering the original divergence minimization objective to measure a divergence between the target expert distribution and the replay buffer distribution. Accordingly, there is guarantee that the learned policy will recover the desired target distribution.
In this work, introduce an algorithm for imitation learning that, on the one hand, performs diver- gence minimization as in the original AIL methods, while on the other hand, is completely off-policy. We begin by providing new formulation of the minimum divergence objective that avoids the use of any explicit on on-policy expectations. While this objective may be used in the traditional way to estimate data distribution ratios that are then input to an RI. algorithm, we go further to show how the specific form of the derived objective renders the use of a separate RL optimization unneces- sary. Rather, gradients of the minimum divergence objective with respect to behavior policy may be computed directly. This way, imitating behavior policy may be learned to minimize the di- vergence without the use of explicit rewards. We call this stream-lined imitation learning algorithm ValueDICE. In addition to being simpler than standard imitation learning methods, show that our proposed algorithm is able to achieve state-of-the thetth-artt performance on a suite of imitation learning benchmarks.
2 BACKGROUND
We consider environments represented as a Markov Decision Process (MDP) (Puterman, 2014). defined by the tuple, (S, (s). (s),
r(s, r(s, a), where S and A the state and action space, respectively, (s(s) is an initial state distribution, p(s'|s, a) defines environment dynamics represented conditional state distribution, r(s, a) is reward function, and is return discount factor. A behavior policy <-|--) interacts with the environment to yield experience (st, (t rt, 8t+1), for 0, I..... where so Po(-). at T--sst, St+1 p(-|s,, at), r. r(st, a.). Without loss of generality, we consider infinite-horizon, to learn behavior policy π (-1ss) to maximize cumulative rewards, based on experience gained from interacting with the environment.
con-terminating environments, In standard RL, one aims
In imitation learning (Pomerleau) 1989; Abbeel & Ng. 2004 Ho Ermon 2016), the environ- ment reward is not observed. Rather, one has access to a set of expert demonstrations D == transitions the environment induced by an un-
{(K,, 0k. N given by state-action-next-se known expert policy Texp and the goal is to learn behavior policy π which recovers Texp- During the learning process, in addition to the finite set of expert demonstrations D, one may also optionally rewards are observed). This setting describes
interact with the environment (in these interactions, a number of real-world applications where rewards are unknown, such as Pomerleau (1989); Muller et al. (2006); Bojarski et al. (2016).
2.1 BEHAVIORAL. CLONING (BC)
Supervised behavioral cloning (BC) is a popular approach for imitation learning. Given a set of expert demonstrations, mapping of state observations actions is fit using regression density estimation. In the simplest case, one simply trains the behavior policy π to minimize the negative log-likelihood of the observed expert actions:
N
min = N
Σ log (an(sk))
Unlike Inverse Reinforcement Learning (IRL) algorithms (e.g. GAIL (Ho & Ermon 2016 BC does not perform any additional policy interactions with the learning environment and hence does not suffer from the same issue of policy sample complexity. However, behavioral cloning suffers from distributional drift (Ross et al. 201T); i..., there is no way for æ to learn how to recover if deviates from the expert behavior state not in the expert demonstrations.
2

---

Published as a conference paper at ICLR 2020
2.2 DISTRIBUTION MATCHING
The distribution matching approach provides a family of methods that are robust to distributional shift. Rather than considering the policy directly as a conditional distribution x x-|ss) over actions, this approach considers the state-action distribution induced by a policy. In particular, under certain conditions (Puterman 2014), there is a one-to-one correspondence between a policy and its state- action distribution d' defined as,
(8, a) = (1 - 7) Σ D'p((s = 8, =,, also po('), 81 p(-1 ~ (-1(+)))- (2)
1=
By the same token, the unknown expert policy Texp also possesses a state-action distribution dxx,, and one may usually assume that the expert demonstrations D == {(5k, 0k. & are sampled as (8%, (&)) ~ dexp, 8/4 p(-1sk; (x))
Accordingly, the distribution matching approach proposes to learn æ to minimize the divergence between df and d=xp, The KL-divergence is typically used to measure the discrepancy between dm
and dxxp (Ho Ermon 2016; Ke et al., 2019))
dexp(s,
DKL (d" [d [(xx)))
E (s.))ad
log d' d=((sa)) a)
The use of the KL-divergence is convenient, as may be expressed as an RL problem where rewards given by log distribution ratios:
dexp a1)
- -DLL (d" () = (1 7)
E spmpol()) (-le)) #641 ppppppsssss pl.
?" log
(4)
d= (s{, ar)
In other words, if one has access to estimates of the distribution ratios of the two policies, then the minimum divergence problem reduces to a max-return RL problem with rewards F(s, = (=,) log (=.aa)) Any on-policy or off-policy RL algorithm can be used to maximize the corresponding expected returns in Equation
Capitalizing on this observation, Ho & Ermon (2016) and Ke et al. (2099) propose algorithms (e.g.,
GAIL) in which the distribution ratio is estimated using GAN-like objective:
max JGAUL (h) == dss (log h(s,, a)] (s,,a)vd (log(1 - h(s,,a))
(5)
k:5xAA>>0.1)
In this objective, the function h acts as discriminator, discriminating between samples (s, a) from dexp and d=, The optimal discriminator satisfies,
ds a)
log h" (hs a) - log(1 - h² (s) = log d= d=
(6)
and so the distribution matching rewards may be computed as f(s, = log h (8, a) - log(1 h*(s)))). In practice, the discriminator is not fully optimized, and instead gradient updates to the discriminator and policy are alternated.
These prior distribution matching approaches possess two limitations which will resolve with our proposed VaIueDICE algorithm:
Arguably the main limitation of these prior approaches is that they require access to on-policy samples from d=, While off-policy RL can be used for learning optimizing the dis- criminator h necessitates having on-policy samples (the second expectation in Equation (5). Thus, in practice, GAIL requires prohibitively large number of environment interactions, making it un- feasible for use in many real-world applications. Attempts to remedy this, such as Discriminator- Actor-Critic (DAC) (Kostrikov et al.) (209)) often do so via ad-hoc methods; for example, chang- ing the on- on-policy expectation E( [log(1-h(s, a))] in Equation 5 to an expectation over the replay buffer (R log(1 h(s, a))]. While DAC achieves good empirical results, it does not guarantee distribution matching of π to Texp. especially when dRB is far from d=
On-policy.
Separate RL optimization. Prior approaches require iteratively taking alternating steps: first estimate the data distribution ratios using the GAN-like objective, then input these into an RL optimization and repeat. The use of a separate RL algorithm introduces complexity to any im- plementation of these approaches, with many additional design choices that need to be made and more function approximators to learn (e.g., value functions). Our introduced ValueDICE will be shown to not need a separate RL optimization.
3

---

Published as a conference paper at ICLR 2020
3 OFF-POLICY FORMULATION OF THE KL-DIVERGENCE
As is standard in distribution matching, we begin with the KI--dierreence between the policy state- action occupancies and the expert. However, in contrast to the form used in Equation 4 or 5 use
the Donsker- Nonsker-Varadhan representation (Donsker& & Varadhan 1983) given by,
DKL (d"
min
log (d. (5.0)) (x,a)=dd (z(ss
Similar to Equation this dual representation of the KL has a property that important for imitation learning. The optimal x² is equal to the log distribution ratio (plus constant)
d' a)
x (s, a) = log dexp (8,a)
+ +.
(8)
In our considered infinite-horizon setting, the constant does not affect optimality and so we will ignore (take C' 0). If one were to take GAlL-like approach, they could use this form of the KL to estimate distribution matching rewards given by f(s,a) = -x" (z' a), and these could then be maximized by any standard RL algorithm. However, there is clear advantage of this objective over GAIL since it still relies on an expectation with respect to on-policy samples from d=.
To make this objective practical for off- off-policy learning, we take inspiration from derivations used in DualDICE (Nachum et al. 2019b), and perform the following change of variables
x(s, v(s, a) 1 v(ssvs (s)
where 8 is the expected Bellman operator with respect to policy and zero reward:
1 v" a) = >IE.' (s. (pss'aa))
(10)
This change of variables is explicitly chosen to take advantage of the linearity of the second expec- tation in Equation 7
Specifically, the representation becomes,
DKL (d (|dxx))
min S.SxA
log E
E [vss,a)-B'v(s,. v(s a)l, (11)
(s.a)
(s.a)~d
where the second expectation conveniently telescopes and reduces to an expectation over initial states (see Nachum et al. (2019b) for details);
min (V) log E (x.p)
so (-), [e - (1-
E [M(So, ao)]. (12)
ag ag-on(1150 (-100
Thus we achieve our ValueDICE objective. It allows us to express the KL-divergence between dm and d in terms of an objective over a fuluu--unctionn y expressed in an off-policy manner, with expectations over expert demonstrations dxx and initial state distribution po(-).
It is clear that the derived objective in Equation 12 possesses two key characteristics missing from prior distribution matching algorithms: First, the objective does not rely on access to samples from the on-policy distribution d"', and so may be used in more realistic, off-policy settings. Second, the objective describes proper divergence between d² and dxx,, as opposed to estimating a divergence between RB and dex,, and thus avoids poor behavior when dRB is far from d². In the following section, we will go further to show how the objective in Equation 12 also renders the use of separate RL optimization unnecessary.
4 VALUEDICE: IMITATION LEARNING WITH IMPLICIT REWARDS
Although it is standard in distribution matching have separate optimizations for estimating the distribution ratios and learning policy, in our case this can be mitigated. Indeed, looking at our
This result is easy to derive by setting the gradient of the Donsker-Varadhan representation to zero and solving for x².
This change of variables valid when one assumes log ds, (s.a))a'*P(. a) E for all s E S, E A. where AC is some hounded subset of IR, and x is restricted to the family of functions $ x A + K. *DIEE Nachum al. 2019h) abhreviation for discounted distribution correction estimation.
4

---

Published as a conference paper at ICLR 2020
formulation of the KL in Equation be easily computed. Specifically, we may express the distribution matching objective for x as max-min optimization:
12
that gradients of this objective with respect to x may
max min JDICE(T, log I ==5xA=B
E [eulaa) (aotsa)) -
E (u(su, an]] (13)
so og agnvx(-180) (-160)
If the inner objective over v is sufficiently optimized, the gradients of may be computed di- rectly (Bertsekas 1999), noting that,
(( -y ( u(,.()))) (esh a' Vog (14)
E Om (e0 (400) (s0, ao)] = Eepr-pol (e0 [ 2o) log ((o]sa))]
(15)
In continuous control environments when is parameterized a Gaussian and is neural net- work, one may use the re-parametterizttion trick (Haarnoja et al. 2018) to compute gradients of the y-values with respect policy mean and variance directly as opposed to computing log x (als). Please see the appendix for full pseudocode implementation of VaCUeDICE. We note that in prac- tice, as GAIL, we do train y until optimality but rather alternate v and updates.
The mechanics of learning according to the ValueDICE objective are straightforward, but what the underlying reason for this more streamlined policy learning? How does it relate the standard protocol of alternating data distribution estimation with RL optimization? To better understand this, we consider the form y when
completely optimized. If we consider the original change of 9 and optimality characterization (Equation [8) we have,
variables (Equation
d" (s a) v v"(s.a) a) B" V' (s, = x x²(s,a) = logg dexp ((xxp(x,a)
(16)
From this characterization of v², realize that v" is sort of Q-value function: v² (s, a) is the future discounted sum of rewards i(s, a) == log d" (s.a) (r<((s.a] when acting according to m. The gradients for æ then encourage the policy to choose actions which minimize v" (s. a), i.e., maximize future (s.a(
discounted log ratios log d= (s.a) Thus we realize that the objective for ı in ValueDICE performs exactly the RL optimization suggested by Equation 4 The streamlined nature ValueDICE comes from the fact that the value function (which would traditionally need to be learned critic in a separate actor-critic RL algorithm) is learned directly from the same objective as that used for distribution matching.
Thus, in addition to estimating a proper divergence between df and dxx in an off-policy manner, ValueDICE also greatly simplifies the implementation of distribution matching algorithms. There is no longer a need to use separate RL algorithm for learning  and moreover, the use of v as a value function removes any use of explicit rewards. Instead, the objective and implementation only in terms of policy π and function v.
SOME PRACTICAL CONSIDERATIONS
In order to make use of the ValueDICE objective (Equation 13) practical scenarios, where one does not have access to dxx or po(-) but rather only limited finite samples, we perform several modifications.
5.1 EMPIRICAL EXPECTATIONS
The objective in Equation 13 contains three expectations:
An expectation over dNP (the first term of the objective). Note that this expectation has logarithm outside of it, which would make any mini-baich approximations of the gradient of this expectation biased.
2. An expectation over Po(+) (the second term of the objective). This term is linear, and so is very amenable to mini-batch optimization.
5

---

Published as a conference paper at ICLR 2020
3. An expectation over the environment transition p(-|s, a) used compute (*, This expectation has log-expected- sreected-expneettt applied to it, so its mini-batch approximated gra- dient would be biased in general.
For the first expectation, previous works have suggested number of remedies reduce the bias of mini-batch gradients, such as maintaining moving averages of various quantities (Belghazi et al.) 2018). In the setting considered, we found this to have a negligible effect on performance. In Tact, simply using the biased mini-batched gradients was sufficient for good performance, and so we used this for our experiments.
For the second expectation, we use standard mini- batch gradients. which are unbiased. Although initial state distributions are usually not used in imitation learning, is easy to record initial states as they are observed, and thus have access to an empirical sample from Po. Furthermore, as detailed in Section (.3] is possible to modify the initial state distribution used in the objective without adverse effects.
Finally, for the third expectation, previous works have suggested the use of Fenchel conjugates to remove the bias (Nachum et al. (001b)). In our case, we found this unnecessary and instead use a biased estimate based on the single sample 8' ~ p(-(s,,) a). This naive approach was enough to achieve good performance on the benchmark domains considered.
In summary, the empirical form the objective is given by,
JDICE(TTV)
E
log
(c(s)a))o((sss0)) - -11-7)- a'men--ls')
(1 [w)) (17)
batch(D)~>D batch(po)~fo
8,0,8 (D).
so basch(po). 00 00~x(-190)
where batch(D) is mini-batch from D and batch(po) is mini-batch from the recorded initial states fo.
5.2 REPLAY BUFFER REGULARIZATION
The original ValueDICE objective only expert samples and the initial state distribution. In practice, the number of expert samples may be small and lack diversity, hampering learning. In order increase the diversity of samples used for training, we consider an alternative objective, with a controllable regularization based on experience in the replay buffer:
Jmix (IC v) log E
je (x ((saa-BBo(s.a)) (1 a) 7)
E
[M an)]
tompol()) ao/o)(-180)
E (d.0)
(1(8, - B ((ss (18)
where d (s, a) (1 as) od a).
The main advantage of this formulation is that introduces v-values into the objective on samples that outside the given expert demonstrations. Thus, if π deviates from the expert trajectory, we will still able to learn optimal actions that return the policy back to the expert behavior. At the same time, can verify that in this formulation the optimal still matches Texps unlike other proposals for incorporating a replay buffer distribution (Kostrikov et al. 2019). Indeed, the objective in Equation
18 corresponds to the
Varadhan representation,
-DLL( ((1 - oz) + RB || (1 a)d ad(RB)
min log E (sia) (+.a)] - (1 a)) ( [x(s,a)] - a- (( (1(8, a)). (19) =.SAA=R
and so the optimal values of i²* satisfy,
(1 a)d (s. a) + RB (s.
- (s, x= (s. log
(20)
(1 - axx a) + (B a)
Therefore, the global optimality of π = Texp is unaffected by any choice of a < We note that in practice use small value = 0.1 for regularization.

---

Published as a conference paper at ICLR 2020
5.3 INITIAL STATE SAMPLING
Recall that dxxp, dd traditionally refer to discounted state-action distributions. That is, sampling from them is equivalent to first sampling a trajectory (st, ag. $1......... ST) and then sampling a time index t from a geometric distribution Geom - ?) (appropriately handling samples that are beyond T). This means that samples far into the trajectory do not contribute much to the objec- tive. To remedy this, we propose treating every state in a trajectory as an 'initial state." That is, we consider a single environment trajectory (so, ao, $1, a1 a1...., .T) as distinct virtual trajectories {(E, de; $1+1, "t+1s-... &T)) We apply this to both dxxp and d',, so that not only does it in- crease the diversity of samples from d'xp, but it also expands the initial state distribution po(-) to encompass every state in trajectory. We note that this does not affect the optimality of the objective with respect π,, since in Markovian environments expert policy should be expert regardless of the state at which it starts (Puterman) 2014).
6 RELATED WORK
In recent years, the development of Adversarial Imitation Learning has been mostly focused on on-policy algorithms. After Ho & Ermon (2016) proposed GAIL to perform imitation learning via adversarial training, number of extensions has been introduced. Many of these applications of the AIL framework (Li et al., 2017]: Hausman et al. 2017 Fu et al.] (017)) maintain the same form of distribution ratio estimation as GAIL which necessitates on-policy samples. In contrast, our work presents an off-policy formulation of the same objective.
Although several works have attempted to apply the AIL framework off-policy settings, these previous approaches are markedly different from our own. For example, Kostrikov el al. (2019) proposed to train the diseriminator in the GAN-like AlL objective using samples from replay buffer instead of samples from a policy. This changes the distribution ratio estimation to measure a divergence between the expert and the replay. Although we introduce a controllable parameter a for incorporating samples from the replay buffer into the data distribution objective, we note that in practice we use a very small a = 0.1. Furthermore, by using samples from the replay buffer in both terms of the objective opposed to just one, the global optimality of the expert policy is not affected.
The off-policy formulation of the KL-divergence we derive is motivated by similar techniques in DualDICE (Nachum et al., 2019b). Still, our use of these techniques provides several novelties. First, Nachum al. (2019h) only use the divergence formulation for data distribution estimation (which is used for off-policy evaluation), assuming a fixed policy. We use the formulation for learning a policy minimize the divergence directly. Moreover, previous works have only ap- plied these derivations to the f-divergence form of the KL-divergence, while we are the first to utilize the Donsker- Varadhan form. Anecdotally in our initial experiments, we found that using the f-divergence form leads to poor performance. We note that our proposed objective follows form similar REPS (Peters al. 2010), which also utilizes log averaeegge- term. However, policy and value learning in REPS are performed via bi-level optimization (i.e., the policy is learned with respect to a different objective), which distinct from our algorithm, which trains values and policy with respect to the same objective. Our proposed ValueDICE is also significant for being able to incorporate arbitrary (non-expert) data into its learning.
7 EXPERIMENTS
We evaluate ValueDICE in variety of settings, starting with a simple synthetic task before contin- uing to an evaluation on a suite of MuJoCo benchmarks.
7.1 RING MDP
We begin by analyzing the behavior of ValueDICE on a simple synthetic MDP (Figure I The states of the MDP are organized in ring. At each state, two actions are possible: move clockwise or counter-clockwise. We first look at the performance of ValueDICE in a situation where the expert data is sparse and does not cover all states and actions. Specifically, we provide expert demonstra-
7

---

Published as a conference paper at ICLR 2020
Sparse Expert
ValueDICE Policy
Stoch, Expert
ValueDICE on Stoch. Data
033
Figure 1: Results of ValueDICE on simple Ring MDP. Left: The expert data is sparse and only covers states 0, I, and 2. Nevertheless, ValueDICE is able to learn policy on all states to best match the observed expert state-action occupancies (the policy learns to always go to states and 2). Right: The expert is stochastie. Value is able learn a policy which successfully minimizes the true KL computed between d and dxx,,
HalfCheetah
Hopper
Walker2d
4200
0002
4000
DAC
3300
Valuu
2020
2200
2000
1:00
1020
1000
-1200
20c
41k
50k
Eru.
Enc.
Ery,
Figure 2: Comparison of algorithms given expert trajectory. We use the original implementation of GAIL (Ho & Ermon 2016) produce GAIL and BC results.
tions which cover only states 0, 1, and (see Figure left). While the problem of recovering the true (unknown) expert is ill-defined, it is still possible find a policy which recovers close to the same occupancies. Indeed, this is the policy found by ValueDICE, which chooses the appropriate actions each state to optimally reach states and 2 (and alternating between states and 2 when at these states). In many practical scenarios, this is the desired outcome if the imitating policy somehow encounters a situation which deviates from the expert demonstrations, we would like it to return to the expert behavior as fast as possible. Notably, technique like behavioral cloning would fail to learn this optimal policy, since its learning is only based on observed expert data.
We also analyzed the behavior of ValueDICE with stochastic expert (Figurerrrghtt By using synthetic MDP, we are able to measure the divergence DKL(d") ddx))) during training. As expected, find that this divergence decreases during ValueDICE training.
7.2 MUJoCo BENCHMARKS
We compare ValueDICE against Discriminator-Actor-CC (DAC) (Kostrikov et al. 2019 which is the ffate-of-the-we in sample-efficient adversarial imitation learning, as well as GAIL (Ho & Ermon 2016). We evaluate the algorithms on the standard MuJoCo environments using expert demonstrations from Ho & Ermon (2016). We plot the average returns for the learned policies
HalfCheetah
Hopper
Ant
Walkerzd
5000
4003
6:000
DAC
2003
Valc-DICC
4200
Expert
2000
1020
1003
2300
0-2
20k.
steps
steps
steps
Env. steps
Figure 3: Comparison of algorithms given 10 expert trajectories. ValueDICE outperforms other methods. However, given this amount of data, BC can recover the expert policy as well.
8

---

Published as a conference paper at ICLR 2020
(using mean action for sampling) every 1000 environment steps using 10 episodes. We perform this procedure for 10 different seeds and compute means and standard deviations (see Fig. and visualize a half of standard deviation these plots).
We present the extremely low-data regime first. In Figure we present the results of the imitation learning algorithms given only single expert trajectory. We find that ValueDICE performs similar or better than DAC an all tasks, with the exception of Walker2d where it converges to a slightly worse policy. Notably, in this low-data regime, behavioral cloning (BC) usually cannot recover the expert policy. We also present the results of these algorithms on a larger number of expert demonstrations (Figure 3). We continue observe strong performance ValueDICE well as faster convergence on all tasks. It is worth mentioning that in this large-data regime, Behavior Cloning can recover the expert performance as well. of these scenarios, GAlL, is too sanple-inefficiet to make any progress.
8 CONCLUSION
We introduced VIC,, an algorithm for imitation learning that outperforms the state-of-the-art standard MuJoCo tasks. In contrast to other algorithms for off-policy imitation learning, the al- gorithm introduced in this paper performs robust divergence minimization in principled off-policy manner and a strong theoretical framework. To the best of our knowledge, this also the first algorithm for adversarial imitation learning that omits learning or defining rewards explicitly and di- rectly learns a Q-function the distribution ratio objective directly. We demonstrate the robustness of DICE in challenging synthetic tabular MDP environment, as well as on standard MuJoCo continuous control benchmark environments, and we show increased performance over baselines in both the low and high data regimes.
REFERENCES
Pieter Abbeel and Andrew Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. ACM, 2004.
Marein Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pa- chocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. arXiv preprint 1tt:::8800000001 2018.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mine: mutual information neural estimation. arXiv preprint arXiv:1801.0406 2018
D. P. Bertsekas. Nonlinear Programming. Athena Scientific, Belmont, MA, second edition, 1999.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, al. End end learning for self-driving cars. arXiv preprint arXiv: 161.00316 2016.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1096, 2018.
Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of certain markov process expectations for large time. iv. Communications on Pure and Applied Mathematics, 36(2):183- 212. 1983.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse rein- forcement learning. arXiv preprint arXiv:1710.11248. 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems. pp. 2672-2680, 2014.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. Advances in neural information processing systems, pp- 5767-5777, 2017.

---

Published as a conference paper at ICLR 2020
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv:1812.05905. 2018.
Karol Hausman, Yevgen Chebotar, Stefan Schaal, Gaurav Sukhatme, and Joseph J Lim, Multi- modal imitation learning from unstructured demonstrations using generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 1235-1245, 2017.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in neural information processing systems, pp. 4565-4573, 2016.
Liyiming Ke, Matt Barnes, Wen Sun, Gilwoo Lee, Sanjiban Choudhury, and Siddhartha Srinivasa. Imitation learning as f-divergence minimization. arXiv preprint a90i: 12888, 2019.
Пlya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tomp- Addressing sample inefficiency and reward bias in adversar-
son. Discriminator-actor--oo ial imitation learning. In International Conference on Learning Representations, 2019. URL
https://epenreviiie net meet/forummiddeteee Ho
Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual demonstrations. In Advances in Neural Information Processing Systems, pp. 38 3812-3822, 2017.
Urs Muller, Jan Ben, Eric Cosatto, Beat Flepp, and Yann L Cun. Off-road obstacle avoidance through end-to-end learning. In Advances in neural information processing systems, pp. 739- 746, 2006.
Ofir Nachum, Michael Ahn, Hugo Ponte, Shixiang Gu, and Vikash Kumar. Multi-agent manipula- tion via locomotion using hierarchical sim2real. arXiv preprint arXiv:1908.05224 2019a.
Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Efficient estimation of off-policy stationary distribution corrections. 2019b.
Andrew Ng, Stuart Russell, et al. Algorithms for inverse reinforcement learning.
Jan Peters, Katharina Mulling. and Yasemin Altun. Relative entropy policy search. In Twenty-Fourhh AAAI Conference on Artificial Intelligence, 2010.
Dean A Pomerleau. Alvinn: An autonomous land vehicle a neural network. In Advances in neural information processing systems, pp. 305-313, 1989.
Martin L Puterman. Markov Decision Processes.: Discrete Stochastic Dynamic Programming. John
Wiley & Sons, 2014.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction no-regret online learning. In Proceedings of the fourteenth international con- ference on artificial intelligence and statisties, pp. 627-635, 2011.
Fumihiro Sasaki, Tetsuya Yohira, and Atsuo Kawaguchi. Sample efficient imitation learning for continuous control. International Conference on Learning Representations, 2019. URL
https://openreview net mett/Foruumid--BNNJoo.oo
10

---

Published as a conference paper at ICLR 2020
A IMPLEMENTATION DETAILS
All algorithms use networks with an MLP architecture with 2 hidden layers and 256 hidden units. For discriminators, critic, // we use Adam optimizer with learning rate 10-3 while for the actors we use the learning rate of 10 10-s. For the discriminator and y networks we use gradient penalties from Gulrajani al. (2017). We also regularize the actor network with the orthogonal regularization Brock et al. 2018) with coefficient 10 10- Also we perform 4 updates per environment step. We handle absorbing states of the environments similarly to Kostrikov et (2019).
B ALGORITHMS
In this section, we present pseudocode for the imitation learning algorithms based on DualDICE.
Algorithm ValueDICE
Input:
expert replay buffer Re
Initialize replay buffer R t 0 for 1....,dd do do Sample (s, 8')) with mo
Add (s, a, s') to the replay buffer
{(s(), (() ) R
Geometric sampling (0) is starting episode state for Geometric sampling, 69
((s E
M1=1 ~ Re
E
(1)
no (-100 (1) for i ..... B
a'(0) no (()) ) for i 1.....
(00 no
). for i = 1,....1. B
Compute loss on expert data:
Jog = log B
1 Σ 1=1 ((1 ( (0) --
(60 aa )a)) (2 (
Compute loss on the replay buffer:
Jtineer π >m ((1 - a1 7) (60
(1) (8)) + a(ps(s(0) a()) (1 ,(10)))
Update v nu (Jloo Stinear)
Update + 1 + 17m VolJoog Jiinear)
end for
C ADDITIONAL EXPERIMENTS
We also compared ValueDICE with behavioral cloning the offline regime, when we sample no additional transitions from the learning environment (see Figure 4. Even given only offline data, VaIueDICE outperforms behavioral cloning. For behavioral cloning we used the same regularization as for actor training in ValueDICE.
HalfCheetah
Happer
Walker2d
2010
4000
3000
3020
2010
2020
1000
1000
Yalus Expert
2500
Updates
Updates
Updates
Figure 4: ValueDICE outperforms behavioral cloning given I trajectory even without replay regu- larization.
11


Reviews:
[{"review_id": "SkxD8yNatH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an algorithm for adversarial imitation that uses off-policy data in a principled manner, unlike prior work. The core idea is to express the KL-divergence between the policy's state-action marginal and the expert's state action marginal using the Donsker-Varadhan representation and then applying the change of variable similar to DualDICE to avoid computing the marginal of the current policy, thus getting rid of the on-policy sampling requirement. The paper then shows how the auxiliary variable (critic) added to the optimization is a value function that maximizes the corresponding induced reward in AIL methods, thus unifying the objectives for policy optimization and reward learning. The authors then present practical considerations needed in getting this formulation to work, including sampling from a replay buffer, biased sampling for the exponentiated term and avoid the double-sampling issue. Finally, the paper presents some results, which show that valueDICE is comparable to most of the other imitation learning methods. \n\nI lean towards accepting this paper. The overall idea seems neat, however, Section 5, and the addition of a replay buffer distribution in the KL-divergence objective, though motivated enough seem to be somewhat not so principled. The experimental section is a bit weak, and I encourage the authors to strengthen this section. Also, in the case of HalfCheetah and Ant (Figure 2), valueDICE usually exhibits overfitting-like trends (the performance drops with more training), why does this happen? Can this be corrected? Overall the idea is neat, I would still say that the components very prominently exist in the literature (f-GANs, DualDICE, etc).   \n\nI would encourage the authors to add some more details experimentally and make the implementation available. For example, optimization of Bellman backup functions without target networks or delays can be unstable, solving saddle point problems can be unstable, etc. How should these factors be tuned? And overall, is the optimization of the ValueDICE objective easy?. DualDICE is known to be unstable (the DualDICE Github implementation mentions this), do similar problems arise with ValueDICE?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HklnbLThtr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The primary contribution of this paper is a principled algorithm for off-policy imitation learning. Generative Adversarial Imitation Learning (GAIL), proposed by Ho and Ermon in 2016, is an on-policy imitation learning method that (provably) minimizes the divergence between a target state-action distribution and the policy state-action distribution. Followups to this work (Kostrikov et al., 2019) show that the same algorithm can be applied to an off-policy setting (replacing the on-policy samples with samples from the replay buffer) and the method still works, but is no longer theoretically justified. I believe using importance ratios would make this approach justified as well, but Kostrikov et al. found that using importance ratios actually degrades the performance of their method (due the difficulty associated with estimating importance ratios). This paper attempts to bridge this gap: a method that is theoretically justified, and still works. \n\nThe paper takes most of its inspiration from the recently proposed DualDICE paper (Nachum et al, 2019), where the authors introduce a method for estimating discounted stationary distribution ratios (i.e. d^{\\pi}/d^{D}, where \\pi is some (known) policy, and D is a given dataset of experience (for example, a replay buffer)). The authors essentially apply the method proposed in DualDICE to estimate d^{\\pi}/d^{exp} instead, where d^{exp} is a dataset of expert trajectories. While it would be possible to simply use this term as a reward and then run reinforcement learning, the authors note that the specific form of estimating d^{\\pi}/d^{exp} allows them to instead directly a train a value function, which can then be used for updating a policy.\n\nThe authors also argue that their method reduces complexity since it does not require a separate RL optimization routine. However, I think having a separate RL optimization routine has its own advantages - it is relatively easy to implement GAIL on top of any existing RL algorithm, which makes it easy to take advantage of recent advances in RL. For the method proposed in this paper, it would not be straightforward to do so. \n\nThe authors also note that they need a number of practical modifications to their original ValueDICE objective in order to make things work (Section 5: Some practical considerations). Notably, the the original ValueDICE objective only needs access to expert samples and the initial state distribution, and does not need access to the the replay buffer samples (apart from the initial state ones). This would likely not work well in practice - similar to how behavior cloning often does worse than GAIL when learning from a small number of expert examples. In order to combat this, the authors incorporate replay buffer regularization \n\nThe authors provide experiments on  a simple synthetic \"ring MDP\", and on four continuous control tasks from OpenAI gym - HalfCheetah, Hopper, Ant and Walker2d. When compared to prior approaches (i.e. Kostrikov et al 2019) in the low-data regime (where behavior cloning fails), the proposed method does significantly better on one task (Ant), slightly worse on one task (walker), and about the same on two tasks (Hopper and HalfCheetah). I do notice the proposed method as being somewhat unstable though - the reward appears to be going down after reaching the max on two of the tasks - HalfCheetah and Ant. Overall, I don't thing experiments are thorough enough to demonstrate that the method is empirically better than competing approaches, but I believe that is not the main point of the paper. \n\nOverall, my recommendation is a weak accept. It is interesting that the authors were able to get a principled method for off-policy imitation learning working (large following from prior work in Nachum et al 2019), but I don't think the method currently offers any significant practical advantages over competing methods.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hkg8cdP6YB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides a novel off policy objective to solve imitation learning. It resolves the limitation of the famous GAIL algorithm that we need on-policy samples to interact with the environment. The new algorithm is simple but efficient, and can handle off-policy settings. The derivation of equation (12) is nice and intuitive, provide a potential on creating new imitation learning algorithm. Empirical results show that the new algorithm can perform as good as the state-of-the-art baseline, under on-policy setting.\n\nClarity:\nThe paper is well written an intuitive. It clearly introduces the previous works and their limitation, and naturally derives the new objective by DualDice trick to resolve the limitation. Section 5 discusses the bias introduce by the exponential of expectation, which in practice does not hurt the performance much. Experimental design is good and informative.\n\nMajor concern:\nIn experiment we only saw the result of on-policy setting using the replay buffer regularizer. However, the first half of the paper focuses on deriving an off-policy objective for imitation learning. A natural question is: how good is the performance if we only use off-policy data? In Figure 3 with enough expert trajectories, how does the off-policy ValueDice perform compared to behavior cloning?\n\nIn sum, I think the paper is clearly above the acceptance threshold. But I will leave it 6 point and raise my point if the authors can answer my question above.\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work addresses new insights in the imitation learning setting, and shows how a popular type of approach can be extended in a principled way to the off-policy learning setting. Several requests for clarification were addressed in the rebuttal phase, in particular regarding the empirical evaluation in off-policy settings. The authors improved the empirical validation and overall clarity of the paper. The resulting manuscript provides valuable new insights, in particular in its principled connections, and extension to previous work.
Decision: Accept (Poster)