Imitation Learning via Off-Policy Distribution Matching

1 INTRODUCTION . Reinforcement learning ( RL ) is typically framed as learning a behavior policy based on reward feedback from trial-and-error experience . Accordingly , many successful demonstrations of RL often rely on carefully handcrafted rewards with various bonuses and penalties designed to encourage intended behavior ( Nachum et al. , 2019a ; Andrychowicz et al. , 2018 ) . In contrast , many real-world behaviors are easier to demonstrate rather than devise explicit rewards . This realization is at the heart of imitation learning ( Ho & Ermon , 2016 ; Ng et al . ; Pomerleau , 1989 ) , in which one aims to learn a behavior policy from a set of expert demonstrations – logged experience data of a near-optimal policy interacting with the environment – without explicit knowledge of rewards . Distribution matching via adversarial learning , or Adversarial Imitation Learning ( AIL ) , has recently become a popular approach for imitation learning ( Ho & Ermon , 2016 ; Fu et al. , 2017 ; Ke et al. , 2019 ; Kostrikov et al. , 2019 ) . These methods interpret the states and actions provided in the expert demonstrations as a finite sample from a target distribution . Imitation learning can then be framed as learning a behavior policy which minimizes a divergence between this target distribution and the state-action distribution induced by the behavior policy interacting with the environment . As derived by Ho & Ermon ( 2016 ) , this divergence minimization may be achieved by iteratively performing two alternating steps , reminiscent of GAN algorithms ( Goodfellow et al. , 2014 ) . First , one estimates the density ratio of states and actions between the target distribution and the behavior policy . Then , these density ratios are used as rewards for a standard RL algorithm , and the behavior policy is updated to maximize these cumulative rewards ( data distribution ratios ) . The main limitation of current distribution matching approaches is that estimating distribution density ratios ( the first step of every iteration ) typically requires samples from the behavior policy distribution . This means that every iteration – every update to the behavior policy – requires new interactions with the environment , precluding the use of these algorithms in settings where interactions with the environment are expensive and limited . Several papers attempt to relax this on-policy ∗Also at NYU . 1Code to reproduce our results is available at https : //github.com/google-research/ google-research/tree/master/value_dice . requirement and resolve the sample inefficiency problem by designing off-policy imitation learning algorithms , which may take advantage of past logged data , usually in the form of a replay buffer ( Kostrikov et al. , 2019 ; Sasaki et al. , 2019 ) . However , these methods do so by altering the original divergence minimization objective to measure a divergence between the target expert distribution and the replay buffer distribution . Accordingly , there is no guarantee that the learned policy will recover the desired target distribution . In this work , we introduce an algorithm for imitation learning that , on the one hand , performs divergence minimization as in the original AIL methods , while on the other hand , is completely off-policy . We begin by providing a new formulation of the minimum divergence objective that avoids the use of any explicit on-policy expectations . While this objective may be used in the traditional way to estimate data distribution ratios that are then input to an RL algorithm , we go further to show how the specific form of the derived objective renders the use of a separate RL optimization unnecessary . Rather , gradients of the minimum divergence objective with respect to behavior policy may be computed directly . This way , an imitating behavior policy may be learned to minimize the divergence without the use of explicit rewards . We call this stream-lined imitation learning algorithm ValueDICE . In addition to being simpler than standard imitation learning methods , we show that our proposed algorithm is able to achieve state-of-the-art performance on a suite of imitation learning benchmarks . 2 BACKGROUND . We consider environments represented as a Markov Decision Process ( MDP ) ( Puterman , 2014 ) , defined by the tuple , ( S , A , p0 ( s ) , p ( s′|s , a ) , r ( s , a ) , γ ) , where S and A are the state and action space , respectively , p0 ( s ) is an initial state distribution , p ( s′|s , a ) defines environment dynamics represented as a conditional state distribution , r ( s , a ) is a reward function , and γ is a return discount factor . A behavior policy π ( ·|· ) interacts with the environment to yield experience ( st , at , rt , st+1 ) , for t = 0 , 1 , . . . , where s0 ∼ p0 ( · ) , at ∼ π ( ·|st ) , st+1 ∼ p ( ·|st , at ) , rt = r ( st , at ) . Without loss of generality , we consider infinite-horizon , non-terminating environments . In standard RL , one aims to learn a behavior policy π ( ·|s ) to maximize cumulative rewards , based on experience gained from interacting with the environment . In imitation learning ( Pomerleau , 1989 ; Abbeel & Ng , 2004 ; Ho & Ermon , 2016 ) , the environment reward is not observed . Rather , one has access to a set of expert demonstrations D : = { ( sk , ak , s′k } Nk=1 given by state-action-next-state transitions in the environment induced by an unknown expert policy πexp and the goal is to learn a behavior policy π which recovers πexp . During the learning process , in addition to the finite set of expert demonstrationsD , one may also optionally interact with the environment ( in these interactions , no rewards are observed ) . This setting describes a number of real-world applications where rewards are unknown , such as Pomerleau ( 1989 ) ; Muller et al . ( 2006 ) ; Bojarski et al . ( 2016 ) . 2.1 BEHAVIORAL CLONING ( BC ) . Supervised behavioral cloning ( BC ) is a popular approach for imitation learning . Given a set of expert demonstrations , a mapping of state observations to actions is fit using regression or density estimation . In the simplest case , one simply trains the behavior policy π to minimize the negative log-likelihood of the observed expert actions : min π JBC ( π ) : = − 1 N N∑ k=1 log π ( ak|sk ) . ( 1 ) Unlike Inverse Reinforcement Learning ( IRL ) algorithms ( e.g . GAIL ( Ho & Ermon , 2016 ) ) , BC does not perform any additional policy interactions with the learning environment and hence does not suffer from the same issue of policy sample complexity . However , behavioral cloning suffers from distributional drift ( Ross et al. , 2011 ) ; i.e. , there is no way for π to learn how to recover if it deviates from the expert behavior to a state s̃ not seen in the expert demonstrations . 2.2 DISTRIBUTION MATCHING . The distribution matching approach provides a family of methods that are robust to distributional shift . Rather than considering the policy directly as a conditional distribution π ( ·|s ) over actions , this approach considers the state-action distribution induced by a policy . In particular , under certain conditions ( Puterman , 2014 ) , there is a one-to-one correspondence between a policy and its stateaction distribution dπ defined as , dπ ( s , a ) = ( 1− γ ) · ∞∑ t=0 γtp ( st = s , at = a|s0 ∼ p0 ( · ) , st ∼ p ( ·|st−1 , at−1 ) , at ∼ π ( ·|st ) ) . ( 2 ) By the same token , the unknown expert policy πexp also possesses a state-action distribution dexp , and one may usually assume that the expert demonstrations D : = { ( sk , ak , s′k } Nk=1 are sampled as ( sk , ak ) ∼ dexp , s′k ∼ p ( ·|sk , ak ) . Accordingly , the distribution matching approach proposes to learn π to minimize the divergence between dπ and dexp . The KL-divergence is typically used to measure the discrepancy between dπ and dexp ( Ho & Ermon , 2016 ; Ke et al. , 2019 ) : −DKL ( dπ||dexp ) = E ( s , a ) ∼dπ [ log dexp ( s , a ) dπ ( s , a ) ] . ( 3 ) The use of the KL-divergence is convenient , as it may be expressed as an RL problem where rewards are given by log distribution ratios : −DKL ( dπ||dexp ) = ( 1− γ ) · E s0∼p0 ( · ) , at∼π ( ·|st ) st+1∼p ( ·|st , at ) [ ∞∑ t=0 γt log dexp ( st , at ) dπ ( st , at ) ] . ( 4 ) In other words , if one has access to estimates of the distribution ratios of the two policies , then the minimum divergence problem reduces to a max-return RL problem with rewards r̃ ( s , a ) = log d exp ( s , a ) dπ ( s , a ) . Any on-policy or off-policy RL algorithm can be used to maximize the corresponding expected returns in Equation 4 . Capitalizing on this observation , Ho & Ermon ( 2016 ) and Ke et al . ( 2019 ) propose algorithms ( e.g. , GAIL ) in which the distribution ratio is estimated using a GAN-like objective : max h : S×A→ ( 0,1 ) JGAIL ( h ) : = E ( s , a ) ∼dexp [ log h ( s , a ) ] + E ( s , a ) ∼dπ [ log ( 1− h ( s , a ) ) ] . ( 5 ) In this objective , the function h acts as a discriminator , discriminating between samples ( s , a ) from dexp and dπ . The optimal discriminator satisfies , log h∗ ( s , a ) − log ( 1− h∗ ( s , a ) ) = log d exp ( s , a ) dπ ( s , a ) , ( 6 ) and so the distribution matching rewards may be computed as r̃ ( s , a ) = log h∗ ( s , a ) − log ( 1 − h∗ ( s , a ) ) . In practice , the discriminator is not fully optimized , and instead gradient updates to the discriminator and policy are alternated . These prior distribution matching approaches possess two limitations which we will resolve with our proposed ValueDICE algorithm : • On-policy . Arguably the main limitation of these prior approaches is that they require access to on-policy samples from dπ . While off-policy RL can be used for learning π , optimizing the discriminator h necessitates having on-policy samples ( the second expectation in Equation 5 ) . Thus , in practice , GAIL requires a prohibitively large number of environment interactions , making it unfeasible for use in many real-world applications . Attempts to remedy this , such as DiscriminatorActor-Critic ( DAC ) ( Kostrikov et al. , 2019 ) , often do so via ad-hoc methods ; for example , changing the on-policy expectation E ( s , a ) ∼dπ [ log ( 1−h ( s , a ) ) ] in Equation 5 to an expectation over the replay buffer E ( s , a ) ∼dRB [ log ( 1 − h ( s , a ) ) ] . While DAC achieves good empirical results , it does not guarantee distribution matching of π to πexp , especially when dRB is far from dπ . • Separate RL optimization . Prior approaches require iteratively taking alternating steps : first estimate the data distribution ratios using the GAN-like objective , then input these into an RL optimization and repeat . The use of a separate RL algorithm introduces complexity to any implementation of these approaches , with many additional design choices that need to be made and more function approximators to learn ( e.g. , value functions ) . Our introduced ValueDICE will be shown to not need a separate RL optimization .

Summary: This paper presents an algorithm for adversarial imitation that uses off-policy data in a principled manner, unlike prior work. The core idea is to express the KL-divergence between the policy's state-action marginal and the expert's state action marginal using the Donsker-Varadhan representation and then applying the change of variable similar to DualDICE to avoid computing the marginal of the current policy, thus getting rid of the on-policy sampling requirement. The paper then shows how the auxiliary variable (critic) added to the optimization is a value function that maximizes the corresponding induced reward in AIL methods, thus unifying the objectives for policy optimization and reward learning. The authors then present practical considerations needed in getting this formulation to work, including sampling from a replay buffer, biased sampling for the exponentiated term and avoid the double-sampling issue. Finally, the paper presents some results, which show that valueDICE is comparable to most of the other imitation learning methods.