Imitation Learning via Off-Policy Distribution Matching

1 INTRODUCTION . Reinforcement learning ( RL ) is typically framed as learning a behavior policy based on reward feedback from trial-and-error experience . Accordingly , many successful demonstrations of RL often rely on carefully handcrafted rewards with various bonuses and penalties designed to encourage intended behavior ( Nachum et al. , 2019a ; Andrychowicz et al. , 2018 ) . In contrast , many real-world behaviors are easier to demonstrate rather than devise explicit rewards . This realization is at the heart of imitation learning ( Ho & Ermon , 2016 ; Ng et al . ; Pomerleau , 1989 ) , in which one aims to learn a behavior policy from a set of expert demonstrations – logged experience data of a near-optimal policy interacting with the environment – without explicit knowledge of rewards . Distribution matching via adversarial learning , or Adversarial Imitation Learning ( AIL ) , has recently become a popular approach for imitation learning ( Ho & Ermon , 2016 ; Fu et al. , 2017 ; Ke et al. , 2019 ; Kostrikov et al. , 2019 ) . These methods interpret the states and actions provided in the expert demonstrations as a finite sample from a target distribution . Imitation learning can then be framed as learning a behavior policy which minimizes a divergence between this target distribution and the state-action distribution induced by the behavior policy interacting with the environment . As derived by Ho & Ermon ( 2016 ) , this divergence minimization may be achieved by iteratively performing two alternating steps , reminiscent of GAN algorithms ( Goodfellow et al. , 2014 ) . First , one estimates the density ratio of states and actions between the target distribution and the behavior policy . Then , these density ratios are used as rewards for a standard RL algorithm , and the behavior policy is updated to maximize these cumulative rewards ( data distribution ratios ) . The main limitation of current distribution matching approaches is that estimating distribution density ratios ( the first step of every iteration ) typically requires samples from the behavior policy distribution . This means that every iteration – every update to the behavior policy – requires new interactions with the environment , precluding the use of these algorithms in settings where interactions with the environment are expensive and limited . Several papers attempt to relax this on-policy ∗Also at NYU . 1Code to reproduce our results is available at https : //github.com/google-research/ google-research/tree/master/value_dice . requirement and resolve the sample inefficiency problem by designing off-policy imitation learning algorithms , which may take advantage of past logged data , usually in the form of a replay buffer ( Kostrikov et al. , 2019 ; Sasaki et al. , 2019 ) . However , these methods do so by altering the original divergence minimization objective to measure a divergence between the target expert distribution and the replay buffer distribution . Accordingly , there is no guarantee that the learned policy will recover the desired target distribution . In this work , we introduce an algorithm for imitation learning that , on the one hand , performs divergence minimization as in the original AIL methods , while on the other hand , is completely off-policy . We begin by providing a new formulation of the minimum divergence objective that avoids the use of any explicit on-policy expectations . While this objective may be used in the traditional way to estimate data distribution ratios that are then input to an RL algorithm , we go further to show how the specific form of the derived objective renders the use of a separate RL optimization unnecessary . Rather , gradients of the minimum divergence objective with respect to behavior policy may be computed directly . This way , an imitating behavior policy may be learned to minimize the divergence without the use of explicit rewards . We call this stream-lined imitation learning algorithm ValueDICE . In addition to being simpler than standard imitation learning methods , we show that our proposed algorithm is able to achieve state-of-the-art performance on a suite of imitation learning benchmarks . 2 BACKGROUND . We consider environments represented as a Markov Decision Process ( MDP ) ( Puterman , 2014 ) , defined by the tuple , ( S , A , p0 ( s ) , p ( s′|s , a ) , r ( s , a ) , γ ) , where S and A are the state and action space , respectively , p0 ( s ) is an initial state distribution , p ( s′|s , a ) defines environment dynamics represented as a conditional state distribution , r ( s , a ) is a reward function , and γ is a return discount factor . A behavior policy π ( ·|· ) interacts with the environment to yield experience ( st , at , rt , st+1 ) , for t = 0 , 1 , . . . , where s0 ∼ p0 ( · ) , at ∼ π ( ·|st ) , st+1 ∼ p ( ·|st , at ) , rt = r ( st , at ) . Without loss of generality , we consider infinite-horizon , non-terminating environments . In standard RL , one aims to learn a behavior policy π ( ·|s ) to maximize cumulative rewards , based on experience gained from interacting with the environment . In imitation learning ( Pomerleau , 1989 ; Abbeel & Ng , 2004 ; Ho & Ermon , 2016 ) , the environment reward is not observed . Rather , one has access to a set of expert demonstrations D : = { ( sk , ak , s′k } Nk=1 given by state-action-next-state transitions in the environment induced by an unknown expert policy πexp and the goal is to learn a behavior policy π which recovers πexp . During the learning process , in addition to the finite set of expert demonstrationsD , one may also optionally interact with the environment ( in these interactions , no rewards are observed ) . This setting describes a number of real-world applications where rewards are unknown , such as Pomerleau ( 1989 ) ; Muller et al . ( 2006 ) ; Bojarski et al . ( 2016 ) . 2.1 BEHAVIORAL CLONING ( BC ) . Supervised behavioral cloning ( BC ) is a popular approach for imitation learning . Given a set of expert demonstrations , a mapping of state observations to actions is fit using regression or density estimation . In the simplest case , one simply trains the behavior policy π to minimize the negative log-likelihood of the observed expert actions : min π JBC ( π ) : = − 1 N N∑ k=1 log π ( ak|sk ) . ( 1 ) Unlike Inverse Reinforcement Learning ( IRL ) algorithms ( e.g . GAIL ( Ho & Ermon , 2016 ) ) , BC does not perform any additional policy interactions with the learning environment and hence does not suffer from the same issue of policy sample complexity . However , behavioral cloning suffers from distributional drift ( Ross et al. , 2011 ) ; i.e. , there is no way for π to learn how to recover if it deviates from the expert behavior to a state s̃ not seen in the expert demonstrations . 2.2 DISTRIBUTION MATCHING . The distribution matching approach provides a family of methods that are robust to distributional shift . Rather than considering the policy directly as a conditional distribution π ( ·|s ) over actions , this approach considers the state-action distribution induced by a policy . In particular , under certain conditions ( Puterman , 2014 ) , there is a one-to-one correspondence between a policy and its stateaction distribution dπ defined as , dπ ( s , a ) = ( 1− γ ) · ∞∑ t=0 γtp ( st = s , at = a|s0 ∼ p0 ( · ) , st ∼ p ( ·|st−1 , at−1 ) , at ∼ π ( ·|st ) ) . ( 2 ) By the same token , the unknown expert policy πexp also possesses a state-action distribution dexp , and one may usually assume that the expert demonstrations D : = { ( sk , ak , s′k } Nk=1 are sampled as ( sk , ak ) ∼ dexp , s′k ∼ p ( ·|sk , ak ) . Accordingly , the distribution matching approach proposes to learn π to minimize the divergence between dπ and dexp . The KL-divergence is typically used to measure the discrepancy between dπ and dexp ( Ho & Ermon , 2016 ; Ke et al. , 2019 ) : −DKL ( dπ||dexp ) = E ( s , a ) ∼dπ [ log dexp ( s , a ) dπ ( s , a ) ] . ( 3 ) The use of the KL-divergence is convenient , as it may be expressed as an RL problem where rewards are given by log distribution ratios : −DKL ( dπ||dexp ) = ( 1− γ ) · E s0∼p0 ( · ) , at∼π ( ·|st ) st+1∼p ( ·|st , at ) [ ∞∑ t=0 γt log dexp ( st , at ) dπ ( st , at ) ] . ( 4 ) In other words , if one has access to estimates of the distribution ratios of the two policies , then the minimum divergence problem reduces to a max-return RL problem with rewards r̃ ( s , a ) = log d exp ( s , a ) dπ ( s , a ) . Any on-policy or off-policy RL algorithm can be used to maximize the corresponding expected returns in Equation 4 . Capitalizing on this observation , Ho & Ermon ( 2016 ) and Ke et al . ( 2019 ) propose algorithms ( e.g. , GAIL ) in which the distribution ratio is estimated using a GAN-like objective : max h : S×A→ ( 0,1 ) JGAIL ( h ) : = E ( s , a ) ∼dexp [ log h ( s , a ) ] + E ( s , a ) ∼dπ [ log ( 1− h ( s , a ) ) ] . ( 5 ) In this objective , the function h acts as a discriminator , discriminating between samples ( s , a ) from dexp and dπ . The optimal discriminator satisfies , log h∗ ( s , a ) − log ( 1− h∗ ( s , a ) ) = log d exp ( s , a ) dπ ( s , a ) , ( 6 ) and so the distribution matching rewards may be computed as r̃ ( s , a ) = log h∗ ( s , a ) − log ( 1 − h∗ ( s , a ) ) . In practice , the discriminator is not fully optimized , and instead gradient updates to the discriminator and policy are alternated . These prior distribution matching approaches possess two limitations which we will resolve with our proposed ValueDICE algorithm : • On-policy . Arguably the main limitation of these prior approaches is that they require access to on-policy samples from dπ . While off-policy RL can be used for learning π , optimizing the discriminator h necessitates having on-policy samples ( the second expectation in Equation 5 ) . Thus , in practice , GAIL requires a prohibitively large number of environment interactions , making it unfeasible for use in many real-world applications . Attempts to remedy this , such as DiscriminatorActor-Critic ( DAC ) ( Kostrikov et al. , 2019 ) , often do so via ad-hoc methods ; for example , changing the on-policy expectation E ( s , a ) ∼dπ [ log ( 1−h ( s , a ) ) ] in Equation 5 to an expectation over the replay buffer E ( s , a ) ∼dRB [ log ( 1 − h ( s , a ) ) ] . While DAC achieves good empirical results , it does not guarantee distribution matching of π to πexp , especially when dRB is far from dπ . • Separate RL optimization . Prior approaches require iteratively taking alternating steps : first estimate the data distribution ratios using the GAN-like objective , then input these into an RL optimization and repeat . The use of a separate RL algorithm introduces complexity to any implementation of these approaches , with many additional design choices that need to be made and more function approximators to learn ( e.g. , value functions ) . Our introduced ValueDICE will be shown to not need a separate RL optimization .

Summary: This paper provides a novel off policy objective to solve imitation learning. It resolves the limitation of the famous GAIL algorithm that we need on-policy samples to interact with the environment. The new algorithm is simple but efficient, and can handle off-policy settings. The derivation of equation (12) is nice and intuitive, provide a potential on creating new imitation learning algorithm. Empirical results show that the new algorithm can perform as good as the state-of-the-art baseline, under on-policy setting.