Support-guided Adversarial Imitation Learning

1 INTRODUCTION . The class of Adversarial Imitation Learning ( AIL ) algorithms learns robust policies that imitate an expert ’ s actions from a small number of expert trajectories , without further access to the expert or environment signals . AIL iterates between refining a reward via adversarial training , and reinforcement learning ( RL ) with the learned adversarial reward . For instance , Generative Adversarial Imitation Learning ( GAIL ) ( Ho & Ermon , 2016 ) shows the equivalence between some settings of inverse reinforcement learning and Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) , and recasts imitation learning as distribution matching between the expert and the RL agent . Similarly , Adversarial Inverse Reinforcement Learning ( AIRL ) ( Fu et al. , 2017 ) modifies the GAIL discriminator to learn a reward function robust to changes in dynamics or environment properties . AIL mitigates the issue of distributional drift from behavioral cloning ( Ross et al. , 2011 ) , a classical imitation learning algorithm , and demonstrates good performance with only a small number of expert demonstrations . However , AIL has several important challenges , including implicit reward bias ( Kostrikov et al. , 2019 ) , potential training instability ( Salimans et al. , 2016 ; Brock et al. , 2018 ) , and potential sample inefficiency with respect to environment interaction ( Sasaki et al. , 2019 ) . In this paper , we propose a principled approach towards addressing these issues . Wang et al . ( 2019 ) demonstrated that imitation learning is also feasible by constructing a fixed reward function via estimating the support of the expert policy . Since support estimation only requires expert demonstrations , the method sidesteps the training instability associated with adversarial training . However , we show in Section 4.2 that the reward learned via support estimation deteriorates when expert data is sparse , and leads to poor policy performances . Support estimation and adversarial reward represent two different yet complementary RL signals for imitation learning , both learnable from expert demonstrations . We unify both signals into Supportguided Adversarial Imitation Learning ( SAIL ) , a generic imitation learning framework . SAIL leverages the adversarial reward to guide policy exploration and constrains the policy search to the estimated support of the expert policy . It is compatible with existing AIL algorithms , such as GAIL and AIRL . We also show that SAIL is at least as efficient as standard AIL . In an extensive evaluation , we demonstrate that SAIL mitigates the implicit reward bias and achieves better performance and training stability against baseline methods over a series of benchmark control tasks . 2 BACKGROUND . We briefly review the Markov Decision Process ( MDP ) , the context of our imitation learning task , followed by related works on imitation learning . Markov Decision Process We consider an infinite-horizon discounted MDP ( S , A , P , r , p0 , γ ) , where S is the set of states , A the set of actions , P : S × A × S → [ 0 , 1 ] the transition probability , r : S × A → R the reward function , p0 : S → [ 0 , 1 ] the distribution over initial states , and γ ∈ ( 0 , 1 ) the discount factor . Let π be a stochastic policy π : S × A → [ 0 , 1 ] with expected discounted reward Eπ ( r ( s , a ) ) , E ( ∑∞ t=0 γ tr ( st , at ) ) where s0 ∼ p0 , at ∼ π ( ·|st ) , and st+1 ∼ P ( ·|st , at ) for t ≥ 0 . We denote πE the expert policy . Behavioral Cloning ( BC ) learns a policy π : S → A directly from expert trajectories via supervised learning . BC is simple to implement , and effective when expert data is abundant . However , BC is prone to distributional drift : the state distribution of expert demonstrations deviates from that of the agent policy , due to accumulation of small mistakes during policy execution . Distributional drift may lead to catastrophic errors ( Ross et al. , 2011 ) . While several methods address the issue ( Ross & Bagnell , 2010 ; Sun et al. , 2017 ) , they often assume further access to the expert during training . Inverse Reinforcement Learning ( IRL ) first estimates a reward from expert demonstrations , followed by RL using the estimated reward ( Ng & Russell , 2000 ; Abbeel & Ng , 2004 ) . Building upon a maximum entropy formulation of IRL ( Ziebart et al. , 2008 ) , Finn et al . ( 2016 ) and Fu et al . ( 2017 ) explore adversarial IRL and its connection to Generative Adversarial Imitation Learning ( Ho & Ermon , 2016 ) . Imitation Learning via Distribution Matching Generative Adversarial Imitation Learning ( GAIL ) ( Ho & Ermon , 2016 ) frames imitation learning as distribution matching between the expert and the RL agent . The authors show the connection between IRL and GANs . Specifically , GAIL imitates the expert by formulating a minimax game : min π max D∈ ( 0,1 ) Eπ ( logD ( s , a ) ) + EπE ( log ( 1−D ( s , a ) ) ) , ( 1 ) where the expectations Eπ and EπE denote the joint distributions over state-actions of the RL agent and the expert , respectively . GAIL is able to achieve expert performance with a small number of expert trajectories on various benchmark tasks . However , GAIL is relatively sample inefficient with respect to environment interaction , and inherits issues associated with adversarial learning , such as vanishing gradients , training instability and overfitting to expert demonstrations ( Arjovsky & Bottou , 2017 ; Brock et al. , 2018 ) . Recent works have improved the sample efficiency and stability of GAIL . For instance , Generative Moment Matching Imitation Learning ( Kim & Park , 2018 ) replaces the adversarial reward with a non-parametric maximum mean discrepancy estimator to sidestep adversarial learning . Baram et al . ( 2017 ) improve sample efficiency with a model-based RL algorithm . Kostrikov et al . ( 2019 ) and Sasaki et al . ( 2019 ) demonstrate significant gain in sample efficiency with off-policy RL algorithms . In addition , Generative Predecessor Models for Imitation Learning ( Schroecker et al. , 2019 ) imitates the expert policy using generative models to reason about alternative histories of demonstrated states . Our proposed method is closely related to the broad family of AIL algorithms including GAIL and adversarial IRL . It is also complementary to many techniques for improving the algorithmic efficiency and stability , as discussed above . In particular , we focus on improving the quality of the learned reward by constraining adversarial reward to the estimated support of the expert policy . Imitation Learning via Support Estimation Alternative to AIL , Wang et al . ( 2019 ) demonstrate the feasibility of using a fixed RL reward via estimating the support of the expert policy from expert demonstrations . Connecting kernel-based support estimation ( De Vito et al. , 2014 ) to Random Network Distillation ( Burda et al. , 2018 ) , the authors propose Random Expert Distillation ( RED ) to learn a reward function based on support estimation . Specifically , RED learns the reward parameter θ̂ by minimizing : min θ̂ Es , a∼πE ||fθ̂ ( s , a ) − fθ ( s , a ) || 2 2 , ( 2 ) where fθ : S ×A→ RK projects ( s , a ) from expert demonstrations to some embedding of size K , with randomly initialized θ . The reward is then defined as : rred ( s , a ) = exp ( −σ||fθ̂ ( s , a ) − fθ ( s , a ) || 2 2 ) , ( 3 ) where σ is a hyperparameter . As optimizing Eq . ( 2 ) only requires expert data , RED sidesteps adversarial learning , and casts imitation learning as a standard RL task using the learned reward . While RED works well given sufficient expert data , we show in the experiments that its performance suffers in the more challenging setting of sparse expert data . 3 METHOD . Formally , we consider the task of learning a reward function r̂ ( s , a ) from a finite set of trajectories { τi } Ni=1 , sampled from the expert policy πE within a MDP . Each trajectory is a sequence of stateaction tuples in the form of τi = { s1 , a1 , s2 , a2 , ... , sT , aT } . Assuming that the expert trajectories are consistent with some latent reward function r∗ ( s , a ) , we aim to learn a policy that achieves good performance with respect to r∗ ( s , a ) by applying RL on the learned reward function r̂ ( s , a ) . In this section , we first discuss the advantages and shortcomings of AIL to motivate our method . We then introduce Support-guided Adversarial Learning ( SAIL ) , and present a theoretical analysis that compares SAIL with the existing methods , specifically GAIL . 3.1 ADVERSARIAL IMITATION LEARNING . A clear advantage of AIL resides in its low sample complexity with respect to expert data . For instance , GAIL requires as little as 200 state-action tuples from the expert to achieve imitation . The reason is that the adversarial reward may be interpreted as an effective exploration mechanism for the RL agent . To see this , consider the learned reward function under the optimality assumption . With the optimal discriminator to Eq . ( 1 ) D∗ ( s , a ) = pπ ( s , a ) pπE ( s , a ) +pπ ( s , a ) , a common reward for GAIL is rgail ( s , a ) = − log ( D∗ ( s , a ) ) = log ( 1 + pπE ( s , a ) pπ ( s , a ) ) = log ( 1 + φ ( s , a ) ) . ( 4 ) Eq . ( 4 ) shows that the adversarial reward only depends on the ratio φ ( s , a ) = pπE ( s , a ) pπ ( s , a ) . Intuitively , rgail incentivizes the RL agent towards under-visited state-actions , where φ ( s , a ) > 1 , and away from over-visited state-actions , where φ ( s , a ) < 1 . When πE and π match exactly , rgail converges to an indicator function for the support of πE , since φ ( s , a ) = 1 ∀ ( s , a ) ∈ supp ( πE ) ( Goodfellow et al. , 2014 ) . In practice , the adversarial reward is unlikely to converge , as pπE is estimated from a finite set of expert demonstrations . Instead , the adversarial reward continuously drives the agent to explore by evolving the reward landscape . However , AIL also presents several challenges . Kostrikov et al . ( 2019 ) demonstrated that the reward − logD ( s , a ) suffers from an implicit survival bias , as the non-negative reward may lead to suboptimal behaviors in goal-oriented tasks where the agent learns to move around the goal to accumulate rewards , instead of completing the tasks . While the authors resolve the issue by introducing absorbing states , the solution assumes extra RL signals from the environment , including access to the time limit of an environment to detect early termination of training episodes . In Section 4.1 , we empirically demonstrate the survival bias on Lunar Lander , a common RL benchmark , by showing that agents trained with GAIL often hover over the goal location1 . We also show that our proposed method is able to robustly imitate the expert . Another challenge with AIL is potential training instability . Wang et al . ( 2019 ) demonstrated empirically that the adversarial reward could be unreliable in regions where the expert data is sparse , causing the agent to diverge from the intended behavior . When the agent policy is substantially different from the expert policy , the discriminator could differentiate them with high confidence , resulting in very low rewards and significant slow down in training , similar to the vanishing gradient problem in GAN training ( Arjovsky & Bottou , 2017 ) .

Summary: This paper proposes an approach for improving adversarial imitation learning, by combining it with support-estimation-based imitation learning. In particular, the paper explores a combination of GAIL (Ho and Ermon, 2016) and RED (Wang et. al., 2019), where the reward for the policy-gradient is a product of the rewards obtained from them separately. The motivation is that, while AIL methods are sample-efficient (in terms of expert data) and implicitly promote useful exploration, they could be unreliable outside the support of the expert policy. Therefore, augmenting them by constraining the imitator to the support of the expert policy (with a method such as RED) could result in an overall better imitation learning algorithm.