Adversarial Imitation via Variational Inverse Reinforcement Learning

1 INTRODUCTION . Reinforcement learning ( RL ) has emerged as a promising tool for solving complex decision-making and control tasks from predefined high-level reward functions ( Sutton et al. , 1998 ) . However , defining an optimizable reward function that inculcates the desired behavior can be challenging for many robotic applications , which include learning social-interaction skills ( Qureshi et al. , 2018 ; 2017 ) , dexterous manipulation ( Finn et al. , 2016b ) , and autonomous driving ( Kuderer et al. , 2015 ) . Inverse reinforcement learning ( IRL ) ( Ng et al. , 2000 ) addresses the problem of learning reward functions from expert demonstrations , and it is often considered as a branch of imitation learning ( Argall et al. , 2009 ) . The prior work in IRL includes maximum-margin ( Abbeel & Ng , 2004 ; Ratliff et al. , 2006 ) and maximum-entropy ( Ziebart et al. , 2008 ) formulations . Currently , maximum entropy ( MaxEnt ) IRL is a widely used approach towards IRL , and has been extended to use non-linear function approximators such as neural networks in scenarios with unknown dynamics by leveraging sampling-based techniques ( Boularias et al. , 2011 ; Finn et al. , 2016b ; Kalakrishnan et al. , 2013 ) . However , designing the IRL algorithm is usually complicated as it requires , to some extent , hand engineering such as deciding domain-specific regularizers ( Finn et al. , 2016b ) . Rather than learning reward functions and solving the IRL problem , imitation learning ( IL ) learns a policy directly from expert demonstrations . Prior work addressed the IL problem through behavior cloning ( BC ) , which learns a policy from expert trajectories using supervised learning ( Pomerleau , 1991 ) . Although BC methods are simple solutions to IL , these methods require a large amount of data because of compounding errors induced by covariate shift ( Ross et al. , 2011 ) . To overcome BC limitations , a generative adversarial imitation learning ( GAIL ) algorithm ( Ho & Ermon , 2016 ) was proposed . GAIL uses the formulation of Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) , i.e. , a generator-discriminator framework , where a generator is trained to generate expert-like trajectories while a discriminator is trained to distinguish between generated and expert trajectories . Although GAIL is highly effective and efficient framework , it does not recover transferable/portable reward functions along with the policies , thus narrowing its use cases to similar problem instances in similar environments . Reward function learning is ultimately preferable , if possible , over direct imitation learning as rewards are portable functions that represent the most basic and complete representation of agent intention , and can be re-optimized in new environments and new agents . Reward learning is challenging as there can be many optimal policies explaining a set of demonstrations and many reward functions inducing an optimal policy ( Ng et al. , 2000 ; Ziebart et al. , 2008 ) . Recently , an adversarial inverse reinforcement learning ( AIRL ) framework ( Fu et al. , 2017 ) , an extension of GAIL , was proposed that offers a solution to the former issue by exploiting the maximum entropy IRL method ( Ziebart et al. , 2008 ) whereas the latter issue is addressed through learning disentangled reward functions by modeling the reward as a function of state only instead of both state and action . However , AIRL fails to recover the ground truth reward when the ground truth reward is a function of both state and action . For example , the reward function in any locomotion or ambulation tasks contains a penalty term that discourages actions with large magnitudes . This need for action regularization is well known in optimal control literature and limits the use cases of a state-only reward function in most practical real-life applications . A more generalizable and useful approach would be to formulate reward as a function of both states and actions , which induces action-driven reward shaping that has been shown to play a vital role in quickly recovering the optimal policies ( Ng et al. , 1999 ) . In this paper , we propose the empowerment-regularized adversarial inverse reinforcement learning ( EAIRL ) algorithm1 . Empowerment ( Salge et al. , 2014 ) is a mutual information-based theoretic measure , like state- or action-value functions , that assigns a value to a given state to quantify the extent to which an agent can influence its environment . Our method uses variational information maximization ( Mohamed & Rezende , 2015 ) to learn empowerment in parallel to learning the reward and policy from expert data . Empowerment acts as a regularizer to policy updates to prevent overfitting the expert demonstrations , which in practice leads to learning robust rewards . Our experimentation shows that the proposed method recovers not only near-optimal policies but also recovers robust , transferable , disentangled , state-action based reward functions that are near-optimal . The results on reward learning also show that EAIRL outperforms several state-of-the-art IRL methods by recovering reward functions that leads to optimal , expert-matching behaviors . On policy learning , results demonstrate that policies learned through EAIRL perform comparably to GAIL and AIRL with non-disentangled ( state-action ) reward function but significantly outperform policies learned through AIRL with disentangled reward ( state-only ) and GAN interpretation of Guided Cost Learning ( GAN-GCL ) ( Finn et al. , 2016a ) . 2 BACKGROUND . We consider a Markov decision process ( MDP ) represented as a tuple ( S , A , P , R , ρ0 , γ ) where S denotes the state-space , A denotes the action-space , P represents the transition probability distribution , i.e. , P : S × A × S → [ 0 , 1 ] , R ( s , a ) corresponds to the reward function , ρ0 is the initial state distribution ρ0 : S → R , and γ ∈ ( 0 , 1 ) is the discount factor . Let q ( a|s , s′ ) be an inverse model that maps current state s ∈ S and next state s′ ∈ S to a distribution over actions A , i.e. , q : S × S × A → [ 0 , 1 ] . Let π be a stochastic policy that takes a state and outputs a distribution over actions such that π : S × A → [ 0 , 1 ] . Let τ and τE denote a set of trajectories , a sequence of state-action pairs ( s0 , a0 , · · · sT , aT ) , generated by a policy π and an expert policy πE , respectively , where T denotes the terminal time . Finally , let Φ ( s ) be a potential function that quantifies a utility of a given state s ∈ S , i.e. , Φ : S → R. In our proposed work , we use an empowerment-based potential function Φ ( · ) to regularize policy update under MaxEnt-IRL framework . Therefore , the following 1Supplementary material is available at https : //sites.google.com/view/eairl sections provide a brief background on MaxEnt-IRL , adversarial reward and policy learning , and variational information-maximization approach to learn the empowerment . 2.1 MAXENT-IRL . MaxEnt-IRL ( Ziebart et al. , 2008 ) models expert demonstrations as Boltzmann distribution using parametrized reward rξ ( τ ) as an energy function , i.e. , pξ ( τ ) = 1 Z exp ( rξ ( τ ) ) ( 1 ) where rξ ( τ ) = ∑T t=0 rξ ( st , at ) is a commutative reward over given trajectory τ , parameterized by ξ , andZ is the partition function . In this framework , the demonstration trajectories are assumed to be sampled from an optimal policy π∗ , therefore , they get the highest likelihood whereas the suboptimal trajectories are less rewarding and hence , are generated with exponentially decaying probability . The main computational challenge in MaxEnt-IRL is to determine Z . The initial work in MaxEnt-IRL computed Z using dynamic programming ( Ziebart et al. , 2008 ) whereas modern approaches ( Finn et al. , 2016b ; a ; Fu et al. , 2017 ) present importance sampling technique to approximate Z under unknown dynamics . 2.2 ADVERSARIAL INVERSE REINFORCEMENT LEARNING . This section briefly describes Adversarial Inverse Reinforcement Learning ( AIRL ) ( Fu et al. , 2017 ) algorithm which forms a baseline of our proposed method . AIRL is the current state-of-the-art IRL method that builds on GAIL ( Ho & Ermon , 2016 ) , maximum entropy IRL framework ( Ziebart et al. , 2008 ) and GAN-GCL , a GAN interpretation of Guided Cost Learning ( Finn et al. , 2016b ; a ) . GAIL is a model-free adversarial learning framework , inspired from GANs ( Goodfellow et al. , 2014 ) , where the policy π learns to imitate the expert policy behavior πE by minimizing the JensenShannon divergence between the state-action distributions generated by π and the expert state-action distribution by πE through following objective min π max D∈ ( 0,1 ) S×A Eπ [ logD ( s , a ) ] + EπE [ log ( 1−D ( s , a ) ) ] − λH ( π ) ( 2 ) where D is the discriminator that performs the binary classification to distinguish between samples generated by π and πE , λ is a hyper-parameter , and H ( π ) is an entropy regularization term Eπ [ log π ] . Note that GAIL does not recover reward ; however , Finn et al . ( 2016a ) shows that the discriminator can be modeled as a reward function . Thus AIRL ( Fu et al. , 2017 ) presents a formal implementation of ( Finn et al. , 2016a ) and extends GAIL to recover reward along with the policy by imposing a following structure on the discriminator : Dξ , ϕ ( s , a , s ′ ) = exp [ fξ , ϕ ( s , a , s ′ ) ] exp [ fξ , ϕ ( s , a , s′ ) ] + π ( a|s ) ( 3 ) where fξ , ϕ ( s , a , s′ ) = rξ ( s ) + γhϕ ( s′ ) − hϕ ( s ) comprises a disentangled reward term rξ ( s ) with training parameters ξ , and a shaping term F = γhϕ ( s′ ) − hϕ ( s ) with training parameters ϕ . The entire Dξ , ϕ ( s , a , s′ ) is trained as a binary classifier to distinguish between expert demonstrations τE and policy generated demonstrations τ . The policy is trained to maximize the discriminative reward r̂ ( s , a , s′ ) = log ( D ( s , a , s′ ) − log ( 1 −D ( s , a , s′ ) ) ) . Note that the function F = γhϕ ( s′ ) − hϕ ( s ) consists of free-parameters as no structure is imposed on hϕ ( · ) , and as mentioned in ( Fu et al. , 2017 ) , the reward function rξ ( · ) and function F are tied upto a constant ( γ − 1 ) c , where c ∈ R ; thus the impact of F , the shaping term , on the recovered reward r is quite limited and therefore , the benefits of reward shaping are not fully realized .

Summary: The paper proposes a method for inverse reinforcement learning based on AIRL. It's main contribution is that the shaping function is not learned while training the discriminator, but separately as an approximation of the empowerment (maximum mutual information). This shaping term aims to learn disentangled rewards without being restricted to learning state-only reward functions, which is a major restriction of AIRL.