Regularized Inverse Reinforcement Learning

1 INTRODUCTION . Reinforcement learning ( RL ) has been successfully applied to many challenging domains including games ( Mnih et al. , 2015 ; 2016 ) and robot control ( Schulman et al. , 2015 ; Fujimoto et al. , 2018 ; Haarnoja et al. , 2018 ) . Advanced RL methods often employ policy regularization motivated by , e.g. , boosting exploration ( Haarnoja et al. , 2018 ) or safe policy improvement ( Schulman et al. , 2015 ) . While Shannon entropy is often used as a policy regularizer ( Ziebart et al. , 2008 ) , Geist et al . ( 2019 ) recently proposed a theoretical foundation of regularized Markov decision processes ( MDPs ) —a framework that uses strongly convex functions as policy regularizers . Here , one crucial advantage is that an optimal policy is shown to uniquely exist , whereas multiple optimal policies may exist in the absence of policy regularization . Meanwhile , since RL requires a given or known reward function ( which can often involve non-trivial reward engineering ) , Inverse Reinforcement Learning ( IRL ) ( Russell , 1998 ; Ng et al. , 2000 ) —the problem of acquiring a reward function that promotes expert-like behavior—is more generally adopted in practical scenarios like robotic manipulation ( Finn et al. , 2016b ) , autonomous driving ( Sharifzadeh et al. , 2016 ; Wu et al. , 2020 ) and clinical motion analysis ( Li et al. , 2018 ) . In these scenarios , defining a reward function beforehand is particularly challenging and IRL is simply more pragmatic . However , complications with IRL in unregularized MDPs relate to the issue of degeneracy , where any constant function can rationalize the expert ’ s behavior ( Ng et al. , 2000 ) . Fortunately , Geist et al . ( 2019 ) show that IRL in regularized MDPs—regularized IRL—does not contain such degenerate solutions due to the uniqueness of the optimal policy for regularized MDPs . Despite this , no tractable solutions of regularized IRL—other than maximum-Shannon-entropy IRL ( MaxEntIRL ) ( Ziebart et al. , 2008 ; Ziebart , 2010 ; Ho & Ermon , 2016 ; Finn et al. , 2016a ; Fu et al. , 2018 ) —have been proposed . In Geist et al . ( 2019 ) , solutions for regularized IRL were introduced . However , they are generally intractable since they require a closed-form relation between the policy and optimal value function and the knowledge on model dynamics . Furthermore , practical algorithms for solving regularized IRL problems have not yet been proposed . We summarize our contributions as follows . Unlike the solutions in Geist et al . ( 2019 ) , we propose tractable solutions for regularized IRL problems that can be derived from policy regularization and ∗Correspondence to : Wonseok Jeon < jeonwons @ mila.quebec > its gradient in discrete control problems ( Section 3.1 ) . We additionally show that our solutions are tractable for Tsallis entropy regularization with multi-variate Gaussian policies in continuous control problems ( Section 3.2 ) . We devise Regularized Adversarial Inverse Reinforcement Learning ( RAIRL ) , a practical sample-based method for policy imitation and reward learning in regularized MDPs , which generalizes adversarial IRL ( AIRL , Fu et al . ( 2018 ) ) ( Section 4 ) . Finally , we empirically validate our RAIRL method on both discrete and continuous control tasks , evaluating RAIRL via episodic scores and from divergence minimization perspective ( Ke et al. , 2019 ; Ghasemipour et al. , 2019 ; Dadashi et al. , 2020 ) ( Section 5 ) . 2 PRELIMINARIES . Notation For finite sets X and Y , Y X is a set of functions from X to Y . ∆X ( ∆XY ) is a set of ( conditional ) probabilities over X ( conditioned on Y ) . Especially for the conditional probabilities pX|Y ∈ ∆XY , we say pX|Y ( ·|y ) ∈ ∆X for y ∈ Y . R is the set of real numbers . For functions f1 , f2 ∈ RX , we define 〈f1 , f2〉X : = ∑ x∈X f1 ( x ) f2 ( x ) . Regularized Markov Decision Processes and Reinforcement Learning We consider sequential decision making problems where an agent sequentially chooses its action after observing the state of the environment , and the environment in turn emits a reward with state transition . Such an interaction between the agent and the environment is modeled as an infinite-horizon Markov Decision Process ( MDP ) , Mr : = 〈S , A , P0 , P , r , γ〉 and the agent ’ s policy π ∈ ∆AS . The terms within the MDP are defined as follows : S is a finite state space , A is a finite action space , P0 ∈ ∆S is an initial state distribution , P ∈ ∆SS×A is a state transition probability , r ∈ RS×A is a reward function , and γ ∈ [ 0 , 1 ) is the discount factor . We also define an MDP without reward asM− : = 〈S , A , P0 , P , γ〉 . The normalized state-action visitation distribution , dπ ∈ ∆S×A , associated with π is defined as the expected discounted state-action visitation of π , i.e. , dπ ( s , a ) : = ( 1−γ ) ·Eπ [ ∑∞ i=0 γ iI { si = s , ai = a } ] , where the subscript π on E means that a trajectory ( s0 , a0 , s1 , a1 , ... ) is randomly generated fromM− and π , and I { · } is an indicator function . Note that dπ satisfies the transposed Bellman recurrence ( Boularias & Chaib-Draa , 2010 ; Zhang et al. , 2019 ) : dπ ( s , a ) = ( 1− γ ) P0 ( s ) π ( a|s ) + γπ ( a|s ) ∑ s̄ , ā P ( s|s̄ , ā ) dπ ( s , a ) . We consider RL in regularized MDPs ( Geist et al. , 2019 ) , where the policy is optimized with a causal policy regularizer . Mathematically for an MDPMr and a strongly convex function Ω : ∆A → R , the objective in regularized MDPs is to seek π that maximizes the expected discounted sum of rewards , or return in short , with policy regularizer Ω : arg max π∈∆AS JΩ ( r , π ) : = Eπ [ ∞∑ i=0 γi { r ( si , ai ) − Ω ( π ( ·|si ) ) } ] = 1 1− γ E ( s , a ) ∼dπ [ r ( s , a ) − Ω ( π ( ·|s ) ] . ( 1 ) It turns out that the optimal solution of Eq . ( 1 ) is unique ( Geist et al. , 2019 ) , whereas multiple optimal policies may exist in unregularized MDPs ( See Appendix A for a detailed explanation ) . In later work ( Yang et al. , 2019 ) , Ω ( p ) = −λEa∼pφ ( p ( a ) ) , p ∈ ∆A was considered for λ > 0 and φ : ( 0 , 1 ] → R satisfying some mild conditions . For example , RL with Shannon entropy regularization ( Haarnoja et al. , 2018 ) can be recovered by φ ( x ) = − log x , while RL with Tsallis entropy regularization ( Lee et al. , 2020 ) can be recovered from φ ( x ) = kq−1 ( 1 − x q−1 ) for k > 0 , q > 1 . The optimal policy π∗ for Eq . ( 1 ) with Ω from Yang et al . ( 2019 ) is shown to be π∗ ( a|s ) = max { gφ ( µ∗ ( s ) −Q∗ ( s , a ) λ ) , 0 } , ( 2 ) Q∗ ( s , a ) = r ( s , a ) + γEs′∼P ( ·|s , a ) V ∗ ( s′ ) , V ∗ ( s ) = µ∗ ( s ) − λ ∑ a∈A π∗ ( a|s ) 2φ′ ( π∗ ( a|s ) ) , ( 3 ) where φ′ ( x ) = ∂∂xφ ( x ) , gφ is an inverse function of f ′ φ for fφ ( x ) : = xφ ( x ) , x ∈ ( 0 , 1 ] , and µ∗ is a normalization term such that ∑ a∈A π ∗ ( a|s ) = 1 . Note that we still need to find out µ∗ to acquire a closed-form relation between optimal policy π∗ and value function Q∗ . However , such relations have not been discovered except for Shannon-entropy regularization ( Haarnoja et al. , 2018 ) and specific instances ( q = 1 , 2 , ∞ ) of Tsallis-entropy regularization ( Lee et al. , 2019 ) to the best of our knowledge . Inverse Reinforcement Learning Given a set of demonstrations from an expert policy π E , IRL ( Russell , 1998 ; Ng et al. , 2000 ) is the problem of seeking a reward function from which we can recover πE through RL . However , IRL in unregularized MDPs has been shown to be an ill-defined problem since ( 1 ) any constant reward function can rationalize every expert and ( 2 ) multiple rewards meet the criteria of being a solution ( Ng et al. , 2000 ) . Maximum entropy IRL ( MaxEntIRL ) ( Ziebart et al. , 2008 ; Ziebart , 2010 ) is capable of solving the first issue by seeking a reward function that maximizes the expert ’ s return along with Shannon entropy of expert policy . Mathematically , for the RL objective JΩ in Eq . ( 1 ) and Ω = −H for negative Shannon entropy H ( p ) = Ea∼p [ − log p ( a ) ] ( Ho & Ermon , 2016 ) , the objective of MaxEntIRL is MaxEntIRL ( π E ) : = arg max r∈RS×A { J−H ( r , πE ) − max π∈∆AS J−H ( r , π ) } . ( 4 ) Another commonly used IRL method is Adversarial Inverse Reinforcement Learning ( AIRL ) ( Fu et al. , 2018 ) which involves generative adversarial training ( Goodfellow et al. , 2014 ; Ho & Ermon , 2016 ) to acquire a solution of MaxEntIRL . AIRL considers the structured discriminator ( Finn et al. , 2016a ) D ( s , a ) = σ ( r ( s , a ) − log π ( a|s ) ) = e r ( s , a ) er ( s , a ) +π ( a|s ) for σ ( x ) : = 1/ ( 1 + e −x ) and iteratively optimizes the following objective : max r∈RS×A E ( s , a ) ∼dπ E [ logDr , π ( s , a ) ] + E ( s , a ) ∼dπ [ log ( 1−Dr , π ( s , a ) ) ] , max π∈∆AS E ( s , a ) ∼dπ [ logDr , π ( s , a ) − log ( 1−Dr , π ( s , a ) ) ] = max π∈∆AS E ( s , a ) ∼dπ [ r ( s , a ) − log π ( a|s ) ] . ( 5 ) It turns out that AIRL minimizes the divergence between visitation distributions dπ and dπ E by solving minπ∈∆AS KL ( dπ||dπE ) for Kullback-Leibler ( KL ) divergence KL ( ·||· ) ( Ghasemipour et al. , 2019 ) .

Summary: This paper proposes a new method for regularized inverse RL. The paper builds upon work by Geist et al. who studied regularized MDPs with convex policy regularizers. The Shannon entropy is a special case of such a policy regularizer. The paper extends the analysis of Geist et al. for regularized IRL and devises tractable solutions to regularized IRL that only depend on the analytic knowledge of the regularizer. The paper further proposes regularized adversarial IRL (RAIRL), an extension of AIRL by Fu et al., as an algorithm for IRL in regularized MDPs. The algorithm is validated on a number of domains.