Regularized Inverse Reinforcement Learning

1 INTRODUCTION . Reinforcement learning ( RL ) has been successfully applied to many challenging domains including games ( Mnih et al. , 2015 ; 2016 ) and robot control ( Schulman et al. , 2015 ; Fujimoto et al. , 2018 ; Haarnoja et al. , 2018 ) . Advanced RL methods often employ policy regularization motivated by , e.g. , boosting exploration ( Haarnoja et al. , 2018 ) or safe policy improvement ( Schulman et al. , 2015 ) . While Shannon entropy is often used as a policy regularizer ( Ziebart et al. , 2008 ) , Geist et al . ( 2019 ) recently proposed a theoretical foundation of regularized Markov decision processes ( MDPs ) —a framework that uses strongly convex functions as policy regularizers . Here , one crucial advantage is that an optimal policy is shown to uniquely exist , whereas multiple optimal policies may exist in the absence of policy regularization . Meanwhile , since RL requires a given or known reward function ( which can often involve non-trivial reward engineering ) , Inverse Reinforcement Learning ( IRL ) ( Russell , 1998 ; Ng et al. , 2000 ) —the problem of acquiring a reward function that promotes expert-like behavior—is more generally adopted in practical scenarios like robotic manipulation ( Finn et al. , 2016b ) , autonomous driving ( Sharifzadeh et al. , 2016 ; Wu et al. , 2020 ) and clinical motion analysis ( Li et al. , 2018 ) . In these scenarios , defining a reward function beforehand is particularly challenging and IRL is simply more pragmatic . However , complications with IRL in unregularized MDPs relate to the issue of degeneracy , where any constant function can rationalize the expert ’ s behavior ( Ng et al. , 2000 ) . Fortunately , Geist et al . ( 2019 ) show that IRL in regularized MDPs—regularized IRL—does not contain such degenerate solutions due to the uniqueness of the optimal policy for regularized MDPs . Despite this , no tractable solutions of regularized IRL—other than maximum-Shannon-entropy IRL ( MaxEntIRL ) ( Ziebart et al. , 2008 ; Ziebart , 2010 ; Ho & Ermon , 2016 ; Finn et al. , 2016a ; Fu et al. , 2018 ) —have been proposed . In Geist et al . ( 2019 ) , solutions for regularized IRL were introduced . However , they are generally intractable since they require a closed-form relation between the policy and optimal value function and the knowledge on model dynamics . Furthermore , practical algorithms for solving regularized IRL problems have not yet been proposed . We summarize our contributions as follows . Unlike the solutions in Geist et al . ( 2019 ) , we propose tractable solutions for regularized IRL problems that can be derived from policy regularization and ∗Correspondence to : Wonseok Jeon < jeonwons @ mila.quebec > its gradient in discrete control problems ( Section 3.1 ) . We additionally show that our solutions are tractable for Tsallis entropy regularization with multi-variate Gaussian policies in continuous control problems ( Section 3.2 ) . We devise Regularized Adversarial Inverse Reinforcement Learning ( RAIRL ) , a practical sample-based method for policy imitation and reward learning in regularized MDPs , which generalizes adversarial IRL ( AIRL , Fu et al . ( 2018 ) ) ( Section 4 ) . Finally , we empirically validate our RAIRL method on both discrete and continuous control tasks , evaluating RAIRL via episodic scores and from divergence minimization perspective ( Ke et al. , 2019 ; Ghasemipour et al. , 2019 ; Dadashi et al. , 2020 ) ( Section 5 ) . 2 PRELIMINARIES . Notation For finite sets X and Y , Y X is a set of functions from X to Y . ∆X ( ∆XY ) is a set of ( conditional ) probabilities over X ( conditioned on Y ) . Especially for the conditional probabilities pX|Y ∈ ∆XY , we say pX|Y ( ·|y ) ∈ ∆X for y ∈ Y . R is the set of real numbers . For functions f1 , f2 ∈ RX , we define 〈f1 , f2〉X : = ∑ x∈X f1 ( x ) f2 ( x ) . Regularized Markov Decision Processes and Reinforcement Learning We consider sequential decision making problems where an agent sequentially chooses its action after observing the state of the environment , and the environment in turn emits a reward with state transition . Such an interaction between the agent and the environment is modeled as an infinite-horizon Markov Decision Process ( MDP ) , Mr : = 〈S , A , P0 , P , r , γ〉 and the agent ’ s policy π ∈ ∆AS . The terms within the MDP are defined as follows : S is a finite state space , A is a finite action space , P0 ∈ ∆S is an initial state distribution , P ∈ ∆SS×A is a state transition probability , r ∈ RS×A is a reward function , and γ ∈ [ 0 , 1 ) is the discount factor . We also define an MDP without reward asM− : = 〈S , A , P0 , P , γ〉 . The normalized state-action visitation distribution , dπ ∈ ∆S×A , associated with π is defined as the expected discounted state-action visitation of π , i.e. , dπ ( s , a ) : = ( 1−γ ) ·Eπ [ ∑∞ i=0 γ iI { si = s , ai = a } ] , where the subscript π on E means that a trajectory ( s0 , a0 , s1 , a1 , ... ) is randomly generated fromM− and π , and I { · } is an indicator function . Note that dπ satisfies the transposed Bellman recurrence ( Boularias & Chaib-Draa , 2010 ; Zhang et al. , 2019 ) : dπ ( s , a ) = ( 1− γ ) P0 ( s ) π ( a|s ) + γπ ( a|s ) ∑ s̄ , ā P ( s|s̄ , ā ) dπ ( s , a ) . We consider RL in regularized MDPs ( Geist et al. , 2019 ) , where the policy is optimized with a causal policy regularizer . Mathematically for an MDPMr and a strongly convex function Ω : ∆A → R , the objective in regularized MDPs is to seek π that maximizes the expected discounted sum of rewards , or return in short , with policy regularizer Ω : arg max π∈∆AS JΩ ( r , π ) : = Eπ [ ∞∑ i=0 γi { r ( si , ai ) − Ω ( π ( ·|si ) ) } ] = 1 1− γ E ( s , a ) ∼dπ [ r ( s , a ) − Ω ( π ( ·|s ) ] . ( 1 ) It turns out that the optimal solution of Eq . ( 1 ) is unique ( Geist et al. , 2019 ) , whereas multiple optimal policies may exist in unregularized MDPs ( See Appendix A for a detailed explanation ) . In later work ( Yang et al. , 2019 ) , Ω ( p ) = −λEa∼pφ ( p ( a ) ) , p ∈ ∆A was considered for λ > 0 and φ : ( 0 , 1 ] → R satisfying some mild conditions . For example , RL with Shannon entropy regularization ( Haarnoja et al. , 2018 ) can be recovered by φ ( x ) = − log x , while RL with Tsallis entropy regularization ( Lee et al. , 2020 ) can be recovered from φ ( x ) = kq−1 ( 1 − x q−1 ) for k > 0 , q > 1 . The optimal policy π∗ for Eq . ( 1 ) with Ω from Yang et al . ( 2019 ) is shown to be π∗ ( a|s ) = max { gφ ( µ∗ ( s ) −Q∗ ( s , a ) λ ) , 0 } , ( 2 ) Q∗ ( s , a ) = r ( s , a ) + γEs′∼P ( ·|s , a ) V ∗ ( s′ ) , V ∗ ( s ) = µ∗ ( s ) − λ ∑ a∈A π∗ ( a|s ) 2φ′ ( π∗ ( a|s ) ) , ( 3 ) where φ′ ( x ) = ∂∂xφ ( x ) , gφ is an inverse function of f ′ φ for fφ ( x ) : = xφ ( x ) , x ∈ ( 0 , 1 ] , and µ∗ is a normalization term such that ∑ a∈A π ∗ ( a|s ) = 1 . Note that we still need to find out µ∗ to acquire a closed-form relation between optimal policy π∗ and value function Q∗ . However , such relations have not been discovered except for Shannon-entropy regularization ( Haarnoja et al. , 2018 ) and specific instances ( q = 1 , 2 , ∞ ) of Tsallis-entropy regularization ( Lee et al. , 2019 ) to the best of our knowledge . Inverse Reinforcement Learning Given a set of demonstrations from an expert policy π E , IRL ( Russell , 1998 ; Ng et al. , 2000 ) is the problem of seeking a reward function from which we can recover πE through RL . However , IRL in unregularized MDPs has been shown to be an ill-defined problem since ( 1 ) any constant reward function can rationalize every expert and ( 2 ) multiple rewards meet the criteria of being a solution ( Ng et al. , 2000 ) . Maximum entropy IRL ( MaxEntIRL ) ( Ziebart et al. , 2008 ; Ziebart , 2010 ) is capable of solving the first issue by seeking a reward function that maximizes the expert ’ s return along with Shannon entropy of expert policy . Mathematically , for the RL objective JΩ in Eq . ( 1 ) and Ω = −H for negative Shannon entropy H ( p ) = Ea∼p [ − log p ( a ) ] ( Ho & Ermon , 2016 ) , the objective of MaxEntIRL is MaxEntIRL ( π E ) : = arg max r∈RS×A { J−H ( r , πE ) − max π∈∆AS J−H ( r , π ) } . ( 4 ) Another commonly used IRL method is Adversarial Inverse Reinforcement Learning ( AIRL ) ( Fu et al. , 2018 ) which involves generative adversarial training ( Goodfellow et al. , 2014 ; Ho & Ermon , 2016 ) to acquire a solution of MaxEntIRL . AIRL considers the structured discriminator ( Finn et al. , 2016a ) D ( s , a ) = σ ( r ( s , a ) − log π ( a|s ) ) = e r ( s , a ) er ( s , a ) +π ( a|s ) for σ ( x ) : = 1/ ( 1 + e −x ) and iteratively optimizes the following objective : max r∈RS×A E ( s , a ) ∼dπ E [ logDr , π ( s , a ) ] + E ( s , a ) ∼dπ [ log ( 1−Dr , π ( s , a ) ) ] , max π∈∆AS E ( s , a ) ∼dπ [ logDr , π ( s , a ) − log ( 1−Dr , π ( s , a ) ) ] = max π∈∆AS E ( s , a ) ∼dπ [ r ( s , a ) − log π ( a|s ) ] . ( 5 ) It turns out that AIRL minimizes the divergence between visitation distributions dπ and dπ E by solving minπ∈∆AS KL ( dπ||dπE ) for Kullback-Leibler ( KL ) divergence KL ( ·||· ) ( Ghasemipour et al. , 2019 ) .

Summary: This work considers a regularized IRL setup, where instead of the entropy regularization used in maximum entropy IRL, an arbitrary convex regularizer $\Omega$ is used. The work presents a number of theoretical results for this general setting, and it is shown that when $\Omega$ is Tsallis entropy, the $RL \cdot IRL$ is equivalent to minimizing a Bregman divergence defined based on the Tsallis entropy and the expert state-action distribution. A practical algorithm is presented for IRL with the Tsallis entropy. A number of experiments are performed to obtain understanding of various components.