Mirror Descent Policy Optimization

1 INTRODUCTION . An important class of RL algorithms consider an additional objective in their policy optimization that aims at constraining the consecutive policies to remain close to each other . These algorithms are referred to as trust region or proximity-based , resonating the fact that they make the new policy to lie within a trust-region around the old one . This class includes the theoretically grounded conservative policy iteration ( CPI ) algorithm [ 15 ] , as well as the state-of-the-art deep RL algorithms , such as trust-region policy optimization ( TRPO ) [ 26 ] and proximal policy optimization ( PPO ) [ 28 ] . The main difference between these algorithms is in the way that they enforce the trust-region constraint . TRPO enforces it explicitly through a line-search procedure that ensures the new policy is selected such that its KL-divergence with the old policy is below a certain threshold . PPO takes a more relaxed approach and updates its policies by solving an unconstrained optimization problem in which the ratio of the new to old policies is clipped to remain bounded . It has been shown that this procedure does not prevent the policy ratios to go out of bound , and only reduces its probability [ 31 , 9 ] . Mirror descent ( MD ) [ 6 , 4 ] is a first-order optimization method for solving constrained convex problems . Although MD is theoretically well-understood in optimization [ 3 , 14 ] , only recently , has it been investigated for policy optimization in RL [ 25 , 12 , 20 , 29 , 1 ] . Despite the progress made by these results in establishing connections between MD and trust-region policy optimization , there are still considerable gaps between the trust-region RL algorithms that have been theoretically analyzed in their tabular form [ 29 ] and those that are used in practice , such as TRPO and PPO . In this paper , motivated by the theory of MD in tabular RL , our goal is to derive scaleable and practical RL algorithms from the MD principles , and to use the MD theory to better understand and explain the popular trust-region policy optimization methods . Going beyond the tabular case , when the policy belongs to a parametric class , the trust-region problems for policy update in RL can not be solved in closed-form . We propose an algorithm , called mirror descent policy optimization ( MDPO ) , that addresses this issue by approximately solving these trust-region problems via taking multiple gradient steps on their objective functions . We derive on-policy and off-policy variants of MDPO ( Section 4 ) . We highlight the connection between on-policy MDPO and TRPO and PPO ( Section 4.1 ) , and empirically compare it against these algorithms on several continuous control tasks from OpenAI Gym [ 7 ] ( Section 5.3 ) . We then show that if we define the trust-region w.r.t . the uniform policy , instead of the old one , our off-policy MDPO coincides with the popular soft actor-critic ( SAC ) algorithm [ 13 ] . We discuss this connection in detail ( Section 4.2 ) and empirically compare these algorithms using the same set of continuous control problems ( Section 5.4 ) . Our observations on the comparison between the MDPO algorithms and TRPO , PPO , and SAC are a result of extensive empirical studies on different versions of these algorithms ( Section 5 and Appendices E and F ) . In particular , we first compare the vanilla versions of these algorithms in order to better understand how the core of these methods work relative to each other . We then add a number of code-level optimization techniques derived from the code-bases of TRPO , PPO , and SAC to these algorithms to compare their best form ( those that obtain the best results reported in the literature ) against each other , while also evaluating MDPO with PPO on 21 Atari games . We address the common belief within the community that explicitly enforcing the trust-region constraint is a necessity for good performance in TRPO , by showing that MDPO , a trust-region method based on the MD principles , does not require enforcing a hard constraint and achieves strong performance by solely solving an unconstrained problem . We address another common belief that PPO is a better performing algorithm than TRPO . By reporting results of both the vanilla version and the version loaded with code-level optimization techniques for all algorithms , we show that in both cases , TRPO consistently outperforms PPO . This is in line with some of the findings from a recent study on PPO and TRPO [ 9 ] . Finally , we provide an optimization perspective for SAC , instead of its initial motivation as an entropy-regularized ( soft ) approximate dynamic programming algorithm . Through comprehensive experiments , we show that on-policy and off-policy MDPO achieve state-of-the-art performance across a number of benchmark tasks , and can be excellent alternatives to popular policy optimization algorithms , such as TRPO , PPO , and SAC . 2 PRELIMINARIES . In this paper , we assume that the agent ’ s interaction with the environment is modeled as a γ-discounted Markov decision process ( MDP ) , denoted byM = ( S , A , P , R , γ , µ ) , where S and A are the state and action spaces ; P ≡ P ( s′|s , a ) is the transition kernel ; R ≡ r ( s , a ) is the reward function ; γ ∈ ( 0 , 1 ) is the discount factor ; and µ is the initial state distribution . Let π : S → ∆A be a stationary Markovian policy , where ∆A is the set of probability distributions onA . The discounted frequency of visiting a state s by following a policy π is defined as ρπ ( s ) ≡ ( 1−γ ) E [ ∑ t≥0 γ tI { st = s } |µ , π ] . The value function of a policy π at a state s ∈ S is defined as V π ( s ) ≡ E [ ∑ t≥0 γ tr ( st , at ) |s0 = s , π ] . Similarly , the action-value function of π is defined as Qπ ( s , a ) = E [ ∑ t≥0 γ tr ( st , at ) |s0 = s , a0 = a , π ] . The difference between the action-value Q and value V functions is referred to as the advantage function Aπ ( s , a ) = Qπ ( s , a ) − V π ( s ) . Since finding an optimal policy for an MDP involves solving a non-linear system of equations and the optimal policy may be deterministic ( less explorative ) , many researchers have proposed to add a regularizer in the form of an entropy term to the reward function , and then solve the entropyregularized ( or soft ) MDP ( e.g. , [ 16 , 30 , 25 ] ) . In this formulation , the reward function is modified as rλ ( s , a ) = r ( s , a ) +λH ( π ( ·|s ) ) , where λ is the regularization parameter and H is an entropy-related term , such as Shannon entropy [ 10 , 23 ] , Tsallis entropy [ 17 , 24 ] , or relative entropy [ 2 , 22 ] . Setting λ = 0 , we return to the original formulation , also referred to as the hard MDP . In what follows , we use the terms ‘ regularized ’ and ‘ soft ’ interchangeably . 2.1 MIRROR DESCENT IN CONVEX OPTIMIZATION . Mirror Descent ( MD ) [ 4 ] is a first-order trust-region optimization method for solving constrained convex problems , i.e. , x∗ ∈ arg minx∈C f ( x ) , where f is a convex function and the constraint set C is convex compact . In each iteration , MD minimizes a sum of two terms : 1 ) a linear approximation of the objective function f at the previous estimate xk , and 2 ) a proximity term that measures the distance between the updated xk+1 and current xk estimates . MD is considered a trust-region method , since the proximity term keeps the updates xk and xk+1 close to each other . We may write the MD update as xk+1 ∈ arg min x∈C 〈∇f ( xk ) , x− xk〉+ 1 tk Bψ ( x , xk ) , ( 1 ) where Bψ ( x , xk ) : = ψ ( x ) − ψ ( xk ) − 〈∇ψ ( xk ) , x− xk〉 is the Bregman divergence associated with a strongly convex potential function ψ , and tk is a step-size determined by the MD analysis . When ψ = 12‖·‖ 2 2 , the Bergman divergence is the Euclidean distance Bψ ( x , xk ) = 1 2‖x − xk‖ 2 2 , and ( 1 ) becomes the projected gradient descent algorithm [ 3 ] . When ψ is the negative Shannon entropy , the Bregman divergence term takes the form of the KL divergence , i.e. , Bψ ( x , xk ) = KL ( x , xk ) . In this case , when the constraint set C is the unit simplex , C = ∆X , MD becomes the exponentiated gradient descent algorithm and ( 1 ) has the following closed form [ 4 ] : xik+1 = xik exp ( − tk∇if ( xk ) ) ∑n j=1 x j k exp ( − tk∇jf ( xk ) ) , ( 2 ) where xik and ∇if are the ith coordinates of xk and ∇f . 3 MIRROR DESCENT IN RL . The goal in RL is to find an optimal policy π∗ . Two common notions of optimality , and as a result , two distinct ways to formulate RL as an optimization problem are as follows : ( a ) π∗ ( ·|s ) ∈ arg max π V π ( s ) , ∀s ∈ S , ( b ) π∗ ∈ arg max π Es∼µ [ V π ( s ) ] . ( 3 ) In ( 3a ) , the value function is optimized over the entire state space S . This formulation is mainly used in value function based RL algorithms . On the other hand , the formulation in ( 3b ) is more common in policy optimization , where a scalar that is the value function at the initial state ( s ∼ µ ) is optimized . Unlike the MD optimization problem , the objective function is not convex in π in either of the above two RL optimization problems . Despite this issue , [ 12 ] and [ 29 ] have shown that we can still use the general MD update rule ( 1 ) and derive MD-style RL algorithms with the update rules πk+1 ( ·|s ) ← arg max π∈Π Ea∼π [ Aπk ( s , a ) ] − 1 tk KL ( s ; π , πk ) , ∀s ∈ S , ( 4 ) πk+1← arg max π∈Π Es∼ρπk [ Ea∼π [ Aπk ( s , a ) ] − 1 tk KL ( s ; π , πk ) ] , ( 5 ) for the optimization problems ( 3a ) and ( 3b ) , respectively . Note that while in ( 4 ) , the policy is optimized uniformly over the state space S , in ( 5 ) , it is optimized over the measure ρπk , i.e. , the state frequency induced by the current policy πk .

Summary: Summary: Inspired by recent theoretical analysis of TRPO and PPO that use mirror descent, this paper proposes two new algorithms that directly minimize a mirror descent objective by taking multiple gradient steps. While similar to TRPO and PPO, MDPO is different in important ways, and happens to perform better in practice. The authors also propose a similar off-policy version of MDPO that happens to be closely related to SAC while also outperforming SAC experimentally.