Near Optimal Policy Optimization via REPS

1 Introduction . Introduced by Peters et al . [ 23 ] , relative entropy policy search ( REPS ) is an algorithm for learning agent policies in a reinforcement learning ( RL ) context . REPS has demonstrated successful policy learning in a variety of challenging simulated and real-world robotic tasks , encompassing table tennis [ 23 ] , tether ball [ 12 ] , beer pong [ 1 ] , and ball-in-a-cup [ 8 ] , among others . Beyond these direct applications of REPS , the mathematical tools and algorithmic components underlying REPS have inspired and been utilized as a foundation for a number of later algorithms , with their own collection of practical successes [ 13 , 25 , 20 , 22 , 15 , 2 , 18 , 21 ] . At its core , the REPS algorithm is derived via an application of convex duality [ 22 , 19 ] , in which a Kullback Leibler ( KL ) -regularized version of the max-return objective in terms of state-action distributions is transformed into an logsumexp objective in terms of state-action advantages ( i.e. , the difference of the value of the state-action pair compared to the value of the state alone , with respect to some learned state value function ) . If this dual objective is optimized , then the optimal policy of the original primal problem may be derived as a softmax of the state-action advantages . This basic derivation may be generalized , using any number of entropic regularizers on the original primal to yield a dual problem in the form of a convex function of advantages , whose optimal values may be transformed back to optimal regularized policies [ 7 ] . While the motivation for the REPS objective through the lens of convex duality is attractive , it leaves two main questions unanswered regarding the theoretical soundness of using such an approach . First , in practice , the dual objective in terms of advantages is likely not optimized fully . Rather , standard gradient-based solvers only provide guarantees on the near-optimality of a returned candidate solution . While convex duality asserts a relationship between primal and dual variables at the exact optimum , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . it is far from clear whether a near-optimal dual solution will be guaranteed to yield a near-optimal primal solution , and this is further complicated by the fact that the primal candidate solution must be transformed to yield an agent policy . The second of the two main practical difficulties is due to the form of the dual objective . Specifically , the form of the dual objective as a convex function of advantages frustrates the use of gradient-based solvers in stochastic settings . That is , the advantage of a state-action pair consists of an expectation over next states – an expectation over the transition function associated with the underlying Markov decision process ( MDP ) . In practical settings , one does not have explicit knowledge of this transition function . Rather , one only has access to stochastic samples from this transition function , and so calculation of unbiased gradients of the REPS objective is not directly feasible . In this paper , we provide solutions to these two main difficulties . To the first issue , we present guarantees on the near-optimality of a derived policy from dual variables optimized via a first-order gradient method , relying on a key property of the REPS objective that ensures near-optimality in terms of gradient norms . To the second issue , we propose and analyze a stochastic gradient descent procedure that makes use of a plug-in estimator of the REPS gradients . Under some mild assumptions on the MDP , our estimators need only sample transitions from a behavior policy rather than full access to a generative model ( where one can uniformly sample transitions ) . We combine these results to yield high-probability convergence rates of REPS to a near-optimal policy . In this way , we show that REPS enjoys not only favorable practical performance but also strong theoretical guarantees . 2 Related Work . As REPS is a popular and influential work , there exist a number of previous papers that have studied its performance guarantees . These previous works predominantly study REPS as an iterative algorithm , where each step comprises of an exact optimization of the REPS objective and then the derived policy is used as the reference distribution for the KL regularization of the next step . This iterative scheme may be interpreted as a form of mirror descent or similar proximal algorithms [ 6 ] , and this interpretation can provide guarantees on convergence to a near-optimal policy [ 29 , 22 ] . However , because this approach assumes the ability to optimize the REPS objective exactly , it still suffers from the practical limitations discussed above ; specifically ( 1 ) translation of near-optimality of advantages to near-optimality of the policy and ( 2 ) ability to compute unbiased gradients when one does not have explicit knowledge of the MDP dynamics . Our analysis attacks these issues head-on , providing guarantees on first-order optimization methods applied to the REPS objective . To maintain focus we do not consider iterative application of REPS , although extending our guarantees to the iterative setting is a promising direction for future research . In a somewhat related vein , a number of works use REPS-inspired derivations to yield dynamic programming algorithms [ 13 , 14 , 26 ] and subsequently provide guarantees on the convergence of approximate dynamic programming in these settings . Our results focus on the use of REPS in a convex programming context , and optimizing these programs via standard gradient-based solvers . The use of convex programming for RL in this way has recently received considerable interest . Works in this area typically propose to learn near-optimal policies through saddle-point optimization [ 9 , 28 , 10 , 4 , 11 , 17 ] . Instead of solving the primal or dual max-return problem directly , these works optimize the Lagrangian in the form of a min-max bilinear problem . The Lagrangian form helps to mitigate the two main issues we identify with advantage learning , since ( 1 ) the candidate primal solution can be used to derive a policy in a significantly more direct fashion than using the candidate dual solution , and ( 2 ) the bilinear form of the Lagrangian is immediately amenable to stochastic gradient computation . In contrast to these works , our analysis focuses on learning exclusively in the dual ( advantage ) space . The first part of our results is most comparable to the work of [ 4 ] , which proposes a saddle-point optimization with runtime O ( 1/✏ ) , assuming access to known dynamics . While our results yield a O ( 1/✏2 ) rate , we show that it can be achieved via optimizing the dual objective alone . More similar to our work is the analysis of Bas-Serrano et al . [ 5 ] , which considers an objective similar to REPS , but which is in terms of Q-values as opposed to state ( V ) values . Beyond these structural differences , our proof techniques also differ . For example , our result on the suboptimality of the policy derived from dual variables ( Lemma 4 ) , is arguably simpler from the analogous result in Bas-Serrano et al . [ 5 ] , which uses a two-step process to first connect suboptimality of the dual variables to constraint violation in the primal , and then connects this to suboptimality of the policy . 3 Contributions . The main contributions of this paper are the following : 1 . We prove several structural results regarding entropy regularized objectives for reinforcement learning and leverage them to prove convergence guarantees for Accelerated Gradient Descent on the dual ( REPS ) objective under mild assumptions on the MDP ( see Theorem 2 ) . For discounted MDPs we show that an ✏-optimal policy can be found after O ( 1/ ( 1 ) 2✏2 ) steps and an ✏ optimal regularized policy can be found in O ( 1/ ( 1 ) 2✏ ) steps . 2 . Similarly we show that a simple version of stochastic gradient descent using biased plug-in gradient estimators can be used to find an ✏ optimal policy after O ( 1/ ( 1 ) 8✏8 ) iterations ( see Theorem 3 ) and an ✏-optimal regularized policy in O ( 1/ ( 1 ) 8✏4 ) steps . Although our rates are short of the ones achievable by alternating optimization methods , we are the first to show meaningful convergence guarantees for a purely dual approach based on on-policy access to samples from the underlying MDP . 3 . In Appendix I , we extend our results beyond the REPS objective and consider the use of Tsallis Entropy regularizers . Similar to our results for the REPS objective we show that for discounted MDPs an ✏ optimal policy can be found after O ( 1/ ( 1 ) 2✏2 ) steps and an ✏ optimal regularized policy can be found in O ( 1/ ( 1 ) 2✏ ) steps . 4 Background . In this section we review the basics of Markov decision processes and their Linear Programming primal and dual formulations ( see section 4.1 ) and some facts about the geometry of convex functions . 4.1 RL as an LP . We consider a discounted Markov decision process ( MDP ) described by a tuple M = ( S , A , P , r , µ , ) , where S is a finite state space , A is a finite action space , P is a transition probability matrix , r is a reward vector , µ is an initial state distribution , and 2 ( 0 , 1 ) is a discount factor . We make the following assumption regarding the reward values { rs , a } . Assumption 1 ( Unit rewards ) . For all s , a 2 S ⇥A , the rewards satisfy , rs , a 2 [ 0 , 1 ] . The agent interacts with M via a policy ⇡ : S ! A . The agent is initialized at a state s0 sampled from an initial state distribution µ and at time k = 0 , 1 , . . . it uses its policy to sample an action ak ⇠ ⇡ ( sk ) . The MDP provides an immediate reward rsk , ak and transitions randomly to a next state sk+1 according to probabilities Pa ( sk+1|sk ) . Given a policy ⇡ we define its infinite-horizon discounted reward as V⇡ : = E⇡ ⇥P1 k=0 krsk , ak ⇤ , where we use E⇡ to denote the expectation over trajectories induced by the MDP M and policy ⇡ . In RL , the agent ’ s objective is to find an optimal policy ⇡ ? ; that is , find a policy maximizing V⇡ over all policy mappings ⇡ : S ! A . We denote the optimal policy as ⇡ ? : = argmax⇡ V⇡ . We now review the definitions of state value functions and visitation distributions : Definition 1 . We define the value vector v⇡ 2 R|S| of a policy ⇡ as v⇡s : = E⇡ ⇥P1 k=0 k rsk , ak |s0 = s ⇤ . Definition 2 . Given a policy ⇡ we define its state-action visitation distribution ⇡ 2 R|S|⇥|A| as , ⇡s , a : = ( 1 ) E⇡ ⇥P1 k=0 k1 ( sk = s , ak = a ) ⇤ . Notice that by definition P s , a s , a = 1 . We note that any vector of nonnegative entries may be used to define a policy ⇡ as : ⇡ ( a|s ) : = s , aP a02A s , a0 . ( 1 ) Note that ⇡ ⇡ = ⇡ , while the visitation distribution ⇡ of ⇡ is not necessarily . Definition 3 . Given a policy ⇡ we define its state visitation distribution as , ⇡s : = ( 1 ) E⇡ ⇥P1 k=0 k1 ( sk = s ) ⇤ . Notice that ⇡s = P a ⇡ s , a and ⇡ s , a = ⇡ s · ⇡ ( a|s ) . The optimal visitation distribution ⇤ is defined as ⇤ : = argmax ⇡ P s , a ⇡ s , ars , a . It can be shown [ 24 , 9 ] that solving for the optimal visitation distribution is equivalent to the following linear program : max s , a2 S⇥A X s , a s , ars , a , s.t . X a s , a = ( 1 ) µs + X s0 , a Pa ( s|s0 ) s0 , a 8s 2 S. ( Primal- ) Where we write P 2 R|S||A|⇥|S| to denote the transition operator . Specifically , the |S| constraints of Primal- restrict any feasible to be the state-action visitations for some policy ⇡ ( given by ⇡ ) . The dual of this LP is given by , min v ( 1 ) X s2S µsvs , s.t . 0 A v s , a 8s 2 S , a 2 A , ( Dual-v ) where Avs , a = rs , a vs + P s0 Pa ( s 0 |s ) vs0 is the advantage evaluated at s , a 2 S ⇥A . It can be shown [ 24 , 9 ] that the unique primal solution ⇤ is exactly ⇡⇤ and the unique dual solution v⇤ is v⇡⇤ . We finalize this section by defining the notion of suboptimality satisfied by the final policy produced by the algorithms that we propose . Definition 4 . Let ✏ > 0 . We say that policy ⇡ is ✏-optimal if maxs2S |v⇡s v⇡ ? s | ✏ . Our objective is to design algorithms such that for any ✏ > 0 , can return an ✏ optimal policy .

Summary: The paper studies the performance of the classic Relative Entropy Policy Search (REPS) algorithm of Peters et al. (2010) in the context of tabular reinforcement learning, focusing on the effect of optimization errors on the quality of the policy extracted from the solution. REPS is based on formulating the policy optimization problem as a regularized linear optimization problem which is equivalent to an unconstrained convex optimization problem, and calculating policy updates via approximately minimizing the dual function. The current paper sets out to understand the optimization issues surrounding solving these problems, and provides the following contributions:  - Showing that the policy suboptimality can be controlled in terms of the gradient norm associated with the approximate solution and some problem-dependent constants. - Showing that the dual REPS objective is smooth, which implies that bounds on the additive optimization errors can be translated to bounds on the gradient norm at the approximate solution. - Providing concrete performance bounds for an accelerated gradient descent method when the gradients of the objective can be evaluated exactly. This setting is rather unrealistic, but the results are insightful in that they shed light on the best attainable performance to be expected from this approach.  - Providing concrete performance bounds for a stochastic gradient descent method that uses a generative model to obtain estimates of the gradients. The challenge here is that one cannot straightforwardly obtain unbiased gradient estimates, and one has to take several sample transitions to make sure that the bias in the updates is small enough. The authors address this challenge thoroughly and obtain guarantees that are polynomial in the desired accuracy level.