Neural Policy Gradient Methods: Global Optimality and Rates of Convergence

1 INTRODUCTION . In reinforcement learning ( Sutton and Barto , 2018 ) , an agent aims to maximize its expected total reward by taking a sequence of actions according to a policy in a stochastic environment , which is modeled as a Markov decision process ( MDP ) ( Puterman , 2014 ) . To obtain the optimal policy , policy gradient methods ( Williams , 1992 ; Baxter and Bartlett , 2000 ; Sutton et al. , 2000 ) directly maximize the expected total reward via gradient-based optimization . As policy gradient methods are easily implementable and readily integrable with advanced optimization techniques such as variance reduction ( Johnson and Zhang , 2013 ; Papini et al. , 2018 ) and distributed optimization ( Mnih et al. , 2016 ; Espeholt et al. , 2018 ) , they enjoy wide popularity among practitioners . In particular , when the policy ( actor ) and action-value function ( critic ) are parameterized by neural networks , policy gradient methods achieve significant empirical successes in challenging applications , such as playing Go ( Silver et al. , 2016 ; 2017 ) , real-time strategy gaming ( Vinyals et al. , 2019 ) , robot manipulation ( Peters and Schaal , 2006 ; Duan et al. , 2016 ) , and natural language processing ( Wang et al. , 2018 ) . See Li ( 2017 ) for a detailed survey . In stark contrast to the tremendous empirical successes , policy gradient methods remain much less well understood in terms of theory , especially when they involve neural networks . More specifically , most existing work analyzes the REINFORCE algorithm ( Williams , 1992 ; Sutton et al. , 2000 ) , which estimates the policy gradient via Monte Carlo sampling . Based on the recent progress in nonconvex optimization , Papini et al . ( 2018 ) ; Shen et al . ( 2019 ) ; Xu et al . ( 2019a ) ; Karimi et al . ( 2019 ) ; ∗equal contribution †Northwestern University ; lingxiaowang2022 @ u.northwestern.edu ‡Northwestern University ; qicai2022 @ u.northwestern.edu §Princeton University ; zy6 @ princeton.edu ¶Northwestern University ; zhaoranwang @ gmail.com 1See https : //arxiv.org/abs/1909.01150 for the full version . Zhang et al . ( 2019 ) establish the rate of convergence of REINFORCE to a first- or second-order stationary point . However , the global optimality of the attained stationary point remains unclear . A more commonly used class of policy gradient methods is equipped with the actor-critic scheme ( Konda and Tsitsiklis , 2000 ) , which alternatingly estimates the action-value function in the policy gradient via a policy evaluation step ( critic update ) , and performs a policy improvement step using the estimated policy gradient ( actor update ) . The global optimality and rate of convergence of such a class are even more challenging to analyze than that of REINFORCE . In particular , the policy evaluation step itself may converge to an undesirable stationary point or even diverge ( Tsitsiklis and Van Roy , 1997 ) , especially when it involves both nonlinear action-value function approximator , such as neural network , and temporal-difference update ( Sutton , 1988 ) . As a result , the estimated policy gradient may be biased , which possibly leads to divergence . Even if the algorithm converges to a stationary point , due to the nonconvexity of the expected total reward with respect to the policy as well as its parameter , the global optimality of such a stationary point remains unclear . The only exception is the linear-quadratic regulator ( LQR ) setting ( Fazel et al. , 2018 ; Malik et al. , 2018 ; Tu and Recht , 2018 ; Yang et al. , 2019a ; Bu et al. , 2019 ) , which is , however , more restrictive than the general MDP setting that possibly involves neural networks . To bridge the gap between practice and theory , we analyze neural policy gradient methods equipped with actor-critic schemes , where the actors and critics are represented by overparameterized twolayer neural networks . In detail , we study two settings , where the policy improvement steps are based on vanilla policy gradient and natural policy gradient , respectively . In both settings , the policy evaluation steps are based on the TD ( 0 ) algorithm ( Sutton , 1988 ) with independent sampling . In the first setting , we prove that neural vanilla policy gradient converges to a stationary point of the expected total reward at a 1/ √ T -rate in the expected squared norm of the policy gradient , where T is the number of policy improvement steps . Meanwhile , through a geometric characterization that relates the suboptimality of the stationary points to the representation power of the neural networks parameterizing the actor and critic , we establish the global optimality of all stationary points under mild regularity conditions . In the second setting , through the lens of Kullback-Leibler ( KL ) divergence regularization , we prove that neural natural policy gradient converges to a globally optimal policy at a 1/ √ T -rate in the expected total reward . In particular , a key to such global optimality and convergence guarantees is a notion of compatibility between the actor and critic , which connects the accuracy of policy evaluation steps with the efficacy of policy improvement steps . We show that such a notion of compatibility is ensured by using shared neural architectures and random initializations for both the actor and critic , which is often used as a practical heuristic ( Mnih et al. , 2016 ) . To our best knowledge , our analysis gives the first global optimality and convergence guarantees for neural policy gradient methods , which corroborate their significant empirical successes . Related Work . In contrast to the huge body of empirical literature on policy gradient methods , theoretical results on their convergence remain relatively scarce . In particular , Sutton et al . ( 2000 ) and Kakade ( 2002 ) analyze vanilla policy gradient ( REINFORCE ) and natural policy gradient with compatible action-value function approximators , respectively , which are further extended by Konda and Tsitsiklis ( 2000 ) ; Peters and Schaal ( 2008 ) ; Castro and Meir ( 2010 ) to incorporate actor-critic schemes . Most of this line of work only establishes the asymptotic convergence based on stochastic approximation techniques ( Kushner and Yin , 2003 ; Borkar , 2009 ) and requires the actor and critic to be parameterized by linear functions . Another line of work ( Papini et al. , 2018 ; Xu et al. , 2019a ; b ; Shen et al. , 2019 ; Karimi et al. , 2019 ; Zhang et al. , 2019 ) builds on the recent progress in nonconvex optimization to establish the nonasymptotic rates of convergence of REINFORCE ( Williams , 1992 ; Baxter and Bartlett , 2000 ; Sutton et al. , 2000 ) and its variants , but only to first- or second-order stationary points , which , however , lacks global optimality guarantees . Moreover , when actor-critic schemes are involved , due to the error of policy evaluation steps and its impact on policy improvement steps , the nonasymptotic rates of convergence of policy gradient methods , even to first- or second-order stationary points , remain rather open . Compared with the convergence of policy gradient methods , their global optimality is even less explored in terms of theory . Fazel et al . ( 2018 ) ; Malik et al . ( 2018 ) ; Tu and Recht ( 2018 ) ; Yang et al . ( 2019a ) ; Bu et al . ( 2019 ) prove that policy gradient methods converge to globally optimal policies in the LQR setting , which is more restrictive . In very recent work , Bhandari and Russo ( 2019 ) establish the global optimality of vanilla policy gradient ( REINFORCE ) in the general MDP setting . However , they require the policy class to be convex , which restricts its applicability to the tabular and LQR settings . In independent work , Agarwal et al . ( 2019 ) prove that vanilla policy gradient and natural policy gradient converge to globally optimal policies at 1/ √ T -rates in the tabular and linear settings . In the tabular setting , their rate of convergence of vanilla policy gradient depends on the size of the state space . In contrast , we focus on the nonlinear setting with the actor-critic scheme , where the actor and critic are parameterized by neural networks . It is worth mentioning that when such neural networks have linear activation functions , our analysis also covers the linear setting , which is , however , not our focus . In addition , Liu et al . ( 2019 ) analyze the proximal policy optimization ( PPO ) and trust region policy optimization ( TRPO ) algorithms ( Schulman et al. , 2015 ; 2017 ) , where the actors and critics are parameterized by neural networks , and establish their 1/ √ T - rates of convergence to globally optimal policies . However , they require solving a subproblem of policy improvement in the functional space using multiple stochastic gradient steps in the parameter space , whereas vanilla policy gradient and natural policy gradient only require a single stochastic ( natural ) gradient step in the parameter space , which makes the analysis even more challenging . There is also an emerging body of literature that analyzes the training and generalization error of deep supervised learning with overparameterized neural networks ( Daniely , 2017 ; Jacot et al. , 2018 ; Wu et al. , 2018 ; Allen-Zhu et al. , 2018a ; b ; Du et al. , 2018a ; b ; Zou et al. , 2018 ; Chizat and Bach , 2018 ; Jacot et al. , 2018 ; Li and Liang , 2018 ; Cao and Gu , 2019a ; b ; Arora et al. , 2019 ; Lee et al. , 2019 ) , especially when they are trained using stochastic gradient . See Fan et al . ( 2019 ) for a detailed survey . In comparison , our focus is on deep reinforcement learning with policy gradient methods . In particular , the policy evaluation steps are based on the TD ( 0 ) algorithm , which uses stochastic semigradient ( Sutton , 1988 ) rather than stochastic gradient . Moreover , the interplay between the actor and critic makes our analysis even more challenging than that of deep supervised learning . Notation . For distribution µ on Ω and p > 0 , we define ‖f ( · ) ‖µ , p = ( ∫ Ω |f |pdµ ) 1/p as the Lp ( µ ) norm of f . We define ‖f ( · ) ‖µ , ∞ = inf { C ≥ 0 : |f ( x ) | ≤ C for µ-almost every x } as the L∞ ( µ ) norm of f . We write ‖f‖µ , p for notational simplicity when the variable of f is clear from the context . We further denote by ‖ · ‖µ the L2 ( µ ) -norm for notational simplicity . For a vector φ ∈ Rn and p > 0 , we denote by ‖φ‖p the ` p-norm of φ . We denote by x = ( [ x ] > 1 , . . . , [ x ] > m ) > a vector in Rmd , where [ x ] i ∈ Rd is the i-th block of x for i ∈ [ m ] . 2 BACKGROUND . In this section , we introduce the background of reinforcement learning and policy gradient methods . Reinforcement Learning . A discounted Markov decision process ( MDP ) is defined by tuple ( S , A , P , ζ , r , γ ) . Here S and A are the sets of all possible states and actions , respectively . Meanwhile , P is the Markov transition kernel and r is the reward function , which is possibly stochastic . Specifically , when taking action a ∈ A at state s ∈ S , the agent receives reward r ( s , a ) and the environment transits into a new state according to transition probability P ( · | s , a ) . Meanwhile , ζ is the distribution of initial state S0 ∈ S and γ ∈ ( 0 , 1 ) is the discount factor . In addition , policy π ( a | s ) gives the probability of taking action a at state s. We denote the state- and action-value functions associated with π by V π : S → R and Qπ : S × A → R , which are defined respectively as V π ( s ) = ( 1− γ ) · E [ ∞∑ t=0 γt · r ( St , At ) ∣∣∣∣ S0 = s ] , ∀s ∈ S , ( 2.1 ) Qπ ( s , a ) = ( 1− γ ) · E [ ∞∑ t=0 γt · r ( St , At ) ∣∣∣∣ S0 = s , A0 = a ] , ∀ ( s , a ) ∈ S ×A , ( 2.2 ) where S0 ∼ ζ ( · ) , At ∼ π ( · |St ) , and St+1 ∼ P ( · |St , At ) for all t ≥ 0 . Also , we define the advantage function of policy π as the difference between Qπ and V π , i.e. , Aπ ( s , a ) = Qπ ( s , a ) − V π ( s ) . By the definitions in ( 2.1 ) and ( 2.2 ) , V π and Qπ are related via V π ( s ) = Eπ [ Qπ ( s , a ) ] = 〈Qπ ( s , · ) , π ( · | s ) 〉 , where 〈· , ·〉 is the inner product in R|A| . Here we write Ea∼π ( · | s ) [ Qπ ( s , a ) ] as Eπ [ Qπ ( s , a ) ] for notational simplicity . Note that policy π together with transition probability P induces a Markov chain over state space S. We denote by % π the stationary state distribution of the Markov chain induced by π . We further define ςπ ( s , a ) = π ( a | s ) · % π ( s ) as the stationary state-action distribution over S × A . Meanwhile , policy π induces a state visitation measure over S and a state-action visitation measure over S × A , which are denoted by νπ and σπ , respectively . Specifically , for all ( s , a ) ∈ S ×A , we define νπ ( s ) = ( 1− γ ) · ∞∑ t=0 γt · P ( St = s ) , σπ ( s , a ) = ( 1− γ ) · ∞∑ t=0 γt · P ( St = s , At = a ) , ( 2.3 ) where S0 ∼ ζ ( · ) , At ∼ π ( · |St ) , and St+1 ∼ P ( · |St , At ) for all t ≥ 0 . By definition , we have σπ ( s , a ) = π ( a | s ) · νπ ( s ) . We define the expected total reward function J ( π ) by J ( π ) = ( 1− γ ) · E [ ∞∑ t=0 γt · r ( St , At ) ] = Eζ [ V π ( s ) ] = Eσπ [ r ( s , a ) ] , ∀π , ( 2.4 ) where we write Eσπ [ r ( s , a ) ] = E ( s , a ) ∼σπ ( · , · ) [ r ( s , a ) ] for notational simplicity . The goal of reinforcement learning is to find the optimal policy that maximizes J ( π ) , which is denoted by π∗ . When state space S is large , a popular approach is to find the maximizer of J ( π ) over a class of parameterized policies { πθ : θ ∈ B } , where θ ∈ B is the parameter and B is the parameter space . In this case , we obtain the optimization problem maxθ∈B J ( πθ ) . Policy Gradient Methods . Policy gradient methods maximize J ( πθ ) using∇θJ ( πθ ) . These methods are based on the policy gradient theorem ( Sutton and Barto , 2018 ) , which states that ∇θJ ( πθ ) = Eσπθ [ Qπθ ( s , a ) · ∇θ log πθ ( a | s ) ] , ( 2.5 ) where σπθ is the state-action visitation measure defined in ( 2.3 ) . Based on ( 2.5 ) , ( vanilla ) policy gradient maximizes the expected total reward via gradient ascent . Specifically , we generate a sequence of policy parameters { θi } i∈ [ T ] via θi+1 ← θi + η · ∇θJ ( πθi ) , ( 2.6 ) where η > 0 is the learning rate . Meanwhile , natural policy gradient ( Kakade , 2002 ) utilizes natural gradient ascent ( Amari , 1998 ) , which is invariant to the parameterization of policies . Specifically , let F ( θ ) be the Fisher information matrix corresponding to policy πθ , which is given by F ( θ ) = Eσπθ [ ∇θ log πθ ( a | s ) [ ∇θ log πθ ( a | s ) ] > ] . ( 2.7 ) At each iteration , natural policy gradient performs θi+1 ← θi + η · F−1 ( θi ) · ∇θJ ( πθi ) , ( 2.8 ) where F−1 ( θi ) is the inverse of F ( θi ) and η is the learning rate . In practice , both Qπθ in ( 2.5 ) and F ( θ ) in ( 2.7 ) remain to be estimated , which yields approximations of the policy improvement steps in ( 2.6 ) and ( 2.8 ) .

Summary: This paper provides theoretical studies for neural policy gradient descents for reinforcement learning problems. The authors prove global optimality and rates of convergence of neural natural/vanilla policy gradient. Their results rely on the key factor for "compatibility" between the actor and critic. This is ensured by sharing neural architectures and random initializations across the actor and critic.