Understanding the Effect of Stochasticity in Policy Optimization

1 Introduction . Policy optimization is a central problem in reinforcement learning ( RL ) that provides a foundation for both policy-based and actor-critic RL methods . Until recently it had generally been assumed that methods based on following the policy gradient ( PG ) [ 1 ] could not be guaranteed to converge to globally optimal solutions , given that the policy value function is not concave . However , this assumption has been contradicted by recent findings that policy gradient methods can indeed prove to converge to global optima , at least in the tabular setting . In particular , the standard softmax PG method with a constant learning rate has been shown to converge to a globally optimal policy at a Θ ( 1/t ) rate for finite MDPs [ 2 ] , albeit with challenging problem and initialization dependent constants [ 3 , 4 ] . Several techniques have been developed to further improve standard PG and achieve better rates and constants . For example , adding entropy regularization has been shown to produce faster O ( e−c·t ) convergence ( c > 0 ) to the optimal regularized policy [ 2 , 5 , 6 ] . By exploiting natural geometries based on Bregman divergences , natural PG ( NPG ) or mirror descent ( MD ) have been shown to achieve better constants than standard PG [ 7 , 5 ] and faster O ( e−c·t ) rates , with [ 5 , 6 ] and without regularization [ 8 ] . Alternative policy parameterizations , such as the escort parameterization , have been shown to improve the constants achieved by softmax and yield faster plateau escaping [ 3 ] . ∗Correspondence to : Jincheng Mei and Csaba Szepesvári { jmei2 , szepesva } @ ualberta.ca 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . More recently , a geometry-aware normalized PG ( GNPG ) approach has been proposed to exploit the non-uniformity of the value function , achieving faster O ( e−c·t ) rates with improved constants [ 9 ] . A key observation is that each of these four techniques— ( i ) entropy regularization , ( ii ) NPG ( or MD ) , ( iii ) alternative policy parameterization , and ( iv ) GNPG—accelerates the convergence of standard softmax PG by better exploiting the geometry of the optimization landscape . In particular , entropy regularization makes the regularized objective behave more like a quadratic [ 2 , 5 , 6 ] , which significantly improves the near-linear character of the softmax policy value [ 2 ] . Natural PG ( or MD ) performs non-Euclidean updates in the parameter space , which is quite different from the Euclidean geometry characterizing standard softmax PG updates [ 7 , 5 , 8 ] . The escort policy parameterization induces an alternative policy-parameter relation [ 3 ] . GNPG exploits the non-uniform smoothness in the optimization landscape via a simple gradient normalization operation [ 9 ] . However , these advantages have only been established for the true gradient setting . A natural question therefore is whether geometry can also be exploited to accelerate convergence to global optimality in stochastic gradient settings . In this paper , we show that in a certain fundamental sense , the answer is no . That is , there exists a fundamental trade-off between leveraging geometry to accelerate convergence and overcoming the noise introduced by stochastic gradients ( possibly infinite ) ; in particular , no uninformed algorithm can improve the O ( 1/t ) convergence rate without incurring a positive probability of failure ( i.e . diverging or converging to a sub-optimal stationary point ) . The conditions used in vanilla stochastic gradient convergence analysis , i.e. , unbiased and variancebounded gradient estimator [ 10 ] , has been exploited to attempt to explain such a trade-off in policy gradients [ 11 , 6 ] . However , the bounded variance requires the sample policy to be bounded away from zero everywhere , which is impractical . Meanwhile , a variant of NPG can converge even with unbounded variance [ 12 ] . These gaps raise the question that if not the bounded variance , then what is the key factor to ensure the convergence of stochastic policy optimization algorithms ? Motivated by this question , we introduce the concept committal rate to characterize the update behaviors , which significantly affect whether convergence to a correct solution can be guaranteed in the stochastic on-policy setting . In particular , we make the following contributions . • First , we illustrate the anomaly that the preferability of policy optimization algorithms ( softmax PG vs. NPG and GNPG ) changes dramatically depending on whether true versus on-policy stochastic gradients are considered , and reveal the impracticality and unnecessity of a bounded variance requirement in Section 2 ; • Second , we introduce the concept of the committal rate in Section 3 to characterize the aggressiveness of an update , which provide us tools for analyzing the stochasticity effect in convergences ; • Third , we use the committal rate to study general stochastic policy optimization behaviors rigorously and reveal the inherent geometry-convergence trade-off in Section 4 ; • Finally , we explain the sensitivity to random initialization in practical policy optimization algorithms . From these results , we then develop an ensemble method that can achieve fast convergence to global optima with high probability in Section 5 . 2 Understanding Algorithm Preferability in On-line Policy Optimization . To illustrate the key aspects of policy optimization methods and their comparative preferability , it suffices to consider deterministic , single-state , finite-action Markov decision processes ( MDPs ) . The main results extend to general finite MDPs , but for clarity of exposition we restrict attention to one-state MDPs . A deterministic , single-state , finite-action MDP can be simply be specified by an action space is [ K ] : = { 1 , 2 , . . .K } and a K-dimensional reward vector r ∈ RK . The problem is to maximize the expected reward of a parametric policy πθ , max θ : [ K ] →R E a∼πθ ( · ) [ r ( a ) ] . ( 1 ) where πθ is parameterized by θ using the standard softmax transform , πθ ( a ) = exp { θ ( a ) } ∑ a′∈ [ K ] exp { θ ( a′ ) } , for all a ∈ [ K ] . ( 2 ) Without loss of generality , we assume there exists a unique optimal action a∗ = arg maxa∈ [ K ] r ( a ) , hence there exists a unique optimal deterministic policy π∗ such that π∗ > r = supθ∈RK π > θ r = r ( a ∗ ) . We make the following assumption on the reward . Assumption 1 ( Positive reward ) . r ( a ) ∈ ( 0 , 1 ] , ∀a ∈ [ K ] . 2.1 Exact Gradient Setting . It is known that Eq . ( 1 ) is a non-concave maximization over the policy parameter θ [ 2 ] . Nevertheless , it has recently become better understood how policy gradient ( PG ) methods still converge to global optima for Eq . ( 1 ) when exact gradients are used . To illustrate the main considerations , we focus on the following three representative algorithms that have recently been proved to achieve convergence to global optima but at different rates : softmax policy gradient ( PG ) , natural PG ( NPG ) , and geometryaware normalized PG ( GNPG ) , while similar conclusions can be drawn for other variants [ 12 , 13 ] . 2.1.1 Softmax PG . The standard softmax PG method is specified by the following update . Update 1 ( Softmax PG , true gradient ) . θt+1 ← θt + η · dπ > θtr dθt , where dπ > θ r dθ = ( diag ( πθ ) − πθπ > θ ) r , and thus dπ > θ r dθ ( a ) = πθ ( a ) · ( r ( a ) − π > θ r ) for all a ∈ [ K ] . As shown in Mei et al . [ 2 ] , the convergence of this update to a globally optimal policy , given exact gradients , can be established by considering the following non-uniform Łojasiewicz ( NŁ ) inequality , Lemma 1 ( NŁ , [ 2 ] ) . ∥∥∥dπ > θ rdθ ∥∥∥ 2 ≥ πθ ( a∗ ) · ( π∗ − πθ ) > r . By considering smoothness of π > θ r , Mei et al . [ 2 ] shows that the progress in each iteration of PG can be lower bounded by the squared norm of the gradient , ∥∥∥dπ > θtrdθt ∥∥∥22 , which leads to a O ( 1/t ) rate . Proposition 1 ( PG upper bound [ 2 ] ) . Using Update 1 with η = 2/5 , we have ( π∗−πθt ) > r ≤ 5/ ( c2·t ) for all t ≥ 1 , such that c = inft≥1 πθt ( a∗ ) > 0 is a constant that depends on r and θ1 , but it does not depend on the time t. In particular , if πθ1 ( a ) = 1/K ∀a then c ≥ 1/K . Proposition 2 ( PG lower bound [ 2 ] ) . For sufficiently large t ≥ 1 , Update 1 with η ∈ ( 0 , 1 ] exhibits ( π∗ − πθt ) > r ≥ ∆2/ ( 6 · t ) , where ∆ = r ( a∗ ) −maxa6=a∗ r ( a ) > 0 is the reward gap of r. Remark 1 . The constant dependence of PG follows a Ω ( 1/c ) lower bound for one-state MDPs [ 3 ] , while c can be exponentially small in terms of the number of states for general finite MDPs [ 4 ] . To summarize , using η ∈ O ( 1 ) , softmax PG achieves convergence to a global optima , but with a Θ ( 1/t ) rate that exhibits poor constant dependence . 2.1.2 Natural PG ( NPG ) . An alternative method , natural PG ( NPG ) [ 14 ] , provides the prototype for many practical policy optimization methods , such as TRPO and PPO [ 15 , 16 ] . NPG is based on the following update . Update 2 ( Natural PG ( NPG ) , true gradient ) . θt+1 ← θt + η · r. For softmax policies , it turns out that Update 2 is identical to mirror descent ( MD ) with a KullbackLeibler ( KL ) divergence . Therefore a standard MD analysis shows that Update 2 achieves convergence to a global optimum at a rate of O ( 1/t ) [ 7 ] . Very recently , work concurrent to this submission [ 8 ] has shown that Update 2 actually enjoys a much fasterO ( e−c·t ) rate . In fact , here too we can establish the same O ( e−c·t ) rate , but using a simpler argument based on the following variant of the NŁ inequality for natural gradients . These results are new to this paper . Due to space limitation , we postpone all the proofs to the appendix . Lemma 2 ( Natural NŁ inequality , continuous ) . 〈dπ > θ r dθ , r 〉 ≥ πθ ( a∗ ) ·∆ · ( π∗ − πθ ) > r. Lemma 3 ( Natural NŁ , discrete ) . Let π′ ( a ) : = π ( a ) ·e η·r ( a ) ∑ a′ π ( a ′ ) ·eη·r ( a′ ) , ∀ a ∈ [ K ] , where η > 0 . Then , ( π′ − π ) > r ≥ [ 1− 1 π ( a∗ ) · ( eη·∆ − 1 ) + 1 ] · ( π∗ − π ) > r. ( 3 ) In particular , by using a non-Euclidean update and analysis , the progress of each iteration of NPG can be lower bounded by the larger bound 〈dπ > θtr dθt , r 〉 instead of the weaker bound ∥∥∥dπ > θtrdθt ∥∥∥22 established for standard PG . Based on this inequality , one can easily establish a much fasterO ( e−c·t ) convergence to a globally optimal solution for NPG , making it far preferable to PG if true gradients are available . Theorem 1 ( NPG upper bound ) . Using Update 2 with any η > 0 , we have , for all t ≥ 1 , ( π∗ − πθt ) > r ≤ ( π∗ − πθ1 ) > r · e−c· ( t−1 ) , ( 4 ) where c : = log ( πθ1 ( a ∗ ) · ( eη·∆ − 1 ) + 1 ) > 0 for any η > 0 , and ∆ = r ( a∗ ) −maxa 6=a∗ r ( a ) > 0 .

Summary: This paper aims to answer a fundamental question: what is the key factor that ensures the convergence of stochastic policy gradient (PG) methods? This question is built on the fact that bounded variance appears to be only a sufficient condition for convergence of the PG methods. To begin with, the paper starts by summarizing the existing convergence results of three PG methods, namely Softmax PG, Natural PG (NPG), and Geometry-aware Normalized PG (GNPG), in both the exact and stochastic gradient settings. Then, the paper proposes the notion of “committal rate” as an indicator for characterizing the convergence behavior of the PG methods. Accordingly, for one-state MDPs, the authors show that $\kappa\leq 1$ for all sub-optimal actions serves as a necessary condition for ensuring convergence as well as provide the committal rates of the three PG methods.