Understanding the Effect of Stochasticity in Policy Optimization

1 Introduction . Policy optimization is a central problem in reinforcement learning ( RL ) that provides a foundation for both policy-based and actor-critic RL methods . Until recently it had generally been assumed that methods based on following the policy gradient ( PG ) [ 1 ] could not be guaranteed to converge to globally optimal solutions , given that the policy value function is not concave . However , this assumption has been contradicted by recent findings that policy gradient methods can indeed prove to converge to global optima , at least in the tabular setting . In particular , the standard softmax PG method with a constant learning rate has been shown to converge to a globally optimal policy at a Θ ( 1/t ) rate for finite MDPs [ 2 ] , albeit with challenging problem and initialization dependent constants [ 3 , 4 ] . Several techniques have been developed to further improve standard PG and achieve better rates and constants . For example , adding entropy regularization has been shown to produce faster O ( e−c·t ) convergence ( c > 0 ) to the optimal regularized policy [ 2 , 5 , 6 ] . By exploiting natural geometries based on Bregman divergences , natural PG ( NPG ) or mirror descent ( MD ) have been shown to achieve better constants than standard PG [ 7 , 5 ] and faster O ( e−c·t ) rates , with [ 5 , 6 ] and without regularization [ 8 ] . Alternative policy parameterizations , such as the escort parameterization , have been shown to improve the constants achieved by softmax and yield faster plateau escaping [ 3 ] . ∗Correspondence to : Jincheng Mei and Csaba Szepesvári { jmei2 , szepesva } @ ualberta.ca 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . More recently , a geometry-aware normalized PG ( GNPG ) approach has been proposed to exploit the non-uniformity of the value function , achieving faster O ( e−c·t ) rates with improved constants [ 9 ] . A key observation is that each of these four techniques— ( i ) entropy regularization , ( ii ) NPG ( or MD ) , ( iii ) alternative policy parameterization , and ( iv ) GNPG—accelerates the convergence of standard softmax PG by better exploiting the geometry of the optimization landscape . In particular , entropy regularization makes the regularized objective behave more like a quadratic [ 2 , 5 , 6 ] , which significantly improves the near-linear character of the softmax policy value [ 2 ] . Natural PG ( or MD ) performs non-Euclidean updates in the parameter space , which is quite different from the Euclidean geometry characterizing standard softmax PG updates [ 7 , 5 , 8 ] . The escort policy parameterization induces an alternative policy-parameter relation [ 3 ] . GNPG exploits the non-uniform smoothness in the optimization landscape via a simple gradient normalization operation [ 9 ] . However , these advantages have only been established for the true gradient setting . A natural question therefore is whether geometry can also be exploited to accelerate convergence to global optimality in stochastic gradient settings . In this paper , we show that in a certain fundamental sense , the answer is no . That is , there exists a fundamental trade-off between leveraging geometry to accelerate convergence and overcoming the noise introduced by stochastic gradients ( possibly infinite ) ; in particular , no uninformed algorithm can improve the O ( 1/t ) convergence rate without incurring a positive probability of failure ( i.e . diverging or converging to a sub-optimal stationary point ) . The conditions used in vanilla stochastic gradient convergence analysis , i.e. , unbiased and variancebounded gradient estimator [ 10 ] , has been exploited to attempt to explain such a trade-off in policy gradients [ 11 , 6 ] . However , the bounded variance requires the sample policy to be bounded away from zero everywhere , which is impractical . Meanwhile , a variant of NPG can converge even with unbounded variance [ 12 ] . These gaps raise the question that if not the bounded variance , then what is the key factor to ensure the convergence of stochastic policy optimization algorithms ? Motivated by this question , we introduce the concept committal rate to characterize the update behaviors , which significantly affect whether convergence to a correct solution can be guaranteed in the stochastic on-policy setting . In particular , we make the following contributions . • First , we illustrate the anomaly that the preferability of policy optimization algorithms ( softmax PG vs. NPG and GNPG ) changes dramatically depending on whether true versus on-policy stochastic gradients are considered , and reveal the impracticality and unnecessity of a bounded variance requirement in Section 2 ; • Second , we introduce the concept of the committal rate in Section 3 to characterize the aggressiveness of an update , which provide us tools for analyzing the stochasticity effect in convergences ; • Third , we use the committal rate to study general stochastic policy optimization behaviors rigorously and reveal the inherent geometry-convergence trade-off in Section 4 ; • Finally , we explain the sensitivity to random initialization in practical policy optimization algorithms . From these results , we then develop an ensemble method that can achieve fast convergence to global optima with high probability in Section 5 . 2 Understanding Algorithm Preferability in On-line Policy Optimization . To illustrate the key aspects of policy optimization methods and their comparative preferability , it suffices to consider deterministic , single-state , finite-action Markov decision processes ( MDPs ) . The main results extend to general finite MDPs , but for clarity of exposition we restrict attention to one-state MDPs . A deterministic , single-state , finite-action MDP can be simply be specified by an action space is [ K ] : = { 1 , 2 , . . .K } and a K-dimensional reward vector r ∈ RK . The problem is to maximize the expected reward of a parametric policy πθ , max θ : [ K ] →R E a∼πθ ( · ) [ r ( a ) ] . ( 1 ) where πθ is parameterized by θ using the standard softmax transform , πθ ( a ) = exp { θ ( a ) } ∑ a′∈ [ K ] exp { θ ( a′ ) } , for all a ∈ [ K ] . ( 2 ) Without loss of generality , we assume there exists a unique optimal action a∗ = arg maxa∈ [ K ] r ( a ) , hence there exists a unique optimal deterministic policy π∗ such that π∗ > r = supθ∈RK π > θ r = r ( a ∗ ) . We make the following assumption on the reward . Assumption 1 ( Positive reward ) . r ( a ) ∈ ( 0 , 1 ] , ∀a ∈ [ K ] . 2.1 Exact Gradient Setting . It is known that Eq . ( 1 ) is a non-concave maximization over the policy parameter θ [ 2 ] . Nevertheless , it has recently become better understood how policy gradient ( PG ) methods still converge to global optima for Eq . ( 1 ) when exact gradients are used . To illustrate the main considerations , we focus on the following three representative algorithms that have recently been proved to achieve convergence to global optima but at different rates : softmax policy gradient ( PG ) , natural PG ( NPG ) , and geometryaware normalized PG ( GNPG ) , while similar conclusions can be drawn for other variants [ 12 , 13 ] . 2.1.1 Softmax PG . The standard softmax PG method is specified by the following update . Update 1 ( Softmax PG , true gradient ) . θt+1 ← θt + η · dπ > θtr dθt , where dπ > θ r dθ = ( diag ( πθ ) − πθπ > θ ) r , and thus dπ > θ r dθ ( a ) = πθ ( a ) · ( r ( a ) − π > θ r ) for all a ∈ [ K ] . As shown in Mei et al . [ 2 ] , the convergence of this update to a globally optimal policy , given exact gradients , can be established by considering the following non-uniform Łojasiewicz ( NŁ ) inequality , Lemma 1 ( NŁ , [ 2 ] ) . ∥∥∥dπ > θ rdθ ∥∥∥ 2 ≥ πθ ( a∗ ) · ( π∗ − πθ ) > r . By considering smoothness of π > θ r , Mei et al . [ 2 ] shows that the progress in each iteration of PG can be lower bounded by the squared norm of the gradient , ∥∥∥dπ > θtrdθt ∥∥∥22 , which leads to a O ( 1/t ) rate . Proposition 1 ( PG upper bound [ 2 ] ) . Using Update 1 with η = 2/5 , we have ( π∗−πθt ) > r ≤ 5/ ( c2·t ) for all t ≥ 1 , such that c = inft≥1 πθt ( a∗ ) > 0 is a constant that depends on r and θ1 , but it does not depend on the time t. In particular , if πθ1 ( a ) = 1/K ∀a then c ≥ 1/K . Proposition 2 ( PG lower bound [ 2 ] ) . For sufficiently large t ≥ 1 , Update 1 with η ∈ ( 0 , 1 ] exhibits ( π∗ − πθt ) > r ≥ ∆2/ ( 6 · t ) , where ∆ = r ( a∗ ) −maxa6=a∗ r ( a ) > 0 is the reward gap of r. Remark 1 . The constant dependence of PG follows a Ω ( 1/c ) lower bound for one-state MDPs [ 3 ] , while c can be exponentially small in terms of the number of states for general finite MDPs [ 4 ] . To summarize , using η ∈ O ( 1 ) , softmax PG achieves convergence to a global optima , but with a Θ ( 1/t ) rate that exhibits poor constant dependence . 2.1.2 Natural PG ( NPG ) . An alternative method , natural PG ( NPG ) [ 14 ] , provides the prototype for many practical policy optimization methods , such as TRPO and PPO [ 15 , 16 ] . NPG is based on the following update . Update 2 ( Natural PG ( NPG ) , true gradient ) . θt+1 ← θt + η · r. For softmax policies , it turns out that Update 2 is identical to mirror descent ( MD ) with a KullbackLeibler ( KL ) divergence . Therefore a standard MD analysis shows that Update 2 achieves convergence to a global optimum at a rate of O ( 1/t ) [ 7 ] . Very recently , work concurrent to this submission [ 8 ] has shown that Update 2 actually enjoys a much fasterO ( e−c·t ) rate . In fact , here too we can establish the same O ( e−c·t ) rate , but using a simpler argument based on the following variant of the NŁ inequality for natural gradients . These results are new to this paper . Due to space limitation , we postpone all the proofs to the appendix . Lemma 2 ( Natural NŁ inequality , continuous ) . 〈dπ > θ r dθ , r 〉 ≥ πθ ( a∗ ) ·∆ · ( π∗ − πθ ) > r. Lemma 3 ( Natural NŁ , discrete ) . Let π′ ( a ) : = π ( a ) ·e η·r ( a ) ∑ a′ π ( a ′ ) ·eη·r ( a′ ) , ∀ a ∈ [ K ] , where η > 0 . Then , ( π′ − π ) > r ≥ [ 1− 1 π ( a∗ ) · ( eη·∆ − 1 ) + 1 ] · ( π∗ − π ) > r. ( 3 ) In particular , by using a non-Euclidean update and analysis , the progress of each iteration of NPG can be lower bounded by the larger bound 〈dπ > θtr dθt , r 〉 instead of the weaker bound ∥∥∥dπ > θtrdθt ∥∥∥22 established for standard PG . Based on this inequality , one can easily establish a much fasterO ( e−c·t ) convergence to a globally optimal solution for NPG , making it far preferable to PG if true gradients are available . Theorem 1 ( NPG upper bound ) . Using Update 2 with any η > 0 , we have , for all t ≥ 1 , ( π∗ − πθt ) > r ≤ ( π∗ − πθ1 ) > r · e−c· ( t−1 ) , ( 4 ) where c : = log ( πθ1 ( a ∗ ) · ( eη·∆ − 1 ) + 1 ) > 0 for any η > 0 , and ∆ = r ( a∗ ) −maxa 6=a∗ r ( a ) > 0 .

Summary: This paper studies the convergence of various policy gradient methods. The authors introduce a novel conception, the committal rate, to characterize the convergence speed and convergence quality of policy gradient methods, which is defined as the speed of the policy on a particular action converges to 1. For a simple MDP with 1 state, the authors show that the committal rate can explain the phenomenon that faster convergent algorithms using true policy gradients can be dominated by slower ones in the stochastic setting.