Title: Ordering-based Conditions for Global Convergence of Policy Gradient Methods
Venue: NeurIPS 2023
Accepted: yes
Keywords: reinforcement learning

policy gradient

policy optimization

function approximation

global convergence
Abstract: We prove that, for finite-arm bandits with linear function approximation, the global convergence of policy gradient (PG) methods depends on inter-related properties between the policy update and the representation. textcolor{blue}{First}, we establish a few key observations that frame the study: \textbf{(i)} Global convergence can be achieved under linear function approximation without policy or reward realizability, both for the standard Softmax PG and natural policy gradient (NPG). \textbf{(ii)} Approximation error is not a key quantity for characterizing global convergence in either algorithm. \textbf{(iii)} The conditions on the representation that imply global convergence are different between these two algorithms. Overall, these observations call into question approximation error as an appropriate quantity for characterizing the global convergence of PG methods under linear function approximation. \textcolor{blue}{Second}, motivated by these observations, we establish new general results: \textbf{(i)} NPG with linear function approximation achieves global convergence \emph{if and only if} the projection of the reward  onto the representable space preserves the optimal action's rank, a quantity that is not strongly related to approximation error. \textbf{(ii)} The global convergence of Softmax PG occurs if the representation satisfies a non-domination condition and can preserve the ranking of rewards, which goes well beyond policy or reward realizability. We provide experimental results to support these theoretical findings.

Ordering-based Conditions for Global Convergence of Policy Gradient Methods

Abstract

We prove that, for finite-arm bandits with linear function approximation, the global convergence of policy gradient (PG) methods depends on inter-related properties between the policy update and the representation. First, we establish a few key observations that frame the study: (i) Global convergence can be achieved under linear function approximation without policy or reward realizability, both for the standard Softmax PG and natural policy gradient (NPG). (ii) Approximation error is not a key quantity for characterizing global convergence in either algorithm. (iii) The conditions on the representation that imply global convergence are different between these two algorithms. Overall, these observations call into question approximation error as an appropriate quantity for characterizing the global convergence of PG methods under linear function approximation. Second, motivated by these observations, we establish new general results: (i) NPG with linear function approximation achieves global convergence if and only if the projection of the reward onto the representable space preserves the optimal action's rank, a quantity that is not strongly related to approximation error. (ii) The global convergence of Softmax PG occurs if the representation satisfies a non-domination condition and can preserve the ranking of rewards, which goes well beyond policy or reward realizability. We provide experimental results to support these theoretical findings.

1 Introduction

Policy gradient (PG) is a foundational concept in reinforcement learning (RL), centrally used in both policy-based and actor-critic methods [25]. Despite the non-convexity of the policy optimization objective [4], global convergence of PG methods has been recently established in the tabular case for standard configurations such as the softmax parameterization [4, 22] and stochastic on-policy sampling [20]. In practice, when an RL agent is faced with a problem with large state and/or action spaces, function approximation is needed to generalize across related states and actions. The behavior of PG methods in these settings is relatively under-explored. In this paper, we study this question for the case of linear function approximation, and establish a surprising result that the classical Softmax PG method converges whenever there exists an adequate linear function that ranks actions in the same order as the ground-truth reward function.

Understanding the behavior of PG methods under function approximation is crucial for describing the behavior of RL in practice, since one rarely faces domains small enough to explicitly enumerate over states and actions in parameterizing the policy. It is well known that, standard Softmax PG converges to stationary points if a 'compatible' function approximation is used [25]; i.e., one that is able to exactly represent policy value functions. However, when exact policy values are non-realizable, 'approximation error' is typically considered to be the key quantity for characterizing how well a function approximation captures relevant problem quantities, including transition dynamics, rewards and policy values. This paper shows that such an approximation error perspective is overly demanding when attempting to characterize the conditions that lead to global convergence of PG methods.

Using the concept of approximation error, global convergence results for PG methods have been recently established in an additive form,

implying that if the approximation error is small, a diminishing optimization error implies a small sub-optimality gap. A representative result is the global convergence of natural policy gradient (NPG) [4, Table 2], where the optimization error will diminish as the algorithm updates. There have also been global convergence results for other PG variants under linear function approximation that follow a similar approximation error analysis [3, 8, 10, 28, 5, 1, 2]. However, an additive bound like Eq. (1) has the inherent weakness that the approximation error will never be zero if the function approximation is not able to perfectly represent the desired quantities. This prevents such a strategy from establishing global convergence in cases where the approximation error is non-zero but a PG method still reaches the best representable solution.

Therefore, in spite of this recent progress, using approximation error in PG global convergence with function approximations has left two major gaps in the literature. First, it has not been investigated whether small approximation error is necessary to achieve convergence to an optimal representable policy [4], diverting attention from feature designs that achieve useful properties beyond small approximation error. Second, it is not clear if standard Softmax PG (other than NPG) converges globally under small approximation errors. In particular, NPG contains a least squares regression step [4, Eq. (17)] that can be naturally characterized with an approximation error quantity. However, standard Softmax PG does not have such a projection step [25], and the results in [4] do not apply to this update. Whether standard Softmax PG can achieve global convergence with even linearly realizable rewards (zero approximation error) is still an open problem.

In this paper, we address the above questions and contribute the following results. First, we provide negative answers to questions on the role of approximation error in determining global convergence of PG methods:

(i) Global convergence can be achieved under linear function approximation with non-zero approximation error, for both the standard Softmax PG and natural policy gradient (NPG) updates.

(ii) Approximation error is not a key quantity for characterizing global convergence in either case.

(iii) The conditions that imply global convergence are different between these two algorithms.

Second, these results lead us to question whether approximation error is an appropriate quantity to consider the global convergence of PG methods under linear function approximation. We establish new general results that characterize the conditions for global convergence of PG methods:

(i) NPG with log-linear function approximation achieves global convergence if and only if the projection of the reward onto the representable space preserves the optimal action's rank. This result significantly extends previous results that use approximation error in the analysis [4, 3], since preserving the rank of the optimal action is not strongly related to approximation error (except in the realizable limit).

(ii) We show that the global convergence of Softmax PG follows if the representation satisfies a non-domination condition and can preserve the ranking of rewards, which goes well beyond policy or reward realizability. As a byproduct, we resolve an open question by showing that even for linearly realizable reward function, Softmax PG cannot always converge to globally optimal policies when the non-domination condition for representation is violated.

2 Settings

We study the policy optimization problem under one state with K actions. Given a reward vector r ∈ R K , the problem is to find a parametric policy π θ to maximize the expected reward,

where θ ∈ R d with d < K is the parameter, and π θ = softmax( Xθ ) is called a 'log-linear policy' [4, 28] such that for all action a ∈ [ K ] := { 1 , 2 , . . . , K } ,

where X ∈ R K × d is the feature matrix with full column rank d < K . There are two major difficulties with the policy optimization problem. First, Eq. (2) is a non-concave maximization w.r.t. θ , due to the softmax transform [22, Proposition 1]. Second, the policy and reward can be unrealizable, in the sense that the parametric log-linear policy π θ = softmax( Xθ ) cannot well approximate every policy π in the K -dimensional probability simplex, and the score Xθ ∈ R K cannot well approximate the true mean reward r ∈ R K . Such limitations arise in the linear function approximation case because π θ and Xθ are restricted to low-dimensional manifolds via θ ∈ R d for d < K .

To solve Eq. (2), we consider the standard Softmax PG [25] and NPG [13, 4] methods, shown in Algorithms 1 and 2. Softmax PG is an instance of gradient ascent, obtained by the chain rule,

On the other hand, NPG conducts updates using least squares regression (i.e., projection),

As representative policy-based methods, in their general forms, Softmax PG and NPG lay the foundation for widely used RL methods, including REINFORCE [26], actor-critic [16, 7, 12], TRPO and PPO [23, 24]. The above Eqs. (4) and (5) are their updates applied to the one-state setting.

To understand the difficulty of the optimization problem in Eq. (2), it is helpful to consider previous work that has analyzed the convergence of PG methods.

In the tabular setting, where d = K , X = Id , and π θ = softmax( θ ) with θ ∈ R K , both the rewards and optimal policy can be arbitrarily well approximated. In this case it is known that NPG enjoys a O (1 /t ) global convergence rate [4, Table 1], which has been recently improved to O ( e -c · t ) [14, 20, 17, 27]. For the case of function approximation, such results have subsequently been extended to log-linear policies, where approximation error is used to characterize the projection step of Eq. (5) [4, 28]. In particular, NPG achieves the following sub-optimality gap for all t ≥ 1 [4, Table 2], √

where c 1 and c 2 are problem specific constants, π ∗ is the globally optimal policy, π θ t is produced by NPG, and glyph[epsilon1] approx is the approximation error, i.e., the minimum error with which the policy values can be approximated using the features [4, Table 2]. The 'optimization error' term c 1 / √ t in Eq. (6) has since been improved to O ( e -c 3 · t ) with c 3 > 0 in [28, 5]. Note that if glyph[epsilon1] approx > 0 then Eq. (6) is insufficient for establishing π glyph[latticetop] θ t r → r ( a ∗ ) := max a ∈ [ K ] r ( a ) as t →∞ even when such global convergence is achieved.

The understanding for the standard Softmax PG is even less clear. In the tabular case, it is known that Softmax PG achieves global convergence asymptotically, i.e., π glyph[latticetop] θ t r → r ( a ∗ ) as t →∞ [4], with an O (1 /t ) rate of convergence that exhibits undesirable problem and initialization dependent constants

[21, 18]. Directly extending this global convergence result to the case of function approximation, i.e., log-linear policies, is impossible without any additional assumptions on the features, since there can be exponentially many sub-optimal local maxima in the worst case [9]. In fact, even with linearly realizable rewards (zero approximation error), whether standard Softmax PG achieves global convergence still remains unsolved [4]. One intuitive reason why this is a difficult result to establish is that standard Softmax PG uses the gradient Eq. (4) rather than projection (regression) to perform updates, which is less directly connected to the concept of approximation error.

3 The Limitations of Approximation Error in Characterizing Convergence

It is known that there exist representations X ∈ R K × d with d < K and r ∈ R K that create exponentially many sub-optimal local maxima in Eq. (2) [9, Theorem 1], which makes it impossible to ensure global convergence of PG methods without imposing any structure on the function approximation. Before identifying specific conditions that ensure global convergence, we first explain how approximation error cannot be a useful structural measure for this purpose, by demonstrating that zero approximation error is not a necessary condition for global convergence, and illustrating problem instances with comparable approximation error that render starkly different convergence behaviors across different PG methods. Specifically, we illustrate these points with a set of concrete scenarios, each with 4 actions and 2 -dimensional feature vectors describing each action. Since d < K , not every policy can be expressed in these representations, hence the problem instances are unrealizable.

3.1 Global Convergence is Achievable with Non-zero Approximation Error

The results of [9, Theorem 1] do not imply that sub-optimal local maxima always appear, as shown in the following.

Note that the approximation error is larger than any sub-optimality gap, i.e., for any policy π ,

hence the bound in Eq. (6) does not imply global convergence for NPG in this example. Yet, despite the non-zero approximation error and the inability of existing results including Eq. (6) to establish global convergence on Example 1, both Algorithms 1 and 2 can be shown to reach a global maximum.

Proposition 1. Denote a ∗ := arg max a ∈ [ K ] r ( a ) . With constant η > 0 and any initialization θ 1 ∈ R d , both Algorithms 1 and 2 guarantee π glyph[latticetop] θ t r → r ( a ∗ ) as t →∞ on Example 1.

All proofs can be found in the appendix due to space limits. The fact that Softmax PG achieves global convergence in Example 1 is much harder to establish than for NPG, since Eq. (4) involves a complex non-linearity given the presence of the softmax, unlike the linear least squares Eq. (5) used in NPG. To illustrate the intuition behind Proposition 1 we use a visualization of the optimization landscape.

Visualization. A visualization of the optimization landscape of Example 1 is shown in Figure 1(a). The bottom two-dimensional plane is the parameter space R d where d = 2 . For each θ ∈ R d , we calculate π θ using Eq. (3) and π glyph[latticetop] θ r using Eq. (2), and use π glyph[latticetop] θ r as the vertical axis value of θ .

PG

PG

PG

NPG

NPG

NPG

9

6

6

8

8

8

Tr

To

7

r

7

7

7

6

-20

6

-20

6

-20

-20

0

-20

0

-20

0

0

20

20

0

20

20

0

20

20

(a)Example 1.

(b)

Example 2.

(c)

Example 3.

To verify Proposition 1, we run Softmax PG and NPG on Example 1 with the same θ 1 = (6 , 8) glyph[latticetop] ∈ R 2 . In Figure 1(a), the optimization trajectories show 85 iterations of NPG and 8 . 5 × 10 6 iterations of

Softmax PG, both with learning rate η = 0 . 2 . It can be clearly seen that both Softmax PG and NPG eventually achieve expected reward π glyph[latticetop] θ t r → 9 = r ( a ∗ ) , demonstrating global convergence (Figure 3(c) later shows that the sub-optimality gap ( π ∗ -π θ t ) glyph[latticetop] r approaches 0 ).

In summary, Example 1 shows that both Softmax PG and NPG are able to achieve global convergence on unrealizable problem instances with non-zero approximation error. This raises the question:

Is non-zero approximation error useful for characterizing global convergence?

3.2 Global Convergence is Irrelevant to Non-zero Approximation Error

We answer the above question negatively. By comparing alternative problem instances with similar approximation errors but different convergence behaviors, we illustrate how approximation error is not able to distinguish between scenarios where global versus local convergence is obtained.

The only difference between Examples 1 and 2 is that the second and third columns of X glyph[latticetop] have been exchanged. The approximation error remains similar to that of Example 1. Using the upper bound of Eq. (6), one might therefore expect similar sub-optimality gaps ( π ∗ -π θ t ) glyph[latticetop] r to be demonstrated by the algorithms, since the r.h.s. contains similar approximation errors. However, as shown in Figure 1(b), using the same initialization and learning rate, Softmax PG obtains π glyph[latticetop] θ t r → 8 = r (2) < r ( a ∗ ) as it converges to a sub-optimal deterministic policy, while NPG continues to succeed.

Lest one believe that NPG is globally convergent, the following example, where the first and second columns of X glyph[latticetop] are swapped, illustrate an analogous failure for NPG but not Softmax PG.

Here again the approximation error is close to that of Example 1. Yet, Figure 1(c) shows that NPG achieves π glyph[latticetop] θ t r → 8 < r ( a ∗ ) as it converges to a sub-optimal solution, while Softmax PG succeeds.

In summary, the Examples 1, 2 and 3 all have similar approximation errors, yet Softmax PG achieves global convergence on Example 1 but reaches a bad local maxima on Example 2, while NPG succeeds on Example 1 and fails on Example 3. Note that these examples can be re-scaled to have exactly the same approximation errors while demonstrating the same convergence behavior of the algorithms. From these findings we conclude that, if there is any quantity that can predict whether global versus local convergence is obtained by Softmax PG or NPG, that the quantity cannot be approximation error alone. This motivates to investigate the question: what is the right quantity to characterize global convergence for unrealizable problems?

3.3 Global Convergence Characterization is Algorithm Dependent

We make one more key point. From Figure 1(b) and Figure 1(c), NPG achieves global convergence on Example 2 but fails on Example 3, while, conversely, Softmax PG succeeds on Example 3 and fails on Example 2. This difference indicates that whatever condition characterizes global convergence, it must be algorithm dependent , even for the closely related algorithms Softmax PG and NPG. Therefore, one has to study the conditions for Softmax PG and NPG respectively (rather than one condition for both algorithms), which motivates the refined question:

What conditions characterize global convergence of Softmax PG and NPG in unrealizable problems?

4 New Characterizations of Global Convergence for PG Methods

From these observations, it is clear that whatever quantity characterizes the global convergence of PG methods, it cannot be based solely on approximation error and it must be algorithm dependent. Therefore, we study distinct global convergence conditions for Softmax PG and NPG respectively.

4.1 Reward Order Preservation with Adequate Features is Sufficient for PG Convergence

We now investigate a global convergence condition for Softmax PG under log-linear policies.

Intuition. Consider Example 1, where Softmax PG achieves global convergence. From the landscape shown in Figure 1(a), there appears to be a monotonic path from any initialization point that allows gradient ascent to reach the optimal plateau with reward r ( a ∗ ) = 9 . Intuitively, this arises because the actions' rewards seem to be nicely 'ordered'. For example, starting from θ 1 = (6 , 8) glyph[latticetop] ∈ R d such that π glyph[latticetop] θ 1 r ≈ 6 , Softmax PG is able to improve its expected reward eventually to π glyph[latticetop] θ t r ≈ 7 , since there exists a sub-optimal plateau with a higher reward 7 right beside the lowest plateau with reward 6 . Next, Softmax PG continues to improve its expected reward eventually to π glyph[latticetop] θ t r ≈ 8 by 'climbing' toward another neighboring plateau with a higher reward. Finally, this process ends with Softmax PG successfully arriving at the optimal plateau with reward r ( a ∗ ) = 9 .

By contrast, in Example 2, as shown in Figure 1(b), Softmax PG gets stuck on a bad plateau with a local maximum reward of 8 . Visually, Softmax PG stops improving its expected reward on this sub-optimal plateau, because it is 'surrounded' by two lower plateaus with rewards 6 and 7 , which breaks the nice 'ordering' of the expected reward landscape and traps the gradient ascent trajectory on a sub-optimal plateau from which there is no monotonic ascent to global optimality.

Verifying reward order preservation. Based on the above intuition and observations, we conjecture that the ordering structure between the different rewards is a key property behind the global convergence of Softmax PG. We can verify this conjecture in each of the Examples 1 to 3 by determining whether the feature matrix X ∈ R K × d allows the same action ordering as the reward vector r ∈ R K to be realized. For Example 1, note that with w = ( -1 , -1) glyph[latticetop] ∈ R d , we have

which preserves the ordering of r ∈ R K , such that for all i, j ∈ [ K ] , r ( i ) > r ( j ) if and only if r ′ ( i ) > r ′ ( j ) . Similarly, for Example 3, if we let w = ( -3 , -1) glyph[latticetop] then we have r ′ := Xw = (3 , 2 , -1 , -6) glyph[latticetop] , which also preserves the order of r over actions. Softmax PG converges to a globally optimal reward in both of these examples.

By contrast, for Example 2, it is impossible to find any w ∈ R d such that Xw preserves the order of the rewards r . To see why, consider any w = ( w (1) , w (2)) glyph[latticetop] and note that

To preserve the reward order, we require both -2 · w (2) > w (2) (which would imply w (2) < 0 ) and -w (1) > 2 · w (1) (which would imply w (1) < 0 ), but these two conditions imply w (2) < 0 < -w (1) , which must reverse the order of the second and third actions. This is an example where PG can fail to reach a global optimum.

Main Softmax PG result. We formalize the above intuition by proving the following main result, which establishes that reward order preservation with adequate representations is a sufficient condition for the global convergence of Softmax PG under log-linear function approximation.

glyph[negationslash]

Theorem 1 (Reward order preservation, non-domination features) . Given any reward r ∈ R K and feature matrix X ∈ R K × d . Denote x i ∈ R d as the i -th row vector of X . If (i) x glyph[latticetop] i x i > x glyph[latticetop] i x j for all j = i , and (ii) there exists at least one w ∈ R d , s.t., r ′ := Xw preserves the order of r , i.e., for all i, j ∈ [ K ] , r ( i ) > r ( j ) if and only if r ′ ( i ) > r ′ ( j ) , then for any initialization θ 1 ∈ R d , Algorithm 1 with a constant learning rate η > 0 achieves global convergence of π glyph[latticetop] θ t r → r ( a ∗ ) as t →∞ .

A few remarks about this theorem are in order.

Examples 1 to 3 all satisfy the non-domination condition (i) on X , and their differences lie in satisfying reward order preservation or not. However, the following example shows that if the condition (i) on X is removed, then global convergence is not always achievable for even linearly realizable rewards (with zero approximation error).

where w = ( -1 , -2) glyph[latticetop] ∈ R d . With initialization θ 1 = ( -ln 2 , ln 2) glyph[latticetop] , Algorithm 1 does not achieve global convergence, i.e., π θ t (1) glyph[negationslash]→ 1 as t →∞ .

Generalization of tabular and linear realizability. When d = K and X = Id , i.e., the softmax tabular parameterization π θ = softmax( θ ) , it is always true that Xr = r preserves the order of r . Consequently, Theorem 1 recovers the global convergence result for PG in the softmax tabular setting [4, 22] as a special case. More generally, for non-domination features, when the reward is linearly realizable, such that Xw = r for some w ∈ R d , the global convergence of Softmax PG also follows from Theorem 1, since r preserves its own order when the approximation error is zero.

glyph[negationslash]

Corollary 1 (Linearly realizable rewards, non-domination features) . Given any reward r ∈ R K and feature matrix X ∈ R K × d . Denote x i ∈ R d as i -th row vector of X . If (i) x glyph[latticetop] i x i > x glyph[latticetop] i x j for all j = i , and (ii) there exists w ∈ R d , s.t., Xw = r , then for any initialization θ 1 ∈ R d , Algorithm 1 with a constant learning rate η > 0 achieves global convergence of π glyph[latticetop] θ t r → r ( a ∗ ) as t →∞ .

It is worth mentioning that Proposition 2 and Corollary 1 together answer a question which still remain unsolved in PG literature [4]: with linearly realizable rewards (zero approximation error), whether standard Softmax PG achieves global convergence? Proposition 2 shows that linearly realizable reward on its own is not enough to guarantee global convergence, while Corollary 1 shows that with adequate features, linearly realizable reward implies global convergence. Note that the NPG global convergence result in [4], such as Eq. (6), does not apply to standard Softmax PG.

Ordering does not determine approximation. As already illustrated in Section 3, approximation error is not adequate for capturing the global convergence of Softmax PG. It is important to emphasize that the existence of an order preserving reward r ′ is very different from having a small approximation error. When the approximation error is zero, then an order preserving reward (equal to r ) always exists. However, in general, r ′ can take very different values than r , and hence have a very large approximation error, yet still enable global convergence as shown in Examples 1 and 3.

Proof idea. The idea behind the proof of the main theorem consists of three parts. We provide a sketch of the proof here; the full proof is given in Appendix A. First , starting from any initialization θ t ∈ R d , Algorithm 1 guarantees that π θ t will approach a (generalized) one-hot policy as t →∞ . To see why, first note that π glyph[latticetop] θ r is β -smooth over θ ∈ R d with some β > 0 (Lemma 3 in Appendix B), since the softmax transform is smooth [4, 22] and the feature matrix X has bounded values. This implies that using a sufficiently small constant learning rate 0 < η ≤ 2 /β we obtain,

Note that π glyph[latticetop] θ r is upper bounded by r ( a ∗ ) . According to the monotone convergence, π glyph[latticetop] θ t r → c ≤ r ( a ∗ ) as t →∞ . This fact combined with Eq. (10) implies ∥ ∥ ∥ d π glyph[latticetop] θ t r dθ t ∥ ∥ ∥ 2 → 0 as t →∞ . Next, a special co-variance structure of softmax PG (Lemma 4) shows that ∥ ∥ ∥ d π glyph[latticetop] θ t r dθ t ∥ ∥ ∥ 2 → 0 implies that ‖ θ t ‖ 2 →∞ and π θ t approaches a (generalized) one-hot policy as t →∞ .

glyph[negationslash]

Lemma 1. Under the same conditions as Theorem 1, and r ( i ) = r ( j ) for all i = j (unique action reward), Algorithm 1 assures ‖ θ t ‖ 2 →∞ and π θ t ( i ) → 1 for an action i ∈ [ K ] as t →∞ .

glyph[negationslash]

Remark 1. Removing the unique action reward condition in Lemma 1 makes π θ t approach a generalized one-hot policy (rather than a strict one-hot in Lemma 1) as t →∞ as a result.

According to Lemma 1, θ t grows unboundedly. Intuitively, this can be seen in Figure 1(a), where there are no stationary points in a finite region.

Second , for any vector r ′ that preserves the order of r , we establish the following key lemma.

Lemma 2 (Non-negative covariance of order preservation) . If r ′ ∈ R K preserves the order of r ∈ R K , i.e., for all i, j ∈ [ K ] , r ( i ) > r ( j ) iff r ′ ( i ) > r ′ ( j ) , then for any policy π ∈ ∆( K ) ,

Now consider the direction w ∈ R d such that r ′ := Xw preserves the order of r . We have,

Third , take a sub-optimal action i ∈ [ K ] with r ( i ) < r ( a ∗ ) , and we show that the assumption π θ t ( i ) → 1 as t →∞ leads to a contradiction.

To that end, first observe that this assumption implies that for all large enough time t ≥ 1 ,

glyph[negationslash]

which means that the sub-optimal action i ∈ [ K ] always has the largest score (since its probability π θ t ( i ) → 1 is always the largest). Moreover, differences between actions' scores are unbounded, due to π θ t ( i ) π θ t ( j ) = exp { [ Xθ t ]( i ) -[ Xθ t ]( j ) } →∞ for all other actions j = i .

glyph[negationslash]

Consider Example 1 for illustration. The top view of Figure 1(a) is shown in Figure 2(a). Take i = 2 and r ( i ) = 8 , and assume θ t ‖ θ t ‖ 2 stays in the green (sub-optimal) region of Figure 2(a), excluding its boundaries. This green region is partitioned in Figure 2(b), where the dark sub-region contains v 2 ∈ R d such that [ Xv 2 ]( a ∗ ) is the second largest component among all a = 2 , and the light sub-region is the remaining.

The argument is completed by addressing the two cases: (i) If θ t ‖ θ t ‖ 2 stays in the dark sub-

0

-1

1

-1

1

region where v 1 ∈ R d belongs to, then π glyph[latticetop] θ t r > r ( i ) = 8 must occur in finite t < ∞ , implying π θ t ( i ) glyph[negationslash]→ 1 , contradicting the assumption. Intuitively, the contradiction occurs because the dark sub-region is closer to a higher plateau with reward 9 , and scaling up θ t 's magnitude in this sub-region eventually ensures π glyph[latticetop] θ t r > r ( i ) = 8 . (ii) If θ t ‖ θ t ‖ 2 stays in the light sub-region which contains v 2 ∈ R d , then w glyph[latticetop] θ t > u glyph[latticetop] θ t must occur in finite time t < ∞ , implying that θ t ‖ θ t ‖ 2 will enter the dark subregion, reducing to the first case. This argument depends on Eq. (12) and a key observation showing that u glyph[latticetop] θ t +1 < u glyph[latticetop] θ t , where u is a 'worse' direction such that [ Xu ]( a -) = max a ∈ [ K ] [ Xu ]( a ) for some a -∈ [ K ] with r ( a -) < r ( i ) .

To summarize, θ t ‖ θ t ‖ 2 cannot always stay in the green sub-optimal region in Figure 2(a), which implies that θ t ‖ θ t ‖ 2 must eventually enter the optimal region that contains w and stay in that region. By Lemma 1 we then obtain π θ t ( a ∗ ) → 1 and π glyph[latticetop] θ t r → r ( a ∗ ) as t →∞ (see appendix).

4.2 Optimal Action Preservation is Necessary and Sufficient for NPG Convergence

Next, we investigate the global convergence conditions for NPG under log-linear policies. Unlike Softmax PG, the key property for determining global convergence of NPG is whether the projection of the rewards r onto the feature representation X preserves the top ranking of the optimal action.

Intuition and demonstration. First consider Example 1 where NPG successfully converges to a global maximum. From Algorithm 2, a simple calculation shows,

which implies that the optimal action a ∗ = 1 always receives the largest update to its score [ Xθ t ]( a ∗ ) in each iteration. Next, take a sub-optimal action a = 2 , as an example, and observe that,

glyph[negationslash]

by Eq. (3), where ˆ r := X ( X glyph[latticetop] X ) -1 X glyph[latticetop] r . Using a constant learning rate η > 0 and applying Eq. (17), we have that π θ t ( a ∗ ) grows exponentially with t , indicating that π θ t ( a ∗ ) → 1 as t →∞ since the same argument works for any sup-optimal action a = a ∗ . Moreover, the rate is O ( e -c · t ) , since ( π ∗ -π θ t ) glyph[latticetop] r ≤ 2 · ‖ r ‖ ∞ · (1 -π θ t ( a ∗ )) . The O ( e -c · t ) rate matches the results in softmax tabular settings [14, 20, 17, 27].

Second, consider Example 2 where NPG fails to converge to a global maximum. Using similar calculations to Eq. (16) we obtain,

which implies that a sub-optimal action a = 2 always receives the largest update on its score [ Xθ t ](2) in each iteration. The failure in Figure 1(b) is then verified by similar arguments around Eq. (17).

Main NPG result. Based on these observations, it is evident that for NPG to converge globally, it is important for the optimal action to eventually always receive the largest update to its score, which makes it critical that the least square projection X ( X glyph[latticetop] X ) -1 X glyph[latticetop] r preserves the top ranking of the optimal action. We formalize this intuition by establishing the following main result.

glyph[negationslash]

Theorem 2 (Optimal action preservation condition) . For a constant learning rate η > 0 , a necessary and sufficient condition for Algorithm 2 to achieve global convergence π glyph[latticetop] θ t r → r ( a ∗ ) as t →∞ from any initialization θ 1 ∈ R d is that ˆ r ( a ∗ ) > ˆ r ( a ) for all a = a ∗ , such that a ∗ := arg max a ∈ [ K ] r ( a ) , and ˆ r := X ( X glyph[latticetop] X ) -1 X glyph[latticetop] r is the least squares projection of r onto the column space of X . If the condition is satisfied, then the rate of convergence is ( π ∗ -π θ t ) glyph[latticetop] r ∈ O ( e -c · t ) for some c > 0 .

Proof idea. When the optimal action preservation is satisfied, similar arguments to Eqs. (16) and (17) guarantee that π θ t ( a ∗ ) grows exponentially with t , indicating that π θ t ( a ∗ ) → 1 as t →∞ .

glyph[negationslash]

The constant c > 0 in Theorem 2 depends on the gap of ˆ r , i.e., ˆ r ( a ∗ ) -max a = a ∗ ˆ r ( a ) , which finds similarities to NPG results in tabular settings [14, 15]. The main difference is that the gap of true reward r in tabular cases is replaced with the gap of least square projection ˆ r in function approximation settings in Theorem 2. This similarity is an evidence for improving the rate to super-linear by using geometrically increasing step sizes, as in tabular settings [17, 27, 19, 28, 6].

One-sided Approximation Error. For NPG, [4, Lemma 6.2] introduces a 'one-sided approximation error' quantity, which aims to overestimate the advantage of the optimal action a ∗ ,

This quantity relaxes the notion of approximation error and still guarantees the global convergence of NPG, since if ∑ t =1 glyph[epsilon1] t ∈ o ( T ) , then NPG with η ∈ O (1 / √ T ) achieves global convergence [4, Lemma 6.2]. We note however that Eq. (19) has two limitations: (i) Eq. (19) depends on the entire update trajectory { θ t } t ≥ 1 , which is hard to verify. By contrast, the optimal action preservation condition in Theorem 2 only involves problem quantities X and r . (ii) It is not clear whether Eq. (19) is a necessary condition for global convergence, while optimal action preservation is proved above to be both necessary and sufficient.

5 Simulation Study

We conducted additional simulations to check the theoretical results. First , we check whether the strict inequality of ˆ r ( a ∗ ) > ˆ r ( a ) for all a = a ∗ in Theorem 2 is required for NPG global convergence.

glyph[negationslash]

glyph[negationslash]

Example 4 has ˆ r ( a ∗ ) = ˆ r (1) = ˆ r (2) , which violates the strict inequality condition of ˆ r ( a ∗ ) > ˆ r ( a ) for all a = a ∗ . The consequence is that NPG guarantees π θ t ( a ∗ ) π θ t (2) = π θ 1 ( a ∗ ) π θ 1 (2) for all t ≥ 1 , which makes it impossible for π θ t ( a ∗ ) → 1 as t →∞ . This is observed in Figure 3(a), supporting that the strictly inequality condition in Theorem 2 is indeed necessary. The initialization is θ 1 = (4 , 10) glyph[latticetop] , and η = 0 . 2 . We run 150 iterations for NPG and 1 . 5 × 10 7 iterations for Softmax PG.

Second , we run 150 iterations of NPG on Example 1. As shown in Figure 3(b), the quantity log ( π ∗ -π θ t ) glyph[latticetop] r is a linear function of time t , implying that ( π ∗ -π θ t ) glyph[latticetop] r ∈ O ( e -c · t ) with c > 0 . This supports the convergence rate results in Theorem 2. Here θ 1 and η are the same as in Figure 1(a).

Third , we check whether the condition in Theorem 1 is required for Softmax PG global convergence.

0

20

40

60

80

100

120

-40

-30

-20

-10

0

NPG

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

-20

-19

-18

-17

-16

-15

-14

-13

-12

-11

-10

-9

-8

-7

-6

-5

-4

-3

-2

-1

0

1

2

3

PG

PG

NPG

7

6

-20

20

20

-20

PG

NPG

8

-20

-20

0

20

20

Similar to Eq. (9), it is impossible to find any w ∈ R d , such that r ′ := Xw preserves the order of r in Example 5. However, as shown in Figure 3(d), Softmax PG achieves π glyph[latticetop] θ t → r ( a ∗ ) = 9 , indicating that the reward order preservation condition in Theorem 1 is sufficient but not necessary for PG to achieve global convergence. The initialization is θ 1 = (10 , -2) glyph[latticetop] , and η = 0 . 2 . We run 100 iterations for NPG and 2 × 10 6 iterations for Softmax PG. Note that NPG behaves erratically on Example 5 (which does not satisfy its global convergence conditions), by first entering then leaving the optimal plateau, eventually approaching a sub-optimal solution.

Finally , we run 10 8 iterations of Softmax PG on Example 1, using the same η and θ 1 as in Figure 1(a). Figure 3(c) shows that the slope of log ( π ∗ -π θ t ) glyph[latticetop] r over log t approaches -1 , indicating that the global convergence rate is ( π ∗ -π θ t ) glyph[latticetop] r ∈ O (1 /t ) , matching the softmax tabular setting results [22].

6 Discussions

Checking ordering-based conditions. Checking the existence of w ∈ R d in Theorem 1 is known as linear feasibility in literature [11], i.e., determining whether a set of inequalities has a non-empty intersection. In particular, suppose X ∈ R K × d , and r ∈ R K is sorted, i.e., r (1) ≥ r (2) ≥ · · · ≥ r ( K ) . Denote x i ∈ R d as the i -th row vector of X . The linear feasibility problem in this case is to check if there exists w ∈ R d , such that for all i ∈ [ K -1] , x glyph[latticetop] i w ≥ x glyph[latticetop] i +1 w . Linear feasibility can be cast as linear programming (LP) using a dummy objective and keeping the constraints, hence any LP technique, such as the ellipsoid method, can be used to solve it [11]. On the other hand, checking the optimal action preservation in Theorem 2 requires the same information as in calculating approximation error ‖ ˆ r -r ‖ 2 = min w ∈ R d ‖ Xw -r ‖ 2 , since arg max a ∈ [ K ] ˆ r ( a ) = arg max a ∈ [ K ] r ( a )

can be immediately verified after calculating the projection ˆ r := X glyph[latticetop] ( X glyph[latticetop] X ) -1 X glyph[latticetop] r .

Generalization to Markov decision processes (MDPs). Our work provides some new and useful insights for understanding more complex settings, but it requires further investigation to resolve this highly non-trivial problem for general MDPs. See Appendix C for detailed discussions.

7 Conclusions and Future Work

We believe this work opens new directions for understanding PG-based methods under function approximation, going well beyond the conventional approximation error based analysis. The major technical findings involve ordering-based conditions and relevant techniques (covariance and global convergence). Identifying exact necessary and sufficient conditions for the global convergence of Softmax PG remains future work. Extending the results and techniques to general MDPs is another important and challenging next step. Combining function approximation with recent results on stochastic on-policy sampling [20] is another interesting direction for agnostic learning. Investigating whether these new global convergence conditions might be used to achieve better representation learning is of great interest for algorithm design. Generalizing the proof techniques to other scenarios where non-linear transforms (activation functions) interact with low-dimensional features through gradient descent, such as in neural networks, is another lofty ambition.

Acknowledgments and Disclosure of Funding

The authors would like to thank anonymous reviewers for their valuable comments. CS, DS gratefully acknowledge funding from the Canada CIFAR AI Chairs Program, Amii and NSERC.

References

Yasin Abbasi-Yadkori, Peter Bartlett, Kush Bhatia, Nevena Lazic, Csaba Szepesvari, and Gellért Weisz. Politex: Regret bounds for policy iteration using expert prediction. In International Conference on Machine Learning , pages 3692-3702. PMLR, 2019.

Yasin Abbasi-Yadkori, Nevena Lazic, Csaba Szepesvari, and Gellert Weisz. Exploration-enhanced politex. arXiv preprint arXiv:1908.10479 , 2019.

Alekh Agarwal, Mikael Henaff, Sham Kakade, and Wen Sun. PC-PG: Policy cover directed exploration for provable policy gradient learning. Advances in neural information processing systems , 33:13399-13412, 2020.

Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research , 22(98):1-76, 2021.

Carlo Alfano and Patrick Rebeschini. Linear convergence for natural policy gradient with log-linear policy parametrization. arXiv preprint arXiv:2209.15382 , 2022.

Carlo Alfano, Rui Yuan, and Patrick Rebeschini. A novel framework for policy mirror descent with general parametrization and linear convergence. arXiv preprint arXiv:2301.13139 , 2023.

Shalabh Bhatnagar, Richard S Sutton, Mohammad Ghavamzadeh, and Mark Lee. Natural actor-critic algorithms. Automatica , 45(11):2471-2482, 2009.

Semih Cayci, Niao He, and Rayadurgam Srikant. Linear convergence of entropy-regularized natural policy gradient with linear function approximation. arXiv preprint arXiv:2106.04096 , 2021.

Minmin Chen, Ramki Gummadi, Chris Harris, and Dale Schuurmans. Surrogate objectives for batch policy optimization in one-step decision making. Advances in Neural Information Processing Systems , 32, 2019.

Zaiwei Chen, Sajad Khodadadian, and Siva Theja Maguluri. Finite-sample analysis of off-policy natural actor-critic with linear function approximation. IEEE Control Systems Letters , 6:2611-2616, 2022.

Martin Grötschel, László Lovász, and Alexander Schrijver. Geometric algorithms and combinatorial optimization , volume 2. Springer Science & Business Media, 2012.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Sham M Kakade. A natural policy gradient. In Advances in neural information processing systems , pages 1531-1538, 2002.

Sajad Khodadadian, Prakirt Raj Jhunjhunwala, Sushil Mahavir Varma, and Siva Theja Maguluri. On the linear convergence of natural policy gradient algorithm. In 2021 60th IEEE Conference on Decision and Control (CDC) , pages 3794-3799. IEEE, 2021.

Sajad Khodadadian, Prakirt Raj Jhunjhunwala, Sushil Mahavir Varma, and Siva Theja Maguluri. On linear and super-linear convergence of natural policy gradient algorithm. Systems & Control Letters , 164:105214, 2022.

Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems , 12, 1999.

Guanghui Lan. Policy mirror descent for reinforcement learning: Linear convergence, new sampling complexity, and generalized problem classes. Mathematical programming , 198(1):1059-1106, 2023.

Gen Li, Yuting Wei, Yuejie Chi, Yuantao Gu, and Yuxin Chen. Softmax policy gradient methods can take exponential time to converge. In Conference on Learning Theory , pages 3107-3110. PMLR, 2021.

Yan Li, Guanghui Lan, and Tuo Zhao. Homotopic policy mirror descent: Policy convergence, implicit regularization, and improved sample complexity. arXiv preprint arXiv:2201.09457 , 2022.

Jincheng Mei, Wesley Chung, Valentin Thomas, Bo Dai, Csaba Szepesvari, and Dale Schuurmans. The role of baselines in policy gradient optimization. Advances in Neural Information Processing Systems , 35:17818-17830, 2022.

Jincheng Mei, Chenjun Xiao, Bo Dai, Lihong Li, Csaba Szepesvári, and Dale Schuurmans. Escaping the gravitational pull of softmax. Advances in Neural Information Processing Systems , 33:21130-21140, 2020.

Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global convergence rates of softmax policy gradient methods. In International Conference on Machine Learning , pages 6820-6829. PMLR, 2020.

John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning , pages 1889-1897, 2015.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems , pages 1057-1063, 2000.

Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , 8(3):229-256, 1992.

Lin Xiao. On the convergence rates of policy gradient methods. Journal of Machine Learning Research , 23(282):1-36, 2022.

Rui Yuan, Simon S Du, Robert M Gower, Alessandro Lazaric, and Lin Xiao. Linear convergence of natural policy gradient methods with log-linear policies. arXiv preprint arXiv:2210.01400 , 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper theoretically proves that, for finite-arm bandits with linear function approximation, the global convergence of policy gradient (PG) methods is not dependent on approximation error, but rather, on the ordering properties of the reward representation. The global convergence is achievable for both standard Softmax PG and natural policy gradient (NPG) under linear function approximation. This result is also verified using simple examples and empirical experiments.\n\nSTRENGTHS:\nThis paper provides a completely new understanding on the convergence of policy gradient method, that is, the global convergence only depends on the specific ordering-based condtions instead of some classical approximation requirements. To my own knowledge, I have not seen such result before. \n\nThis novel insight is also significant. There are many situations where the approximation error could never be sufficiently small. This work provides a key to understand the covnergence of PG-based algorithms. And as the author claims, I agree with that this paper will open a new directions for PG-based methods under function approximation. \n\nThis work is well-written. I can clearly understand which problem this paper is solving; especially, examples given in Section 3 are very helpful to understand the motivation of proposing the ordering-based conditions. This paper also has rigorously defined two new preservation condtions on the ordering, and proved the necessarity or sufficiency for those proposed conditions.\n\nWEAKNESSES:\nThe studied case only contains one state, so it is a simple bandit optimization problem. Though the main result of this paper is inspiring and interesting, this paper lacks of (1) persuading evidence showing that these understandings will also be valid for more general cases, and (2) sufficient discussions on how to generalize this result in a more general Markov decision process.\n\nQUESTIONS:\nThe single-state MDP is a very special MDP; the proposed idea may or may not work for a more general senario. How this work will be generalized to a multi-state MDP? Is there any simulation results on that?\n\nLIMITATIONS:\nThis is a theoretical work so there is no negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the global convergence of standard softmax policy gradient and natural policy gradient for finite-arm bandits with linear function approximation (i.e., considering a log-linear policy). It is shown that the approximation error is not crucial for characterizing global convergence since the latter can be achieved even with non-zero approximation error. Ordering-based conditions are provided instead to guarantee the global convergence of softmax PG and natural PG while the approximation error is non-zero. More precisely, this paper establishes that both NPG and softmax PG converge globally when (a)  for NPG, the optimal action’s rank is preserved by the projection of the true reward onto the space of representable rewards and (b) for softmax PG, there exists a linear function preserving the ranking of actions provided by the true reward function. The case of a linearly realizable reward function is a particular case of the aforementioned reward rank preservation condition. Numerical examples illustrate all these results throughout the paper.\n\nSTRENGTHS:\nThe main results of this paper provide very interesting insights on the global convergence of some PG methods under the function approximation setting for finite-arm bandits. The paper provides strong and solid contributions.\n\n**(a) Originality**: the paper challenges common existing analysis featuring the approximation error as a natural limit when considering function approximation. Results are new to the best of my knowledge.\n\n**(b) Significance**: as also highlighted by the paper in the conclusion, the results are likely to inspire further important developments for function approximation in the more general setting of MDPs, nonlinear approximation and representation learning. \n\n**(c) Correctness**: To the best of my knowledge, except for the proof of Theorem 1 which is more involved and may deserve some clarifications in my opinion (see my comments below), the proofs in the appendix are correct, complete and relatively easy to follow. I checked the appendix in details. \n\n**(d) Clarity and writing**: This paper is very well-written, the structure is clear, the exposition is progressive, the story line is very nice and the illustrations are insightful. This is a solid body of work.\n\nWEAKNESSES:\n- While the proof of Theorem 2 is very clear to me, some parts of the proof of Theorem 1 (and its sketch l. 266 to 293) deserve some clarifications in my opinion. Please see my comments in the questions section. \n\n- Comment: The proof of Theorem 2 seems to follow similar lines to the proofs in Khodadadian et al. 2022 even if the latter work does not consider linear function approximation nor does it propose ordering-based conditions (while it applies however to MDPs).  The main difference seems in replacing the  true gap $\\Delta$ by the gap induced by the linear approximation $\\hat{\\Delta}$. For similarities, see for e.g., Appendix C in Khodadadian et al. 2022 for the bandits case or the main part of the paper (Theorem 1, Lemma 1, Proposition 1 p. 3-4 and their proofs). \n\nKhodadadian, S., Jhunjhunwala, P. R., Varma, S. M., & Maguluri, S. T. (2022). On linear and super-linear convergence of Natural Policy Gradient algorithm. Systems & Control Letters, 164, 105214.\n\n**Minor:**\n\n-  It is not very clear what is $v_2$ in l. 278-279 (especially that there is also $v$ in l. 278) and what is $v_1$ in l. 283 and how do they relate to $v$ in l. 268, I find the notations and the formulation a bit confusing in l. 266-293. \n\n- Please give the precise reference to Theorem 1 when you reference [8] in lines 116 and 123. \n\n- $r(a^*)$ introduced in l. 110 does not seem to be defined before (definition of $a^*$ comes later in l. 140).  \n\n- Numerical legends in all the figures are almost invisible in a printed format (although visible when zooming on a computer), it would be nice to increase the size of the numerical characters (and save the figure in pdf format if not already done) to ease the reading. \n\n- Eq. (27): Why is the inequality strict? This (strict) positivity does not seem to be used anyway. \n\n- Suggestion: Eq. (102) maybe add $>0$ for clarification. \n\n- Suggestion: Eq. (149) to Eq. (150) maybe add the splitting sum steps with the order summing exchange to detail a bit more for the convenience of the reader. \n\n**Typos:** \n- l. 446: $\\theta’ \\in \\mathbb{R}^d$ instead of $\\mathbb{R}^K$?\n\n- l. 474: $a \\in [K]$ instead of $i \\in [a]$. \n\n- l. 481: Eq. (58) instead of (54).  \n\n- l. 493: Eq. (68) instead of (65). \n\n- Eq. (97): Eq. (94) instead of (91). \n\n- Eq. (101): Eq. (97) instead of (95). \n\n- Eqs. (138) and (139): $\\max_a$ instead of $\\max_i$. \n\n- l. 539, Eq. (144): say there exists $\\theta_{\\zeta} \\in [\\theta, \\theta']$ such that ...\n\nQUESTIONS:\nAbout the proof of Theorem 1, I have the following technical questions (1. (a)-(b)) which constitute my main concerns and that I would like to be addressed for clarification before updating my evaluation:  \n\n**1. (a)** Second part/Lemma 1 (l. 455): Why does the fact $\\|\\| \\frac{d \\pi_{\\theta_t}^T r}{d \\theta_t}\\| \\|_2 \\to 0$ imply that $\\|\\|\\theta_t\\|\\| \\to \\infty$? I understand that there is no stationary point for any $\\theta \\in \\mathbb{R}^d$ following the proof but how do you conclude properly that then $\\|\\|\\theta_t\\|\\| \\to \\infty$? In particular in the proof, l. 446: what does the assumption \nsuppose there exists $\\theta' \\in \\mathbb{R}^K$ with $\\|\\|\\theta’\\|\\|_2 < \\infty$ mean? \n\nIt is proven (by contradiction) that for every $\\theta’ \\in \\mathbb{R}^d$, we have $\\frac{d \\pi_{\\theta’}^T r}{d \\theta’} \\neq 0$. \nTo conclude that $\\|\\|\\theta_t\\|\\| \\to \\infty$, I would expect that the proof assumes that the sequence $(\\theta_t)$ is bounded and then finds a contradiction. Could you please clarify the reasoning here? \n\n**1. (b)** Third part: the general reasoning and the conclusion l. 498 to 501 are not very clear to me.\n\n   (i) It is proven by contradiction that every suboptimal action $i$ s.t. (47) is satisfied, $\\pi_{\\theta_t}(i)$ does not converge to 1. How does this help concluding that for the optimal action $a$ (or any optimal action if non unique) we have $\\pi_{\\theta_t}(a)$ converges to 1? Could you clarify the reasoning and the guiding intuition? \n\n (ii) Could you also elaborate more on how (60), (69), (71) and (72) are used to conclude? \n\n(iii) Is the third case $j \\in \\mathcal{A}(i)$ (i.e., $i,j$ are mapped to the same reward value) handled in the proof? If trivial, maybe a word about it can be added. \n\n (iv) Minor: what do you mean by $X$ and $u$ are bounded in l. 492? $X$ is the feature matrix and $u$ is a fixed vector in $\\mathbb{R}^d$. \n\n**2.** Could you clarify why the possibility of having exponentially many suboptimal local maxima renders global convergence (to the optimal reward) impossible without further structure on function approximation as stated in l. 123-124? Is it because there is more chance to be stuck at a local suboptimal local maxima? How does the ordering-based condition preclude such a case in Theorem 1? \n\n**3.** Do the examples of section 3 allude to the fact that the ordering of the features matters since the examples only differ between them by column permutations? \n\n**4.** In proposition 1 (and more generally in the paper), do you assume that there exists a unique optimal action? If there are many, I guess any optimal action would work. You may want to add a comment regarding this.\n\nLIMITATIONS:\nLimitations and opportunities for future work are briefly discussed in the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper challenges (arguably) the current best known policy gradient (PG) convergence analysis, which is the conventional approximation error based analysis originally proposed by the seminal work of Agarwal et al. (2021). To this end, the authors consider the finite-arm bandits with log-linear policy and study the conditions of the global convergence of PG and natural policy gradient (NPG). \n\n**First**, by carefully designing numerical simulations, the authors show that global convergence can be achieved even if the parameterized policy space can not cover the full policy space, or the approximation error is not zero. Consequently, the approximation error is not a key quantity for characterizing global convergence in either algorithm under linear function approximation.\n\n**Second**, the authors establish new conditions of the global convergence of PG and NPG for the same setting, separately. For NPG, the necessary and sufficient condition of the global convergence is whether the projection of the reward vector onto the feature map strictly preserves the top ranking of the optimal action. For PG, the sufficient but not necessary condition of the global convergence is whether there exists a point in the image of the feature map such that it preserves the entire ranking of the reward vector. These conditions are again well supported by numerical simulations.\n\n\n\nAgarwal, Alekh, Sham M. Kakade, Jason D. Lee, and Gaurav Mahajan (2021). On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift. Journal of Machine Learning Research 22.98, pp. 1–76.\n\nSTRENGTHS:\nThe paper is revolutionary and the results are surprising. This paper may have the same impact on the theoretical RL community as that of Agarwal et al. (2021). The strengths of the paper can be summarized as follows.\n\n- First, the research question itself on the approximation error assumption is important and revolutionary. Challenging the previous pioneering work is always not easy.\n- However, the argumentation of the paper is impeccable. Readers will probably be surprised at first, but will quickly be convinced by several simple but sophisticated numerical simulations.\n- Not only the authors are able to find negative answers to questions about the role of approximation error for the global convergence of PG methods, they also establish new results characterizing the conditions for the global convergence of PG methods and draw the connection between the new conditions and the approximation error assumption. The novelty of the paper is significant.\n- In addition, the paper is very well written. The research question is well formulated. The new conditions are well presented. And the reasoning is detailed with intuitive explanations, figures, many examples and the proof sketch.\n\nI agree with the authors that this work will open many new directions for understanding PG-based methods in the function approximation regime, especially considering the general Markov decision processes (MDPs).\n\nWEAKNESSES:\nAlthough the paper is well written, I still find some minor points for improvement.\n\n- In the figures, the authors can specify that y-axis is the reward / value function.\n- Line 52-53: \"... approximation error ..., diverting attention from feature designs that achieve useful properties beyond small approximation error.\" Although one goal of the paper is to claim that approximation error is not necessary, I find this sentence to be a little too dismissive of approximation error. It may leave the reader with the impression that approximation error is rarely useful outside of the tabular case where the approximation error is zero. It is true that, zero approximation error does not fit well with linear function approximation in general, which is the original motivation for the paper. However, when the approximation error is zero, the conventional approximation error based analysis becomes useful. For instance, another interesting case of zero approximation error is the use of neural networks, recently studied by Alfano et al. (2023) in Section 4.2. That is, a sufficiently wide and shallow ReLU network can infinitely approximate the Q-function such that the approximation error is zero. Consequently, their approximation error based analysis leads to a new SOTA sample complexity of PG method under neural network parameterization.\n- Line 361-362: \"Extending the results and techniques to general Markov decision processes (MDPs) is another important and challenging next step.\" The authors can use an extra page in the revised version to discuss the intuition of possible obstacles to such an extension. See also my question below.\n\nAlfano, Carlo, Rui Yuan, and Patrick Rebeschini (2023). A Novel Framework for Policy Mirror Descent with General Parametrization and Linear Convergence.\n\nQUESTIONS:\nFirst, I have a clarification question. The authors claim that they solve an open problem left by Agarwal et al. (2021). If I understand correctly, the authors refer to this one: _if zero approximation error holds, does softmax PG achieve global convergence ?_ However, I couldn't find this claim in Agarwal et al. (2021). Instead, I found two open problems left by Agarwal et al. (2021). \n\n- __(1)__ In their Remark 11 (journal version) / Remark 5.1 (arxiv version), the first open problem is that, whether or not softmax PG will converge globally if the initial state distribution is not state-wise strictly positive. Since this paper only considers bandit problems with one single state, the initial state distribution is constant 1 with that state. The paper does not resolve this open problem. \n- __(2)__ In their Remark 14 (journal version / Remark 5.2 (arxiv version), the second open problem is whether a polynomial global convergence rate is achievable for softmax PG with the entropy regularizer. This problem envolves entropy regularization, which is outside the scope of the paper. Thus, the paper dose not resolve this problem neither.\n\nCan the authors point me specifically to where Agarwal et al. (2021) claim  the open problem mentioned right after Corollary 1 in this paper ?\n\nNext, I have some more open questions that the authors can decide to include them in the paper or not.\n\n- Do the results of softmax PG and NPG in this paper still hold if the optimal action is not unique ?\n- Same question for entropy regularized bandit case ?\n- Do the results for NPG with non-adaptive geometrically increasing step size still hold ? Note that NPG and variants with non-adaptive geometrically increasing step size have been studied extensively recently (Lan, 2022; Xiao, 2022; Li et al., 2022; Yuan et al., 2023; Alfano et al., 2023). Can we obtain superlinear convergence as discussed in Xiao (Section 4.3, 2022) and Li et al. (2022) ?\n- As for the general MDP, one can think about the compatible function approximation framework of Agarwal et al. (2021). By similarity, the ranking condition of the reward vector for the softmax PG becomes the ranking condition of the advantage function / Q-function. And the projection of the reward vector onto the image of the feature map for NPG becomes the projection of the advantage function / Q-function onto the image of the matrix $[\\nabla_\\theta \\log \\pi_{\\theta}(a \\mid s)]_{s, a}$. What do you think about this idea ? Can this idea and the techniques used in this paper be enough to be extended to general MDP ? If not, what is the missing factor here and what is the challenging obstacle ? For simplicity, one can consider the exact NPG instead of stochastic NPG.\n\nXiao, Lin (2022). On the Convergence Rates of Policy Gradient Methods. Journal of Machine Learning Research 23.282, pp. 1–36.\n\nLi, Yan, Tuo Zhao, and Guanghui Lan (2022). Homotopic Policy Mirror Descent: Policy Convergence, Implicit Regularization, and Improved Sample Complexity.\n\nLan, Guanghui (Apr. 2022). Policy mirror descent for reinforcement learning: linear convergence, new sampling complexity, and generalized problem classes. Mathematical Programming.\n\nYuan, Rui, Simon Shaolei Du, Robert M. Gower, Alessandro Lazaric, and Lin Xiao (2023). Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policies. In International Conference on Learning Representations.\n\nAlfano, Carlo, Rui Yuan, and Patrick Rebeschini (2023). A Novel Framework for Policy Mirror Descent with General Parametrization and Linear Convergence.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 9, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies softmax policy gradient and natural policy gradient methods for multi-arm bandits problems using linear function approximation. The authors provide examples to illustrate the global convergence of these methods when the standard function approximation error is not zero. To better characterize the global convergence, the authors provide the ordering-based conditions on rewards. Some numerical examples are provided to verify the proposed conditions.\n\nSTRENGTHS:\n**orginality**\n\n- The studied deterministic policy gradient methods are well-known in the literature. The authors re-revisit the convergence of these methods for multi-arm bandits with linear function approximation. Such convergence hasn't been studied directly. \n\n- The authors show necessary (and sufficient) conditions for two policy gradient methods to converge in the linear function approximation setting. The analysis generalizes some similar analysis, e.g., reference [18] to linear function approximation. Technical comparisons with the existing analysis are not very clear. \n\n**quality**\n\n- It seems that the global convergence is abused in some way, since the existing convergence studies in the linear function approximation investigate different sub-optimality gaps, e.g., [4] is based on the regret analysis while [24] utilizes the mirror descent analysis. \n\n- It is not clear how to interpret 'Approximation error is not a key quantity for characterizing global convergence in either algorithm'. The reference [24] shows that zero approximation error leads to global convergence. Although this result does not show necessity, approximation error is still an important quantity we use in practice.\n\n-  Compared with approximation error, it is less clear how to check optimal action perseveration condition, especially when the action space is large. So, weaknesses haven't been clearly stated. \n\n**clarity**\n\n- The paper is structured well, but it lacks of technical comparisons with existing results. \n\n- The visualization is not clearly stated, e.g., axes, error bars, speed. \n\n**significance**\n\n- Since the function approximation is widely used in reinforcement learning, this work is important. It provides new understandings of policy gradient methods in bandit cases.\n\nWEAKNESSES:\n- Technical comparisons with existing results are not detailed, sometimes vague. For instance, in line 96 why (4) and (5) for bandits can be used as RL methods; in line 102 which work provides $1/\\sqrt{t}$ rate; line 121, why 'insufficiency'; line 152, how to check global convergence; line 191, what of algorithms the quantity must depend on? Please state claims with concrete justifications. \n\n- Optimal action perseveration generalizes the analysis in reference [18]. It is similar to the gap of optimal action and second optimal action used in literature, e.g., reference [12] and the paper: Regret Analysis of a Markov Policy Gradient Algorithm for Multiarm Bandits. These quantities seem to be known important for global convergence and the authors generalize them to linear function approximation. Therefore, it is important to clarify the connections and position the work in the literature properly. \n\n- The usefulness of proposed ordering-based conditions is still questionable. First, the generalization to MDPs is not provided. Second, it is not clear how to check such conditions when state/action spaces are large, even infinite. \n\n- The paper focuses on multi-arm bandits and deterministic policy gradient methods, which have a large gap with reinforcement learning. This work seems to be still in progress and significant effort is needed to generalize this work for serious publication.\n\nQUESTIONS:\nPlease see questions in Strengths and Weaknesses. \n\nHere are some other questions.\n\n- Why is it fair to compare function approximation error with ordering-based conditions or optimal action gap conditions? The function approximation error is a general quantity that does not depend on MDP structures, while ordering-based conditions depend on the MDPs. In a basic tabular case, ordering-based conditions do not necessarily hold, e.g., equal rewards at any action, but the softmax function approximation error is zero. Hence, such conditions do not explain the basic case.\n\n- The convergence rate seems to be vague in the main paper, which has a dependence on the gap between optimal action and second optimal action, which is known in the tabular case [12, 18]. How much does your rate analysis go beyond the existing analysis? any rate improvement? \n\n- For softmax PG, reference [15] shows that it can take exponential time to converge in a hard MDP instance. How does ordering-based condition rule out the hard MDP instance? \n\n- The setup is limited to deterministic algorithms with exact gradients, which is an ideal setting. Can the authors apply them to practical stochastic bandit problems?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considered the problem of global convergence condition of policy gradient (PG) methods with linear function approximation motivated by three observations: i) global convergence under linear function approximation can be achieved without policy or reward realizability; 2) approximation error is not a critical factor for global convergence; and 3) conditions for characterizing global conference should be algorithm-dependent. Based on these observations, the authors developed new ordering-based conditions for global convergence of PG methods: i) For Softmax PG, a sufficient condition for global convergence to occur is that the representation preserves the ranking of the rewards; and ii) For natural PG (NPG), the necessary and sufficient condition of global convergence is that the projection of the reward onto the representation space preserves the optimal action’s rank.\n\nSTRENGTHS:\n1. This paper develops a new set of global convergence conditions for PG methods with linear function approximation, which advances the state of the art of understanding PG methods (Softmax PG in particular).\n\n2. The proof strategies and algorithm analysis techniques are novel.\n\n3. Motivating examples in this paper are insightful.\n\nWEAKNESSES:\n1. The paper could benefit from constructing a bit more larger-scale experiments.\n\n2. This paper could have some further discussions on the implications of the ordering-based conditions.\n\nQUESTIONS:\nI appreciate the new findings and fresh insights of the ordering-based global convergence conditions for PG methods with linear function approximation. One immediate question that comes to my mind after reading this paper is how restrictive (or non-restrictive) these ordering-based conditions are. For example, for the Softmax PG method, it appears to me that given $r$ and $X$, checking the existence of a $w$ that preserves the reward ranking may not be a simple task. Is there a systematic way to check the existence of such a $w$? Is there any other condition equivalent to order-preserving but easier to identify and check? Could the authors discuss how large the subspace of $X$ in $\\mathbb{R}^{d}$ that satisfies the order-preserving condition is? The same questions can also be asked for NPG.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-25"}