Title: Optimal Convergence Rate for Exact Policy Mirror Descent in Discounted Markov Decision Processes
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reinforcement Learning Theory

Policy Mirror Descent

Policy Gradient
Abstract: Policy Mirror Descent (PMD) is a general family of algorithms that covers a wide range of novel and fundamental methods in reinforcement learning. Motivated by the instability of policy iteration (PI) with inexact policy evaluation, unregularised PMD algorithmically regularises the policy improvement step of PI without regularising the objective function. With exact policy evaluation, PI is known to converge linearly with a rate given by the discount factor $\gamma$ of a Markov Decision Process. In this work, we bridge the gap between PI and PMD with exact policy evaluation and show that the dimension-free $\gamma$-rate of PI can be achieved by the general family of unregularised PMD algorithms under an adaptive step-size. We show that both the rate and step-size are unimprovable for PMD: we provide matching lower bounds that demonstrate that the $\gamma$-rate is optimal for PMD methods as well as PI and that the adaptive step-size is necessary to achieve it. Our work is the first to relate PMD to rate-optimality and step-size necessity. Our study of the convergence of PMD avoids the use of the performance difference lemma, which leads to a direct analysis of independent interest. We also extend the analysis to the inexact setting and establish the first dimension-optimal sample complexity for unregularised PMD under a generative model, improving upon the best-known result.

Optimal Convergence Rate for Exact Policy Mirror Descent in Discounted Markov Decision Processes

Abstract

Policy Mirror Descent (PMD) is a general family of algorithms that covers a wide range of novel and fundamental methods in reinforcement learning. Motivated by the instability of policy iteration (PI) with inexact policy evaluation, PMD algorithmically regularises the policy improvement step of PI. With exact policy evaluation, PI is known to converge linearly with a rate given by the discount factor γ of a Markov Decision Process. In this work, we bridge the gap between PI and PMD with exact policy evaluation and show that the dimension-free γ -rate of PI can be achieved by the general family of unregularised PMD algorithms under an adaptive step-size. We show that both the rate and step-size are unimprovable for PMD: we provide matching lower bounds that demonstrate that the γ -rate is optimal for PMD methods as well as PI, and that the adaptive step-size is necessary for PMD to achieve it. Our work is the first to relate PMD to rate-optimality and step-size necessity. Our study of the convergence of PMD avoids the use of the performance difference lemma, which leads to a direct analysis of independent interest. We also extend the analysis to the inexact setting and establish the first dimension-optimal sample complexity for unregularised PMD under a generative model, improving upon the best-known result.

1 Introduction

The problem of finding an optimal policy in tabular discounted Markov Decision Processes (MDPs) was classically solved using dynamic programming approaches such as policy iteration (PI) and value iteration (VI) [29, 35]. These methods are well understood theoretically and are guaranteed to converge linearly to the optimal policy in the tabular setting with a rate equal to the discount factor γ of the MDP [9]. Recently, increased interest has been devoted to the study of policy-gradient (PG) approaches based on optimising a parameterised policy with respect to an objective [36, 23, 20].

Given their popularity, it is of interest to better understand PG methods and determine if their guarantees match those of classical algorithms in tabular MDPs. Among the recent works focused on understanding these methods in the tabular setting, [38] study a general family of algorithms known as Policy Mirror Descent (PMD). PMD algorithmically regularises the policy improvement step of PI and as such can be seen as a version of regularised PI, without actually regularising the objective of interest. It is also viewed as a policy-gradient method through its connection to mirror descent [8]. Linear convergence of PMD was established by [38], though their rate depends on an instance-

dependent factor that can scale with the dimension of the problem, such as the size of the state space. For a specific instance of PMD known as Natural Policy Gradient (NPG), [22] showed that an instance-independent γ -rate is achievable, although their results do not cover general PMD. In MDPs where the objective is regularised, the γ -rate has been established for PMD [13, 24, 41]. The classical approaches (PI and VI) achieve the γ -rate without regularisation, revealing that regularisation is, in general, not necessary for algorithms to reach the γ -rate. This motivates the following questions:

Can the classical linear γ -rate be matched by unregularised policy-gradient algorithms? And what is the best rate that unregularised policy-gradient methods can achieve ?

For PMD, our work answers the first question positively and answers the second by establishing that the γ -rate is in fact the best rate achievable for PMD as well as for a more general family of algorithms (see Section 4.1). PMD allows for a choice of a mirror map that specifies different algorithms. Among these, NPG and PI are two ubiquitous instances of PMD each corresponding to their own mirror map. However, PMD is much more general and other mirror maps will lead to alternative algorithms endowed with the guarantees of PMD that we establish in this paper. In particular, the correspondence of mirror maps with exponential families [7] allows us to specify a wealth of valid mirror maps. This illustrates that PMD is a general framework that encompasses a wide range of novel but also fundamental algorithms, and motivates the study of its convergence guarantees. In this work, we make the following contributions and summarise them in Table 1,

We recover the γ -rate for the general family of PMD algorithms with exact policy evaluations under an adaptive size (see the third bullet point below). In particular, Theorem 4.1 establishes the following bound in ℓ ∞ -norm for the value V π k of the policy π k after k iterations of PMD compared to the value V π ⋆ of an optimal policy π ⋆ ,

providing guarantees for any starting-state distribution. This matches the rate of VI and PI as well as the best known rates for PMD on regularised MDPs. This is also the first fully dimension-independent linear convergence result for unregularised PMD, by which we mean that there is no dependence on the size of the state space or the action space.

We provide a matching non-asymptotic lower-bound in Theorem 4.2, establishing the γ -rate as the optimal rate for PMD methods in early iterations. Our results show that a particular choice of learning rate allows PMD to reach this lower-bound exactly. Note that an asymptotic lower-bound is not possible due to the exact convergence of PI in finite iterations (see Section 4.1 for a discussion of the significance of this non-asymptotic lower-bound).

The γ -rate for PMD in Theorem 4.1 relies on an adaptive step-size, where the adaptivity comes from the fact that the step-size depends on the policy at the current iteration (see Section 4). In Theorem 4.3 we show that this adaptivity is necessary for PMD to achieve the γ -rate, establishing our step-size as both sufficient and necessary.

We establish a novel theoretical analysis that avoids the use of the performance difference lemma [19]. This leads to a simple analysis and avoids needing to deal with visitation distribution mismatches that lead to dimension dependence in prior work.

By extending our analysis to the inexact setting, with an approach similar to that of in [38], we establish an instance-independent sample complexity of ˜ O ( |S||A| (1 -γ ) -8 ε -2 ) under a generative model, where the notation ˜ O () hides poly-logarithmic factors, S is the state space of the MDP, A is the action space and ε is the required accuracy. This improves on the previous best known sample complexity for PMD by removing the dependence on a distribution mismatch coefficient that can scale with problem-dependent quantities such as the size of the state space. More generally, we highlight that the analysis we establish in the exact setting can easily be combined with any other scheme for estimating the Q functions (see Section 5), paving the way for further improvements in instance-independent sample complexity results should more efficient estimation procedures be developed.

Our contributions are primarily on establishing the optimal rate for general (not just NPG) exact PMD where we assume access to the true action-values of policies, and the simplicity of the analysis. The sample complexity result in the inexact setting illustrates how our analysis can be easily extended to

obtain improved results for inexact PMD. These contributions advance the theoretical understanding of PMD and rule out searching for instances of PMD that may improve on PI or NPG in the exact setting. The algorithmic differences between instances of PMD do not affect the performance of the algorithm beyond the step-size condition.

2 Related work

2.1 Convergence rates for exact policy mirror descent

We first consider the setting where exact policy evaluation is assumed. In this setting, several earlier works have sub-linear convergence results for PMD [17, 33] and NPG specifically [3, 1], though these have since been improved to linear convergence results as discussed below. Note that the work of [1] refers to their algorithm as "Politex", which shares the same update as NPG.

A line of work has considered PG methods applied to regularised MDPs. In this setting, linear convergence has been established for NPG with entropy regularisation [13], PMD with strongly-convex regularisers [24] and PMD with convex non-smooth regularisers [41]. The rates of convergence are either exactly γ or can be made arbitrarily close to γ by letting the step-size go to infinity.

In the setting of unregularised MDPs, which is the focus of this paper, linear convergence of the special case of NPG was established [11, 22] under an adaptive step-size similar to ours that depends on the current policy at each step. The bound of [11] has an additive asymptotic error-floor that can be made arbitrarily small by making the step-size larger, while for a similar step-size [22] does not have this term so we focus on this work. Their analysis relies on a link between NPG and PI and consists of bounding the difference in value between iterates of both methods. [11] also establish linear convergence for a number of algorithms including PMD, although it is in the idealised setting of choosing the step size at each iteration that leads to the largest increase in value. This step-size choice will make PMD at least as good as PI since arbitrarily large step-sizes can be chosen and PMD with an infinite step-size converges to a PI update. Since PI converges linearly, so will PMD. This does not establish linear convergence of PMD for step-sizes with a closed-form expression. However, linear convergence for unregularised general PMD was recently established by [38] under a geometrically increasing step-size. In general, their rate is instance-dependent and may scale with problem dependent quantities such as the size of the state space. For general starting-state distributions, this same instance-dependent rate was established by [27] for a variant of PMD which augments the update with an added regularisation term. We focus our comparison on the work of [38] rather than this work as the guarantees are equivalent in both but [27] do not directly study PMD and have a more complicated algorithm. A summary of our results compared to those of [22] and [38] is presented in Table 1 and discussed in more detail in Section 4. In terms of optimality, [22] provide a lower-bound for constant step-size NPG, though it only applies to MDPs with a single-state, which can be solved in a single iteration with exact policy evaluation as the step-size goes to infinity (for which the lower-bound goes to 0). We provide a lower-bound in Theorem 4.2 that applies to PMD with arbitrary step-size on an MDP with any finite state space. To the best of our knowledge, prior to this work no lower-bound has been established in this general setting.

2.2 Sample complexity of inexact policy mirror descent

Sample complexity in the inexact policy evaluation setting refers to the number of samples needed to guarantee an ε -optimal policy is returned when we no longer have access to the Q-values exactly. We here give an outline of results, typically established in high-probability, under a generative model that we formally present in Section 5. The lower bound on the sample complexity in this setting was shown to be of ˜ Ω ( |S||A| (1 -γ ) 3 ε 2 ) by [6]. This lower-bound can be reached by model-based approches [2, 26] and model-free approaches [34, 37].

The sample-complexity for PG methods has been recently studied in [39]. Under a generative model, some works have considered PMD or NPG under various types of regularisation [13, 24, 41]. We focus on unregularised methods, for which results for PMD or its instances on tabular MDPs under a generative model are limited. There are works that obtain sample complexity results for NPG [3, 28] and for PMD [33] though they do not attain the optimal ε -dependence of O ( ε -2 ) . [25] show that a variant of PI, a special case of PMD, achieves the optimal ε -dependence of O ( ε -2 ) . More recently, [38] show that the general family of PMD methods match the O ( ε -2 ) sample complexity with a factor of (1 -γ ) -8 . Our result for the inexact setting shares the same dependence on ε and 1 -γ as [38] but removes an instance-dependent quantity which can depend on the size of the state space. Further comparison to the result in [38] is given in Section 5. Beyond tabular MDPs and generative models, [15] study NPG under linear function approximation and off-policy sampling, though their results imply worse sample complexities when restricted to tabular MDPs under a generative model. PMD under linear function approximation [40, 4] and general function approximation [5] have also been studied and results similar to [38] were obtained in those settings.

3 Preliminaries

A Markov Decision Process (MDP) is a discrete-time stochastic process, comprised of a set of states S , a set of actions A , a discount factor γ ∈ [0 , 1) and for each state-action pair ( s, a ) a next-state transition function p ( ·| s, a ) ∈ ∆( S ) and a (assumed here deterministic) reward function r ( s, a ) ∈ [0 , 1] . ∆( X ) denotes the probability simplex over a set X . We consider both S and A to be finite, which is known as the tabular setting. In a state s , an agent chooses an action a , which gives them a reward r ( s, a ) and transitions them to a new state according to the transition function p ( ·| s, a ) . Once they are in a new state, the process continues. The actions chosen by an agent are formalised through policies. A policy π : S → ∆( A ) is a mapping from a state to a distribution over actions. We will often write it as an element in Π = ∆( A ) |S| . In each state s ∈ S , an agent following policy π chooses an action a ∈ A according to π s = π ( ·| s ) ∈ ∆( A ) .

In this work, the goal is to learn how to behave in a γ -discounted infinite-horizon MDP. We measure the performance of a policy with respect to the value function V π : S → R ,

where s t , a t are the state and action in time-step t and the expectation is with respect to both the randomness in the transitions and the choice of actions under policy π . This is a notion of long-term reward that describes the discounted rewards accumulated over future time-steps when following policy π and starting in state s . For a distribution over starting states ρ ∈ ∆( S ) , we write V π ( ρ ) = ∑ s ∈S ρ ( s ) V π ( s ) for the expected value when starting in a state distributed according to ρ . It is also useful to work with the state-action value Q π : S × A → R :

which is similar to V π , with the additional constraint of taking action a in the first time-step. We will often write V π ∈ R |S| (resp. Q π ∈ R |S|×|A| ) to refer to the vector form, where each entry represents the value (resp. action-value) in that state (resp. state-action pair). Similarly, we write Q π s ∈ R |A| for the vector of action-values in state s . The following expressions, which relate Q π and V π in terms of each other and when combined give the Bellman equations [9], follow from their definitions above,

We now define the discounted visitation-distribution for starting state s ′ and policy π ,

which plays an important part in the study of PG methods. Note that P π ( s t = s | s 0 = s ′ ) is the probability of being in state s at time t when starting in state s ′ and following policy π . We also write d π ρ ( s ) = ∑ s ′ ∈S ρ ( s ′ ) d π s ′ ( s ) .

One of the main aims of reinforcement learning is to find a policy π that maximises V π . It is known that there exists a deterministic policy that simultaneously maximises V π and Q π for all states and actions [9]. We call such a policy an optimal policy and denote it by π ⋆ . We are interested in finding an ε -optimal policy, i.e a policy π such that ∥ V π ⋆ -V π ∥ ∞ < ε .

3.1 Exact policy mirror descent

We are interested in PG methods that are based on optimising a parameterised policy π θ with respect to V π θ ( ρ ) for some ρ ∈ ∆( S ) . In the tabular setting, we can use the direct parameterisation of a policy π θ , which associates a parameter to each state-action pair, i.e. we have π θ ( a | s ) = θ s,a . We will drop the subscript θ for notational convenience. The gradient of the value function with respect to this parameterisation [36] is given for each state-action pair (s,a) by

Mirror Descent (MD, [8]) carries out gradient descent in a geometry that is non-Euclidean. Using -V π ( ρ ) as the minimising objective, the proximal perspective of MD gives an update of the form

where h : dom h → R is the mirror map (with Π ⊂ dom h ) and D h is the Bregman divergence generated by h . We require h to be of Legendre type [30], i.e strictly convex and essentially smooth (differentiable and ∥∇ h ( x k ) ∥ → ∞ for any sequence x k converging to a point on the boundary of dom h ) on the relative interior (rint) of dom h . The Bregman Divergence is defined as

As the objective V π ( ρ ) is non-convex in general [3], usual techniques from convex theory [12] are not applicable.

The presence of the visitation-distribution term d π ρ ( s ) in the gradient of the objective in (2) can slow down learning because it can lead to vanishingly small gradients when states are infrequently visited under the current policy π [3]. To circumvent this issue, Policy Mirror Descent (PMD) [24, 33, 38] applies a variant of update (3) with a weighted Bregman divergence D PMD h that matches the visitation distribution factors of the gradient D PMD h ( p, π k ) = ∑ s d π k ρ ( s ) D h ( p s , π k s ) where the mirror map h is now defined on a subset of R |A| . The resulting update has for all states a factor of d π k ρ ( s ) in both terms. The minimisation can then be applied for each state individually to get the PMD update

for all states s . We will often add a superscript k to any quantity that is associated to π k . For example, V k ( s ) = V π k ( s ) . Similarly for π ⋆ and the superscript ⋆ . Exact PMD iteratively applies update (4) for some sequence of step-sizes η k > 0 and initial policy π 0 ∈ rint Π . We call this algorithm exact because we assume access to the true state-action values Q k .

The update (4) of PMD considered in this work uses the true action-value Q π . In prior work, PMD has sometimes been applied to regularised MDPs [24] where the action-value is augmented with regularisation and is no longer the true action-value. This is a different algorithm that converges to a policy that is optimal in the regularised MDP but not in the original unregularised MDP.

PMD is a general family that covers many algorithms, specified by the choice of mirror map h . These will inherit the guarantees of PMD, which motivates the study of the convergence guarantees of

PMD beyond specific instances. Taking h to be the negative entropy yields NPG, whose theoretical properties have attracted a lot of interest [3, 13, 22]. With a null Bregman Divergence, PMD recovers PI. PI is generated by a constant mirror map, which is not of Legendre type but the analysis still applies so all results on PMD remain valid for PI. In fact, PMD can be viewed as a form of regularised PI since the update (4) converges to a PI update as η k →∞ , regardless of the mirror map. Beyond these, providing mirror maps that generate other Bregman Divergences will lead to different algorithms. In particular, every exponential family has a corresponding mirror map generating a unique Bregman Divergence [7], highlighting the generality of PMD.

4 Main results for exact policy mirror descent

In this section, we present our main results on the convergence of exact PMD. We first introduce some relevant notation. Fix a state s ∈ S and an integer k ≥ 0 . Let A k s = { a ∈ A : Q k ( s, a ) = max a ′ ∈A Q k ( s, a ′ ) } denote the set of optimal actions in state s under policy π k . Denote by ˜ Π k +1 s the set of greedy policies w.r.t Q k s in state s, i.e ˜ Π k +1 s = { p ∈ ∆( A ) : ∑ a ∈A k s p ( a ) = 1 } . We are now ready to state our main result in the setting of exact PMD, which is proved in Section 6.

Theorem 4.1. Let { c k } k ∈ Z ≥ 0 be a sequence of positive reals. Consider applying iterative updates of (4) with π 0 ∈ rint Π and step-sizes satisfying for all k ≥ 0 ,

Then we have for all k ≥ 0 ,

The sequence { c k } k ∈ Z ≥ 0 plays an important role in both the step-size constraint (5) and the bound (6). In particular, different choices will lead to different guarantees. We focus on c i = γ 2( i +1) c 0 for some c 0 > 0 , giving a step-size with a geometrically increasing component. The resulting bound is

which converges linearly with the γ -rate, and matches the bounds of PI and VI as c 0 goes to 0. PMD cannot do better as we will show in Theorem 4.2. We discuss other choices of { c k } in Appendix C. We provide some simulations that demonstrate the behaviour of our step-size (5) and validate its benefits compared to the step-size of [38] in Appendix D.

Comparison to [38]: Linear convergence of unregularised PMD was first established by [38] under a geometrically increasing step-size η k = η 0 /γ k . We discuss this step-size further and show the necessity of adaptivity to achieve the γ -rate in Section 4.2. Their rate of convergence is 1 -1 θ ρ where θ ρ is an instance-dependent term defined as follows

where d ⋆ ρ is the visitation distribution defined in (1) under an optimal policy and ρ is the starting-state distribution to which the bound applies, i.e the bound is on V ⋆ ( ρ ) -V k ( ρ ) . This θ ρ is at best γ when we use ρ to be the stationary distribution of the optimal policy. However, in this case, the guarantee only applies to states on the support of this stationary distribution and provides no guarantees for other states. In general, it is unclear how θ ρ may scale in a specific MDP. In particular, it is possible to construct an MDP where θ ρ scales linearly with the size of the state space |S| (Appendix H.1). Though this MDP is somewhat trivial, it nonetheless illustrates how θ ρ can easily be large leading to slow rates of convergence. It is also not straightforward to obtain convergence in individual states from the bound in [38] due to the presence of ρ in the denominator of the mismatch coefficient in θ ρ . In contrast, we obtain the optimal γ -rate of convergence and our result holds in ℓ ∞ -norm over all states so avoids having to deal with a starting-state distribution ρ altogether.

This distribution mismatch coefficient commonly appears in convergence bounds in the literature [19, 32, 10, 33, 3], both under exact and inexact policy evaluation. For many of these papers mentioned, removing it would be of great interest though often does not appear possible. Our results show that it is removable for the general family of PMD algorithms to obtain dimension-free linear convergence. The techniques we use may be of interest for removing this coefficient in other settings.

Comparison to [22]: The γ -rate was established by [22] for NPG, a specific instance of PMD for which the Bregman Divergence is the KL-divergence. The bound shown in their work is similar to the one implied by our result with c i = γ 2( i +1) c 0 . Defining ∆ k ( s ) = max a ∈A Q k ( s, a ) -max a/ ∈A k s Q k ( s, a ) , the minimal sub-optimality gap in state s under π k , then the step-size corresponding to their bound with the KL as Bregman Divergence is

where L k = Lk for some constant L > 0 . This highlights the connection with our step-size condition (5). In particular, they both have an adaptive component that depends linearly on the Bregman divergence between the current policy and the greedy policy and a non-adaptive component on which the bound depends. An important difference is that our step-size is independent of the sub-optimality gap ∆ k ( s ) , and will be robust to situations where this gap is small. We can construct a general family of MDPs for which we can make ∆ k ( s ) arbitrarily small and the step-size of [22] will correspondingly become arbitrarily large (Appendix H.2). Despite the apparent similarities with our results, their analysis is significantly different to ours as it exploits the specific closed-form update of NPG to bound the difference in value with an update of PI. Our analysis applies to PMD for a general mirror map (not just NPG) and as such does not utilize specific properties of the mirror map and does not require the analytic solution of the update to be known. Our analysis also easily extends to inexact PMD (see Section 5), which theirs does not.

Computing the step-size: The expression (5) for the step-size can be simplified. The minimum over ˜ π k +1 s ∈ ˜ Π k +1 s can be removed by taking any ˜ π k +1 s ∈ ˜ Π k +1 s . It was stated with the minimum to have the smallest condition but that is not necessary. In fact, there will often only be one greedy action, i.e. A k s will contain a single action and there will just be one policy in ˜ Π k +1 s . As for the maximum over s ∈ S , this condition is imposed because we are using the same step-size in all states for simplicity. If we allow different step-sizes in each state, then the step-size in state s , denoted η k ( s ) would just have to satisfy (choosing any ˜ π k +1 s ∈ ˜ Π k +1 s )

The computational complexity of the step-size is then that of the Bregman divergence between two policies. Note that the step-size (5) is always finite. It can be unbounded when h is of Legendre type on the relative interior of ∆( A ) (as for the negative entropy) but then all iterates will belong to the relative interior of ∆( A ) and the Bregman divergences are well defined so the step-size is finite.

4.1 Optimality of PMD

We have established in Theorem 4.1 that PMD achieves a linear γ -rate. The following result shows that this rate is in fact optimal in a worst-case sense. The proof can be found in Appendix E.

Theorem 4.2. Fix n > 0 . There exists a class of MDPs with |S| = 2 n +1 and a policy π 0 ∈ rint Π such that running iterative updates of (4) for any positive step-size regime, we have for k < n :

A key feature of this result is that the bound holds for k < n . For a fixed iteration budget, Theorem 4.2 implies that there exists an MDP on which PMD will not do better than the linear γ -rate for any step-size. The γ -rate for PMD that we prove in Theorem 4.1 is optimal in this sense.

Lower-bound beyond k < n ? For a fixed state-space size |S| = 2 n +1 , it is not known what lower-bound holds when k ≥ n . However, PI is an instance of PMD (see Section 3.1) and thus a lower bound that scales with γ k for all k > 0 cannot hold since PI converges exactly in finite-iterations

(in fact with a number of iterations that scales linearly with |S| [31, Theorem 3]). However, an asymptotic lower-bound for PMD under arbitrary step-size will not hold even if we exclude PI from the class of considered algorithms. As the step-size tends to infinity, any PMD update recovers a PI update. This implies that general PMD can be arbitrarily close to PI's exact convergence for the same finite number of iterations. Thus, any lower-bound on the convergence of PMD must be limited to finite iterations. Since the finite iteration convergence of PI only scales linearly with |S| [31, Theorem 3], the number of iterations guaranteed by Theorem 4.2 has the same dependence on |S| as the number of iterations needed for exact convergence of PI. To the best of our knowledge, this lower bound on the value convergence of PMD scaling with γ k is new. We expect this result may have been known for the special case of PI, though we could not find a proof of it in the literature. The works that establish a lower bound for PI do so in the setting of exact convergence to the optimal policy [18], not ε -accurate convergence, and for undiscounted MDPs [16].

Rate-Optimality Theorem 4.2 establishes that the γ -rate is optimal in the first |S| / 2 iterations. This non-asymptotic optimality is justified by the discussion above, which highlights that a rate for PMD under arbitrary step-size can only be optimal for a finite number of iterations, and specifically up until the exact convergence of PI.

There are some results on the super-linear convergence of NPG in the literature, though these apply once you have a policy within some neighbourhood of the optimal policy or value. [13] establish such a result for NPG in the regularised case, and [22] in the unregularised case under additional conditions. Theorem 4.2 does not contradict this latter result as for the MDP considered in the proof, the super-linear convergence would kick-in for iterations beyond the k < n considered here.

Lower-bound for general PG methods: The lower bound of Theorem 4.2 in fact applies to any algorithm that at each iteration increases the probability of the current greedy action. The greedy action is the action with highest action-value for the current policy. This covers algorithms more general than PMD and in particular, includes the vanilla PG algorithm.

4.2 Adaptive step-size necessity

We have established in Theorem 4.1 that PMD under an adaptive step-size achieves a linear γ -rate and in Theorem 4.2 that this rate is optimal for PMD. We now show the adaptivity is in fact necessary to achieve the γ -rate. This strengthens the notion of optimality from the previous section - both the rate and step-size are unimprovable. The proof can be found in Appendix F.

Theorem 4.3. Fix n > 0 and γ > 0 . 2 . There exists an MDP with state-space of size |S| = 2 n +1 and a policy π 0 ∈ rint Π such that running iterative updates of NPG (PMD with h as the negative entropy) that satisfy ∥ V ⋆ -V k ∥ ∞ ≤ γ k ( ∥ V ⋆ -V 0 ∥ ∞ + 1 -γ 8 ) requires

This theorem states that there are at least n distinct iterations with n distinct states where the step-size has to be bigger than a quantity depending on the Bregman divergence between the current policy and its greedy policy in the considered state in order to achieve a linear γ -rate. This is precisely the notion of adaptivity that appears in the step-size condition of Theorem 4.1 and [22]. Theorem 4.3 shows we cannot improve on this in general and provides justification for using an adaptive step-size instead of the one from [38]. Beyond its necessity, the adaptivity of our step-size can be a strength: it is large when needed, small when not. This is illustrated by the simulations we provide in Appendix D.

Theorem 4.3 only holds for the particular case of the negative entropy (NPG). This limitation is due to technical reasons rather than fundamental ones. The proof relies on the closed form update of NPG and we don't have this for general mirror map (see Appendix F). However, this result for NPG is enough to establish the necessity of the step-size for PMD in general.

5 Sample complexity of inexact policy mirror descent under generative model

In the previous sections, we have assumed access to the state-action values Q k s to carry out the PMD update. In Inexact PMD (IPMD), we replace Q k s with an estimate ̂ Q k s giving the update

Similarly to the exact case, IPMD iteratively applies update (9) for some sequence of η k > 0 and initial policy π 0 ∈ rint Π , this time only assuming access to an inexact estimator of Q k .

We consider the setting of a generative model [21], which is a sampling model where we can draw samples from the transition probabilities p ( ·| s, a ) for any pair ( s, a ) . Weborrow an estimator common in the literature (see e.g. [38], [25]): for all state-actions pairs ( s, a ) , draw M k trajectories of length or horizon H , i.e samples of the form ( ( s ( i ) 0 , a ( i ) 0 ) , ( s ( i ) 1 , a ( i ) 1 ) , ..., ( s ( i ) H -1 , a ( i ) H -1 ) ) i =1 ,...,M k , where a ( i ) t is drawn from π k ( ·| s ( i ) t ) , s ( i ) t +1 is drawn from p ( ·| s ( i ) t , a ( i ) t ) and ( s ( i ) 0 , a ( i ) 0 ) = ( s, a ) . Using these samples, we can define a truncated Monte-Carlo estimate of the values as follows,

We use these ̂ Q k ( s, a ) to replace Q k ( s, a ) in the PMD update step. [38] present a bound on the accuracy of this estimator which is restated in Appendix G. Following the same ideas as [38], we can extend Theorem 4.1 to the inexact setting. The following theorem establishes a sample complexity result, which is the sufficient number of calls to the generative model to obtain an ε -optimal policy. For simplicity, we focus on the step-size following from the choice c i = γ 2( i +1) .

Theorem 5.1. Consider applying iterative updates of (9) using the Q-estimator in (10) given access to a generative model with π 0 ∈ rint Π and step-sizes satisfying for all k ≥ 0 (with the definitions of A k s and ˜ Π k +1 s suitably adjusted with ̂ Q k s instead of Q k s ),

Fix ε > 0 . For any δ ∈ (0 , 1) , suppose the following are satisfied for all k ≥ 0 ,

Then with probability at least 1 -δ , ∥ V ⋆ -V k ∥ ∞ ≤ γ k ( ∥ V ⋆ -V 0 ∥ ∞ + 1 1 -γ ) + 8 (1 -γ ) 3 γ H < ε . Choosing K , H and M to be tight to their lower-bounds, the corresponding sample complexity is ˜ O ( |S||A| (1 -γ ) 8 ε 2 ) , where the notation ˜ O () hides poly-logarithmic factors.

The proof can be found in Appendix G.1. The sample complexity established by [38] (Theorem 16) under a generative model and the same Q-estimator is ˜ O ( |S||A| (1 -γ ) 8 ε 2 ∥ ∥ ∥ d ⋆ ρ ρ ∥ ∥ ∥ 3 ∞ ) .

In their work, [38] stresses the interest in reducing the dependence on both 1 / (1 -γ ) and the distribution mismatch coefficient in order to scale the PMD guarantees to more relevant settings such as function approximation. Theorem 5.1 partially resolves this matter by removing the dependence on the distribution mismatch coefficient, which may scale with the size of the state space (Appendix H.1). This makes the result dimension-optimal, which is crucial when scaling the results to large or infinite state or action spaces. The dependence on 1 / (1 -γ ) remains distant from the 1 / (1 -γ ) 3 lower-bound of [6] (see Section 2). Whether this can be reached by PMD methods remains open, though using a more suitable Q-estimator than (10) with our step size regime and analysis, which extends to arbitrary Q-estimators, could bring the sample complexity closer to this.

6 Analysis

In this section, we present the proof of Theorem 4.1. A key component in establishing the γ -rate is avoiding the performance difference lemma that we state in Appendix B. In prior works, the quantity

that we are looking to bound V ⋆ ( ρ ) -V k ( ρ ) arises through the performance difference lemma. In particular, [38] use the lemma on E s ∼ d ⋆ ρ [ ⟨ Q k s , π ⋆ s -π k +1 s ⟩ ] , which introduces a distribution mismatch coefficient in order to get a recursion. On the other hand, we extract the value sub-optimalities V ⋆ ( s ) -V k ( s ) and ∥ V ⋆ -V k +1 ∥ ∞ directly from ⟨ Q k s , π ⋆ s -π k +1 s ⟩ in (12). This leads to an elegant analysis that may be of interest in the study of other methods, and ultimately allows us to remove distribution mismatch factors and obtain an exact γ -rate.

Proof of Theorem 4.1: Fix s ∈ S and k ≥ 0 . From Lemma A.2, we have that ⟨ Q k s , π k +1 s ⟩ ≤ ⟨ Q k +1 s , π k +1 s ⟩ = V k +1 ( s ) . This decouples the dependencies on π k and π k +1 below and is one of the ingredients that allows us to bypass the performance difference lemma. Using this,

where the last step uses Hölder's inequality. Now we use that the difference in state-action values of different policies for the same state-action pair propagates the error to the next time-step, which is discounted by a factor of γ . Formally, for any state-action pair ( s, a ) ∈ S × A ,

which is the same phenomenon that is responsible for the contraction of the Bellman operator. This gives ∥ Q ⋆ s -Q k s ∥ ∞ ≤ γ ∥ V ⋆ -V k ∥ ∞ . Plugging into Equation (11),

The rest of the proof relies on making the right-hand side of the above arbitrarily small by taking a large enough step size. Choose any greedy policy with respect to Q k s , ˜ π k +1 s ∈ ˜ Π k +1 s ,

where we use that ˜ π k +1 s is greedy with respect to Q k s . We then apply Lemma A.1 or (16) to p = ˜ π k +1 s ,

Combining with (13) and noting that this holds for any ˜ π k +1 s ∈ ˜ Π k +1 s , we have

from the step-size condition in the statement of the theorem. Rearranging and recalling that s and k were arbitrary, we can choose s where V ⋆ ( s ) -V k +1 ( s ) reaches its maximum value. We get

and unravelling this recursion completes the proof.

7 Conclusion

In this paper, we have shown that the general family of exact policy mirror descent algorithms in tabular MDPs under an adaptive step-size achieve the same dimension-free linear γ -rate of convergence of classical algorithms such as policy iteration. We provide matching lower-bounds that establish this rate as optimal for PMD and the adaptive step-size as necessary. We exploit a new approach to study the convergence of PMD, for which avoiding the performance difference lemma is a key element. Though the focus of our work is on the exact policy evaluation setting, the analysis naturally extends to the inexact setting, given access to an estimator of the action-value of a policy. We provide a result for a simple estimator under a generative model that improves upon the best-known sample complexity, although it still does not match the lower bound. Our method is general and applies to any estimator, meaning our result could be improved by a better estimator. Exploiting further algorithmic properties of PMD in the inexact setting may be needed to bridge the gap to the optimal sample complexity, and determine if PMD can match the lower bound in the inexact setting.

■

Acknowledgments and Disclosure of Funding

EJ is funded by EPSRC through the Modern Statistics and Statistical Machine Learning (StatML) CDT (grant no. EP/S023151/1).

We would like to thank the reviewers and meta-reviewers for their time and feedback, which led to a better version of this paper.

References

Yasin Abbasi-Yadkori, Peter Bartlett, Kush Bhatia, Nevena Lazic, Csaba Szepesvari, and Gellert Weisz. POLITEX: Regret bounds for policy iteration using expert prediction. In Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 3692-3702. PMLR, 2019.

Alekh Agarwal, Sham Kakade, and Lin F. Yang. Model-based reinforcement learning with a generative model is minimax optimal. In Proceedings of Thirty Third Conference on Learning Theory , volume 125 of Proceedings of Machine Learning Research , pages 67-83. PMLR, 2020.

Alekh Agarwal, Sham M. Kakade, Jason D. Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research , 22(98):1-76, 2021.

Carlo Alfano and Patrick Rebeschini. Linear convergence for natural policy gradient with log-linear policy parametrization. 2022. Preprint, arXiv: 2209.15382.

Carlo Alfano, Rui Yuan, and Patrick Rebeschini. A novel framework for policy mirror descent with general parametrization and linear convergence. 2023. Preprint, arXiv: 2301.13139.

Mohammad Gheshlaghi Azar, Rémi Munos, and Hilbert Kappen. Minimax PAC bounds on the sample complexity of reinforcement learning with a generative model. Machine Learning , 91 (3):325-349, 2013.

Arindam Banerjee, Srujana Merugu, Inderjit S. Dhillon, and Joydeep Ghosh. Clustering with bregman divergences. Journal of Machine Learning Research , 6(58):1705-1749, 2005.

Amir Beck and Marc Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters , 31(3):167-175, 2003.

Richard Bellman. Dynamic Programming . Dover Publications, 1957.

Jalaj Bhandari and Daniel Russo. Global optimality guarantees for policy gradient methods. 2019. Preprint, arXiv: 1906.01786.

Jalaj Bhandari and Daniel Russo. On the linear convergence of policy gradient methods for finite mdps. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics , volume 130 of Proceedings of Machine Learning Research , pages 2386-2394. PMLR, 2021.

Sébastien Bubeck. Convex optimization: Algorithms and complexity. Foundations and Trends in Machine Learning , 8(3-4), 2015.

Shicong Cen, Chen Cheng, Yuxin Chen, Yuting Wei, and Yuejie Chi. Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularization. Operations Research , 70(4): 2563-2578, 2021.

Gong Chen and Marc Teboulle. Convergence Analysis of a Proximal-Like Minimization Algorithm Using Bregman Functions. SIAM Journal on Optimization , 3(3):538-543, 1993.

Zaiwei Chen and Siva Theja Maguluri. Sample complexity of policy-based methods under off-policy sampling and linear function approximation. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics , volume 151 of Proceedings of Machine Learning Research , pages 11195-11214. PMLR, 2022.

John Fearnley. Exponential lower bounds for policy iteration. In Automata, Languages and Programming , pages 551-562, 2010.

Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized Markov decision processes. In Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 2160-2169. PMLR, 2019.

Romain Hollanders, Jean-Charles Delvenne, and Raphaël M. Jungers. The complexity of policy iteration is exponential for discounted markov decision processes. In 51st IEEE Conference on Decision and Control (CDC) , pages 5997-6002, 2012.

Sham Kakade and John Langford. Approximately Optimal Approximate Reinforcement Learning. In Proceedings of the 19th International Conference on Machine Learning , volume 2, pages 267-274, 2002.

Sham M Kakade. A natural policy gradient. In Advances in Neural Information Processing Systems , volume 14, 2001.

Michael Kearns and Satinder Singh. Finite-sample convergence rates for q-learning and indirect algorithms. In Advances in Neural Information Processing Systems , volume 11, 1998.

Sajad Khodadadian, Prakirt Raj Jhunjhunwala, Sushil Mahavir Varma, and Siva Theja Maguluri. On the linear convergence of natural policy gradient algorithm. In 60th IEEE Conference on Decision and Control (CDC) , pages 3794-3799, 2021.

Vijay Konda and John Tsitsiklis. Actor-critic algorithms. In Advances in Neural Information Processing Systems , volume 12, 1999.

Guanghui Lan. Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classes. Mathematical Programming , 2021.

Alessandro Lazaric, Mohammad Ghavamzadeh, and Rémi Munos. Analysis of classificationbased policy iteration algorithms. Journal of Machine Learning Research , 17(19):1-30, 2016.

Gen Li, Yuting Wei, Yuejie Chi, Yuantao Gu, and Yuxin Chen. Breaking the sample size barrier in model-based reinforcement learning with a generative model. In Advances in Neural Information Processing Systems , volume 33, pages 12861-12872, 2020.

Yan Li, Guanghui Lan, and Tuo Zhao. Homotopic policy mirror descent: Policy convergence, implicit regularization, and improved sample complexity. Mathematical Programming , 2023.

Yanli Liu, Kaiqing Zhang, Tamer Basar, and Wotao Yin. An improved analysis of (variancereduced) policy gradient and natural policy gradient methods. In Advances in Neural Information Processing Systems , volume 33, pages 7624-7636, 2020.

Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming . Wiley Series in Probability and Statistics. Wiley, 1994.

R. Tyrrell Rockafellar. Convex Analysis . Princeton Landmarks in Mathematics and Physics. Princeton University Press, 1970.

Bruno Scherrer. Improved and generalized upper bounds on the complexity of policy iteration. In Advances in Neural Information Processing Systems , volume 26, 2013.

Bruno Scherrer. Approximate policy iteration schemes: A comparison. In Proceedings of the 31st International Conference on Machine Learning , volume 32 of Proceedings of Machine Learning Research , pages 1314-1322. PMLR, 2014.

Lior Shani, Yonathan Efroni, and Shie Mannor. Adaptive trust region policy optimization: Global convergence and faster rates for regularized mdps. Proceedings of the AAAI Conference on Artificial Intelligence , 34(04):5668-5675, 2020.

Aaron Sidford, Mengdi Wang, Xian Wu, Lin Yang, and Yinyu Ye. Near-optimal time and sample complexities for solving markov decision processes with a generative model. In Advances in Neural Information Processing Systems , volume 31, 2018.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, second edition, 2018.

Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems , volume 12, 1999.

M. J. Wainwright. Variance-reduced Q -learning is minimax optimal. Technical report, UC Berkeley, 2019. Preprint, arxiv:1906.04697.

Lin Xiao. On the convergence rates of policy gradient methods. Journal of Machine Learning Research , 23(282):1-36, 2022.

Rui Yuan, Robert M. Gower, and Alessandro Lazaric. A general sample complexity analysis of vanilla policy gradient. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics , volume 151 of Proceedings of Machine Learning Research , pages 3332-3380. PMLR, 2022.

Rui Yuan, Simon Shaolei Du, Robert M. Gower, Alessandro Lazaric, and Lin Xiao. Linear convergence of natural policy gradient methods with log-linear policies. In The Eleventh International Conference on Learning Representations , 2023.

Wenhao Zhan, Shicong Cen, Baihe Huang, Yuxin Chen, Jason D. Lee, and Yuejie Chi. Policy mirror descent for regularized reinforcement learning: A generalized framework with linear convergence. SIAM Journal on Optimization , 33(2):1061-1091, 2023.

A Properties of PMD

We present lemmas relevant to the analysis of PMD. Key to the analysis is the Three-Point Descent Lemma, that relates the improvement of the proximal gradient update compared to an arbitrary point. It originally comes from [14] (Lemma 3.2) where a proof can be found, though we use a slightly modified version from [38] (Lemma 6).

Lemma A.1 (Three-Point Descent Lemma, Lemma 6 in [38]) . Suppose that C ⊂ R n is a closed convex set, ϕ : C → R is a proper, closed convex function, D h ( · , · ) is the Bregman divergence generated by a function h of Legendre type and rint dom h ∩ C ̸ = ∅ . For any x ∈ rint dom h , let

Then x + ∈ rint dom h ∩ C and ∀ u ∈ C ,

The update (4) of PMD is an instance of the proximal minimisation (14) with C = ∆( A ) , x = π k s and ϕ ( x ) = -η k ⟨ Q k s , x ⟩ . Plugging these into (15), Lemma A.1 relates the decrease in the proximal objective of π k +1 s to any other policy, i.e. ∀ p ∈ ∆( A ) ,

This equation is key to the analysis in Section 6. In particular, it allows us to prove the following lemma regarding the monotonic improvement in action-value of PMD iterates. This is an extension of Lemma 7 in [38].

Lemma A.2. Consider the policies produced by the iterative updates of PMD in (4). Then for any k ≥ 0 ,

A.1 Proof of Lemma A.2

We first present Lemma 7 from [38], from which Lemma A.2 almost immediately follows.

Lemma A.3 (Descent Property of PMD, Lemma 7 in [38]) . Consider the policies produced by the iterative updates of PMD in (4). Then for any k ≥ 0

Proof. From [38]. Recall that the Three-Point Descent Lemma states that ∀ p ∈ ∆( A ) ,

Using this with p = π k s ,

and since the Bregman divergences are none-negative and η k > 0 ,

and the result follows by an application of the performance difference lemma (Appendix B)

Note that we use the performance difference lemma here because it gives a simple concise proof, but we do not actually need to. To maintain our claim that we avoid the use of the performance difference lemma, we can get the same result without it. We sketch how to do this as follows. From the first part of the lemma, we have

in all states s . Now note that the left hand side above is

a s ) and we can then apply ⟨ Q k s ′ , π k +1 s ′ ⟩ ≥ V k ( s ′ ) at state s ′ : V k ( s ) ≤ ∑ a π k +1 ( a | s ) ( r ( s, a ) + γ ∑ s ′ p ( s ′ | s, a ) V k ( s ′ ) ) ≤ ∑ a π k +1 ( a | s ) ( r ( s, a ) + γ ∑ s ′ p ( s ′ | s, a ) ∑ a ′ π k +1 ( a ′ | s ′ ) ( r ( s ′ , a ′ ) + γ ∑ s ′′ p ( s ′′ | s ′ , a ′ ) V k ( s ′′ ) and proceeding iteratively in the limit you get exactly V k +1 ( s ) . ■

Since Lemma A.3 holds for any ρ ∈ ∆( S ) , it guarantees that the value in each state is non-decreasing for an update of PMD, i.e for all s ∈ S ,

Using this, we get

which concludes the proof.

A.2 Extension of Lemma A.2 to inexact setting:

As in the exact case, we first present Lemma 12 from [38] which is the extension of Lemma A.3 to the inexact case. We note that in the inexact case, we lose the monotonic increase of values due to the inaccuracy in our estimate ̂ Q k of Q k s .

Lemma A.4. Consider the policies produced by the iterative updates of IPMD in (9). For any k ≥ 0 , if ∥ ̂ Q k s -Q k s ∥ ∞ ≤ τ , then

Proof. From [38]. The Three-Point Descent Lemma applied to the IPMD update (9) gives ∀ p ∈ ∆( A ) ,

Using this with p = π s ,

and since the Bregman divergences are none-negative and η k > 0 ,

which proves the first inequality. Now we cannot use the above inequality directly with the performance difference lemma since ̂ Q k s is not the true action-value. Instead, we have

which concludes the proof.

■

Using the above lemma, we can state and prove the extension of Lemma A.2 to the inexact setting. Lemma A.5. Consider the policies produced by the iterative updates of IPMD in (9). For any k ≥ 0 , if ∥ ̂ Q k s -Q k s ∥ ∞ ≤ τ , then

Proof. As in the exact case, since Lemma A.4 holds for any ρ ∈ ∆( S ) , it applies to each state, i.e for all s ∈ S ,

Using this, we immediately have

which concludes the proof.

B Performance difference lemma

Lemma B.1 (Performance Difference Lemma) . For any π, π ′ ∈ Π , we have

The performance difference lemma [19] is a property that relates the difference in values of policies to the policies themselves. The proof can be found in their paper under Lemma 6.1.

C Guarantees of Theorem 4.1 for various step-size choices

We give here two more choices of { c k } k ∈ Z ≥ 0 for the step-size 5 of PMD and their corresponding guarantees from Theorem 4.1:

c i = c 0 for some c 0 > 0 yields a step-size with a constant component. The resulting bound is

which converges linearly up to some accuracy controlled by c 0 .

c i = γ i +1 c 0 for some initial c 0 > 0 will yield a step-size with a component that is geometrically increasing as in [38], though at a slower rate than the one discussed in Section 4. The resulting bound is

which converges linearly with the sought-for γ -rate, though in early iterations the k factor may dominate.

■

D Simulations

In this Section, we present simulations (see Figures 1-4 below) made by running NPG (PMD with negative-entropy mirror map, see [22]) on the hard MDP used to prove Theorem 4.2 - see Appendix E for the details of the construction. We use n = 25 so |S| = 51 and |A| = 2 . We use δ = (1 -γ ) γ n / 100 , γ = 0 . 99 , π 0 ( a 1 | s ) = α (note that a 1 is the optimal action in each state). In each of the plots, we compare the performance of NPG using our step-size (5) from Theorem 4.1 (denoted Adaptive in the plots and by ADA in the discussion below) with c k = γ 2 k and c 0 = 1 and the step-size of [38] η k = η 0 /γ k (denoted by INC in the discussion below) for a fixed η 0 = 1 . In each of the plots, the left plot is ∥ V ⋆ -V k ∥ ∞ against iterations and the right plot is η k against iterations.

In all plots, the green curve represents the γ -rate. In particular it is the curve of y = γ x .

Discussion

With fixed initial step-size η 0 = 1 for INC, the convergence of NPG with INC can be made arbitrarily slow in early iterations by choosing α (= π 0 ( a 1 | s ) ) closer to 0 (Figures 1-3). The convergence of NPG with ADA is unaffected by small values of α because it adapts to them.

These observations agree with the theoretical results we established. Our adaptive step-size (5) achieves the γ -rate (Theorem 4.1) and this is optimal for iterations k up until n = 25 (Theorem 4.2). The adaptivity in the step-size is necessary to achieve the γ -rate in these early iterations (Theorem 4.3).

In later iterations (Figure 1: iterations ≈ 100 . Figure 2: iterations ≈ 200 ), INC has faster convergence than ADA. In these later iterations, acting greedily with respect to the action value of the current policy gives the optimal policy so INC converges faster because the step-size is larger (see Figure 1-2 right plot). This suggests combining ADA and INC by taking the maximum of both (Figure 4), yielding faster convergence than both. This is encouraging for future-work on super-linear convergence (as discussed in [38]) beyond the optimal γ -rate valid up to a finite iteration (see Section 4.1).

0

100

200

300

400

500

Iterations

10

10

10

8

10

6

10

4

10

2

10 0

V

V k

Adaptive (ADA)

k

=

0/

k

(INC)

0

100

200

300

400

500

Iterations

10

1

10 0

10 1

10 2

Step-Size

Adaptive (ADA)

k

=

0/

k

(INC)

0

100

200

300

400

500

Iterations

10

10

10

8

10

6

10

4

10

2

10 0

V

V k

Adaptive (ADA)

k

=

0/

k

(INC)

0

100

200

300

400

500

Iterations

10

3

10

2

10

1

10 0

10 1

10 2

Step-Size

Adaptive (ADA)

k

=

0/

k

(INC)

0

100

200

300

400

500

Iterations

10

9

10

7

10

5

10

3

10

1

V

V k

Adaptive (ADA)

k

=

0/

k

(INC)

0

100

200

300

400

500

Iterations

10 0

10 1

10 2

Step-Size

Adaptive (ADA)

k

=

0/

k

(INC)

0

100

200

300

400

500

Iterations

10

10

10

8

10

6

10

4

10

2

10 0

V

V k

k

= max(ADA,

0/

k

)

k

=

0/

k

(INC)

0

100

200

300

400

500

Iterations

10 0

10 1

10 2

Step-Size

k

= max(ADA,

0/

k

)

k

=

0/

k

(INC)

E Proof of Theorem 4.2

Fix n > 0 and δ ∈ (0 , (1 -γ ) γ n ) . Consider the MDP shown in Figure 5. The state space is S = { s 0 , s 1 , s ′ 1 , ..., s n , s ′ n } and the action space is A = { a 1 , a 2 } . There is a chain of states of length n +1 with the states indexed from 0 to n . The left-most state ( s 0 ) is absorbing with reward +1. In the other states in the chain ( s i for i = 1 , ..., n ), the agent can take action a 1 and move left (to s i -1 ) with reward of 0, or take action a 2 and move to an additional absorbing state unique to the state it is currently in ( s ′ i ) with reward r i = γ i +1 + δ (that the agent also receives in that state for all future time-steps). Summarising, we have for 1 ≤ i ≤ n

The value of δ is carefully restricted so that the optimal action in all the states of the chain is a 1 . The proof will consist in showing that if the agent starts with an initial policy that places most probability mass on the sub-optimal action a 2 , then it has to learn that a 1 is the optimal action in the state directly to the left before it can start switching from action a 2 to a 1 in the current state. And this can at best happen one iteration at a time starting starting from the left-most state. In particular, we consider π 0 s.t π 0 ( a 1 | s ) = α , π 0 ( a 2 | s ) = 1 -α for all states and some α s.t 0 < α ≤ δ (1 -γ ) . We make the following claim from which the result will follow straightforwardly.

Claim: Fix k < n . The policies produced by PMD satisfy π k ( a 1 | s i ) ≤ α for k < i ≤ n .

We prove this claim by induction.

Base Case: We want to show that π 1 ( a 1 | s i ) ≤ α for i > 1 . We do this by showing that Q 0 ( s i , a 1 ) ≤ Q 0 ( s i , a 2 ) for i > 1 so that the probability of π 1 ( a 1 | s i ) cannot increase w.r.t π 0 ( a 1 | s i ) , which is α

(this follows from ⟨ Q k s , π k +1 s -π k s ⟩ ≥ 0 for all iterations of PMD). We have:

where we used α ≤ δ (1 -γ ) in (a) and γ (1 + γ i -1 -γ i ) < 1 for γ ∈ [0 , 1) in (b). This concludes the base case.

Inductive Step: Now assume that the claim is true for k and we want to show that π k +1 ( a 1 | s i ) ≤ α for i > k +1 . We do this in the same way as the base case by showing that Q k ( s i , a 1 ) ≤ Q k ( s i , a 2 ) for i > k +1 so that the probability of π k +1 ( a 1 | s i ) cannot increase w.r.t π k ( a 1 | s i ) , which is less than or equal to α by the inductive hypothesis. We have:

where we used in (a) that π k ( a 1 | s i -1 ) ≤ α for i > k +1 , which is true by the inductive hypothesis since i -1 > k , in (b) that α ≤ δ (1 -γ ) and in (c) that γ (1 + γ i -1 -γ i ) < 1 for γ ∈ [0 , 1) . This concludes the proof of the claim.

Now using the claim

so

where we used that α ≤ δ . Now note that

so

and by induction we can show this is the case for all states (above is base case), the inductive step is as follows (assuming V ⋆ ( s k ) -V 0 ( s k ) ≤ γ ),

and so which combining with (17) gives,

Choosing δ = (1 -γ ) 4 γ n ∥ V ⋆ -V 0 ∥ ∞ ≤ (1 -γ ) 4 γ n +1 (which is in the valid range of δ ) concludes the proof since 2 δ 1 -γ ≤ 1 2 γ k ∥ V ⋆ -V 0 ∥ ∞ . ■

F Proof of Theorem 4.3

Consider the same MDP as in the proof of Theorem 4.2 in Appendix E (see Figure 5). Denote c = 1 -γ 8 and note that c < √ γ 1+ √ γ 1 -γ 2 since 1 4 < √ γ 1+ √ γ for γ > 0 . 2 .

Suppose you consider NPG updates with initial policy π 0 ( a 1 | s i ) = α . Recall that NPG is the instance of PMD with relative entropy as the mirror map. It can be shown that NPG has the closed form update

We know from the proof of Theorem E that for any step-size regime, for i > k +1

Now, ∥ V ⋆ -V 0 ∥ ∞ = V ⋆ ( s 1 ) -V 0 ( s 1 ) ≤ γ -δ 1 -γ (see Section F.1 below). The idea of the proof is to show that satisfying the bound given in the statement of the theorem will imply that a certain condition on the step-size.

Fix a state s k and let k 0 be the first iteration where Q k 0 ( s k , a 1 ) > Q k 0 ( s k , a 2 ) . By the above, we must have k ≤ k 0 +1 , or k 0 ≥ k -1 . By the proof of Theorem E, we also have π k 0 ( a 1 | s k ) ≤ α (before iteration k 0 , Q ( s k , · ) favors a 2 , so π k 0 ( a 1 | s k ) has not increased compared to π 0 ( a 1 | s k ) = α ).

We want a γ -contraction at every iteration, i.e. we assume the following is satisfied:

Now, by direct computation,

Putting this together with the above (this is an implication as this is about the necessity rather than sufficiency), we must have:

If we choose δ < 1 2 (1 -γ )(1 - √ γ ) γ k then β < √ γ and require

To see this, start from β ≤ √ γ , this is equivalent to

which is the condition for δ we imposed initially.

To achieve the above condition π k 0 +1 ( a 2 | s 2 ) ≤ √ γ , recalling that π k 0 ( a 2 | s 2 ) ≥ 1 -α , η k 0 has to satisfy

To see this, again start from π k 0 +1 ( a 2 | s 2 ) ≤ √ γ , this is equivalent to (use k 0 = m for simplicity of notation) using the closed-form update of NPG:

As we take α → 0 , the KL term will dominate. In particular, note α < 1 -γ so 1 -α > γ and

and if we further impose the condition α < γ 2 (1 - √ γ ) 2 then

and the step-size needs to satisfy the following condition:

Distinct Iterations: Note that the iteration k 0 ( s k ) where Q ( · , s k ) starts becoming bigger at a 1 that a 2 is distinct for each s k . Fix any s k and k 0 = k 0 ( s k ) . We have

then π k 0 +1 ( a 1 | s k ) ≤ π k 0 ( a 1 | s k ) ≤ α (since Q t points towards a 2 in s k for all t ≤ k 0 ). Then applying exactly the same steps as in the proof of Theorem 4.2, we have

meaning that k 0 ( s k ) is disctinct to k 0 ( s k +1 ) .

Upper Bounding Q-value difference: We want to upper-bound the Q-value difference appearing in the step-size condition above. We have,

Plugging this into the above bound (18), if the iterates of NPG are to satisfy the bound with the γ -rate in the statement of the theorem, the step-size must at least satisfy the following condition:

which concludes the proof.

F.1 Largest sub-optimality gap at iteration 0

In this section, we prove the claim that

Proof: First of all, V ⋆ ( s 1 ) -V 0 ( s 1 ) = π 0 ( a 2 | s 1 ) γ -r 1 1 -γ = (1 -α )( γ -δ 1 -γ ) ≤ γ -δ 1 -γ . For the first part, we proceed by induction. We will use throughout that

So,

This is true if (when LHS is the largest)

which holds when

Base Case:

Inductive Step: Assume true for k . Then,

which concludes the proof.

■

G Inexact policy mirror descent and the generative model

The following Lemma from [38] controls the accuracy of the estimator ̂ Q k s specified in (10) of Section 5 with respect to H and M k :

Lemma G.1 (Lemma 15 in [38]) . Consider using (10) to estimate Q k s for all state-action pairs for K iterations of IPMD. Then for δ ∈ (0 , 1) , if for all k ≤ K ,

Then with probability at least 1 -δ , we have for all k ≤ K ,

The proof of this result can be found in Lemma 15 of [38].

G.1 Proof of Theorem 5.1

This proof is similar to that of [38] (Theorem 14). It is also similar in structure to the proof of Theorem 4.1 in Section 6.

Fix a state s ∈ S and an integer k ≥ 0 . For now let's assume that our Q-estimates are τ -accurate ( τ > 0 ), i.e.

for all k ≥ 0 . With this assumption, we have from Lemma A.5 in Appendix A.1,

Now proceeding in a similar way to Section 6,

Now again proceeding exactly as in Section 6 with this extra τ -term using the step-size condition ( c k = γ 2 k +1 ), we end up with

Unravelling this recursion yields

Now using the properties of the estimator (10) in Lemma G.1, we have with probability 1 -δ for all 0 ≤ k ≤ K ,

giving

This establishes the first bound. Now

as required. In terms of M, we have

and the corresponding number of calls to the sampling model, i.e. the sample complexity is (what we have shown above is actually a lower bound but can choose K,H,M so that it is of the following order),

where the notation ˜ O () hides poly-logarithmic factors. This completes the proof.

giving

■

H MDPexamples

H.1 MDPon which distribution-mismatch coefficient scales with size of state space

We construct an MDP on which

̸

scales with |S| , and hence so does the iteration complexity of the bound of [38] for exact PMD.

Consider an MDP with state-space S = { s 1 , s 2 , ..., s n } of size n and arbitrary action space A . s 1 is an absorbing state giving out rewards of 1 at each time-step, regardless of the action taken, i.e

All others states have an action, say a 1 , that gives out a reward of 1 and with probability 1 -δ brings the agent to state s 1 for some δ > 0 and spreads the remaining δ probability arbitrarily amongst the other states. The other actions have arbitrary rewards strictly less than 1 associated to them, and arbitrary transition probabilities that place 0 mass on state s 1 , i.e

̸

̸

̸

̸

Denote r max = max s = s 1 ,a = a 1 r ( s, a ) < 1 . The following condition ensures that a 1 is the optimal action in all states,

̸

so that π ⋆ ( s ) = a 1 for all states s . To see this, consider s i = s 1 , a m = a 1 and an arbitrary policy π ,

̸

and solving

will yield the condition above.

Then for t ≥ 1 (abusing notation, s t denotes the state at time t),

and

Now

and depending on what ρ you consider, θ ρ can be arbitrarily large. In particular, the natural choice of the uniform starting-state distribution ρ ( s ) = 1 /n leads to

which gives an iteration complexity under the result of [38] for an ε -optimal policy that is

Recall that n = |S| , so this iteration complexity scales linearly with the size of the state space.

H.2 Family of MDPs on which sub-optimality gaps can be made arbitrarily small

We present how to construct a family of MDPs on which ∆ k ( s ) defined in Section 4 can be made arbitrarily small.

Consider an arbitrary MDP M with state space S and action space A . For each state-action pair ( s, a ) ∈ S × A , create a duplicate action a ′ s.t the transitions from that action in that state are the same as for the original pair, i.e

and the reward is shifted down by δ > 0 from the original reward, i.e

This results in a new MDP M ′ with an augmented action space A ′ , that is twice the size of the action space of the original MDP M . In terms of action-value of an arbitrary policy π , this results in

where the notation Q π M ′ refers to action-values in the MDP M ′ . In terms of sub-optimality gaps, this gives

Choosing δ small enough, we can make the step-size of [22] arbitrarily large, at least in early iterations. The step-size condition (5) of Theorem 4.1 will be less affected by this issue as it does not depend directly on ∆ k ( s ) , and not at all in the first iteration. Beyond its generality to PMD, this illustrates the benefit of our result restricted to NPG over the result of [22].

Peer reviews:
comments: SUMMARY:
This paper proposes a new analysis of the policy mirror descent algorithm and achieves a linear convergence rate with exact $Q$-function that does not contain any distribution mismatch coefficient. Furthermore, it also proposes lower bounds showing that the linear convergence rate is optimal under specific setting and policy-dependent learning rates are necessary to achieve this rate.

STRENGTHS:
- The convergence result for policy mirror descent is strong while the analysis is concise.
- The proposed two lower bounds are also considered to be novel.

WEAKNESSES:
One weakness of this paper lies in the lower bound stated in Theorem 4.2 only holds for $k<n$, which does not rule out possibility of stronger convergence rate when $k\geq n$. Therefore, it seems not very appropriate to claim that this result is optimal without any qualifiers.

Another weakness is on the lack of experiments. Since the major results of this paper strongly rely on a delibrately designed learning rate, it will be much better to empirically (in some tabular MDPs) see that this choice of learning rate is indeed better than other choices.

QUESTIONS:
- What are the difficulties of extending the result in Theorem 4.2 to the setting of $k\geq n$?
- Is that possible to strengthen the Theorem 4.3 in a form saying that (under the same or similar setting for $n$, $\gamma$ and $\mathcal{S}$) policy-dependent learning rates are necessary whenever we hope to achieve $\Vert V^\star-V^k\Vert_{\infty}\leq c_0\gamma^k(\Vert V^\star-V^0\Vert_{\infty}+c_1)$ for any $c_0, c_1>0$?
- Is $\eta_k$ always guaranteed to be finite?

LIMITATIONS:
The limitations are not addressed very well as explained in the weakness section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the convergence rate of a general family of policy gradient methods called Policy Mirror Descent (PMD) in the setting of discounted MDP, unregularized objective, and tabular policy model. In prior work ((Xiao, 2022), [7] in the paper), it has been known that with a geometrically increasing step size schedule, all PMD algorithms enjoy nice convergence rate and sample complexity with respect to the discounting factor $\\gamma$, but the bounds proved in [7] also depend on a \"distribution mismatch coefficient\" term. This paper shows that with a more adaptive step-size schedule, this \"distribution mismatch coefficient\" term can be removed from the bounds. The results are derived by simplifying the analysis in [7] with a value improvement property of PMD algorithms (Lemma A.2). The paper also presents theorems to argue that the convergence rate they proved is tight, and that the adaptive step-size schedule is necessary for achieving this rate.\n\nSTRENGTHS:\n1. This paper seems to be a solid follow-up work after (Xiao, 2022) [7]. The paper gives a nice analysis to the PMD algorithms (removing unnecessary and \"ugly\" terms in previous results, simplifying the analysis framework, providing optimality/necessity arguments, etc.).\n\n2. The results presented in this paper are nicely compared and contrasted with previous works.\n\nWEAKNESSES:\n1. I am not very sure about the significance of the results presented by the paper. While it is nice to see that with carefully scheduled step sizes every PMD algorithm could match the performance of PI and NPG, the step-size schedule required does not look very practically feasible to me, and it's still not clear if any concrete and practical PMD algorithm can demonstrate better performance than PI and NPG. Because PI and NPG are two special cases of PMD, bounds applying to all PMD algorithms are necessarily subject to the bounds of PI and NPG, so I wonder if aiming at proving bounds that \"universally\" apply to all PMD algorithms, as this paper does, can be truly helpful in identifying *better* PMD algorithms. Maybe the authors can consider to better motivate this work.\n\n2. I have some concern about whether Theorem 4.2 can indeed establish the optimality of the convergence rate as proved in the paper. If I understand correctly, Theorem 4.2 says that for any given state-space size $|S|$, there exists a MDP with $|S|$ states such that no PMD algorithm can be faster than the linear $\\gamma$-rate in the first $|S|/2$ PMD iterations. But this result does not bound the asymptotic performance of PMD, right? As the paper already mentioned (Line 265), PI can converge exactly to the optimal policy in finite iterations, this fact means PI, as one instance in the PMD family, must converge faster than the linear $\\gamma$-rate after the first $|S|/2$ iterations. In this case I am not sure why the linear bound in Theorem 4.1 must be optimal.\n\nQUESTIONS:\nA. Table 1: Why isn't \"linear $\\gamma$-rate\" checked for [7], while at Line 31 you said \"*linear convergence of PMD was established by [7]*\"?\n\nB. Line 264-266: I wonder if extending (7) to k>n will necessarily contradict with the fact that PI converges in finite iterations. Specifically, the latter implies that for large enough k, we have $||V^*-V^k||=0$. To have (7) hold for such k, we \"only\" need to have $\\gamma^k ||V^*-V^0||\\leq \\frac{2\\delta}{1-\\gamma}$, which does not look impossible to me unless $\\delta$ and $V^0$ have some special constraints -- can you explain a bit more here? Although (7) is linear to $\\gamma^k$, does the negative constant term $-\\frac{2\\delta}{1-\\gamma}$ in (7) actually allows exact convergence to $V^*$?\n\nC. Line 308: $(s^0_t, a^0_t)$ should be $(s^i_0, a^i_0)$ ?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work considers the class of Policy Mirror Descent (PMD) methods and establishes a linear $\\gamma$-rate which is dimension-free using adaptive step sizes. In particular, the convergence rate does not depend on visitation distribution mismatch coefficients or the state/action space sizes. It is further shown that this rate is optimal for PMD methods as it is the case for Policy Iteration via a matching lower bound. Adaptive step sizes are also shown to be necessary for this result to hold. Improving over prior work, the analysis which does not make use of the performance difference lemma is further extended to the inexact setting where the action value function needs to be estimated under a generative model.\n\nSTRENGTHS:\n- The independence of the rate on the visitation distribution mismatch coefficients is interesting and the novel analysis avoids using the performance difference lemma to achieve this. The flexibility of the analysis offers the possibility to close the gap between the convergence rates of PG methods and value-based methods in terms of $1/(1-\\gamma)$ and match the known lower bound. \n\n- The convergence rate which holds for the $l_{\\infty}$ norm is shown to be optimal and the adaptive step size is also necessary.  \n\n- The paper is well-written and the contributions are clearly presented.\n\n- The results are sound and the proof of the main result is clean.\n\nWEAKNESSES:\n- The interesting dimension-free rate seems to come with the price of more complicated adaptive step sizes compared to the increasing steps in [7]. \n\n- Some recent related works going beyond the tabular setting are missing in the discussion, especially regarding l. 131-133.  \n\nYuan, R., Du, S. S., Gower, R. M., Lazaric, A., & Xiao, L. Linear convergence of natural policy gradient methods with log-linear policies. ICLR 2023\n\nAlfano, C., & Rebeschini, P. (2022). Linear convergence for natural policy gradient with log-linear policy parametrization. arXiv preprint arXiv:2209.15382.\n\nAlfano, C., Yuan, R., & Rebeschini, P. (2023). A novel framework for policy mirror descent with general parametrization and linear convergence. arXiv preprint arXiv:2301.13139.\n\n- While a matching lower bound is provided for the convergence rate, the dependence with respect to $1/(1-\\gamma)$ does not improve over prior work and does not match the known lower bound as acknowledged by the paper. \n\n-  Leveraging a connection with Policy Iteration, a dimension free rate was previously established in the literature in [9] for the NPG method with adaptive step sizes which is a particular case of the PMD class of methods. The present work extends somehow this result with adaptive step sizes which are independent on the minimal sub-optimality gap, using a different analysis (as discussed in l. 237 to 253). The extension to the inexact setting follows the same lines as prior work [7].    \n\n**Minor:** \n\n- l. 174: The function $h$ is continuously differentiable on the relative interior of the simplex $\\Delta(\\mathcal{A})$, so I guess $\\pi'$ belongs to the relative interior of $\\Delta(\\mathcal{A})$. \n\n**Typos:** \n- l. 184: mirror map h -> $h$\n- Eq. (4) (and Eq. (9)): $p \\in \\Delta(\\mathcal{A})$ and then $p_s$ is used in the argmin.\n\nQUESTIONS:\n1.  How easy is it to compute the adaptive step sizes in (5)? Is there any computational tradeoff? The step sizes seem to potentially involve dependence on the state/action space sizes and would require some computational efforts to be computed. For instance, computing the set $\\mathcal{A}_s^k$ requires screening the action space. Could you comment on this and compare to the step sizes in [7] regarding this aspect?  \n\n2. Does Theorem 4.3 about adaptive step-size necessity only hold for the particular case of the negative entropy for $h$ unlike all other results which hold for the general class of PMD methods? Could you comment on the reasons for that?  \n\n3. In Eq. (8), is the definition of $\\tilde{\\pi}_{s_i}^{k_i + 1}$ the same as in (5)? What are the states ${s_i}$ in l. 289?\n\nLIMITATIONS:\nThe suboptimality in terms of the dependence on $1/(1-\\gamma)$ (which does not match the known lower bound) is discussed both in l. 330-332 and in the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper studies the convergence rate of Policy Mirror Descent. The authors prove a $\gamma$-rate for exact PMD update and a matching lower bound for this class of algorithm. Then they prove a sample complexity bound of $O(\epsilon^{-2}(1-\gamma)^{-8})$.

STRENGTHS:
The paper analyses an interesting problem on the convergence rate for a general class of RL algorithms. The work is nicely structured and the theoretical results seems not trivial to obtain.

WEAKNESSES:
In the reviewer's opinion the paper does not convey clearly the contribution of the paper. More specifically it's hard to understand if the paper advances theoretically the field of RL or just the theoretical understanding of PMD.

Minor:
The following sentence in the abstract: "Motivated by the instability of policy iteration (PI) with inexact policy evaluation, unregularised PMD algorithmically regularises the policy improvement step of PI without regular5 ising the objective function" sounds convoluted and maybe even grammatically incorrect. Please have a look

QUESTIONS:
1) A lower bound on a class of algorithm and not on the problem seems strange. Could please the author elaborate on this point? Is this standard in RL literature?
2) The terminology: Unregularized PMD is not explicitly explained in text. It would be helpful to clarity it explicitly.

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07