Title: Accelerating Value Iteration with Anchoring
Venue: NeurIPS 2023
Accepted: yes
Keywords: Value Iteration

Reinforcement Learning

Reinforcement Learning Theory

Dynamic Programming

Acceleration

Anchoring mechanism
Abstract: Value Iteration (VI) is foundational to the theory and practice of modern reinforcement learning, and it is known to converge at a $\mathcal{O}(\gamma^k)$-rate. Surprisingly, however, the optimal rate for the VI setup was not known, and finding a general acceleration mechanism has been an open problem. In this paper, we present the first accelerated VI for both the Bellman consistency and optimality operators. Our method, called Anc-VI, is based on an \emph{anchoring} mechanism (distinct from Nesterov's acceleration), and it reduces the Bellman error faster than standard VI. In particular, Anc-VI exhibits a $\mathcal{O}(1/k)$-rate for $\gamma\approx 1$ or even $\gamma=1$, while standard VI has rate $\mathcal{O}(1)$ for $\gamma\ge 1-1/k$, where $k$ is the iteration count. We also provide a complexity lower bound matching the upper bound up to a constant factor of $4$, thereby establishing optimality of the accelerated rate of Anc-VI. Finally, we show that the anchoring mechanism provides the same benefit in the approximate VI and Gauss--Seidel VI setups as well.

Accelerating Value Iteration with Anchoring

Abstract

Value Iteration (VI) is foundational to the theory and practice of modern reinforcement learning, and it is known to converge at a O ( γ k ) -rate, where γ is the discount factor. Surprisingly, however, the optimal rate in terms of Bellman error for the VI setup was not known, and finding a general acceleration mechanism has been an open problem. In this paper, we present the first accelerated VI for both the Bellman consistency and optimality operators. Our method, called Anc-VI, is based on an anchoring mechanism (distinct from Nesterov's acceleration), and it reduces the Bellman error faster than standard VI. In particular, Anc-VI exhibits a O (1 /k ) -rate for γ ≈ 1 or even γ = 1 , while standard VI has rate O (1) for γ ≥ 1 -1 /k , where k is the iteration count. We also provide a complexity lower bound matching the upper bound up to a constant factor of 4 , thereby establishing optimality of the accelerated rate of Anc-VI. Finally, we show that the anchoring mechanism provides the same benefit in the approximate VI and Gauss-Seidel VI setups as well.

1 Introduction

Value Iteration (VI) is foundational to the theory and practice of modern dynamic programming (DP) and reinforcement learning (RL). It is well known that when a discount factor γ < 1 is used, (exact) VI is a contractive iteration in the ∥·∥ ∞ -norm and therefore converges. The progress of VI is measured by the Bellman error in practice (as the distance to the fixed point is not computable), and much prior work has been dedicated to analyzing the rates of convergence of VI and its variants.

Surprisingly, however, the optimal rate in terms of Bellman error for the VI setup was not known, and finding a general acceleration mechanism has been an open problem. The classical O ( γ k ) -rate of VI is inadequate as many practical setups use γ ≈ 1 or γ = 1 for the discount factor. (Not to mention that VI may not converge when γ = 1 .) Moreover, most prior works on accelerating VI focused on the Bellman consistency operator (policy evaluation) as its linearity allows eigenvalue analyses, but the Bellman optimality operator (control) is the more relevant object in modern RL.

Contribution. In this paper, we present the first accelerated VI for both the Bellman consistency and optimality operators. Our method, called Anc-VI, is based on an 'anchoring' mechanism (distinct from Nesterov's acceleration), and it reduces the Bellman error faster than standard VI. In particular, Anc-VI exhibits a O (1 /k ) -rate for γ ≈ 1 or even γ = 1 , while standard VI has rate O (1) for γ ≥ 1 -1 /k , where k is the iteration count. We also provide a complexity lower bound matching the upper bound up to a constant factor of 4 , thereby establishing optimality of the accelerated rate of Anc-VI. Finally, we show that the anchoring mechanism provides the same benefit in the approximate VI and Gauss-Seidel VI setups as well.

1.1 Notations and preliminaries

We quickly review basic definitions and concepts of Markov decision processes (MDP) and reinforcement learning (RL). For further details, refer to standard references such as [69, 84, 81].

Markov Decision Process. Let M ( X ) be the space of probability distributions over X . Write ( S , A , P, r, γ ) to denote the MDP with state space S , action space A , transition probability P : S × A → M ( S ) , reward r : S × A → R , and discount factor γ ∈ (0 , 1] . Denote π : S → M ( A ) for a policy, V π ( s ) = E π [ ∑ ∞ t =0 γ t r ( s t , a t ) | s 0 = s ] and Q π ( s, a ) = E π [ ∑ ∞ t =0 γ t r ( s t , a t ) | s 0 = s, a 0 = a ] for V - and Q -value functions, where E π denotes the expected value over all trajectories ( s 0 , a 0 , s 1 , a 1 , . . . ) induced by P and π . We say V ⋆ and Q ⋆ are optimal V - and Q - value functions if V ⋆ = sup π V π and Q ⋆ = sup π Q π . We say π ⋆ V and π ⋆ Q are optimal policies if π ⋆ V = argmax π V π and π ⋆ Q = argmax π Q π . (If argmax is not unique, break ties arbitrarily.)

Value Iteration. Let F ( X ) denote the space of bounded measurable real-valued functions over X . With the given MDP ( S , A , P, r, γ ) , for V ∈ F ( S ) and Q ∈ F ( S × A ) , define the Bellman consistency operators T π as

for all s ∈ S , a ∈ A , and the Bellman optimality operators T ⋆ as

for all s ∈ S , a ∈ A . For notational conciseness, we write T π V = r π + γ P π V and T π Q = r + γ P π Q , where r π ( s ) = E a ∼ π ( · | s ) [ r ( s, a )] is the reward induced by policy π and P π ( s ) and P π ( s, a ) defined as

are the transition probabilities induced by policy π . We define VI for Bellman consistency and optimality operators as

where V 0 , Q 0 are initial points. VI for control, after executing K iterations, returns the near-optimal policy π K as a greedy policy satisfying

For γ < 1 , both Bellman consistency and optimality operators are contractions, and, by Banach's fixed-point theorem [5], the VIs converge to the unique fixed points V π , Q π , V ⋆ , and Q ⋆ with O ( γ k ) -rate. For notational unity, we use the symbol U when both V and Q can be used. Since ∥ ∥ TU k -U k ∥ ∥ ∞ ≤ ∥ ∥ TU k -U ⋆ ∥ ∥ ∞ + ∥ ∥ U k -U ⋆ ∥ ∥ ∞ ≤ (1 + γ ) ∥ ∥ U k -U ⋆ ∥ ∥ ∞ , VI exhibits the rate on the Bellman error:

where T is Bellman consistency or optimality operator, U 0 is a starting point, and U ⋆ is fixed point of T . We say V ≤ V ′ or Q ≤ Q ′ if V ( s ) ≤ V ′ ( s ) or Q ( s, a ) ≤ Q ′ ( s, a ) for all s ∈ S and a ∈ A , respectively.

Fixed-point iterations. Given an operator T , we say x ⋆ is fixed point if Tx ⋆ = x ⋆ . Since Banach [5], the standard fixed-point iteration

has been commonly used to find fixed points. Note that VI for policy evaluation and control are fixed-point iterations with Bellman consistency and optimality operators. In this work, we also consider the Halpern iteration

where x 0 is an initial point and { β k } k ∈ N ∈ (0 , 1) .

1.2 Prior works

Value Iteration. Value iteration (VI) was first introduced in the DP literature [8] for finding optimal value function, and its variant approximate VI [11, 30, 56, 32, 19, 90, 81] considers approximate evaluations of the Bellman optimality operator. In RL, VI and approximate VI have served as the basis of RL algorithms such as fitted value iteration [29, 57, 52, 87, 50, 36] and temporal difference learning [80, 89, 41, 94, 54]. There is a line of research that emulates VI by learning a model of the MDP dynamics [85, 83, 62] and applying a modified Bellman operator [7, 33]. Asynchronous VI, another variation of VI updating the coordinate of value function in asynchronous manner, has also been studied in both RL and DP literature [11, 9, 88, 100].

Fixed-point iterations. The Banach fixed-point theorem [5] establishes the convergence of the standard fixed-point iteration with a contractive operator. The Halpern iteration [39] converges for nonexpansive operators on Hilbert spaces [96] and uniformly smooth Banach spaces [70, 97]. (To clarify, the ∥ · ∥ ∞ -norm in R n is not uniformly smooth.)

The fixed-point residual ∥ Tx k -x k ∥ is a commonly used error measure for fixed-point problems. In general normed spaces, the Halpern iteration was shown to exhibit O (1 / log( k )) -rate for (nonlinear) nonexpansive operators [48] and O (1 /k ) -rate for linear nonexpansive operators [17] on the fixedpoint residual. In Hilbert spaces, [72] first established a O (1 /k ) -rate for the Halpern iteration and the constant was later improved by [49, 43]. For contractive operators, [65] proved exact optimality of Halpern iteration through an exact matching complexity lower bound.

Acceleration. Since Nesterov's seminal work [61], there has been a large body of research on acceleration in convex minimization. Gradient descent [15] can be accelerated to efficiently reduce function value and squared gradient magnitude for smooth convex minimization problems [61, 44, 45, 46, 102, 21, 60] and smooth strongly convex minimization problems [59, 91, 64, 86, 73]. Motivated by Nesterov acceleration, inertial fixed-point iterations [51, 22, 75, 70, 42] have also been suggested to accelerate fixed-point iterations. Anderson acceleration [2], another acceleration scheme for fixed-point iterations, has recently been studied with interest [6, 74, 93, 101].

In DP and RL, prioritized sweeping [55] is a well-known method that changes the order of updates to accelerate convergence, and several variants [68, 53, 95, 3, 18] have been proposed. Speedy Qlearning [4] modifies the update rule of Q-learning and uses aggressive learning rates for acceleration. Recently, there has been a line of research that applies acceleration techniques of other areas to VI: [34, 79, 28, 67, 27, 76] uses Anderson acceleration of fixed-point iterations, [92, 37, 38, 12, 1] uses Nesterov acceleration of convex optimization, and [31] uses ideas inspired by PID controllers in control theory. Among those works, [37, 38, 1] applied Nesterov acceleration to obtain theoretically accelerated convergence rates, but those analyses require certain reversibility conditions or restrictions on eigenvalues of the transition probability induced by the policy.

The anchor acceleration , a new acceleration mechanism distinct from Nesterov's, lately gained attention in convex optimization and fixed-point theory. The anchoring mechanism, which retracts iterates towards the initial point, has been used to accelerate algorithms for minimax optimization and fixed-point problems [71, 47, 98, 65, 43, 20, 99, 78], and we focus on it in this paper.

Complexity lower bound. With the information-based complexity analysis [58], complexity lower bound on first-order methods for convex minimization problem has been thoroughly studied [59, 23, 25, 13, 14, 24]. If a complexity lower bound matches an algorithm's convergence rate, it establishes optimality of the algorithm [58, 44, 73, 86, 26, 65]. In fixed-point problems, [16] established Ω(1 /k 1 -√ 2 /q ) lower bound on distance to solution for Halpern iteration with a nonexpansive operator in q -uniformly smooth Banach spaces. In [17], a Ω(1 /k ) lower bound on the fixed-point residual for the general Mann iteration with a nonexpansive linear operator, which includes standard fixed-point iteration and Halpern iterations, in the ℓ ∞ -space was provided. In Hilbert spaces, [65] showed exact complexity lower bound on fixed-point residual for deterministic fixed-point iterations with γ -contractive and nonexpansive operators. Finally, [37] provided lower bound on distance to optimal value function for fixed-point iterations satisfying span condition with Bellman consistency and optimality operators and we discussed this lower bound in section 4.

2 Anchored Value Iteration

Let T be a γ -contractive (in the ∥ · ∥ ∞ -norm) Bellman consistency or optimality operator. The Anchored Value Iteration (Anc-VI) is

for k = 1 , 2 , . . . , where β k = 1 / ( ∑ k i =0 γ -2 i ) and U 0 is an initial point. In this section, we present accelerated convergence rates of Anc-VI for both Bellman consistency and optimality operators for both V - and Q -value iterations. For the control setup, where the Bellman optimality operator is used, Anc-VI returns the near-optimal policy π K as a greedy policy satisfying T π K U K = T ⋆ U K after executing K iterations.

Notably, Anc-VI obtains the next iterate as a convex combination between the output of T and the starting point U 0 . We call the β k U 0 term the anchor term since, loosely speaking, it serves to pull the iterates toward the starting point U 0 . The strength of the anchor mechanism diminishes as the iteration progresses since β k is a decreasing sequence.

The anchor mechanism was introduced [39, 72, 49, 65, 17, 48] for general nonexpansive operators and ∥ · ∥ 2 -nonexpansive and contractive operators. The optimal method for ∥ · ∥ 2 -nonexpansive and contractive operators in [65] shares the same coefficients with Anc-VI, and convergence results for general nonexapnsive operators in [17, 48] are applicable to Anc-VI for nonexpansive Bellman optimality and consistency operators. While our anchor mechanism does bear a formal resemblance to those of prior works, our convergence rates and point convergence are neither a direct application nor a direct adaptation of the prior convergence analyses. The prior analyses for ∥ · ∥ 2 -nonexpansive and contractive operators do not apply to Bellman operators, and prior analyses for general nonexpansive operators have slower rates and do not provide point convergence while our Theorem 3 does. Our analyses specifically utilize the structure of Bellman operators to obtain the faster rates and point convergence.

The accelerated rate of Anc-VI for the Bellman optimality operator is more technically challenging and is, in our view, the stronger contribution. However, we start by presenting the result for the Bellman consistency operator because it is commonly studied in the prior RL theory literature on accelerating value iteration [37, 38, 1, 31] and because the analysis in the Bellman consistency setup will serve as a good conceptual stepping stone towards the analysis in the Bellman optimality setup.

2.1 Accelerated rate for Bellman consistency operator

First, for general state-action spaces, we present the accelerated convergence rate of Anc-VI for the Bellman consistency operator.

Theorem 1. Let 0 < γ < 1 be the discount factor and π be a policy. Let T π be the Bellman consistency operator for V or Q . Then, Anc-VI exhibits the rate

where ϵ = 1 -γ and the bigO notation considers the limit ϵ → 0 . If, furthermore, U 0 ≤ T π U 0 or U 0 ≥ T π U 0 , then Anc-VI exhibits the rate

If γ ≥ 1 2 , both rates of Theorem 1 are strictly faster than the standard rate ( 1 ) of VI, since

The second rate of Theorem 1, which has the additional requirement, is faster than the standard rate ( 1 ) of VI for all 0 < γ < 1 . Interestingly, in the γ ≈ 1 regime, Anc-VI achieves O (1 /k ) -rate while VI has a O (1) -rate. We briefly note that the condition U 0 ≤ TU 0 and U 0 ≥ TU 0 have been used in analyses of variants of VI [69, Theorem 6.3.11], [77, p.3].

In the following, we briefly outline the proof of Theorem 1 while deferring the full description to Appendix B. In the outline, we highlight a particular step, labeled ▲ , that crucially relies on the linearity of the Bellman consistency operator. In the analysis for the Bellman optimality operator of Theorem 2, resolving the ▲ step despite the nonlinearity is the key technical challenge.

Proof outline of Theorem 1. Recall that we can write Bellman consistency operator as T π V = r π + γ P π V and T π Q = r + γ P π Q . Since T π is a linear operator 1 , we get

where the first equality follows from the definition of Anc-VI and the property of fixed point, while the last equality follows from induction. Taking the ∥·∥ ∞ -norm of both sides, we conclude

2.2 Accelerated rate for Bellman optimality operator

We now present the accelerated convergence rate of Anc-VI for the Bellman optimality operator.

Our analysis uses what we call the Bellman anti-optimality operator , defined as

for all s ∈ S and a ∈ A . (The sup is replaced with a inf.) When 0 < γ < 1 , the Bellman antioptimality operator is γ -contractive and has a unique fixed point ˆ U ⋆ by the exact same arguments that establish γ -contractiveness of the standard Bellman optimality operator.

Theorem 2. Let 0 < γ < 1 be the discount factor. Let T ⋆ and ˆ T ⋆ respectively be the Bellman optimality and anti-optimality operators for V or Q . Let U ⋆ and ˆ U ⋆ respectively be the fixed points of T ⋆ and ˆ T ⋆ . Then, Anc-VI exhibits the rate

for k = 0 , 1 , . . . . If, furthermore, U 0 ≤ T ⋆ U 0 or U 0 ≥ T ⋆ U 0 , then Anc-VI exhibits the rate

for k = 0 , 1 , . . . .

Anc-VI with the Bellman optimality operator exhibits the same accelerated convergence rate as Anc-VI with the Bellman consistency operator. As in Theorem 1, the rate of Theorem 2 also becomes O (1 /k ) when γ ≈ 1 , while VI has a O (1) -rate.

Proof outline of Theorem 2. The key technical challenge of the proof comes from the fact that the Bellman optimality operator is non-linear. Similar to the Bellman consistency operator case, we have

where π k is the greedy policy satisfying T π k U k = T ⋆ U k , we define Π i l = k γ P π l = γ P π k γ P π k -1 · · · γ P π i , and last inequality follows by induction and monotonicity of Bellman optimality operator. The key step ▲ uses greedy policies { π l } l =0 , 1 ,...,k , which are well defined when the action space is finite. When the action space is infinite, greedy policies may not exist, so we use the Hahn-Banach extension theorem to overcome this technicality. The full argument is provided in Appendix B.

To lower bound T ⋆ U k -U k , we use a similar line of reasoning with the Bellman anti-optimality operator. Combining the upper and lower bounds of T ⋆ U k -U k , we conclude the accelerated rate of Theorem 2.

For γ < 1 , the rates of Theorems 1 and 2 can be translated to a bound on the distance to solution:

for k = 1 , 2 , . . . . This O ( γ k ) rate is worse than the rate of (classical) VI by a constant factor. Therefore, Anc-VI is better than VI in terms of the Bellman error, but it is not better than VI in terms of distance to solution.

3 Convergence when γ = 1

Undiscounted MDPs are not commonly studied in the DP and RL theory literature due to the following difficulties: Bellman consistency and optimality operators may not have fixed points, VI is a nonexpansive (not contractive) fixed-point iteration and may not convergence to a fixed point even if one exist, and the interpretation of a fixed point as the (optimal) value function becomes unclear when the fixed point is not unique. However, many modern deep RL setups actually do not use discounting, 2 and this empirical practice makes the theoretical analysis with γ = 1 relevant.

In this section, we show that Anc-VI converges to fixed points of the Bellman consistency and optimality operators of undiscounted MDPs. While a full treatment of undiscounted MDPs is beyond the scope of this paper, we show that fixed points, if one exists, can be found, and we therefore argue that the inability to find fixed points should not be considered an obstacle in studying the γ = 1 setup.

We first state our convergence result for finite state-action spaces.

Theorem 3. Let γ = 1 . Let T : R n → R n be the nonexpansive Bellman consistency or optimality operator for V or Q . Assume a fixed point exists (not necessarily unique). If, U 0 ≤ TU 0 , then Anc-VI exhibits the rate

for any fixed point U ⋆ satisfying U 0 ≤ U ⋆ . Furthermore, U k → U ∞ for some fixed point U ∞ .

If rewards are nonnegative, then the condition U 0 ≤ TU 0 is satisfied with U 0 = 0 . So, under this mild condition, Anc-VI with γ = 1 converges with O (1 /k ) -rate on the Bellman error. To clarify, the convergence U k → U ∞ has no rate, i.e., ∥ U k -U ∞ ∥ ∞ = o (1) , while ∥ ∥ TU k -U k ∥ ∥ ∞ = O (1 /k ) . In contrast, standard VI does not guarantee convergence in this setup.

We also point out that the convergence of Bellman error does not immediately imply point convergence, i.e., TU k -U k → 0 does not immediately imply U k → U ⋆ , when γ = 1 . Rather, we show (i) U k is a bounded sequence, (ii) any convergent subsequence U k j converges to a fixed point U ∞ , and (iii) U k is elementwise monotonically nondecreasing and therefore has a single limit.

Next, we state our convergence result for general state-action spaces.

Theorem 4. Let γ = 1 . Let the state and action spaces be general (possibly infinite) sets. Let T be the nonexpansive Bellman consistency or optimality operator for V or Q , and assume T is well defined. 3 Assume a fixed point exists (not necessarily unique). If U 0 ≤ TU 0 , then Anc-VI exhibits the rate

for any fixed point U ⋆ satisfying U 0 ≤ U ⋆ . Furthermore, U k → U ∞ pointwise monotonically for some fixed point U ∞ .

The convergence U k → U ∞ pointwise in infinite state-action spaces is, in our view, a non-trivial contribution. When the state-action space is finite, pointwise convergence directly implies convergence in ∥ · ∥ ∞ , and in this sense, Theorem 4 is generalization of Theorem 3. However, when the state-action space is infinite, pointwise convergence does not necessarily imply uniform convergence, i.e., U k → U ∞ pointwise does not necessarily imply U k → U ∞ in ∥ · ∥ ∞ .

4 Complexity lower bound

We now present a complexity lower bound establishing optimality of Anc-VI.

Theorem 5. Let k ≥ 0 , n ≥ k + 2 , 0 < γ ≤ 1 , and U 0 ∈ R n . Then there exists an MDP with |S| = n and |A| = 1 (which implies the Bellman consistency and optimality operator for V and Q all coincide as T : R n → R n ) such that T has a fixed point U ⋆ satisfying U 0 ≤ U ⋆ and

for any iterates { U i } k i =0 satisfying

Proof outline of Theorem 5. Without loss of generality, assume n = k +2 and U 0 = 0 . Consider the MDP ( S , A , P, r, γ ) such that

Then, T = γ P π U +[0 , 1 , 0 , . . . , 0] ⊺ , U ⋆ = [0 , 1 , γ, . . . , γ k ] ⊺ , and ∥ ∥ U 0 -U ⋆ ∥ ∥ ∞ = 1 . Under the span condition, we can show that ( U k ) 1 = ( U k ) k +2 = 0 . Then, we get

and this implies

Taking the absolute value on both sides,

Note that the case γ = 1 is included in Theorem 5. When γ = 1 , the lower bound of Theorem 5 exactly matches the upper bound of Theorem 3.

Since

the lower bound establishes optimality of the second rates Theorems 1 and 2 up to a constant of factor 4 . Theorem 5 improves upon the prior state-of-the-art complexity lower bound established in the proof of [37, Theorem 3] by a factor 1 -γ k +1 . (In [37, Theorem 3], a lower bound on the distance to optimal value function is provided. Their result has an implicit dependence on the initial distance to optimal value function ∥ U 0 -U ⋆ ∥ ∞ , so we make the dependence explicit, and we translate their result to a lower bound on the Bellman error. Once this is done, the difference between our lower bound of Theorem 5 and of [37, Theorem 3] is a factor of 1 -γ k +1 . The worst-case MDP of [37, Theorem 3] and our worst-case MDP primarily differ in the rewards, while the states and the transition probabilities are almost the same.)

The so-called 'span condition' of Theorem 5 is arguably very natural and is satisfied by standard VI and Anc-VI. The span condition is commonly used in the construction of complexity lower bounds on first-order optimization methods [59, 23, 25, 13, 14, 65] and has been used in the prior state-of-the-art lower bound for standard VI [37, Theorem 3]. However, designing an algorithm that breaks the lower bound of Theorem 5 by violating the span condition remains a possibility. In optimization theory, there is precedence of lower bounds being broken by violating seemingly natural and minute conditions [40, 35, 98].

5 Approximate Anchored Value Iteration

In this section, we show that the anchoring mechanism is robust against evaluation errors of the Bellman operator, just as much as the standard approximate VI.

Let 0 < γ < 1 and let T ⋆ be the Bellman optimality operator. The Approximate Anchored Value Iteration (Apx-Anc-VI) is

for k = 1 , 2 , . . . , where β k = 1 / ( ∑ k i =0 γ -2 i ) , U 0 is an initial point, and the { ϵ k } ∞ k =0 is the error sequence modeling approximate evaluations of T ⋆ .

Of course, the classical Approximate Value Iteration (Apx-VI) is

for k = 1 , 2 , . . . , where U 0 is an initial point.

Fact 1 (Classical result, [11, p.333]) . Let 0 < γ < 1 be the discount factor. Let T ⋆ be the Bellman optimality for V or Q . Let U ⋆ be the fixed point of T ⋆ . Then Apx-VI exhibits the rate

Therefore, we conclude

Theorem 6. Let 0 < γ < 1 be the discount factor. Let T ⋆ and ˆ T ⋆ respectively be the Bellman optimality and anti-optimality operators for V or Q . Let U ⋆ and ˆ U ⋆ respectively be the fixed points of T ⋆ and ˆ T ⋆ . Then Apx-Anc-VI exhibits the rate

If, furthermore, U 0 ≥ T ⋆ U 0 , then (Apx-Anc-VI) exhibits the rate

for k = 1 , 2 , . . . .

The dependence on max ∥ ϵ i ∥ ∞ of Apx-Anc-VI is no worse than that of Apx-VI. In this sense, Apx-Anc-VI is robust against evaluation errors of the Bellman operator, just as much as the standard Apx-VI. Finally, we note that a similar analysis can be done for Apx-Anc-VI with the Bellman consistency operator.

6 Gauss-Seidel Anchored Value Iteration

In this section, we show that the anchoring mechanism can be combined with Gauss-Seidel-type updates in finite state-action spaces. Let 0 < γ < 1 and let T ⋆ : R n → R n be the Bellman optimality operator. Define T ⋆ GS : R n → R n as

where T ⋆ j : R n → R n is defined as

for j = 1 , . . . , n .

Fact 2. [Classical result, [69, Theorem 6.3.4]] T ⋆ GS is a γ -contractive operator and has the same fixed point as T ⋆ .

The Gauss-Seidel Anchored Value Iteration (GS-Anc-VI) is

for k = 1 , 2 , . . . , where β k = 1 / ( ∑ k i =0 γ -2 i ) and U 0 is an initial point.

Theorem 7. Let the state and action spaces be finite sets. Let 0 < γ < 1 be the discount factor. Let T ⋆ and ˆ T ⋆ respectively be the Bellman optimality and anti-optimality operators for V or Q . Let U ⋆ and ˆ U ⋆ respectively be the fixed points of T ⋆ and ˆ T ⋆ . Then GS-Anc-VI exhibits the rate

for k = 0 , 1 , . . . . If, furthermore, U 0 ≤ T ⋆ GS U 0 or U 0 ≥ T ⋆ GS U 0 , then GS-Anc-VI exhibits the rate

for k = 0 , 1 , . . . .

We point out that GS-Anc-VI cannot be directly extended to infinite action spaces since Hahn-Banach extension theorem is not applicable in the Gauss-Seidel setup. Furthermore, we note that a similar analysis can be carried out for GS-Anc-VI with the Bellman consistency operator.

7 Conclusion

Weshow that the classical value iteration (VI) is, in fact, suboptimal and that the anchoring mechanism accelerates VI to be optimal in the sense that the accelerated rate matches a complexity lower bound up to a constant factor of 4 . We also show that the accelerated iteration provably converges to a fixed point even when γ = 1 , if a fixed point exists. Being able to provide a substantive improvement upon the classical VI is, in our view, a surprising contribution.

One direction of future work is to study the empirical effectiveness of Anc-VI. Another direction is to analyze Anc-VI in a model-free setting and, more broadly, to investigate the effectiveness of the anchor mechanism in more practical RL methods.

Our results lead us to believe that many of the classical foundations of dynamic programming and reinforcement learning may be improved with a careful examination based on an optimization complexity theory perspective. The theory of optimal optimization algorithms has recently enjoyed significant developments [44, 43, 45, 98, 66], the anchoring mechanism being one such example [49, 65], and the classical DP and RL theory may benefit from a similar line of investigation on iteration complexity.

Acknowledgments and Disclosure of Funding

This work was supported by the the Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) [NO.2021-0-01343, Artificial Intelligence Graduate School Program (Seoul National University)] and the Samsung Science and Technology Foundation (Project Number SSTF-BA2101-02). We thank Jisun Park for providing valuable feedback.

References

M. Akian, S. Gaubert, Z. Qu, and O. Saadi. Multiply accelerated value iteration for nonsymmetric affine fixed point problems and application to markov decision processes. SIAM Journal on Matrix Analysis and Applications , 43(1):199-232, 2022.

D. G. Anderson. Iterative procedures for nonlinear integral equations. Journal of the Association for Computing Machinery , 12(4):547-560, 1965.

D. Andre, N. Friedman, and R. Parr. Generalized prioritized sweeping. Neural Information Processing Systems , 1997.

M. G. Azar, R. Munos, M. Ghavamzadeh, and H. Kappen. Speedy Q-learning. Neural Information Processing Systems , 2011.

S. Banach. Sur les opérations dans les ensembles abstraits et leur application aux équations intégrales. Fundamenta Mathematicae , 3(1):133-181, 1922.

M. Barré, A. Taylor, and A. d'Aspremont. Convergence of a constrained vector extrapolation scheme. SIAM Journal on Mathematics of Data Science , 4(3):979-1002, 2022.

M. G. Bellemare, G. Ostrovski, A. Guez, P. Thomas, and R. Munos. Increasing the action gap: New operators for reinforcement learning. Association for the Advancement of Artificial Intelligence , 2016.

R. Bellman. A Markovian decision process. Journal of Mathematics and Mechanics , 6(5):679684, 1957.

D. Bertsekas and J. Tsitsiklis. Parallel and Distributed Computation: Numerical Methods . Athena Scientific, 2015.

D. P. Bertsekas. Dynamic Programming and Optimal Control, volume II . 4th edition, 2012.

D. P. Bertsekas and J. N. Tsitsiklis. Neuro-Dynamic Programming . Athena Scientific, 1995.

W. Bowen, X. Huaqing, Z. Lin, L. Yingbin, and Z. Wei. Finite-time theory for momentum Q-learning. Conference on Uncertainty in Artificial Intelligence , 2021.

Y. Carmon, J. C. Duchi, O. Hinder, and A. Sidford. Lower bounds for finding stationary points I. Mathematical Programming , 184(1-2):71-120, 2020.

Y. Carmon, J. C. Duchi, O. Hinder, and A. Sidford. Lower bounds for finding stationary points II: first-order methods. Mathematical Programming , 185(1-2):315-355, 2021.

A.-L. Cauchy. Méthode générale pour la résolution des systemes d'équations simultanées. Comptes rendus de l'Académie des Sciences , 25:536-538, 1847.

V. Colao and G. Marino. On the rate of convergence of Halpern iterations. Journal of Nonlinear and Convex Analysis , 22(12):2639-2646, 2021.

J. P. Contreras and R. Cominetti. Optimal error bounds for non-expansive fixed-point iterations in normed spaces. Mathematical Programming , 199(1-2):343-374, 2022.

P. Dai, D. S. Weld, J. Goldsmith, et al. Topological value iteration algorithms. Journal of Artificial Intelligence Research , 42:181-209, 2011.

D. P. De Farias and B. Van Roy. On the existence of fixed points for approximate value iteration and temporal-difference learning. Journal of Optimization theory and Applications , 105:589-608, 2000.

J. Diakonikolas. Halpern iteration for near-optimal and parameter-free monotone inclusion and strong solutions to variational inequalities. Conference on Learning Theory , 2020.

J. Diakonikolas and P. Wang. Potential function-based framework for minimizing gradients in convex and min-max optimization. SIAM Journal on Optimization , 32(3):1668-1697, 2022.

Q. Dong, H. Yuan, Y. Cho, and T. M. Rassias. Modified inertial Mann algorithm and inertial CQ-algorithm for nonexpansive mappings. Optimization Letters , 12(1):87-102, 2018.

Y. Drori. The exact information-based complexity of smooth convex minimization. Journal of Complexity , 39:1-16, 2017.

Y. Drori and O. Shamir. The complexity of finding stationary points with stochastic gradient descent. International Conference on Machine Learning , 2020.

Y. Drori and A. Taylor. On the oracle complexity of smooth strongly convex minimization. Journal of Complexity , 68, 2022.

Y. Drori and M. Teboulle. An optimal variant of Kelley's cutting-plane method. Mathematical Programming , 160(1-2):321-351, 2016.

M. Ermis, M. Park, and I. Yang. On Anderson acceleration for partially observable Markov decision processes. IEEE Conference on Decision and Control , 2021.

M. Ermis and I. Yang. A3DQN: Adaptive Anderson acceleration for deep Q-networks. IEEE Symposium Series on Computational Intelligence , 2020.

D. Ernst, P. Geurts, and L. Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research , 6:503-556, 2005.

D. Ernst, M. Glavic, P. Geurts, and L. Wehenkel. Approximate value iteration in the reinforcement learning context. Application to electrical power system control. International Journal of Emerging Electric Power Systems , 3(1), 2005.

A.-m. Farahmand and M. Ghavamzadeh. PID accelerated value iteration algorithm. International Conference on Machine Learning , 2021.

A.-m. Farahmand, C. Szepesvári, and R. Munos. Error propagation for approximate policy and value iteration. Neural Information Processing Systems , 2010.

M. Fellows, K. Hartikainen, and S. Whiteson. Bayesian Bellman operators. Neural Information Processing Systems , 2021.

M. Geist and B. Scherrer. Anderson acceleration for reinforcement learning. European Workshop on Reinforcement Learning , 2018.

N. Golowich, S. Pattathil, C. Daskalakis, and A. Ozdaglar. Last iterate is slower than averaged iterate in smooth convex-concave saddle point problems. Conference on Learning Theory , 2020.

G. J. Gordon. Stable function approximation in dynamic programming. International Conference on Machine Learning , 1995.

V. Goyal and J. Grand-Clément. A first-order approach to accelerated value iteration. Operations Research , 71(2):517-535, 2022.

J. Grand-Clément. From convex optimization to MDPs: A review of first-order, second-order and quasi-newton methods for MDPs. arXiv:2104.10677 , 2021.

B. Halpern. Fixed points of nonexpanding maps. Bulletin of the American Mathematical Society , 73(6):957-961, 1967.

R. Hannah, Y. Liu, D. O'Connor, and W. Yin. Breaking the span assumption yields fast finite-sum minimization. Neural Information Processing Systems , 2018.

M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning. Association for the Advancement of Artificial Intelligence , 2018.

F. Iutzeler and J. M. Hendrickx. A generic online acceleration scheme for optimization algorithms via relaxation and inertia. Optimization Methods and Software , 34(2):383-405, 2019.

D. Kim. Accelerated proximal point method for maximally monotone operators. Mathematical Programming , 190(1-2):57-87, 2021.

D. Kim and J. A. Fessler. Optimized first-order methods for smooth convex minimization. Mathematical Programming , 159(1-2):81-107, 2016.

D. Kim and J. A. Fessler. Optimizing the efficiency of first-order methods for decreasing the gradient of smooth convex functions. Journal of Optimization Theory and Applications , 188(1):192-219, 2021.

J. Lee, C. Park, and E. K. Ryu. A geometric structure of acceleration and its role in making gradients small fast. Neural Information Processing Systems , 2021.

S. Lee and D. Kim. Fast extra gradient methods for smooth structured nonconvex-nonconcave minimax problems. Neural Information Processing Systems , 2021.

L. Leustean. Rates of asymptotic regularity for Halpern iterations of nonexpansive mappings. Journal of Universal Computer Science , 13(11):1680-1691, 2007.

F. Lieder. On the convergence rate of the Halpern-iteration. Optimization Letters , 15(2):405418, 2021.

M. Lutter, S. Mannor, J. Peters, D. Fox, and A. Garg. Value iteration in continuous actions, states and time. International Conference on Machine Learning , 2021.

P.-E. Maingé. Convergence theorems for inertial KM-type algorithms. Journal of Computational and Applied Mathematics , 219(1):223-236, 2008.

A. massoud Farahmand, M. Ghavamzadeh, C. Szepesvári, and S. Mannor. Regularized fitted Q-iteration for planning in continuous-space Markovian decision problems. American Control Conference , 2009.

H. B. McMahan and G. J. Gordon. Fast exact planning in Markov decision processes. International Conference on Automated Planning and Scheduling , 2005.

V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nature , 518(7540):529-533, 2015.

A. W. Moore and C. G. Atkeson. Prioritized sweeping: Reinforcement learning with less data and less time. Machine Learning , 13:103-130, 1993.

R. Munos. Error bounds for approximate value iteration. Association for the Advancement of Artificial Intelligence , 2005.

R. Munos and C. Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research , 9(27):815-857, 2008.

A. S. Nemirovski. Information-based complexity of linear operator equations. Journal of Complexity , 8(2):153-175, 1992.

Y. Nesterov. Lectures on Convex Optimization . Springer, 2nd edition, 2018.

Y. Nesterov, A. Gasnikov, S. Guminov, and P. Dvurechensky. Primal-dual accelerated gradient methods with small-dimensional relaxation oracle. Optimization Methods and Software , 36(4):773-810, 2021.

Y. E. Nesterov. A method for solving the convex programming problem with convergence rate O (1 /k 2 ) . Doklady Akademii Nauk SSSR , 269(3):543-547, 1983.

S. Niu, S. Chen, H. Guo, C. Targonski, M. Smith, and J. Kovaˇ cevi´ c. Generalized value iteration networks: Life beyond lattices. Association for the Advancement of Artificial Intelligence , 2018.

C. Nota and P. Thomas. Is the policy gradient a gradient? International Conference on Autonomous Agents and Multiagent Systems , 2020.

√

C. Park, J. Park, and E. K. Ryu. Factor2 acceleration of accelerated gradient methods. Applied Mathematics & Optimization , 2023.

J. Park and E. K. Ryu. Exact optimal accelerated complexity for fixed-point iterations. International Conference on Machine Learning , 2022.

J. Park and E. K. Ryu. Accelerated infeasibility detection of constrained optimization and fixed-point iterations. International Conference on Machine Learning , 2023.

M. Park, J. Shin, and I. Yang. Anderson acceleration for partially observable Markov decision processes: A maximum entropy approach. arXiv:2211.14998 , 2022.

J. Peng and R. J. Williams. Efficient learning and planning within the Dyna framework. Adaptive Behavior , 1(4):437-454, 1993.

M. L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming . John Wiley and Sons, 1994.

S. Reich. Strong convergence theorems for resolvents of accretive operators in Banach spaces. Journal of Mathematical Analysis and Applications , 75(1):287-292, 1980.

E. K. Ryu, K. Yuan, and W. Yin. Ode analysis of stochastic gradient methods with optimism and anchoring for minimax problems. arXiv:1905.10899 , 2019.

S. Sabach and S. Shtern. A first order method for solving convex bilevel optimization problems. SIAM Journal on Optimization , 27(2):640-660, 2017.

A. Salim, L. Condat, D. Kovalev, and P. Richtárik. An optimal algorithm for strongly convex minimization under affine constraints. International Conference on Artificial Intelligence and Statistics , 2022.

D. Scieur, A. d'Aspremont, and F. Bach. Regularized nonlinear acceleration. Mathematical Programming , 179(1-2):47-83, 2020.

Y. Shehu. Convergence rate analysis of inertial Krasnoselskii-Mann type iteration with applications. Numerical Functional Analysis and Optimization , 39(10):1077-1091, 2018.

W. Shi, S. Song, H. Wu, Y.-C. Hsu, C. Wu, and G. Huang. Regularized Anderson acceleration for off-policy deep reinforcement learning. Neural Information Processing Systems , 2019.

O. Shlakhter, C.-G. Lee, D. Khmelev, and N. Jaber. Acceleration operators in the value iteration algorithms for Markov decision processes. Operations Research , 58(1):193-202, 2010.

J. J. Suh, J. Park, and E. K. Ryu. Continuous-time analysis of anchor acceleration. Neural Information Processing Systems , 2023.

K. Sun, Y. Wang, Y. Liu, B. Pan, S. Jui, B. Jiang, L. Kong, et al. Damped Anderson mixing for deep reinforcement learning: Acceleration, convergence, and stabilization. Neural Information Processing Systems , 2021.

R. S. Sutton. Learning to predict by the methods of temporal differences. Machine Learning , 3:9-44, 1988.

R. S. Sutton and A. G. Barto. Reinforcement Learning: An introduction . MIT press, 2nd edition, 2018.

R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. Neural Information Processing Systems , 1999.

Q. Sykora, M. Ren, and R. Urtasun. Multi-agent routing value iteration network. International Conference on Machine Learning , 2020.

C. Szepesvári. Algorithms for Reinforcement Learning . Springer, 1st edition, 2010.

A. Tamar, Y. Wu, G. Thomas, S. Levine, and P. Abbeel. Value iteration networks. Neural Information Processing Systems , 2016.

A. Taylor and Y. Drori. An optimal gradient method for smooth strongly convex minimization. Mathematical Programming , 199(1-2):557-594, 2023.

S. Tosatto, M. Pirotta, C. d'Eramo, and M. Restelli. Boosted fitted Q-iteration. International Conference on Machine Learning , 2017.

J. N. Tsitsiklis. Asynchronous stochastic approximation and Q-learning. Machine Learning , 16:185-202, 1994.

H. Van Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double Q-learning. Association for the Advancement of Artificial Intelligence , 2016.

B. Van Roy. Performance loss bounds for approximate value iteration with state aggregation. Mathematics of Operations Research , 31(2):234-244, 2006.

B. Van Scoy, R. A. Freeman, and K. M. Lynch. The fastest known globally convergent first-order method for minimizing strongly convex functions. IEEE Control Systems Letters , 2(1):49-54, 2018.

N. Vieillard, B. Scherrer, O. Pietquin, and M. Geist. Momentum in reinforcement learning. International Conference on Artificial Intelligence and Statistics , 2020.

H. F. Walker and P. Ni. Anderson acceleration for fixed-point iterations. SIAM Journal on Numerical Analysis , 49(4):1715-1735, 2011.

C. J. Watkins and P. Dayan. Q-learning. Machine Learning , 8:279-292, 1992.

D. Wingate, K. D. Seppi, and S. Mahadevan. Prioritization methods for accelerating MDP solvers. Journal of Machine Learning Research , 6(25):851-881, 2005.

R. Wittmann. Approximation of fixed points of nonexpansive mappings. Archiv der Mathematik , 58(5):486-491, 1992.

H.-K. Xu. Iterative algorithms for nonlinear operators. Journal of the London Mathematical Society , 66(1):240-256, 2002.

T. Yoon and E. K. Ryu. Accelerated algorithms for smooth convex-concave minimax problems with O (1 /k 2 ) rate on squared gradient norm. International Conference on Machine Learning , 2021.

T. Yoon and E. K. Ryu. Accelerated minimax algorithms flock together. arXiv:2205.11093 , 2022.

Y. Zeng, F. Feng, and W. Yin. AsyncQVI: Asynchronous-parallel Q-value iteration for discounted Markov decision processes with near-optimal sample complexity. International Conference on Artificial Intelligence and Statistics , 2020.

J. Zhang, B. O'Donoghue, and S. Boyd. Globally convergent type-I Anderson acceleration for nonsmooth fixed-point iterations. SIAM Journal on Optimization , 30(4):3170-3197, 2020.

K. Zhou, L. Tian, A. M.-C. So, and J. Cheng. Practical schemes for finding near-stationary points of convex finite-sums. International Conference on Artificial Intelligence and Statistics , 2022.

A Preliminaries

For notational unity, we use the symbol U when both V and Q can be used.

Lemma 1. [10, Lemma 1.1.1] Let 0 < γ ≤ 1 . If U ≤ ˜ U , then T π U ≤ T π ˜ U,T ⋆ U ≤ T ⋆ ˜ U .

Lemma 2. Let 0 < γ ≤ 1 . For any policy π , P π is a nonexpansive linear operator such that if U ≤ ˜ U , P π U ≤ P π ˜ U .

Proof. If r ( s, a ) = 0 for all s ∈ S and a ∈ A , T π = γ P π . Then by Lemma 1 and γ -contraction of T π , we have the desired result.

Lemma 3. Let 0 < γ < 1 . Let T ⋆ and ˆ T ⋆ respectively be the Bellman optimality and anti-optimality operators. Let U ⋆ and ˆ U ⋆ respectively be the fixed points of T ⋆ and ˆ T ⋆ . Then ˆ U ⋆ ≤ U ⋆ .

B Omitted proofs in Section 2

First, we prove the following lemma by induction.

Lemma4. Let 0 < γ ≤ 1 , and if γ = 1 , assume a fixed point U π exists. For the iterates { U k } k =0 , 1 ,... of Anc-VI,

where ( Π k j = k +1 (1 -β j ) ) = 1 and β 0 = 1 .

Proof. If k = 0 , we have

If k = m , since T π is a linear operator,

Now, we prove the first rate of Theorem 1.

Proof of first rate in Theorem 1. Taking ∥·∥ ∞ -norm both sides of equality in Lemma 4, we get

where the first inequality comes from triangular inequality, second inequality is from Lemma 2, and equality come from calculations.

For the second rate of Theorem 1, we introduce following lemma.

Lemma 5. Let 0 < γ < 1 . Let T be Bellman consistency or optimality operator. For the iterates { U k } k =0 , 1 ,... of Anc-VI, if U 0 ≤ TU 0 , then U k -1 ≤ U k ≤ TU k -1 ≤ TU k ≤ U ⋆ for 1 ≤ k . Also, if U 0 ≥ TU 0 , then U k -1 ≥ U k ≥ TU k -1 ≥ TU k ≥ U ⋆ for 1 ≤ k .

Proof. First, let U 0 ≤ TU 0 . If k = 1 , U 0 ≤ β 1 U 0 +(1 -β 1 ) TU 0 = U 1 ≤ TU 0 by assumption. Since U 0 ≤ U 1 , TU 0 ≤ TU 1 by monotonicity of Bellman consistency and optimality operators.

By induction, and since β k ≤ β k -1 ,

Also, U k -1 ≤ U k implies TU k -1 ≤ TU k by monotonicity of Bellman consistency and optimality operators, and U k ≤ TU k implies that U k ≤ lim m →∞ ( T ) m U k = U ⋆ for all k = 0 , 1 , . . . .

Now, suppose U 0 ≥ TU 0 . If k = 1 , U 0 ≥ β 1 U 0 +(1 -β 1 ) TU 0 = U 1 ≥ TU 0 by assumption. Since U 0 ≥ U 1 , TU 0 ≥ TU 1 by monotonicity of Bellman consistency and optimality operators.

By induction, and since β k ≤ β k -1 ,

Also, U k -1 ≥ U k implies TU k -1 ≥ TU k by monotonicity of Bellman consistency and optimality operators, and U k ≥ TU k implies that U k ≥ lim m →∞ ( T ) m U k = U ⋆ for all k = 0 , 1 , . . . .

Now, we prove following key lemmas.

Lemma 6. Let 0 < γ ≤ 1 , and assume a fixed point U π exists if γ = 1 . For the iterates { U k } k =0 , 1 ,... of Anc-VI, if U 0 ≤ U π ,

where ( Π k j = k +1 (1 -β j ) ) = 1 and β 0 = 1 .

Lemma 7. Let 0 < γ < 1 . For the iterates { U k } k =0 , 1 ,... of Anc-VI, if U 0 ≥ T π U 0 ,

where ( Π k j = k +1 (1 -β j ) ) = 1 and β 0 = 1 .

Proof of Lemma 6. If U 0 ≤ U π , we get

by Lemma 4 and the fact that ( Π k j =1 (1 -β j ) ) ( γ P π ) k +1 ( U 0 -U π ) ≤ 0 .

Proof of Lemma 7. If U 0 ≥ TU 0 , U 0 -U π ≥ 0 by Lemma 5. Hence, by Lemma 4, we have

Now, we prove the second rates of Theorem 1.

Proof of second rates in Theorem 1. Let 0 < γ < 1 . By Lemma 5, if U 0 ≤ T π U 0 , then U 0 ≤ U π . Hence,

by Lemma 6. Taking ∥·∥ ∞ -norm both sides, we have

Otherwise, if U 0 ≥ TU 0 , U k ≥ TU k by Lemma 5. Since

by Lemma 7, taking ∥·∥ ∞ -norm both sides, we obtain same rate as before.

Lastly, Taylor series expansion for both rates at γ = 1 is

For the analyses of Anc-VI for Bellman optimality operator, we first prove following two lemmas.

Lemma 8. Let 0 < γ ≤ 1 . If γ = 1 , assume a fixed point U ⋆ exists. Then, if 0 ≤ α ≤ 1 and U -(1 -α ) ˜ U -αU ⋆ ≤ ¯ U , there exist nonexpansive linear operator P H such that

Lemma 9. Let 0 < γ < 1 . If 0 ≤ α ≤ 1 and ¯ U ≤ U -(1 -α ) ˜ U -α ˆ U ⋆ , then there exist nonexpansive linear operator ˆ P H such that

Proof of Lemma 8. First, let U = V, ˜ U = ˜ V , U ⋆ = V ⋆ , ¯ U = ¯ V , and V -(1 -α ) ˜ V -αV ⋆ ≤ ¯ V . If action space is finite,

where π is the greedy policy satisfying T π V = T ⋆ V , first inequality is from T π ˜ V ≤ T ⋆ ˜ V and T π V ⋆ ≤ T ⋆ V ⋆ , and second inequality comes from Lemma 1. Thus, we can conclude P H = P π .

Otherwise, if action space is infinite, define P ( c ¯ V ) = c sup s ∈S ¯ V ( s ) for c ∈ R and previously given ¯ V . Let M be linear space spanned by ¯ V with ∥·∥ ∞ -norm. Then, P is linear functional on M and ∥P∥ op ≤ 1 since | c sup s ∈S ¯ V ( s ) | ∥ c ¯ V ∥ ∞ ≤ 1 . Due to Hahn-Banach extension Theorem, there exist linear functional P h : F ( S ) → R with P h ( ¯ V ) = sup s ∈S ¯ V ( s ) and ∥P h ∥ op ≤ 1 . Furthermore, we can define P H : F ( S ) → F ( S ) such that P H V ( s ) = P h ( V ) for all s ∈ S . Then, since ∥P H ( V ) ∥ ∞ = |P h ( V ) | ≤ ∥P h ∥ op ≤ 1 for ∥ V ∥ ∞ ≤ 1 , we have ∥P H ∥ ∞ ≤ 1 . Therefore, P H is

nonexpansive linear operator in ∥·∥ ∞ -norm. Then,

for all s ∈ S . Therefore, we have

Similarly, let U = Q, ˜ U = ˜ Q,U ⋆ = Q ⋆ , ¯ U = ¯ Q , and Q -(1 -α ) ˜ Q -αQ ⋆ ≤ ¯ Q .

If action space is finite,

where π is the greedy policy satisfying T π Q = T ⋆ Q , first inequality is from T π ˜ Q ≤ T ⋆ ˜ Q and T π Q ⋆ ≤ T ⋆ Q ⋆ , and second inequality comes from Lemma 1. Then, we can conclude P H = P π .

Otherwise, if action space is infinite, define P ( c ¯ Q ) = c sup ( s ′ ,a ′ ) ∈S×A ¯ Q ( s ′ , a ′ ) for c ∈ R and previously given ¯ Q . Let M be linear space spanned by ¯ Q with ∥·∥ ∞ -norm. Then, P is linear functional on M and ∥P∥ op ≤ 1 . Due to Hahn-Banach extension Theorem, there exist linear functional P h : F ( S × A ) → R with P h ( ¯ Q ) = sup ( s ′ ,a ′ ) ∈S×A ¯ Q ( s ′ , a ′ ) and ∥P h ∥ op ≤ 1 . Furthermore, we can define P H : F ( S × A ) →F ( S × A ) such that P H Q ( s, a ) = P h ( Q ) for all ( s, a ) ∈ S × A and ∥ P H ∥ ∞ ≤ 1 . Therefore, P H is nonexpansive linear operator in ∥·∥ ∞ -norm. Then,

for all ( s, a ) ∈ S × A . Therefore, we have

Proof of Lemma 9. Note that ˆ T ⋆ is Bellman anti-optimality operators for V or Q , and ˆ U ⋆ is the fixed point of ˆ T ⋆ . First, let U = V, ˜ U = ˜ V , ˆ U ⋆ = ˆ V ⋆ , ¯ U = ¯ V , and ¯ V ≤ V -(1 -α ) ˜ V -α ˆ V ⋆ . Then,

Then, if action space is finite,

where ˆ π is the policy satisfying ˆ π ( · | s ) = argmin a ∈A E s ′ ∼ P ( · | s,a ) [ V ( s ′ ) -(1 -α ) ˜ V ( s ′ ) -α ˆ V ⋆ ( s ′ ) ] and second inequality comes from Lemma 1. Thus, we can conclude P H = P π .

Otherwise, if action space is infinite, define ˆ P ( c ¯ V ) = c inf s ∈S ¯ V ( s ) for c ∈ R and previously given ¯ V . Let M be linear space spanned by ¯ V with ∥·∥ ∞ -norm. Then, ˆ P is linear functional on M and ∥ ˆ P∥ op ≤ 1 since | c inf s ∈S ¯ V ( s ) | ∥ c ¯ V ∥ ∞ ≤ 1 . Due to Hahn-Banach extension Theorem, there exist linear functional ˆ P h : F ( S ) → R with ˆ P h ( ¯ V ) = inf s ∈S ¯ V ( s ) and ∥ ˆ P h ∥ op ≤ 1 . Furthermore, we can define ˆ P H : F ( S ) → F ( S ) such that ˆ P H V ( s ) = ˆ P h ( V ) for all s ∈ S . Then ∥ ˆ P H ∥ ∞ ≤ 1 since ∥ ˆ P H ( V ) ∥ ∞ = | ˆ P h ( V ) | ≤ ∥ ˆ P h ∥ op ≤ 1 for ∥ V ∥ ∞ ≤ 1 . . Thus, ˆ P H is nonexpansive linear operator in ∥·∥ ∞ -norm. Then, we have

for all s ∈ S . Therefore, we have

Hence, if action space is finite,

where ˆ π is the policy satisfying ˆ π ( · | s ) = argmin a ∈A E s ′ ∼ P ( · | s,a ) [ Q ( s ′ ) -(1 -α ) ˜ Q ( s ′ ) -αQ ⋆ ( s ′ ) ] and second inequality comes from Lemma 1. Then, we can conclude P H = P ˆ π .

Otherwise, if action space is infinite, define ˆ P ( c ¯ Q ) = c inf ( s ′ ,a ′ ) ∈S×A ¯ Q ( s ′ , a ′ ) for c ∈ R n and previously given ¯ Q . Let M be linear space spanned by ¯ Q with ∥·∥ ∞ -norm. Then, P is linear functional on M with ∥ ˆ P∥ op ≤ 1 . Due to Hahn-Banach extension Theorem, there exist linear functional ˆ P h : F ( S × A ) → R with ˆ P h ( ¯ Q ) = inf ( s ′ ,a ′ ) ∈S×A ¯ Q ( s ′ , a ′ ) and ∥ ˆ P h ∥ op ≤ 1 . Furthermore, we can define ˆ P H : F ( S × A ) → F ( S × A ) such that P H Q ( s, a ) = ˆ P h ( Q ) for all ( s, a ) ∈ S × A and ∥ ˆ P H ∥ ∞ ≤ 1 . Thus ˆ P H is nonexpansive linear operator in ∥·∥ ∞ -norm. Then, we have

for all ( s, a ) ∈ S × A . Therefore, we have

Now, we present our key lemmas for the first rate of Theorem 2.

Lemma 10. Let 0 < γ ≤ 1 . If γ = 1 , assume a fixed point U ⋆ exists. For the iterates { U k } k =0 , 1 ,... of Anc-VI, there exist nonexpansive linear operators {P l } l =0 , 1 ,...,k such that

where Π k j = k +1 (1 -β j ) = 1 and β 0 = 1 .

Lemma 11. Let 0 < γ < 1 . For the iterates { U k } k =0 , 1 ,... of Anc-VI, there exist nonexpansive linear operators { ˆ P l } l =0 , 1 ,...,k such that

where Π k j = k +1 (1 -β j ) = 1 and β 0 = 1 .

We prove previous lemmas by induction.

Proof of Lemma 10. If k = 0 ,

where inequality comes from first inequality in Lemma 8 with α = 1 , U = U 0 , ¯ U = U 0 -U ⋆ . By induction,

and let ¯ U be the entire right hand side of inequality. Then, we have

where inequality comes from first inequality in Lemma 8 with α = β k , U = U k , ˜ U = U k -1 , and previously defined ¯ U .

Proof of Lemma 11. Note that ˆ T ⋆ is Bellman anti-optimality operators for V or Q , and ˆ U ⋆ is the fixed point of ˆ T ⋆ . If k = 0 ,

where inequality comes from second inequality in Lemma 9 with α = 1 , U = U 0 , ¯ U = U 0 -ˆ U ⋆ .

By induction,

and let ¯ U be the entire right hand side of inequality. Then, we have

where inequality comes from second inequality in Lemma 9 with α = β k , U = U k , ˜ U = U k -1 , and previously defined ¯ U .

Now, we prove the first rate of Theorem 2.

Proof of first rate in Theorem 2. Since B 1 ≤ A ≤ B 2 implies ∥ A ∥ ∞ ≤ sup {∥ B 1 ∥ ∞ , ∥ B 2 ∥ ∞ } for A,B ∈ F ( X ) , if we take ∥·∥ ∞ right side first inequality of Lemma 10, we have

where the first inequality comes from triangular inequality, second inequality is from nonexpansiveness of P l , and last equality comes from calculations.

If we take ∥·∥ ∞ right side of second inequality of Lemma 10, similarly, we have

where the first inequality comes from triangular inequality, second inequality is from from nonexpansiveness of ˆ P l , and last equality comes from calculations. Therefore, we conclude

Next, for the second rate in Theorem 2, we prove following lemmas by induction.

Lemma 12. Let 0 < γ ≤ 1 . If γ = 1 , assume a fixed point U ⋆ exists. For the iterates { U k } k =0 , 1 ,... of Anc-VI, if T ⋆ U 0 ≤ U ⋆ , there exist nonexpansive linear operators {P l } l =0 , 1 ,...,k such that

where Π k j = k +1 (1 -β j ) = 1 and β 0 = 1 .

Lemma 13. Let 0 < γ < 1 . For the iterates { U k } k =0 , 1 ,... of Anc-VI, if U 0 ≥ T ⋆ U 0 , there exist nonexpansive linear operators { ˆ P l } l =0 , 1 ,...,k such that

where Π k j = k +1 (1 -β j ) = 1 and β 0 = 1 .

Proof of Lemma 12. If k = 0 ,

where the second inequality is from the condition.

By induction,

and let ¯ U be the entire right hand side of inequality. Then, we have

where inequality comes from first inequality in Lemma 8 with α = β k , U = U k , ˜ U = U k -1 , and previously defined ¯ U .

Proof of Lemma 13. If k = 0 ,

where the second inequality is from the fact that U 0 ≥ T ⋆ U 0 implies T ⋆ U 0 ≥ U ⋆ by Lemma 5 and U ⋆ ≥ ˆ U ⋆ by Lemma 3.

By induction,

and let ¯ U be the entire right hand side of inequality. Then, we have

where inequality comes from second inequality in Lemma 9 with α = β k , U = U k , ˜ U = U k -1 , and previously defined ¯ U .

Now, we prove the second rates of Theorem 2.

Proof of second rates in Theorem 2. Let 0 < γ < 1 . Then, if U 0 ≤ T ⋆ U 0 , then T ⋆ U 0 ≤ U ⋆ and U k ≤ T ⋆ U k by Lemma 5. Hence, taking ∥·∥ ∞ -norm both sides of first inequality in Lemma 12, we have

Otherwise, if U 0 ≥ TU 0 , U k ≥ TU k by Lemma 5. taking ∥·∥ ∞ -norm both sides of second inequality in Lemma 13, we have

C Omitted proofs in Section 3

First, we present the following lemma.

Lemma 14. Let γ = 1 . Assume a fixed point U ⋆ exists. For the iterates { U k } k =0 , 1 ,... of Anc-VI, ∥ ∥ U k -U ⋆ ∥ ∥ ∞ ≤ ∥ ∥ U 0 -U ⋆ ∥ ∥ ∞ .

Proof. If k = 0 , it is obvious. By induction,

where the second inequality comes form nonexpansiveness of T .

Now, we present the proof of Theorem 3.

Proof of Theorem 3. First, if U 0 ≤ TU 0 , with same argument in proof of Lemma 5, we can show that U k -1 ≤ U k ≤ TU k -1 ≤ TU k for k = 1 , 2 , . . . .

Since fixed point U ⋆ exists by assumption, Lemma 4 and 10 hold. Note that γ = 1 implies β k = 1 k +1 and if we take ∥·∥ ∞ -norm both sides for those inequalities in lemmas, by simple calculation, we have

for any fixed point U ⋆ (since 0 ≤ T ⋆ U k -U k , we can get upper bound of ∥ ∥ T ⋆ U k -U k ∥ ∥ ∞ from Lemma 10).

Suppose that there exist { k j } j =0 , 1 ,... such that U k j converges to some ˜ U ⋆ . Then, lim j →∞ ( T -I ) U k j = ( T -I ) ˜ U ⋆ = 0 since T -I is continuous. This implies that ˜ U ⋆ is a fixed point. By Lemma 14 and previous argument, U k is increasing and bounded sequence in R n . Thus, U k has single limit point, some fixed point ˜ U ⋆ . Furthermore, the fact that U 0 ≤ TU 0 ≤ ˜ U ⋆ implies that Lemma 6 and 12 hold. Therefore, we have

Next, we prove the Theorem 4.

Proof of Theorem 4. By same argument in the proof of Theorem 3, if U 0 ≤ TU 0 , we can show that U k -1 ≤ U k ≤ TU k -1 ≤ TU k for k = 1 , 2 , . . . . , and

for any fixed point U ⋆ . Since U k is increasing and bounded by Lemma 14 and previous argument, U k converges pointwise to some ˜ U ⋆ in general action-state space. We now show that TU k also converges pointwise to T ˜ U ⋆ . First, let T be Bellman consistency operator and U = V, ˜ U ⋆ = ˜ V π . By monontone convergence theorem,

for any fixed s ∈ S . With same argument, case U = Q also holds. If T is Bellman optimality operator, we use following lemma.

Lemma 15. Let W,W k ∈ F ( X ) for k = 0 , 1 , . . . . If W k ( x ) ≤ W k +1 ( x ) for all x ∈ X , and { W k } k =0 , 1 ,..., converge pointwise to W , then lim k →∞ { sup x W k ( x ) } = sup x W ( x ) .

Proof. W k ( x ) ≤ W ( x ) implies that sup x W k ( x ) ≤ sup x W ( x ) . If sup x W ( x ) = a , there exist x which satisfying a -W ( x ) < ϵ 2 , and by definition of W , there exist W k such that a -W k ( x ) < ϵ for any ϵ > 0 .

If U = V and ˜ U ⋆ = ˜ V ⋆ , by previous lemma and monotone convergence theorem, we have

for any fixed s ∈ S . With similar argument, case U = Q also holds.

Since TU k → T ˜ U ⋆ and U k → ˜ U ⋆ pointwisely, TU k -U k converges pointwise to T ˜ U ⋆ -˜ U ⋆ = 0 . Thus, ˜ U ⋆ is indeed fixed point of T . Furthermore, the fact that U 0 ≤ TU 0 ≤ ˜ U ⋆ implies that Lemma 6 and 12 hold. Therefore, we have

D Omitted proofs in Section 4

We present the proof of Theorem 5.

Proof of Theorem 5. First, we prove the case U 0 = 0 for n ≥ k +2 . Consider the MDP ( S , A , P, r, γ ) such that

Then, T = γ P π U +[0 , 1 , 0 , . . . , 0] ⊺ , U ⋆ = [0 , 1 , γ, . . . , γ n -2 ] ⊺ , and ∥ ∥ U 0 -U ⋆ ∥ ∥ ∞ = 1 . Under the span condition, we can show that ( U k ) 1 = ( U k ) l = 0 for k +2 ≤ l ≤ n by following lemma.

Lemma 16. Let T : R n → R n be defined as before. Then, under span condition, ( U i ) 1 = 0 for 0 ≤ i ≤ k , and ( U i ) j = 0 for 0 ≤ i ≤ k and i +2 ≤ j ≤ n .

Proof. Case k = 0 is obvious. By induction, ( U l ) 1 = 0 for 0 ≤ l ≤ i -1 . Then ( TU l ) 1 = 0 for 0 ≤ l ≤ i -1 . This implies that ( TU l -U l ) 1 = 0 for 0 ≤ l ≤ i -1 . Hence ( U i ) 1 = ( U 0 ) 1 = 0 . Again, by induction, ( U l ) j = 0 for 0 ≤ l ≤ i -1 , l +2 ≤ j ≤ n . Then ( TU l ) j = 0 for 0 ≤ l ≤ i -1 , l +3 ≤ j ≤ n and this implies that ( TU l -U l ) j = 0 for 0 ≤ l ≤ i -1 , l +3 ≤ j ≤ n . Therefore, ( U i ) j = 0 for i +2 ≤ j ≤ n .

Then, we get

and this implies

Taking the absolute value on both sides,

Now, we show that for any initial point U 0 ∈ R n , there exists an MDP which exhibits same lower bound with the case U 0 = 0 . Denote by MDP( 0 ) and T 0 the worst-case MDP and Bellman consistency or opitmality operator constructed for U 0 = 0 . Define an MDP( U 0 ) ( S , A , P, r, γ ) for U 0 = 0 as

̸

Then, Bellman consistency or optimality operator T satisfies

Let ˜ U ⋆ be fixed point of T 0 . Then, if U ⋆ = ˜ U ⋆ + U 0 , U ⋆ is fixed point of T . Furthermore, if { U i } k i =0 satisfies span condition

˜ U i = U i -U 0 is a sequence satisfying

which is the same span condition in Theorem 5 with respect to T 0 . This is because

for i = 0 , . . . , k . Thus, { ˜ U i } k i =0 is a sequence starting from 0 and satisfy the span condition for T 0 . This implies that

Hence, MDP( U 0 ) is indeed our desired worst-case instance. Lastly, the fact that U 0 -U ⋆ = ˜ U 0 -˜ U ⋆ = -(0 , 1 , γ, . . . , γ n -2 ) implies U 0 ≤ U ⋆ .

Therefore, we conclude

E Omitted proofs in Section 5

First, we prove following key lemma.

Lemma 17. Let 0 < γ < 1 . For the iterates { U k } k =0 , 1 ,... of Anc-VI, there exist nonexpansive linear operators {P l } l =0 , 1 ,...,k and { ˆ P l } l =0 , 1 ,...,k such that

Proof of Lemma 17. First, we prove the first inequality in Lemma 17 by induction. If k = 1 ,

where inequality comes from Lemma 8 with α = 1 , U = U 0 , ¯ U = U 0 -U ⋆ , and let ¯ U be the entire right hand side of inequality. Then, we have

where inequality comes from Lemma 8 with α = β 1 , U = U 1 , ˜ U = U 0 , and previously defined ¯ U . By induction,

and let ¯ U be the entire right hand side of inequality. Then, we have

where inequality comes from Lemma 8 with α = β k , U = U k , ˜ U = U k -1 , and previously defined ¯ U .

Now, we prove second inequality in Lemma 17 by induction.

If k = 1 ,

where inequality comes from Lemma 9 with α = 1 , U = U 0 , ¯ U = U 0 -ˆ U ⋆ , and let ¯ U be the entire right hand side of inequality. Then, we have

where inequality comes from Lemma 9 with α = β 1 , U = U 1 , ˜ U = U 0 , and previously defined ¯ U . By induction,

and let ¯ U be the entire right hand side of inequality. Then, we have

where inequality comes from Lemma 9 with α = β k , U = U k , ˜ U = U k -1 , and previously defined ¯ U

Now, we prove the first rate in Theorem 6.

Proof of first rate in Theorem 6. Since B 1 ≤ A ≤ B 2 implies ∥ A ∥ ∞ ≤ sup {∥ B 1 ∥ ∞ , ∥ B 2 ∥ ∞ } for A,B ∈ F ( X ) , if we take ∥·∥ ∞ right side of first inequality in Lemma 17, we have

If we apply second inequality of Lemma 17 and take ∥·∥ ∞ -norm right side, we have

Therefore, we get

Now, for the second rate in Theorem 6, we present following key lemma.

Lemma 18. Let 0 < γ < 1 . For the iterates { U k } k =0 , 1 ,... of Anc-VI, if U 0 ≥ T ⋆ U 0 , there exist nonexpansive linear operators {P l } l =0 , 1 ,...,k and { ˆ P l } l =0 , 1 ,...,k such that

for 1 ≤ k , where Π k j = k +1 (1 -β j ) = 1 , Π k +1 l = k γ P l = Π k +1 l = k γ ˆ P l = I , and β 0 = 1 .

Proof of Lemma 18. If U 0 ≥ T ⋆ U 0 , U 0 ≥ lim m →∞ ( T ⋆ ) m U 0 = U ⋆ by Lemma 1. By Lemma 3, this also implies U 0 ≥ ˆ U ⋆ .

First, we prove first inequality in Lemma 18 by induction. If k = 1 ,

where the second inequality is from the (2 β 1 -1)( U 0 -U ⋆ ) ≤ 0 , and first inequality comes from Lemma 8 with α = 1 , U = U 0 , ¯ U = U 0 -U ⋆ , and let ¯ U be the entire right hand side of inequality. Then, we have

where inequality comes from Lemma 8 with α = β 1 , U = U 1 , ˜ U = U 0 , and previously defined ¯ U . By induction,

where the second inequality is from β k -(1 -β k ) β k -1 ≤ 0 and let ¯ U be the entire right hand side of inequality. Then, we have

where the first inequality comes from Lemma 8 with α = β k , U = U k , ˜ U = U k -1 , and previously defined ¯ U .

For the second inequality in Lemma 18, if k = 1 ,

where the second inequality is from U 0 ≥ T ⋆ U 0 ≥ ˆ U ⋆ , and let ¯ U be the entire right hand side of inequality. Then, we have

where inequality comes from Lemma 9 with α = β 1 , U = U 1 , ˜ U = U 0 , and previously defined ¯ U . By induction,

and let ¯ U be the entire right hand side of inequality. Then, we have

where inequality comes from Lemma 9 with α = β k , U = U k , ˜ U = U k -1 , and previously defined ¯ U .

Now, we prove the second rate in Theorem 6.

Proof of second rate in Theorem 6. If we take ∥·∥ ∞ right side of first inequality in Lemma 18, we have

If we apply second inequality of Lemma 18 and take ∥·∥ ∞ -norm right side, we have

Therefore, we get

since ˆ U ⋆ ≤ U ⋆ ≤ U 0 implies that

F Omitted proofs in Section 6

For the analyses, we first define ˆ T ⋆ GS : R n → R n as

where ˆ T ⋆ j : R n → R n is defined as

for j = 1 , . . . , n , where ˆ T ⋆ is Bellman anti-optimality operator.

Fact 3. [Classical result, [10, Proposition 1.3.2]] ˆ T ⋆ GS is a γ -contractive operator and has the same fixed point as ˆ T ⋆ .

Now, we introduce the following lemmas.

Lemma 19. Let 0 < γ < 1 . If 0 ≤ α ≤ 1 , then there exist γ -contractive nonnegative matrix P GS such that

Lemma 20. Let 0 < γ < 1 . If 0 ≤ α ≤ 1 , then there exist γ -contractive nonnegative matrix ˆ P GS such that

Proof of Lemma 19. First let U = V, ˜ U = ˜ V , U ⋆ = V ⋆ . For 1 ≤ i ≤ n , we have

where π i is the greedy policy satisfying T π i V = T ⋆ V and first inequality is from T π i ˜ V ≤ T ⋆ ˜ V and T π i V ⋆ ≤ T ⋆ V ⋆ . Then, define matrix P i as

for i = 1 , . . . , n . Note that P i is nonnegative matrix since P π i is nonnegative matrix. Then, we have

By induction, there exist a sequence of matrices {P i } i =1 ,...,n satisfying

since T ⋆ i V ⋆ = V ⋆ for all i . Denote P GS as P n · · · P 1 . Then, P GS is γ -contractive nonnegative matrix since

for 1 ≤ i ≤ n , where first equality is from definition of P l for i +1 ≤ l ≤ n , inequality comes from definition of P l for 1 ≤ l ≤ i -1 , and last equality is induced by definition of P i . Therefore, this implies that ∥ P GS ∥ ∞ ≤ γ .

If U = Q , with similar argument of case U = V , let π i be the greedy policy, define matrix P i as

and denote P GS as P n · · · P 1 . Then, P GS is γ -contractive nonnegative matrix satisfying

Proof of Lemma 20. First let U = V, ˜ U = ˜ V , ˆ U ⋆ = ˆ V ⋆ . For 1 ≤ i ≤ n , we have

Let ˆ π i ( · | s ) = argmin a ∈A E s ′ ∼ P ( · | s,a ) [ V ( s ′ ) -(1 -α ) ˜ V ( s ′ ) -α ˆ V ⋆ ( s ′ ) ] and define matrix ˆ P i as

for i = 1 , . . . , n . Note that ˆ P i is nonnegative matrix since P ˆ π i is nonnegative matrix. Then, we have

By induction, there exist a sequence of matrices { ˆ P i } i =1 ,...,n satisfying

and denote ˆ P GS as ˆ P n · · · ˆ P 1 . With same argument in proof of Lemma 19, ˆ P GS is γ -contractive nonnegative matrix.

If U = Q , with similar argument, let ˆ π i ( · | s ) = argmin a ∈A { Q ( s, a ) -(1 -α ) ˜ Q ( s, a ) -α ˆ Q ⋆ ( s, a ) } and define matrix ˆ P i as

Denote ˆ P GS as ˆ P n · · · ˆ P 1 . Then, with same argument in proof of Lemma 19, ˆ P GS is γ -contractive nonnegative matrix satisfying

Next, we prove following key lemma.

Lemma 21. Let 0 < γ < 1 . For the iterates { U k } k =0 , 1 ,... of (GS-Anc-VI) , there exist γ -contractive nonnegative matrices {P l GS } l =0 , 1 ,...,k and { ˆ P l GS } l =0 , 1 ,...,k such that

where Π k j = k +1 (1 -β j ) = 1 and β 0 = 1 .

Proof of Lemma 21. First, we prove first inequality in Lemma 21 by induction.

If k = 0 ,

where inequality comes from Lemma 19 with α = 1 , U = U 0 .

By induction,

where the first inequality comes from Lemma 19 with α = β k , U = U k , ˜ U = U k -1 , and second inequality comes from nonnegativeness of P k GS .

First, we prove second inequality in Lemma 21 by induction.

If k = 0 ,

where inequality comes from Lemma 20 with α = 1 , U = U 0 . By induction,

where the first inequality comes from Lemma 20 with α = β k , U = U k , ˜ U = U k -1 , and nonnegativeness of ˆ P k GS .

Now, we prove the first rate in Theorem 7.

Proof of first rate in Theorem 7. Since B 1 ≤ A ≤ B 2 implies ∥ A ∥ ∞ ≤ sup {∥ B 1 ∥ ∞ , ∥ B 2 ∥ ∞ } for A,B ∈ F ( X ) , if we take ∥·∥ ∞ right side of first inequality in Lemma 21, we have

where the first inequality comes from triangular inequality, second inequality is from γ -contraction of P l GS , and last equality comes from calculations. If we take ∥·∥ ∞ right side of second inequality in Lemma 21, we have

where the first inequality comes from triangular inequality, second inequality is from from γ -contraction of ˆ P l GS , and last equality comes from calculations. Therefore, we conclude

For the second rates of Theorem 7, we introduce following lemma.

Lemma 22. Let 0 < γ < 1 . For the iterates { U k } k =0 , 1 ,... of (GS-Anc-VI) , if U 0 ≤ T ⋆ GS U 0 , then U k -1 ≤ U k ≤ T ⋆ GS U k -1 ≤ T ⋆ GS U k ≤ U ⋆ for 1 ≤ k . Also, if U 0 ≥ T ⋆ GS U 0 , then U k -1 ≥ U k ≥ T ⋆ GS U k -1 ≥ T ⋆ GS U k ≥ U ⋆ for 1 ≤ k .

Proof. By Fact 3, lim m →∞ T ⋆ GS U = U ⋆ . By definition, if U ≤ ˜ U , T ⋆ i U ≤ T ⋆ i ˜ U for any 1 ≤ i ≤ n and this implies that if U ≤ ˜ U , then T ⋆ GS U ≤ T ⋆ GS ˜ U . Hence, with same argument in proof of Lemma 5, we can obtain desired results.

Now, we prove the second rates in Theorem 7.

Proof of second rates in Theorem 7. If U 0 ≤ T ⋆ GS U 0 , then U 0 -U ⋆ ≤ 0 and U k ≤ T ⋆ GS U k by Lemma 22. Hence, by Lemma 21, we get

where the second inequality follows from ( Π k j =1 (1 -β j ) ) ( Π i l = k P l GS ) ( U 0 -U ⋆ ) ≤ 0 . Taking ∥·∥ ∞ -norm both sides, we have

Otherwise, if U 0 ≥ T ⋆ GS U 0 , U k ≥ T ⋆ GS U k and U 0 ≥ U ⋆ ≥ ˆ U ⋆ by Lemma 22 and 3. Thus, by Lemma 21, we get

where the second inequality follows from 0 ≤ ( Π k j =1 (1 -β j ) ) ( Π 0 l = k ˆ P l GS ) ( U 0 -ˆ U ⋆ ) . Taking ∥·∥ ∞ -norm both sides, we have

G Broader Impacts

Our work focuses on the theoretical aspects of reinforcement learning. There are no negative social impacts that we anticipate from our theoretical results.

H Limitations

Our analysis concerns value iteration. While value iteration is of theoretical interest, the analysis of value iteration is not sufficient to understand modern deep reinforcement learning practices.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nNew variants of value iterations based on anchoring\nComplexity lower-bound that shows that their algorithms is optimal in a worst-case sense\nNice extensions for various important cases ($\\gamma =1$, approximate VI, gauss-seidel, infinite state-action spaces).\n\nSTRENGTHS:\n* Strong theorems, able to overcome the shortcomings of the analysis of some prior work that only prove acceleration for linear operators [1, 37]\n\n* Well-written and easy to follow\n\nWEAKNESSES:\n* A bit sketchy on the terminology: “rate” is not defined, some inaccurate statements (“O(1) rate for VI when $\\gamma \\approx 1$”), see my questions below\n* No numerical comparisons with existing methods\n* No analysis in the model-free setup (using sampling)\n\nQUESTIONS:\np1, l20: “the optimal rate for the VI setup was not known” : isn’t it Theorem 3 in [37] ? This provides a lower-bound on the rate for both value iteration and value evaluation, and shows that VI achieves this rate.\n\np1, l26 - contributions: I am a bit puzzled by the formulations of “$O(1/k)$ rate for $\\gamma \\approx 1$” etc. While this becomes clear when the exact theorems are stated, it may be worth expanding here on the precise formulations of your results.\n\np2, l45, definition of Bellman operator and Bellman consistency: please mention that you overload the notation. Right now $T^\\pi$ is both an operator of $R^{|S|}$ and $R^{|S|\\times|A|}$.\n\nSection 1.1: please define what is the rate of an algorithm. There may be inconsistencies since usually rate suggests that the convergence is linear and the rate $\\rho$ means that error decreases as $\\rho^k$ after k iterations; whereas I believe that the rate of p3, l83 $O(1/k)$ is for the error to decrease as $1/k$ after iterations. Please clarify.\n\nDistinctions between anchoring and Nesterov’s acceleration: some recent works show that in some setup, anchoring and Nesterov’s acceleration are exactly equivalent - they are the same algorithm up to some reformulation [0]. Does this connection hold in your setup?\n\n[0] Tran-Dinh, Q. (2022). The Connection Between Nesterov's Accelerated Methods and Halpern Fixed-Point Iterations. arXiv preprint arXiv:2203.04869.\n\np4, l122: why not simplify the expression of $\\beta_k$ ? The denominator is just a partial geometric sum.\n\np4, lines 151-152: this is quite sketchy, since “rate” has not been defined (see my previous comment). The comments about “$O(1)$ rate for VI when $\\gamma \\approx 1$” do not make sense to me. I understand $\\gamma \\approx 1$ means that VI will converge slowly, but it is more convincing to compare the number of iterations before returning an $\\epsilon$ optimal solution. \n\nChoice of U0: to satisfy the inequalities of line 152, one can choose either $U_0 = 0$ or $U_0 = R / (1-\\gamma)$, with $R$ an upper bound on the rewards. Would that be a good choice in practice though?\n\nTheorem 1: is your choice of \\beta_j the optimal choice?\n\nTheorem 5: is your MDP instance different from the MDP instance from Theorem 3 in [37]? Please clarify.\n\nLIMITATIONS:\n* No significant limitations apart from the absence of numerical simulations. It is not clear if the algorithms proposed in this paper perform well; beating vanilla value iteration is not really hard in practice.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors focus their study on the theoretical analysis of a (simple) variation of the Value Iteration (VI) algorithm, a classical and grounding tool behind many modern (deep)-RL algorithms.\nThe variation considered by the authors incorporates anchor acceleration mechanisms leading to what the authors call Anchored Value Iteration (Anc-VI). More specifically, rather than repeatedly applying the bellman consistency/optimality operators $T$ to reach the fixed point (i.e., $x_{k+1} = T x_k$), Anc-VI obtains the next point as a convex combination between the initial point and the result of applying the operator on the previous point, namely $x_{k+1} = \\beta_k x_0 + (1-\\beta_k) T x_{k}$. Naturally, the sequence $ \\beta_k$ is a vanishing sequence. \n\nAfter introducing the algorithm, the authors proceed with an in-depth analysis of the convergence rates of Anc-VI.\n\nMore specifically:\n1) They study the convergence rate of the Bellman error (i.e, $||Tx_k - x_k||_{\\infty}$) both for the consistency and the optimal operators. The authos show that Anc-VI converges at rate $\\mathcal{O}\\left( 1/k \\right)$ for $\\gamma \\approx 1$ (while standard VI has rate $\\mathcal{O}(1)$). \n2) Under mild assumptions, the authors derive a lower bound that shows that the convergence rates of Anc-VI are tight up to a constant factor of 4. \n3) The author extends their study to the case in which $\\gamma=1$, where VI may not converge to a fixed point even if one exists. Anc-VI, on the other hand, converges to some fixed point asymptotically, and the bellman error shrinks to 0 at a linear rate.\n4) Finally, results are extended to the approximated value iteration setting and to the Gauss-Seidel variation of VI.\n\nSTRENGTHS:\nVI is a grounding tool of many modern RL algorithms, and its analysis has gathered the community's attention for a long time. The authors thoroughly review existing works and properly contextualize their results within the field. \nApplying the idea of anchoring to Value Iteration is, to the best of my knowledge, novel and leads to some surprising results, which have all been stated in summary above. \n\nMore specifically, the most relevant ones are:\n1) Optimality of the accelerated rate of Anc-VI (i.e., upper bound matches lower bound up to constant factor). \n2) Convergence in the undiscounted setting. \n\nGiven the fact that VI is the grounding core of many modern methods, I think these results are of interest to the RL community at NeurIPS. For this reason, I recommend acceptance. \n\nOn the clarity. Overall, the paper is a nice read, and not particularly hard to follow.\n\nWEAKNESSES:\nNot much.\n\nQUESTIONS:\n1) Do you have any intuitive explanation of why such stronger results can be obtained with anchoring? Why should be want to anchor our search close to the initial point? I think this point might add value for possible future development of RL algorithms that might take inspiration from this work. \n2) Why should we anchor only on the initial value rather than doing a, e.g., linear combination of the past value functions? (except for memory requirements). \n3) Is there any existing variation of VI whose upper-bound matches the lower-bound?\n\nLIMITATIONS:\nThe authors discuss Limitations in Appendix H. Furthermore, the value of each theoretical result is properly discussed after presenting each result.\n\nPotential negative societal impact. The paper deals with foundational research on the convergence rates of Value Iteration. I don't see a direct path to negative applications.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers an anchored version of Value Iteration and derives accelerated rates in terms of the Bellman error for both the Bellman consistency and optimality operators. Then, the work addresses the particular case of $\\gamma =1$ with a $O(1/k)$ rate that VI fails to guarantee via the standard contraction argument. A complexity lower bound matching the upper-bound (up to a constant numerical factor) is then established. The paper further proposes an error propagation analysis of approximate anchored VI, extending the results of the exact case and a Gauss-Seidel version of the algorithm is also analyzed.\n\nSTRENGTHS:\n- I find the fact that the Anc-VI algorithm is able to address the case $\\gamma  =1$ interesting and the anchoring mechanism seems to be well suited for this case unlike VI.  \nIt is interesting to see that the anchoring idea which was recently investigated in depth in optimization is also interesting in the DP setting. While the idea may seem natural as DP involves fixed point iterations, the execution requires to address several technical challenges that are more specific to the DP setting. \n\n- The contributions of the paper are solid and somehow comprehensive with accelerated convergence rates, a lower bound and extensions to inexact and Gauss-Seidel variants. \n\n- The anchoring mechanism can also be used for further extensions in DP/RL for the design of learning algorithms beyond the deterministic setting.   \n\n- The paper is well-written and the contributions are overall clearly stated.\n\nWEAKNESSES:\n1. Even if the Bellman error is a natural quantity as a performance measure for fixed point iteration schemes, it would be nice if the paper can comment more if possible on the motivation to consider convergence guarantees in terms of this performance measure compared to the distance to the optimal value function. It is mentioned in l. 18 that the distance to the optimum is not computable and that the Bellman error can be monitored which I find a valid and fair interesting point. However, as a matter of fact, the lower bound provided in [37, Theorem 3] (which is mentioned in the paper) in terms of distance to the optimal value function is actually achieved for Value Iteration. If we would like to approximate the optimal value function as fast as possible to find an optimal policy in a control task for instance, why would we use anchored VI instead of VI? See also the related question 5 in the ‘Questions’ section below. \n\n2.  It is not discussed how to find a near-optimal policy using Anchored VI.  I guess you can just output a policy that is greedy with respect to the output of the Anchored VI algorithm after a certain number of iterations as it is usually done with VI for which there are policy error bounds. Given that Section 1.1 mentions optimal policies, it would be nice to mention that the Anchored VI could be further enhanced to find near-optimal policies as an approximate planning algorithm. \n \n3. The paper only discusses guarantees in terms of the Bellman error comparing with the translated result for VI from distance to optimal value function $\\|\\|U^k - U^{\\star}\\|\\|_{\\infty}$ to Bellman error. It is also similarly possible to translate the Bellman error guarantee to a distance to optimality result. See question 5 below. \n\n4. Case $\\gamma = 1$: How can the (action) state value functions be defined in that case? Definitions of Section 1.1 may not be relevant anymore. While the paper states that that ‘a full treatment of undiscounted MDPs is beyond the scope of this paper’, I think at least adding references would be useful to give a meaning to the problem and hint to the fact that fixed points can be guaranteed to exist (as currently assumed in the paper) under some technical assumptions. This would justify that the assumption is reasonable and allows to avoid some technicalities.\n\n5. I think the paper can comment on the advantages of the Gauss-Seidel Anc-VI and why this extension is considered. Section 6 does not discuss the motivation for considering such an algorithm. Is it for the possibility to perform asynchronous updates via its coordinate-wise update rule? \n\n**Minor:**\n \n- I find the terminology ‘linear operator’ for the Bellman (consistency) operator a little confusing even if Puterman 2005 (p. 144, Eq. (6.1.7)) uses for e.g. the terminology ‘linear transformation’. A linear operator T would satisfy T(0) = 0 according to the standard mathematical definition of a linear operator, which is not the case for the Bellman consistency operator. Maybe affine would be more appropriate.  \n\n- l. 137-141: I would say the result for the Bellman ‘consistency’ operator can also be relevant for policy evaluation which is useful as a subroutine for several algorithms in RL beyond value iteration. \n\n- As a suggestion, the paper could add a figure for the hard MDP instance to ease the reading (see for e.g., Figure 1 in [37]). It also seems that the hard MDP instance is the same as the one in [37], it may be worth it to mention this in that case. \n\n- There is sometimes some redundancy in the proofs. For instance the Hahnn-Banach argument is used identically 2 times in l. 568, l. 590. It is again used for Q functions in l. 578 and l.  600. This also happens for some inductions throughout the proofs which are very similar.  \n\n- Lemma 15 in appendix: in the proof, ‘by definition W, there exist W’, I guess you mean there exists $k$.  \n\n- l. 513 and later: $\\lim …  \\to $,  I guess $\\to$ can be replaced by $=$ here. \n\n- Several grammatical articles are missing in writing, especially in the appendix: l. 225 (a complexity …), l. 507 (a nonexpansive), 510, 515, 520, 664, 700, 759, 760, 764 and several other places.  \n\n**Typos:**\nl. 111, 113: ‘nonexpensive’; \nl. 116: ‘operator’.\n\nQUESTIONS:\nBesides the comments above, please find some questions below for clarifications: \n\n1. l. 149 - 151: is the comparison of the rates with VI straightforward from the rates or does it require some algebraic manipulation? It does not seem to be immediate from the expression, please provide more details to clarify in the appendix if needed.  \n \n2. It is mentioned in the paper that the anchoring mechanism is ‘distinct from Nesterov’s acceleration’. The latter mechanism has inspired the work [37] which requires for instance some reversibility conditions as discussed in related works. A recent work (Tran-Dinh 2022) draws connections between Nesterov’s accelerated methods and Halpern fixed point iterations. Could you comment on this? \n\nQuoc Tran-Dinh, 2022. The Connection Between Nesterov’s Accelerated Methods and Halpern Fixed-Point Iterations. \nhttps://arxiv.org/pdf/2203.04869.pdf\n\n3. In the abstract, it is mentioned that ‘the optimal rate for the VI setup was not known’. Do you mean for the Bellman error (as the rest of the abstract and the paper state results for the Bellman error)?  Please precise in that case, I found this a bit confusing at first read. \n\n4. How do you come up with the specific way you set the parameter $\\beta_t$ in l. 122? It seems to match the usual $\\beta_t = 1/(t+1)$ used for anchoring/Halpern iterations in the case where $\\gamma = 1$. Could you provide more intuition about this? \n \n5. Results on the Bellman error can also be translated to the optimality measure $\\|\\|U^k - U^{\\star}\\|\\|$\n using the triangle inequality at least for $0<\\gamma<1$ \nto obtain $\\|\\|U^k - U^{\\star}\\|\\| \\leq \\frac{1}{1-\\gamma} \\|\\|U^k - T U^k\\|\\|$. How would this result compare to VI? Is it meaningful to include a comment along these lines for comprehensiveness given that you discuss translating the result for VI to control the Bellman error? I understand though that the anchoring mechanism is better suited to control the Bellman error. \n\n6. l. 290 - 292: How would the GS update rule would even be defined in infinite dimensions even before talking about the Hahn-Banach theorem for the analysis? Why Hahn-Banach theorem would not be applicable?\n\n7. Minor comment, l. 240: for the upper bound, it seems that one can obtain a constant equal to 2 using the following derivations:  \n$$\\frac{(\\gamma^{-1} - \\gamma)(1+\\gamma - \\gamma^{k+1})}{(\\gamma^{k+1})^{-1} - \\gamma^{k+1}} \n= \\frac{\\gamma^k (1-\\gamma) (1+\\gamma - \\gamma^{k+1}) }{1-\\gamma^{2(k+1)}}\n= \\gamma^k \\frac{1}{\\sum_{i=0}^{2k+1} \\gamma^i} (1+\\gamma - \\gamma^{k+1}) \n\\leq \\frac{2 \\gamma^k}{\\sum_{i=0}^k \\gamma^i}.$$\n\n**Proof related questions for clarifications:** \n\n 8. About the Hahn-Banach argument (for infinite state action space): Could you please provide clarifications regarding the following questions? \n- Lemma 8, 9 in the appendix: What are $U$, $\\tilde{U}$, $\\bar{U}$ in the statements? If these are arbitrary (as they are instantiated later in the proofs of Lemma 10,11 for example), please precise the statement of the lemmas for clarity. \n\n- The definition of the operator $\\mathcal{P}$ (l. 600 for e.g., and others) is not very clear to me. Do you define it only for multiples of $\\bar{Q}$ where $\\bar{Q}$ is defined in the lemma statement or do you define it for every $Q$? Why do you introduce $c$ in the definition? I guess this is because you need homogeneity to invoke the Hahn-Banach theorem but I would expect that this would be verified once you define the sublinear function for the application of the theorem. Also, it is not clear to me why is the $\\mathcal{P}$ as defined a linear functional on M (because of the inf) with $\\|\\|\\mathcal{P}\\|\\|$ = 1, which norm is this? Is it the operator norm (please define it in this case)? The notation $\\bar{Q}$ is a little confusing. If $\\mathcal{P}$ is defined by the inf (without c) for any $Q$, then you can show sublinearity (or rather ‘superlinearity’ with the inf) and homogeneity. The restriction to the span of $\\bar{Q}$ would be a linear functional (as you mention it) and then you can conclude I guess. If this is the reasoning you conduct, please clarify. I suggest to state the Hahn-Banach theorem and its application or at least to clarify what is the sublinear function $p$ used and what is the linear functional dominated by $p$, especially the use of the notation $\\bar{Q}$ for defining $\\mathcal{P}$. \n\n- Minor, l. 568, l. 590: ‘if action space is infinite’? Is it rather state space here? Same comment for other occurrences.   \n\n9. Case $\\gamma =1$: l. 672, 673 in the proofs in appendix, it is stated that the $O(1/(k+1))$ rate is obtained after taking the infinity norm in Lemmas 4 and 10. Does the $O(1/(k+1))$ rate just follow from using $\\beta_k = 1/(k+1)$ or is there any particular upper bounding manipulation? I think more clarifications would be useful for the reader.\n\nLIMITATIONS:\nThe paper does list the fact that the analysis of VI is not sufficient to understand modern deep RL (in appendix H in the end of the paper) and comments on the potential of future work in the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces an accelerated version of the Value Iteration (VI) algorithm, called Anc-VI, based on the anchoring mechanism.  The proposed method achieves faster reduction in the Bellman error compared to standard VI even when the discounting factor is close to 1. Meanwhile, a complexity lower bound is also provided which match with the upper bound up to a constant 4. Meanwhile, this work also shows the benefits of anchoring mechanism in approximate VI and Gauss-Seidel VI.\n\nSTRENGTHS:\n1. This work proposed the acceleration of VI by leveraging the anchoring mechanism\n2. The proposed Anc-VI has provable faster convergence rate comparing with classic VI\n3. This work provide the acceleration rate for both the bellman optimality operator and consistency operator\n\nWEAKNESSES:\n1. The performance of anchoring mechanism has reliance on the starting point $U_0$ and discounting factor. In order to have the fast acceleration, we need to set $\\gamma\\approx1$. However, one of the reason that we set discounting factor $\\gamma < 1$ is for faster computing in practice.  The author need to clarify whether the anchoring mechanism still can benefit the learning acceleration in the practical case, e.g., in Theorem 1, when $\\gamma \\to 0$, how does the upper bound show the acceleration. Same question apply to Theorem 2\n2. In the Apx-Anc-VI, when $U_0$ is far from optimal, according to Theorem 6, the first few iteration does not necessary results in policy improvement (e.g., by setting $k=1$, $\\gamma=1$). Thus, what does Theorem 6 can tell on the benefits of using Anchoring mechanism?\n\nQUESTIONS:\n1.  If it is possible to derive any practical algorithm based on the theoretical findings in this work?\n2. What is the computing complexity of the proposed method comparing with VI?\n3. What is the guidance on choosing $\\gamma$ in practice in order to achieve faster convergence, e.g., let $\\gamma \\approx 1$?\n\nLIMITATIONS:\nThe authors addressed the limitations in the Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}