SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jongmin Lee<sup>1</sup>  
Ernest K. Ryu<sup>1,2</sup>
$^{1}$ Department of Mathematical Science, Seoul National University  $^{2}$ Interdisciplinary Program in Artificial Intelligence, Seoul National University
Value Iteration (VI) is foundational to the theory and practice of modern reinforcement learning, and it is known to converge at a  $\mathcal{O}(\gamma^k)$ -rate, where  $\gamma$  is the discount factor. Surprisingly, however, the optimal rate in terms of Bellman error for the VI setup was not known, and finding a general acceleration mechanism has been an open problem. In this paper, we present the first accelerated VI for both the Bellman consistency and optimality operators. Our method, called Anc-VI, is based on an anchoring mechanism (distinct from Nesterov's acceleration), and it reduces the Bellman error faster than standard VI. In particular, Anc-VI exhibits a  $\mathcal{O}(1/k)$ -rate for  $\gamma \approx 1$  or even  $\gamma = 1$ , while standard VI has rate  $\mathcal{O}(1)$  for  $\gamma \geq 1 - 1/k$ , where  $k$  is the iteration count. We also provide a complexity lower bound matching the upper bound up to a constant factor of 4, thereby establishing optimality of the accelerated rate of Anc-VI. Finally, we show that the anchoring mechanism provides the same benefit in the approximate VI and Gauss-Seidel VI setups as well.
Value Iteration (VI) is foundational to the theory and practice of modern dynamic programming (DP) and reinforcement learning (RL). It is well known that when a discount factor  $\gamma < 1$  is used, (exact) VI is a contractive iteration in the  $\| \cdot \|_{\infty}$ -norm and therefore converges. The progress of VI is measured by the Bellman error in practice (as the distance to the fixed point is not computable), and much prior work has been dedicated to analyzing the rates of convergence of VI and its variants.
Surprisingly, however, the optimal rate in terms of Bellman error for the VI setup was not known, and finding a general acceleration mechanism has been an open problem. The classical  $\mathcal{O}(\gamma^k)$ -rate of VI is inadequate as many practical setups use  $\gamma \approx 1$  or  $\gamma = 1$  for the discount factor. (Not to mention that VI may not converge when  $\gamma = 1$ .) Moreover, most prior works on accelerating VI focused on the Bellman consistency operator (policy evaluation) as its linearity allows eigenvalue analyses, but the Bellman optimality operator (control) is the more relevant object in modern RL.
Contribution. In this paper, we present the first accelerated VI for both the Bellman consistency and optimality operators. Our method, called Anc-VI, is based on an "anchoring" mechanism (distinct from Nesterov's acceleration), and it reduces the Bellman error faster than standard VI. In particular, Anc-VI exhibits a  $\mathcal{O}(1 / k)$ -rate for  $\gamma \approx 1$  or even  $\gamma = 1$ , while standard VI has rate  $\mathcal{O}(1)$  for  $\gamma \geq 1 - 1 / k$ , where  $k$  is the iteration count. We also provide a complexity lower bound matching the upper bound up to a constant factor of 4, thereby establishing optimality of the accelerated rate of Anc-VI. Finally, we show that the anchoring mechanism provides the same benefit in the approximate VI and Gauss-Seidel VI setups as well.
1.1 Notations and preliminaries
We quickly review basic definitions and concepts of Markov decision processes (MDP) and reinforcement learning (RL). For further details, refer to standard references such as [69, 84, 81].
Markov Decision Process. Let  $\mathcal{M}(\mathcal{X})$  be the space of probability distributions over  $\mathcal{X}$ . Write  $(\mathcal{S}, \mathcal{A}, P, r, \gamma)$  to denote the MDP with state space  $\mathcal{S}$ , action space  $\mathcal{A}$ , transition probability  $P: \mathcal{S} \times \mathcal{A} \to \mathcal{M}(\mathcal{S})$ , reward  $r: \mathcal{S} \times \mathcal{A} \to \mathbb{R}$ , and discount factor  $\gamma \in (0,1]$ . Denote  $\pi: \mathcal{S} \to \mathcal{M}(\mathcal{A})$  for a policy,  $V^{\pi}(s) = \mathbb{E}_{\pi}[\sum_{t=0}^{\infty} \gamma^{t} r(s_{t}, a_{t}) | s_{0} = s]$  and  $Q^{\pi}(s, a) = \mathbb{E}_{\pi}[\sum_{t=0}^{\infty} \gamma^{t} r(s_{t}, a_{t}) | s_{0} = s, a_{0} = a]$  for  $V$ - and  $Q$ -value functions, where  $\mathbb{E}_{\pi}$  denotes the expected value over all trajectories  $(s_{0}, a_{0}, s_{1}, a_{1}, \ldots)$  induced by  $P$  and  $\pi$ . We say  $V^{\star}$  and  $Q^{\star}$  are optimal  $V$ - and  $Q$ -value functions if  $V^{\star} = \sup_{\pi} V^{\pi}$  and  $Q^{\star} = \sup_{\pi} Q^{\pi}$ . We say  $\pi_{V}^{\star}$  and  $\pi_{Q}^{\star}$  are optimal policies if  $\pi_{V}^{\star} = \operatorname{argmax}_{\pi} V^{\pi}$  and  $\pi_{Q}^{\star} = \operatorname{argmax}_{\pi} Q^{\pi}$ . (If  $\operatorname{argmax}$  is not unique, break ties arbitrarily.)
Value Iteration. Let  $\mathcal{F}(\mathcal{X})$  denote the space of bounded measurable real-valued functions over  $\mathcal{X}$ . With the given MDP  $(S, \mathcal{A}, P, r, \gamma)$ , for  $V \in \mathcal{F}(S)$  and  $Q \in \mathcal{F}(S \times \mathcal{A})$ , define the Bellman consistency operators  $T^{\pi}$  as
$$
T ^ {\pi} V (s) = \mathbb {E} _ {a \sim \pi (\cdot | s), s ^ {\prime} \sim P (\cdot | s, a)} [ r (s, a) + \gamma V (s ^ {\prime}) ],
$$
$$
T ^ {\pi} Q (s, a) = r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a), a ^ {\prime} \sim \pi (\cdot | s ^ {\prime})} [ Q (s ^ {\prime}, a ^ {\prime}) ]
$$
for all  $s \in S, a \in \mathcal{A}$ , and the Bellman optimality operators  $T^{\star}$  as
$$
T ^ {\star} V (s) = \sup  _ {a \in \mathcal {A}} \left\{r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ V (s ^ {\prime}) \right] \right\},
$$
$$
T ^ {\star} Q (s, a) = r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ \sup  _ {a ^ {\prime} \in \mathcal {A}} Q \left(s ^ {\prime}, a ^ {\prime}\right) \right]
$$
for all  $s \in S, a \in \mathcal{A}$ . For notational conciseness, we write  $T^{\pi}V = r^{\pi} + \gamma \mathcal{P}^{\pi}V$  and  $T^{\pi}Q = r + \gamma \mathcal{P}^{\pi}Q$ , where  $r^{\pi}(s) = \mathbb{E}_{a \sim \pi(\cdot|s)}[r(s,a)]$  is the reward induced by policy  $\pi$  and  $\mathcal{P}^{\pi}(s)$  and  $\mathcal{P}^{\pi}(s,a)$  defined as
$$
\mathcal {P} ^ {\pi} (s \rightarrow s ^ {\prime}) = \operatorname {P r o b} (s \rightarrow s ^ {\prime} \mid a \sim \pi (\cdot \mid s), s ^ {\prime} \sim P (\cdot \mid s, a))
$$
$$
\mathcal {P} ^ {\pi} \left(\left(s, a\right)\rightarrow \left(s ^ {\prime}, a ^ {\prime}\right)\right) = \operatorname {P r o b} \left(\left(s, a\right)\rightarrow \left(s ^ {\prime}, a ^ {\prime}\right) \mid s ^ {\prime} \sim P (\cdot \mid s, a), a ^ {\prime} \sim \pi (\cdot \mid s ^ {\prime})\right),
$$
are the transition probabilities induced by policy  $\pi$ . We define VI for Bellman consistency and optimality operators as
$$
V ^ {k + 1} = T ^ {\pi} V ^ {k}, \quad Q ^ {k + 1} = T ^ {\pi} Q ^ {k}, \quad V ^ {k + 1} = T ^ {\star} V ^ {k}, \quad Q ^ {k + 1} = T ^ {\star} Q ^ {k} \qquad \text {f o r} k = 0, 1, \dots ,
$$
where  $V^0, Q^0$  are initial points. VI for control, after executing  $K$  iterations, returns the near-optimal policy  $\pi_K$  as a greedy policy satisfying
$$
T ^ {\pi_ {K}} V ^ {K} = T ^ {\star} V ^ {K}, \quad T ^ {\pi_ {K}} Q ^ {K} = T ^ {\star} Q ^ {K}.
$$
For  $\gamma < 1$ , both Bellman consistency and optimality operators are contractions, and, by Banach's fixed-point theorem [5], the VIs converge to the unique fixed points  $V^{\pi}$ ,  $Q^{\pi}$ ,  $V^{\star}$ , and  $Q^{\star}$  with  $\mathcal{O}(\gamma^{k})$ -rate. For notational unity, we use the symbol  $U$  when both  $V$  and  $Q$  can be used. Since  $\left\|TU^{k}-U^{k}\right\|_{\infty} \leq\left\|TU^{k}-U^{\star}\right\|_{\infty}+\left\|U^{k}-U^{\star}\right\|_{\infty} \leq(1+\gamma)\left\|U^{k}-U^{\star}\right\|_{\infty}$ , VI exhibits the rate on the Bellman error:
$$
\left\| T U ^ {k} - U ^ {k} \right\| _ {\infty} \leq (1 + \gamma) \gamma^ {k} \left\| U ^ {0} - U ^ {\star} \right\| _ {\infty} \quad \text {f o r} k = 0, 1, \dots , \tag {1}
$$
where  $T$  is Bellman consistency or optimality operator,  $U^0$  is a starting point, and  $U^{\star}$  is fixed point of  $T$ . We say  $V \leq V'$  or  $Q \leq Q'$  if  $V(s) \leq V'(s)$  or  $Q(s,a) \leq Q'(s,a)$  for all  $s \in S$  and  $a \in A$ , respectively.
Fixed-point iterations. Given an operator  $T$ , we say  $x^{\star}$  is fixed point if  $Tx^{\star} = x^{\star}$ . Since Banach [5], the standard fixed-point iteration
$$
x ^ {k + 1} = T x ^ {k} \quad \text {f o r} k = 0, 1, \dots
$$
has been commonly used to find fixed points. Note that VI for policy evaluation and control are fixed-point iterations with Bellman consistency and optimality operators. In this work, we also consider the Halpern iteration
$$
x ^ {k + 1} = \beta_ {k + 1} x ^ {0} + (1 - \beta_ {k + 1}) T x ^ {k} \quad \text {f o r} k = 0, 1, \dots ,
$$
where  $x^0$  is an initial point and  $\{\beta_k\}_{k\in \mathbf{N}}\in (0,1)$
1.2 Prior works
Value Iteration. Value iteration (VI) was first introduced in the DP literature [8] for finding optimal value function, and its variant approximate VI [11, 30, 56, 32, 19, 90, 81] considers approximate evaluations of the Bellman optimality operator. In RL, VI and approximate VI have served as the basis of RL algorithms such as fitted value iteration [29, 57, 52, 87, 50, 36] and temporal difference learning [80, 89, 41, 94, 54]. There is a line of research that emulates VI by learning a model of the MDP dynamics [85, 83, 62] and applying a modified Bellman operator [7, 33]. Asynchronous VI, another variation of VI updating the coordinate of value function in asynchronous manner, has also been studied in both RL and DP literature [11, 9, 88, 100].
Fixed-point iterations. The Banach fixed-point theorem [5] establishes the convergence of the standard fixed-point iteration with a contractive operator. The Halpern iteration [39] converges for nonexpansive operators on Hilbert spaces [96] and uniformly smooth Banach spaces [70, 97]. (To clarify, the  $\| \cdot \|_{\infty}$ -norm in  $\mathbb{R}^n$  is not uniformly smooth.)
The fixed-point residual  $\| Tx_k - x_k \|$  is a commonly used error measure for fixed-point problems. In general normed spaces, the Halpern iteration was shown to exhibit  $\mathcal{O}(1 / \log(k))$ -rate for (nonlinear) nonexpansive operators [48] and  $\mathcal{O}(1/k)$ -rate for linear nonexpansive operators [17] on the fixed-point residual. In Hilbert spaces, [72] first established a  $\mathcal{O}(1/k)$ -rate for the Halpern iteration and the constant was later improved by [49, 43]. For contractive operators, [65] proved exact optimality of Halpern iteration through an exact matching complexity lower bound.
Acceleration. Since Nesterov's seminal work [61], there has been a large body of research on acceleration in convex minimization. Gradient descent [15] can be accelerated to efficiently reduce function value and squared gradient magnitude for smooth convex minimization problems [61, 44, 45, 46, 102, 21, 60] and smooth strongly convex minimization problems [59, 91, 64, 86, 73]. Motivated by Nesterov acceleration, inertial fixed-point iterations [51, 22, 75, 70, 42] have also been suggested to accelerate fixed-point iterations. Anderson acceleration [2], another acceleration scheme for fixed-point iterations, has recently been studied with interest [6, 74, 93, 101].
In DP and RL, prioritized sweeping [55] is a well-known method that changes the order of updates to accelerate convergence, and several variants [68, 53, 95, 3, 18] have been proposed. Speedy Q-learning [4] modifies the update rule of Q-learning and uses aggressive learning rates for acceleration. Recently, there has been a line of research that applies acceleration techniques of other areas to VI: [34, 79, 28, 67, 27, 76] uses Anderson acceleration of fixed-point iterations, [92, 37, 38, 12, 1] uses Nesterov acceleration of convex optimization, and [31] uses ideas inspired by PID controllers in control theory. Among those works, [37, 38, 1] applied Nesterov acceleration to obtain theoretically accelerated convergence rates, but those analyses require certain reversibility conditions or restrictions on eigenvalues of the transition probability induced by the policy.
The anchor acceleration, a new acceleration mechanism distinct from Nesterov's, lately gained attention in convex optimization and fixed-point theory. The anchoring mechanism, which retracts iterates towards the initial point, has been used to accelerate algorithms for minimax optimization and fixed-point problems [71, 47, 98, 65, 43, 20, 99, 78], and we focus on it in this paper.
Complexity lower bound. With the information-based complexity analysis [58], complexity lower bound on first-order methods for convex minimization problem has been thoroughly studied [59, 23, 25, 13, 14, 24]. If a complexity lower bound matches an algorithm's convergence rate, it establishes optimality of the algorithm [58, 44, 73, 86, 26, 65]. In fixed-point problems, [16] established  $\Omega(1/k^{1-\sqrt{2/q}})$  lower bound on distance to solution for Halpern iteration with a nonexpansive operator in  $q$ -uniformly smooth Banach spaces. In [17], a  $\Omega(1/k)$  lower bound on the fixed-point residual for the general Mann iteration with a nonexpansive linear operator, which includes standard fixed-point iteration and Halpern iterations, in the  $\ell^{\infty}$ -space was provided. In Hilbert spaces, [65] showed exact complexity lower bound on fixed-point residual for deterministic fixed-point iterations with  $\gamma$ -contractive and nonexpansive operators. Finally, [37] provided lower bound on distance to optimal value function for fixed-point iterations satisfying span condition with Bellman consistency and optimality operators and we discussed this lower bound in section 4.
2 Anchored Value Iteration
Let  $T$  be a  $\gamma$ -contractive (in the  $\| \cdot \|_{\infty}$ -norm) Bellman consistency or optimality operator. The Anchored Value Iteration (Anc-VI) is
$$
U ^ {k} = \beta_ {k} U ^ {0} + (1 - \beta_ {k}) T U ^ {k - 1} \tag {Anc-VI}
$$
for  $k = 1,2,\ldots$ , where  $\beta_{k} = 1 / (\sum_{i = 0}^{k}\gamma^{-2i})$  and  $U^0$  is an initial point. In this section, we present accelerated convergence rates of Anc-VI for both Bellman consistency and optimality operators for both  $V$ - and  $Q$ -value iterations. For the control setup, where the Bellman optimality operator is used, Anc-VI returns the near-optimal policy  $\pi_K$  as a greedy policy satisfying  $T^{\pi_K}U^K = T^\star U^K$  after executing  $K$  iterations.
Notably, Anc-VI obtains the next iterate as a convex combination between the output of  $T$  and the starting point  $U^0$ . We call the  $\beta_k U_0$  term the anchor term since, loosely speaking, it serves to pull the iterates toward the starting point  $U_0$ . The strength of the anchor mechanism diminishes as the iteration progresses since  $\beta_k$  is a decreasing sequence.
The anchor mechanism was introduced [39, 72, 49, 65, 17, 48] for general nonexpansive operators and  $\| \cdot \| _2$ -nonexpansive and contractive operators. The optimal method for  $\| \cdot \| _2$ -nonexpansive and contractive operators in [65] shares the same coefficients with Anc-VI, and convergence results for general nonexpansive operators in [17, 48] are applicable to Anc-VI for nonexpansive Bellman optimality and consistency operators. While our anchor mechanism does bear a formal resemblance to those of prior works, our convergence rates and point convergence are neither a direct application nor a direct adaptation of the prior convergence analyses. The prior analyses for  $\| \cdot \| _2$ -nonexpansive and contractive operators do not apply to Bellman operators, and prior analyses for general nonexpansive operators have slower rates and do not provide point convergence while our Theorem 3 does. Our analyses specifically utilize the structure of Bellman operators to obtain the faster rates and point convergence.
The accelerated rate of Anc-VI for the Bellman optimality operator is more technically challenging and is, in our view, the stronger contribution. However, we start by presenting the result for the Bellman consistency operator because it is commonly studied in the prior RL theory literature on accelerating value iteration [37, 38, 1, 31] and because the analysis in the Bellman consistency setup will serve as a good conceptual stepping stone towards the analysis in the Bellman optimality setup.
2.1 Accelerated rate for Bellman consistency operator
First, for general state-action spaces, we present the accelerated convergence rate of Anc-VI for the Bellman consistency operator.
Theorem 1. Let  $0 < \gamma < 1$  be the discount factor and  $\pi$  be a policy. Let  $T^{\pi}$  be the Bellman consistency operator for  $V$  or  $Q$ . Then, Anc-VI exhibits the rate
$$
\begin{array}{l} \left\| T ^ {\pi} U ^ {k} - U ^ {k} \right\| _ {\infty} \leq \frac {\left(\gamma^ {- 1} - \gamma\right) \left(1 + 2 \gamma - \gamma^ {k + 1}\right)}{\left(\gamma^ {k + 1}\right) ^ {- 1} - \gamma^ {k + 1}} \left\| U ^ {0} - U ^ {\pi} \right\| _ {\infty} \\ = \left(\frac {2}{k + 1} + \frac {k - 1}{k + 1} \epsilon + O \left(\epsilon^ {2}\right)\right) \left\| U ^ {0} - U ^ {\pi} \right\| _ {\infty} \quad f o r k = 0, 1, \dots , \\ \end{array}
$$
where  $\epsilon = 1 - \gamma$  and the big- $\mathcal{O}$  notation considers the limit  $\epsilon \to 0$ . If, furthermore,  $U^0 \leq T^\pi U^0$  or  $U^0 \geq T^\pi U^0$ , then Anc-VI exhibits the rate
$$
\begin{array}{l} \left\| T ^ {\pi} U ^ {k} - U ^ {k} \right\| _ {\infty} \leq \frac {\left(\gamma^ {- 1} - \gamma\right) \left(1 + \gamma - \gamma^ {k + 1}\right)}{\left(\gamma^ {k + 1}\right) ^ {- 1} - \gamma^ {k + 1}} \left\| U ^ {0} - U ^ {\pi} \right\| _ {\infty} \\ = \left(\frac {1}{k + 1} + \frac {k}{k + 1} \epsilon + O (\epsilon^ {2})\right) \left\| U ^ {0} - U ^ {\pi} \right\| _ {\infty} \quad f o r k = 0, 1, \dots . \\ \end{array}
$$
If  $\gamma \geq \frac{1}{2}$ , both rates of Theorem 1 are strictly faster than the standard rate (1) of VI, since
$$
\frac {\left(\gamma^ {- 1} - \gamma\right) \left(1 + 2 \gamma - \gamma^ {k + 1}\right)}{\left(\gamma^ {k + 1}\right) ^ {- 1} - \gamma^ {k + 1}} = \gamma^ {k} \frac {\left(1 - \gamma^ {2}\right) \left(1 + 2 \gamma - \gamma^ {k + 1}\right)}{\left(1 - \gamma^ {2 k + 2}\right)} <   \gamma^ {k} (1 + \gamma).
$$
The second rate of Theorem 1, which has the additional requirement, is faster than the standard rate (1) of VI for all  $0 < \gamma < 1$ . Interestingly, in the  $\gamma \approx 1$  regime, Anc-VI achieves  $\mathcal{O}(1 / k)$ -rate while VI has a  $\mathcal{O}(1)$ -rate. We briefly note that the condition  $U^0 \leq TU^0$  and  $U^0 \geq TU^0$  have been used in analyses of variants of VI [69, Theorem 6.3.11], [77, p.3].
In the following, we briefly outline the proof of Theorem 1 while deferring the full description to Appendix B. In the outline, we highlight a particular step, labeled  $\triangle$ , that crucially relies on the linearity of the Bellman consistency operator. In the analysis for the Bellman optimality operator of Theorem 2, resolving the  $\triangle$  step despite the nonlinearity is the key technical challenge.
Proof outline of Theorem 1. Recall that we can write Bellman consistency operator as  $T^{\pi}V = r^{\pi} + \gamma \mathcal{P}^{\pi}V$  and  $T^{\pi}Q = r + \gamma \mathcal{P}^{\pi}Q$ . Since  $T^{\pi}$  is a linear operator<sup>1</sup>, we get
$$
\begin{array}{l} T ^ {\pi} U ^ {k} - U ^ {k} = T ^ {\pi} U ^ {k} - \left(1 - \beta_ {k}\right) T ^ {\pi} U ^ {k - 1} - \beta_ {k} T ^ {\pi} U ^ {\pi} - \beta_ {k} \left(U ^ {0} - U ^ {\pi}\right) \\ \triangleq \gamma \mathcal {P} ^ {\pi} \left(U ^ {k} - \left(1 - \beta_ {k}\right) U ^ {k - 1} - \beta_ {k} U ^ {\pi}\right) - \beta_ {k} \left(U ^ {0} - U ^ {\pi}\right) \\ = \gamma \mathcal {P} ^ {\pi} \left(\beta_ {k} \left(U ^ {0} - U ^ {\pi}\right) + \left(1 - \beta_ {k}\right) \left(T ^ {\pi} U ^ {k - 1} - U ^ {k - 1}\right)\right) - \beta_ {k} \left(U ^ {0} - U ^ {\pi}\right) \\ = \sum_ {i = 1} ^ {k} \left[ \left(\beta_ {i} - \beta_ {i - 1} \left(1 - \beta_ {i}\right)\right) \left(\Pi_ {j = i + 1} ^ {k} \left(1 - \beta_ {j}\right)\right) \left(\gamma \mathcal {P} ^ {\pi}\right) ^ {k - i + 1} \left(U ^ {0} - U ^ {\pi}\right) \right] \\ - \beta_ {k} (U ^ {0} - U ^ {\pi}) + \left(\Pi_ {j = 1} ^ {k} (1 - \beta_ {j})\right) (\gamma \mathcal {P} ^ {\pi}) ^ {k + 1} (U ^ {0} - U ^ {\pi}), \\ \end{array}
$$
where the first equality follows from the definition of Anc-VI and the property of fixed point, while the last equality follows from induction. Taking the  $\| \cdot \|_{\infty}$  -norm of both sides, we conclude
$$
\left\| T ^ {\pi} U ^ {k} - U ^ {k} \right\| _ {\infty} \leq \frac {\left(\gamma^ {- 1} - \gamma\right) \left(1 + 2 \gamma - \gamma^ {k + 1}\right)}{\left(\gamma^ {k + 1}\right) ^ {- 1} - \gamma^ {k + 1}} \left\| U ^ {0} - U ^ {\pi} \right\| _ {\infty}.
$$
2.2 Accelerated rate for Bellman optimality operator
We now present the accelerated convergence rate of Anc-VI for the Bellman optimality operator.
Our analysis uses what we call the Bellman anti-optimality operator, defined as
$$
\hat {T} ^ {\star} V (s) = \inf  _ {a \in \mathcal {A}} \left\{r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ V (s ^ {\prime}) \right] \right\}
$$
$$
\hat {T} ^ {\star} Q (s, a) = r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ \inf  _ {a ^ {\prime} \in \mathcal {A}} Q (s ^ {\prime}, a ^ {\prime}) \right],
$$
for all  $s \in S$  and  $a \in \mathcal{A}$ . (The sup is replaced with a inf.) When  $0 < \gamma < 1$ , the Bellman anti-optimality operator is  $\gamma$ -contractive and has a unique fixed point  $\hat{U}^\star$  by the exact same arguments that establish  $\gamma$ -contractiveness of the standard Bellman optimality operator.
Theorem 2. Let  $0 < \gamma < 1$  be the discount factor. Let  $T^{\star}$  and  $\hat{T}^{\star}$  respectively be the Bellman optimality and anti-optimality operators for  $V$  or  $Q$ . Let  $U^{\star}$  and  $\hat{U}^{\star}$  respectively be the fixed points of  $T^{\star}$  and  $\hat{T}^{\star}$ . Then, Anc-VI exhibits the rate
$$
\left\| T ^ {\star} U ^ {k} - U ^ {k} \right\| _ {\infty} \leq \frac {\left(\gamma^ {- 1} - \gamma\right) \left(1 + 2 \gamma - \gamma^ {k + 1}\right)}{\left(\gamma^ {k + 1}\right) ^ {- 1} - \gamma^ {k + 1}} \max \left\{\left\| U ^ {0} - U ^ {\star} \right\| _ {\infty}, \left\| U ^ {0} - \hat {U} ^ {\star} \right\| _ {\infty} \right\}
$$
for  $k = 0,1,\ldots$ . If, furthermore,  $U^0\leq T^\star U^0$  or  $U^0\geq T^\star U^0$ , then Anc-VI exhibits the rate
$$
\left\| T ^ {\star} U ^ {k} - U ^ {k} \right\| _ {\infty} \leq \frac {\left(\gamma^ {- 1} - \gamma\right) \left(1 + \gamma - \gamma^ {k + 1}\right)}{\left(\gamma^ {k + 1}\right) ^ {- 1} - \gamma^ {k + 1}} \left\| U ^ {0} - U ^ {\star} \right\| _ {\infty} \quad i f U ^ {0} \leq T ^ {\star} U ^ {0}
$$
$$
\left\| T ^ {\star} U ^ {k} - U ^ {k} \right\| _ {\infty} \leq \frac {\left(\gamma^ {- 1} - \gamma\right) \left(1 + \gamma - \gamma^ {k + 1}\right)}{\left(\gamma^ {k + 1}\right) ^ {- 1} - \gamma^ {k + 1}} \left\| U ^ {0} - \hat {U} ^ {\star} \right\| _ {\infty} \quad i f U ^ {0} \geq T ^ {\star} U ^ {0}
$$
for  $k = 0,1,\ldots$
<sup>1</sup>Arguably,  $T^{\pi}$  is affine, not linear, but we follow the convention of [69] say  $T^{\pi}$  is linear.
Anc-VI with the Bellman optimality operator exhibits the same accelerated convergence rate as Anc-VI with the Bellman consistency operator. As in Theorem 1, the rate of Theorem 2 also becomes  $\mathcal{O}(1 / k)$  when  $\gamma \approx 1$ , while VI has a  $\mathcal{O}(1)$ -rate.
Proof outline of Theorem 2. The key technical challenge of the proof comes from the fact that the Bellman optimality operator is non-linear. Similar to the Bellman consistency operator case, we have