SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} T ^ {\star} U ^ {k} - U ^ {k} = T ^ {\star} U ^ {k} - \left(1 - \beta_ {k}\right) T ^ {\star} U ^ {k - 1} - \beta_ {k} T ^ {\star} U ^ {\star} - \beta_ {k} \left(U ^ {0} - U ^ {\star}\right) \\ \leq \gamma \mathcal {P} ^ {\pi_ {k}} \left(U ^ {k} - \left(1 - \beta_ {k}\right) U ^ {k - 1} - \beta_ {k} U ^ {\star}\right) - \beta_ {k} \left(U ^ {0} - U ^ {\star}\right) \\ = \gamma \mathcal {P} ^ {\pi_ {k}} \left(\beta_ {k} \left(U ^ {0} - U ^ {\star}\right) + \left(1 - \beta_ {k}\right) \left(T ^ {\star} U ^ {k - 1} - U ^ {k - 1}\right)\right) - \beta_ {k} \left(U ^ {0} - U ^ {\star}\right) \\ \leq \sum_ {i = 1} ^ {k} \left[ \left(\beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i})\right) \left(\Pi_ {j = i + 1} ^ {k} (1 - \beta_ {j})\right) \left(\Pi_ {l = k} ^ {i} \gamma \mathcal {P} ^ {\pi_ {l}}\right) \left(U ^ {0} - U ^ {\star}\right) \right] \\ - \beta_ {k} (U ^ {0} - U ^ {\star}) + \left(\Pi_ {j = 1} ^ {k} (1 - \beta_ {j})\right) \left(\Pi_ {l = k} ^ {0} \gamma \mathcal {P} ^ {\pi_ {l}}\right) (U ^ {0} - U ^ {\star}), \\ \end{array}
$$
where  $\pi_k$  is the greedy policy satisfying  $T^{\pi_k}U^k = T^\star U^k$ , we define  $\Pi_{l=k}^{i}\gamma\mathcal{P}^{\pi_l} = \gamma\mathcal{P}^{\pi_k}\gamma\mathcal{P}^{\pi_{k-1}}\cdots\gamma\mathcal{P}^{\pi_i}$ , and last inequality follows by induction and monotonicity of Bellman optimality operator. The key step  $\triangleleft$  uses greedy policies  $\{\pi_l\}_{l=0,1,\ldots,k}$ , which are well defined when the action space is finite. When the action space is infinite, greedy policies may not exist, so we use the Hahn-Banach extension theorem to overcome this technicality. The full argument is provided in Appendix B.
To lower bound  $T^{\star}U^{k} - U^{k}$ , we use a similar line of reasoning with the Bellman anti-optimality operator. Combining the upper and lower bounds of  $T^{\star}U^{k} - U^{k}$ , we conclude the accelerated rate of Theorem 2.
For  $\gamma < 1$ , the rates of Theorems 1 and 2 can be translated to a bound on the distance to solution:
$$
\left\| U ^ {k} - U ^ {\star} \right\| _ {\infty} \leq \gamma^ {k} \frac {(1 + \gamma) (1 + 2 \gamma - \gamma^ {k + 1})}{(1 - \gamma^ {2 k + 2})} \left\| U ^ {0} - U ^ {\star} \right\| _ {\infty}
$$
for  $k = 1,2,\ldots$ . This  $O(\gamma^k)$  rate is worse than the rate of (classical) VI by a constant factor. Therefore, Anc-VI is better than VI in terms of the Bellman error, but it is not better than VI in terms of distance to solution.
3 Convergence when  $\gamma = 1$
Undiscounted MDPs are not commonly studied in the DP and RL theory literature due to the following difficulties: Bellman consistency and optimality operators may not have fixed points, VI is a nonexpansive (not contractive) fixed-point iteration and may not converge to a fixed point even if one exist, and the interpretation of a fixed point as the (optimal) value function becomes unclear when the fixed point is not unique. However, many modern deep RL setups actually do not use discounting, $^{2}$  and this empirical practice makes the theoretical analysis with  $\gamma = 1$  relevant.
In this section, we show that Anc-VI converges to fixed points of the Bellman consistency and optimality operators of undiscounted MDPs. While a full treatment of undiscounted MDPs is beyond the scope of this paper, we show that fixed points, if one exists, can be found, and we therefore argue that the inability to find fixed points should not be considered an obstacle in studying the  $\gamma = 1$  setup.
We first state our convergence result for finite state-action spaces.
Theorem 3. Let  $\gamma = 1$ . Let  $T: \mathbb{R}^n \to \mathbb{R}^n$  be the nonexpansive Bellman consistency or optimality operator for  $V$  or  $Q$ . Assume a fixed point exists (not necessarily unique). If,  $U^0 \leq TU^0$ , then Anc-VI exhibits the rate
$$
\left\| T U ^ {k} - U ^ {k} \right\| _ {\infty} \leq \frac {1}{k + 1} \left\| U ^ {0} - U ^ {\star} \right\| _ {\infty} \quad f o r k = 0, 1, \dots .
$$
As a specific example, the classical policy gradient theorem [82] calls for the use of  $\nabla J(\theta) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^{t} \nabla_{\theta} \log \pi_{\theta}(a_{t} \mid s_{t}) Q_{\gamma}^{\phi}(s_{t}, a_{t})\right]$ , but many modern deep policy gradient methods use  $\gamma = 1$  in the first instance of  $\gamma$  (so  $\gamma^{t} = 1$ ) while using  $\gamma < 1$  in  $Q_{\gamma}^{\phi}(s_{t}, a_{t})$  [63].
for any fixed point  $U^{\star}$  satisfying  $U^{0} \leq U^{\star}$ . Furthermore,  $U^{k} \to U^{\infty}$  for some fixed point  $U^{\infty}$ .
If rewards are nonnegative, then the condition  $U^0 \leq TU^0$  is satisfied with  $U^0 = 0$ . So, under this mild condition, Anc-VI with  $\gamma = 1$  converges with  $\mathcal{O}(1 / k)$ -rate on the Bellman error. To clarify, the convergence  $U^k \to U^\infty$  has no rate, i.e.,  $\| U^k - U^\infty \|_\infty = o(1)$ , while  $\left\| TU^k - U^k \right\|_\infty = \mathcal{O}(1 / k)$ . In contrast, standard VI does not guarantee convergence in this setup.
We also point out that the convergence of Bellman error does not immediately imply point convergence, i.e.,  $TU^{k} - U^{k}\to 0$  does not immediately imply  $U^k\rightarrow U^\star$ , when  $\gamma = 1$ . Rather, we show (i)  $U^{k}$  is a bounded sequence, (ii) any convergent subsequence  $U^{k_j}$  converges to a fixed point  $U^{\infty}$ , and (iii)  $U^{k}$  is elementwise monotonically nondecreasing and therefore has a single limit.
Next, we state our convergence result for general state-action spaces.
Theorem 4. Let  $\gamma = 1$ . Let the state and action spaces be general (possibly infinite) sets. Let  $T$  be the nonexpansive Bellman consistency or optimality operator for  $V$  or  $Q$ , and assume  $T$  is well defined. Assume a fixed point exists (not necessarily unique). If  $U^0 \leq TU^0$ , then Anc-VI exhibits the rate
$$
\left\| T U ^ {k} - U ^ {k} \right\| _ {\infty} \leq \frac {1}{k + 1} \left\| U ^ {0} - U ^ {\star} \right\| _ {\infty} \quad \text {f o r} k = 0, 1, \dots
$$
for any fixed point  $U^{\star}$  satisfying  $U^0 \leq U^{\star}$ . Furthermore,  $U^{k} \to U^{\infty}$  pointwise monotonically for some fixed point  $U^{\infty}$ .
The convergence  $U^k \to U^\infty$  pointwise in infinite state-action spaces is, in our view, a non-trivial contribution. When the state-action space is finite, pointwise convergence directly implies convergence in  $\| \cdot \|_\infty$ , and in this sense, Theorem 4 is generalization of Theorem 3. However, when the state-action space is infinite, pointwise convergence does not necessarily imply uniform convergence, i.e.,  $U^k \to U^\infty$  pointwise does not necessarily imply  $U^k \to U^\infty$  in  $\| \cdot \|_\infty$ .
4 Complexity lower bound
We now present a complexity lower bound establishing optimality of Anc-VI.
Theorem 5. Let  $k \geq 0$ ,  $n \geq k + 2$ ,  $0 < \gamma \leq 1$ , and  $U^0 \in \mathbb{R}^n$ . Then there exists an MDP with  $|\mathcal{S}| = n$  and  $|\mathcal{A}| = 1$  (which implies the Bellman consistency and optimality operator for  $V$  and  $Q$  all coincide as  $T \colon \mathbb{R}^n \to \mathbb{R}^n$ ) such that  $T$  has a fixed point  $U^\star$  satisfying  $U^0 \leq U^\star$  and
$$
\left\| T U ^ {k} - U ^ {k} \right\| _ {\infty} \geq \frac {\gamma^ {k}}{\sum_ {i = 0} ^ {k} \gamma^ {i}} \left\| U ^ {0} - U ^ {\star} \right\| _ {\infty}
$$
for any iterates  $\{U^i\}_{i = 0}^k$  satisfying
$$
U ^ {i} \in U ^ {0} + \operatorname {s p a n} \{T U ^ {0} - U ^ {0}, T U ^ {1} - U ^ {1}, \dots , T U ^ {i - 1} - U ^ {i - 1} \} \quad f o r i = 1, \dots , k.
$$
Proof outline of Theorem 5. Without loss of generality, assume  $n = k + 2$  and  $U^0 = 0$ . Consider the MDP  $(\mathcal{S},\mathcal{A},P,r,\gamma)$  such that
$$
\mathcal {S} = \{s _ {1}, \ldots , s _ {k + 2} \}, \quad \mathcal {A} = \{a _ {1} \}, \quad P (s _ {i} | s _ {j}, a _ {1}) = \mathbb {1} _ {\{i = j = 1, j = i + 1 \}}, \quad r (s _ {i}, a _ {1}) = \mathbb {1} _ {\{i = 2 \}}.
$$
Then,  $T = \gamma \mathcal{P}^{\pi}U + [0,1,0,\dots,0]^{\intercal}, U^{\star} = [0,1,\gamma,\dots,\gamma^{k}]^{\intercal}$ , and  $\left\| U^{0} - U^{\star}\right\|_{\infty} = 1$ . Under the span condition, we can show that  $\left(U^{k}\right)_{1} = \left(U^{k}\right)_{k+2} = 0$ . Then, we get
$$
T U ^ {k} - U ^ {k} = \left(0, 1 - \left(U ^ {k}\right) _ {2}, \gamma \left(U ^ {k}\right) _ {2} - \left(U ^ {k}\right) _ {3}, \ldots , \gamma \left(U ^ {k}\right) _ {k} - \left(U ^ {k}\right) _ {k + 1}, \gamma \left(U ^ {k}\right) _ {k + 1}\right)
$$
and this implies
$$
\left(T U ^ {k} - U ^ {k}\right) _ {1} + \left(T U ^ {k} - U ^ {k}\right) _ {2} + \gamma^ {- 1} \left(T U ^ {k} - U ^ {k}\right) _ {3} + \dots + \gamma^ {- k} \left(T U ^ {k} - U ^ {k}\right) _ {k + 2} = 1.
$$
<sup>3</sup>Well-definedness of  $T$  requires a  $\sigma$ -algebra on state and action spaces, expectation with respect to transition probability and policy to be well defined, boundedness and measurability of the output of Bellman operators, etc.
Taking the absolute value on both sides,
$$
\left(1 + \dots +\gamma^ {- k}\right)\max_{1\leq i\leq k + 2}\left\{\left|T U^{k} - U^{k}\right|_{i}\right\} \geq 1.
$$
Therefore, we conclude
$$
\| T U ^ {k} - U ^ {k} \| _ {\infty} \geq \frac {\gamma^ {k}}{\sum_ {i = 0} ^ {k} \gamma^ {i}} \left\| U ^ {0} - U ^ {\star} \right\| _ {\infty}.
$$
Note that the case  $\gamma = 1$  is included in Theorem 5. When  $\gamma = 1$ , the lower bound of Theorem 5 exactly matches the upper bound of Theorem 3.
$$
\frac {\gamma^ {k}}{\sum_ {i = 0} ^ {k} \gamma^ {i}} \leq \frac {\left(\gamma^ {- 1} - \gamma\right) \left(1 + \gamma - \gamma^ {k + 1}\right)}{\left(\gamma^ {k + 1}\right) ^ {- 1} - \gamma^ {k + 1}} \leq \frac {4 \gamma^ {k}}{\sum_ {i = 0} ^ {k} \gamma^ {i}} \quad \text {f o r a l l} 0 <   \gamma <   1,
$$
the lower bound establishes optimality of the second rates Theorems 1 and 2 up to a constant of factor 4. Theorem 5 improves upon the prior state-of-the-art complexity lower bound established in the proof of [37, Theorem 3] by a factor  $1 - \gamma^{k + 1}$ . (In [37, Theorem 3], a lower bound on the distance to optimal value function is provided. Their result has an implicit dependence on the initial distance to optimal value function  $\| U^0 -U^\star \|_\infty$ , so we make the dependence explicit, and we translate their result to a lower bound on the Bellman error. Once this is done, the difference between our lower bound of Theorem 5 and of [37, Theorem 3] is a factor of  $1 - \gamma^{k + 1}$ . The worst-case MDP of [37, Theorem 3] and our worst-case MDP primarily differ in the rewards, while the states and the transition probabilities are almost the same.)
The so-called "span condition" of Theorem 5 is arguably very natural and is satisfied by standard VI and Anc-VI. The span condition is commonly used in the construction of complexity lower bounds on first-order optimization methods [59, 23, 25, 13, 14, 65] and has been used in the prior state-of-the-art lower bound for standard VI [37, Theorem 3]. However, designing an algorithm that breaks the lower bound of Theorem 5 by violating the span condition remains a possibility. In optimization theory, there is precedence of lower bounds being broken by violating seemingly natural and minute conditions [40, 35, 98].
5 Approximate Anchored Value Iteration
In this section, we show that the anchoring mechanism is robust against evaluation errors of the Bellman operator, just as much as the standard approximate VI.
Let  $0 < \gamma < 1$  and let  $T^{\star}$  be the Bellman optimality operator. The Approximate Anchored Value Iteration (Apx-Anc-VI) is
$$
U _ {\epsilon} ^ {k} = T ^ {\star} U ^ {k - 1} + \epsilon^ {k - 1}
$$
$$
U ^ {k} = \beta_ {k} U ^ {0} + (1 - \beta_ {k}) U _ {\epsilon} ^ {k}
$$
(Apx-Anc-VI)
for  $k = 1,2,\ldots$ , where  $\beta_{k} = 1 / (\sum_{i = 0}^{k}\gamma^{-2i})$ ,  $U^0$  is an initial point, and the  $\{\epsilon^k\}_{k = 0}^\infty$  is the error sequence modeling approximate evaluations of  $T^{\star}$ .
Of course, the classical Approximate Value Iteration (Apx-VI) is
$$
U ^ {k} = T ^ {\star} U ^ {k - 1} + \epsilon^ {k - 1} \tag {Apx-VI}
$$
for  $k = 1,2,\ldots$  , where  $U^0$  is an initial point.
Fact 1 (Classical result, [11, p.333]). Let  $0 < \gamma < 1$  be the discount factor. Let  $T^{\star}$  be the Bellman optimality for  $V$  or  $Q$ . Let  $U^{\star}$  be the fixed point of  $T^{\star}$ . Then Apx-VI exhibits the rate
$$
\left\| T ^ {\star} U ^ {k} - U ^ {k} \right\| _ {\infty} \leq (1 + \gamma) \gamma^ {k} \left\| U ^ {0} - U ^ {\star} \right\| _ {\infty} + (1 + \gamma) \frac {1 - \gamma^ {k}}{1 - \gamma} \max  _ {0 \leq i \leq k - 1} \left\| \epsilon^ {i} \right\| _ {\infty} \quad f o r k = 1, 2, \dots .
$$
Theorem 6. Let  $0 < \gamma < 1$  be the discount factor. Let  $T^{\star}$  and  $\hat{T}^{\star}$  respectively be the Bellman optimality and anti-optimality operators for  $V$  or  $Q$ . Let  $U^{\star}$  and  $\hat{U}^{\star}$  respectively be the fixed points of  $T^{\star}$  and  $\hat{T}^{\star}$ . Then Apx-Anc-VI exhibits the rate
$$
\begin{array}{l} \left\| T ^ {\star} U ^ {k} - U ^ {k} \right\| _ {\infty} \leq \frac {\left(\gamma^ {- 1} - \gamma\right) \left(1 + 2 \gamma - \gamma^ {k + 1}\right)}{\left(\gamma^ {k + 1}\right) ^ {- 1} - \gamma^ {k + 1}} \max  \left\{\left\| U ^ {0} - U ^ {\star} \right\| _ {\infty}, \left\| U ^ {0} - \hat {U} ^ {\star} \right\| _ {\infty} \right\} \\ + \frac {1 + \gamma}{1 + \gamma^ {k + 1}} \frac {1 - \gamma^ {k}}{1 - \gamma} \max  _ {0 \leq i \leq k - 1} \left\| \epsilon^ {i} \right\| _ {\infty} \quad f o r k = 1, 2, \dots . \\ \end{array}
$$
If, furthermore,  $U^0 \geq T^\star U^0$ , then (Apx-Anc-VI) exhibits the rate
$$
\left\| T ^ {\star} U ^ {k} - U ^ {k} \right\| _ {\infty} \leq \frac {\left(\gamma^ {- 1} - \gamma\right) \left(1 + \gamma - \gamma^ {k + 1}\right)}{\left(\gamma^ {k + 1}\right) ^ {- 1} - \gamma^ {k + 1}} \left\| U ^ {0} - \hat {U} ^ {\star} \right\| _ {\infty} + \frac {1 + \gamma}{1 + \gamma^ {k + 1}} \frac {1 - \gamma^ {k}}{1 - \gamma} \max  _ {0 \leq i \leq k - 1} \left\| \epsilon^ {i} \right\| _ {\infty}
$$
for  $k = 1,2,\ldots$
The dependence on  $\max \| \epsilon_i\|_{\infty}$  of Apx-Anc-VI is no worse than that of Apx-VI. In this sense, Apx-Anc-VI is robust against evaluation errors of the Bellman operator, just as much as the standard Apx-VI. Finally, we note that a similar analysis can be done for Apx-Anc-VI with the Bellman consistency operator.
6 Gauss-Seidel Anchored Value Iteration
In this section, we show that the anchoring mechanism can be combined with Gauss-Seidel-type updates in finite state-action spaces. Let  $0 < \gamma < 1$  and let  $T^{\star} \colon \mathbb{R}^{n} \to \mathbb{R}^{n}$  be the Bellman optimality operator. Define  $T_{GS}^{\star} \colon \mathbb{R}^{n} \to \mathbb{R}^{n}$  as
$$
T _ {G S} ^ {\star} = T _ {n} ^ {\star} \dots T _ {2} ^ {\star} T _ {1} ^ {\star},
$$
where  $T_{j}^{\star}:\mathbb{R}^{n}\to \mathbb{R}^{n}$  is defined as
$$
T _ {j} ^ {\star} (U) = \left(U _ {1}, \dots , U _ {j - 1}, \left(T ^ {\star} (U)\right) _ {j}, U _ {j + 1}, \dots , U _ {n}\right)
$$
for  $j = 1,\dots ,n$
Fact 2. [Classical result, [69, Theorem 6.3.4]]  $T_{GS}^{\star}$  is a  $\gamma$ -contractive operator and has the same fixed point as  $T^{\star}$ .
The Gauss-Seidel Anchored Value Iteration (GS-Anc-VI) is
$$
U ^ {k} = \beta_ {k} U ^ {0} + (1 - \beta_ {k}) T _ {G S} ^ {\star} U ^ {k - 1} \tag {GS-Anc-VI}
$$
for  $k = 1,2,\ldots$ , where  $\beta_{k} = 1 / (\sum_{i = 0}^{k}\gamma^{-2i})$  and  $U^0$  is an initial point.
Theorem 7. Let the state and action spaces be finite sets. Let  $0 < \gamma < 1$  be the discount factor. Let  $T^{\star}$  and  $\hat{T}^{\star}$  respectively be the Bellman optimality and anti-optimality operators for  $V$  or  $Q$ . Let  $U^{\star}$  and  $\hat{U}^{\star}$  respectively be the fixed points of  $T^{\star}$  and  $\hat{T}^{\star}$ . Then GS-Anc-VI exhibits the rate
$$
\left\| T _ {G S} ^ {\star} U ^ {k} - U ^ {k} \right\| _ {\infty} \leq \frac {\left(\gamma^ {- 1} - \gamma\right) \left(1 + 2 \gamma - \gamma^ {k + 1}\right)}{\left(\gamma^ {k + 1}\right) ^ {- 1} - \gamma^ {k + 1}} \max  \left\{\left\| U ^ {0} - U ^ {\star} \right\| _ {\infty}, \left\| U ^ {0} - \hat {U} ^ {\star} \right\| _ {\infty} \right\}
$$
for  $k = 0,1,\ldots$ . If furthermore,  $U^0\leq T_{GS}^\star U^0$  or  $U^0\geq T_{GS}^\star U^0$ , then GS-Anc-VI exhibits the rate
$$
\left\| T _ {G S} ^ {\star} U ^ {k} - U ^ {k} \right\| _ {\infty} \leq \frac {\left(\gamma^ {- 1} - \gamma\right) \left(1 + \gamma - \gamma^ {k + 1}\right)}{\left(\gamma^ {k + 1}\right) ^ {- 1} - \gamma^ {k + 1}} \left\| U ^ {0} - U ^ {\star} \right\| _ {\infty} \quad i f U ^ {0} \leq T _ {G S} ^ {\star} U ^ {0}
$$
$$
\left\| T _ {G S} ^ {\star} U ^ {k} - U ^ {k} \right\| _ {\infty} \leq \frac {\left(\gamma^ {- 1} - \gamma\right) \left(1 + \gamma - \gamma^ {k + 1}\right)}{\left(\gamma^ {k + 1}\right) ^ {- 1} - \gamma^ {k + 1}} \left\| U ^ {0} - \hat {U} ^ {\star} \right\| _ {\infty} \quad i f U ^ {0} \geq T _ {G S} ^ {\star} U ^ {0}
$$
We point out that GS-Anc-VI cannot be directly extended to infinite action spaces since Hahn-Banach extension theorem is not applicable in the Gauss-Seidel setup. Furthermore, we note that a similar analysis can be carried out for GS-Anc-VI with the Bellman consistency operator.
We show that the classical value iteration (VI) is, in fact, suboptimal and that the anchoring mechanism accelerates VI to be optimal in the sense that the accelerated rate matches a complexity lower bound up to a constant factor of 4. We also show that the accelerated iteration provably converges to a fixed point even when  $\gamma = 1$ , if a fixed point exists. Being able to provide a substantive improvement upon the classical VI is, in our view, a surprising contribution.
One direction of future work is to study the empirical effectiveness of Anc-VI. Another direction is to analyze Anc-VI in a model-free setting and, more broadly, to investigate the effectiveness of the anchor mechanism in more practical RL methods.
Our results lead us to believe that many of the classical foundations of dynamic programming and reinforcement learning may be improved with a careful examination based on an optimization complexity theory perspective. The theory of optimal optimization algorithms has recently enjoyed significant developments [44, 43, 45, 98, 66], the anchoring mechanism being one such example [49, 65], and the classical DP and RL theory may benefit from a similar line of investigation on iteration complexity.
This work was supported by the Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) [NO.2021-0-01343, Artificial Intelligence Graduate School Program (Seoul National University)] and the Samsung Science and Technology Foundation (Project Number SSTF-BA2101-02). We thank Jisun Park for providing valuable feedback.
[1] M. Akian, S. Gaubert, Z. Qu, and O. Saadi. Multiply accelerated value iteration for nonsymmetric affine fixed point problems and application to markov decision processes. SIAM Journal on Matrix Analysis and Applications, 43(1):199-232, 2022.
[2] D. G. Anderson. Iterative procedures for nonlinear integral equations. Journal of the Association for Computing Machinery, 12(4):547-560, 1965.
[3] D. Andre, N. Friedman, and R. Parr. Generalized prioritized sweeping. Neural Information Processing Systems, 1997.
[4] M. G. Azar, R. Munos, M. Ghavamzadeh, and H. Kappen. Speedy Q-learning. Neural Information Processing Systems, 2011.
[5] S. Banach. Sur les opérations dans les ensembles abstraits et leur application aux équations intégrales. Fundamenta Mathematicae, 3(1):133-181, 1922.
[6] M. Barre, A. Taylor, and A. d'Aspremont. Convergence of a constrained vector extrapolation scheme. SIAM Journal on Mathematics of Data Science, 4(3):979-1002, 2022.
[7] M. G. Bellemare, G. Ostrovski, A. Guez, P. Thomas, and R. Munos. Increasing the action gap: New operators for reinforcement learning. Association for the Advancement of Artificial Intelligence, 2016.
[8] R. Bellman. A Markovian decision process. Journal of Mathematics and Mechanics, 6(5):679-684, 1957.
[9] D. Bertsekas and J. Tsitsiklis. Parallel and Distributed Computation: Numerical Methods. Athena Scientific, 2015.
[10] D. P. Bertsekas. Dynamic Programming and Optimal Control, volume II. 4th edition, 2012.
[11] D. P. Bertsekas and J. N. Tsitsiklis. Neuro-Dynamic Programming. Athena Scientific, 1995.
[33] M. Fellows, K. Hartikainen, and S. Whiteson. Bayesian Bellman operators. Neural Information Processing Systems, 2021.
[34] M. Geist and B. Scherrer. Anderson acceleration for reinforcement learning. European Workshop on Reinforcement Learning, 2018.
[35] N. Golowich, S. Pattathil, C. Daskalakis, and A. Ozdaglar. Last iterate is slower than averaged iterate in smooth convex-concave saddle point problems. Conference on Learning Theory, 2020.
[36] G. J. Gordon. Stable function approximation in dynamic programming. International Conference on Machine Learning, 1995.
[37] V. Goyal and J. Grand-Clément. A first-order approach to accelerated value iteration. Operations Research, 71(2):517-535, 2022.
[38] J. Grand-Clement. From convex optimization to MDPs: A review of first-order, second-order and quasi-newton methods for MDPs. arXiv:2104.10677, 2021.
[39] B. Halpern. Fixed points of nonexpanding maps. Bulletin of the American Mathematical Society, 73(6):957-961, 1967.
[40] R. Hannah, Y. Liu, D. O'Connor, and W. Yin. Breaking the span assumption yields fast finite-sum minimization. Neural Information Processing Systems, 2018.
[41] M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning. Association for the Advancement of Artificial Intelligence, 2018.
[42] F. Iutzeler and J. M. Hendrickx. A generic online acceleration scheme for optimization algorithms via relaxation and inertia. Optimization Methods and Software, 34(2):383-405, 2019.
[43] D. Kim. Accelerated proximal point method for maximally monotone operators. Mathematical Programming, 190(1-2):57-87, 2021.
[44] D. Kim and J. A. Fessler. Optimized first-order methods for smooth convex minimization. Mathematical Programming, 159(1-2):81-107, 2016.
[45] D. Kim and J. A. Fessler. Optimizing the efficiency of first-order methods for decreasing the gradient of smooth convex functions. Journal of Optimization Theory and Applications, 188(1):192-219, 2021.
[46] J. Lee, C. Park, and E. K. Ryu. A geometric structure of acceleration and its role in making gradients small fast. Neural Information Processing Systems, 2021.
[47] S. Lee and D. Kim. Fast extra gradient methods for smooth structured nonconvex-nonconcave minimax problems. Neural Information Processing Systems, 2021.
[48] L. Leustean. Rates of asymptotic regularity for Halpern iterations of nonexpansive mappings. Journal of Universal Computer Science, 13(11):1680-1691, 2007.
[49] F. Lieder. On the convergence rate of the Halpern-iteration. Optimization Letters, 15(2):405–418, 2021.
[50] M. Lutter, S. Mannor, J. Peters, D. Fox, and A. Garg. Value iteration in continuous actions, states and time. International Conference on Machine Learning, 2021.
[51] P.-E. Maingé. Convergence theorems for inertial KM-type algorithms. Journal of Computational and Applied Mathematics, 219(1):223-236, 2008.
[52] A. massoud Farahmand, M. Ghavamzadeh, C. Szepesváři, and S. Mannor. Regularized fitted Q-iteration for planning in continuous-space Markovian decision problems. American Control Conference, 2009.