SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[74] D. Scieur, A. d'Aspremont, and F. Bach. Regularized nonlinear acceleration. Mathematical Programming, 179(1-2):47-83, 2020.
[75] Y. Shehu. Convergence rate analysis of inertial Krasnoselskii-Mann type iteration with applications. Numerical Functional Analysis and Optimization, 39(10):1077-1091, 2018.
[76] W. Shi, S. Song, H. Wu, Y.-C. Hsu, C. Wu, and G. Huang. Regularized Anderson acceleration for off-policy deep reinforcement learning. Neural Information Processing Systems, 2019.
[77] O. Shlakhter, C.-G. Lee, D. Khmelev, and N. Jaber. Acceleration operators in the value iteration algorithms for Markov decision processes. Operations Research, 58(1):193-202, 2010.
[78] J. J. Suh, J. Park, and E. K. Ryu. Continuous-time analysis of anchor acceleration. Neural Information Processing Systems, 2023.
[79] K. Sun, Y. Wang, Y. Liu, B. Pan, S. Jui, B. Jiang, L. Kong, et al. Damped Anderson mixing for deep reinforcement learning: Acceleration, convergence, and stabilization. Neural Information Processing Systems, 2021.
[80] R. S. Sutton. Learning to predict by the methods of temporal differences. Machine Learning, 3:9-44, 1988.
[81] R. S. Sutton and A. G. Barto. Reinforcement Learning: An introduction. MIT press, 2nd edition, 2018.
[82] R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. Neural Information Processing Systems, 1999.
[83] Q. Sykora, M. Ren, and R. Urtasun. Multi-agent routing value iteration network. International Conference on Machine Learning, 2020.
[84] C. Szepesvári. Algorithms for Reinforcement Learning. Springer, 1st edition, 2010.
[85] A. Tamar, Y. Wu, G. Thomas, S. Levine, and P. Abbeel. Value iteration networks. Neural Information Processing Systems, 2016.
[86] A. Taylor and Y. Drori. An optimal gradient method for smooth strongly convex minimization. Mathematical Programming, 199(1-2):557-594, 2023.
[87] S. Tosatto, M. Pirotta, C. d'Eramo, and M. Restelli. Boosted fitted Q-iteration. International Conference on Machine Learning, 2017.
[88] J. N. Tsitsiklis. Asynchronous stochastic approximation and Q-learning. Machine Learning, 16:185-202, 1994.
[89] H. Van Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double Q-learning. Association for the Advancement of Artificial Intelligence, 2016.
[90] B. Van Roy. Performance loss bounds for approximate value iteration with state aggregation. Mathematics of Operations Research, 31(2):234-244, 2006.
[91] B. Van Scoy, R. A. Freeman, and K. M. Lynch. The fastest known globally convergent first-order method for minimizing strongly convex functions. IEEE Control Systems Letters, 2(1):49-54, 2018.
[92] N. Vieillard, B. Scherrer, O. Pietquin, and M. Geist. Momentum in reinforcement learning. International Conference on Artificial Intelligence and Statistics, 2020.
[93] H. F. Walker and P. Ni. Anderson acceleration for fixed-point iterations. SIAM Journal on Numerical Analysis, 49(4):1715-1735, 2011.
[94] C. J. Watkins and P. Dayan. Q-learning. Machine Learning, 8:279-292, 1992.
[95] D. Wingate, K. D. Seppi, and S. Mahadevan. Prioritization methods for accelerating MDP solvers. Journal of Machine Learning Research, 6(25):851-881, 2005.
[96] R. Wittmann. Approximation of fixed points of nonexpansive mappings. Archiv der Mathematik, 58(5):486-491, 1992.
[97] H.-K. Xu. Iterative algorithms for nonlinear operators. Journal of the London Mathematical Society, 66(1):240-256, 2002.
[98] T. Yoon and E. K. Ryu. Accelerated algorithms for smooth convex-concave minimax problems with  $\mathcal{O}(1 / k^2)$  rate on squared gradient norm. International Conference on Machine Learning, 2021.
[99] T. Yoon and E. K. Ryu. Accelerated minimax algorithms flock together. arXiv:2205.11093, 2022.
[100] Y. Zeng, F. Feng, and W. Yin. AsyncQVI: Asynchronous-parallel Q-value iteration for discounted Markov decision processes with near-optimal sample complexity. International Conference on Artificial Intelligence and Statistics, 2020.
[101] J. Zhang, B. O'Donoghue, and S. Boyd. Globally convergent type-I Anderson acceleration for nonsmooth fixed-point iterations. SIAM Journal on Optimization, 30(4):3170-3197, 2020.
[102] K. Zhou, L. Tian, A. M.-C. So, and J. Cheng. Practical schemes for finding near-stationary points of convex finite-sums. International Conference on Artificial Intelligence and Statistics, 2022.
A Preliminaries
For notational unity, we use the symbol  $U$  when both  $V$  and  $Q$  can be used.
Lemma 1. [10, Lemma 1.1.1] Let  $0 < \gamma \leq 1$ . If  $U \leq \tilde{U}$ , then  $T^{\pi}U \leq T^{\pi}\tilde{U}, T^{\star}U \leq T^{\star}\tilde{U}$ .
Lemma 2. Let  $0 < \gamma \leq 1$ . For any policy  $\pi$ ,  $\mathcal{P}^{\pi}$  is a nonexpansive linear operator such that if  $U \leq \tilde{U}$ ,  $\mathcal{P}^{\pi}U \leq \mathcal{P}^{\pi}\tilde{U}$ .
Proof. If  $r(s, a) = 0$  for all  $s \in S$  and  $a \in A$ ,  $T^{\pi} = \gamma \mathcal{P}^{\pi}$ . Then by Lemma 1 and  $\gamma$ -contraction of  $T^{\pi}$ , we have the desired result.
Lemma 3. Let  $0 < \gamma < 1$ . Let  $T^{\star}$  and  $\hat{T}^{\star}$  respectively be the Bellman optimality and anti-optimality operators. Let  $U^{\star}$  and  $\hat{U}^{\star}$  respectively be the fixed points of  $T^{\star}$  and  $\hat{T}^{\star}$ . Then  $\hat{U}^{\star} \leq U^{\star}$ .
Proof. By definition,  $\hat{U}^{\star} = \hat{T}^{\star}\hat{U}^{\star}\leq T^{\star}\hat{U}^{\star}$ . Thus,  $\hat{U}^{\star}\leq \lim_{m\to \infty}(T^{\star})^{m}\hat{U}^{\star} = U^{\star}$
B Omitted proofs in Section 2
First, we prove the following lemma by induction.
Lemma 4. Let  $0 < \gamma \leq 1$ , and if  $\gamma = 1$ , assume a fixed point  $U^{\pi}$  exists. For the iterates  $\{U^k\}_{k=0,1,\ldots}$  of  $\text{Anc-}VI$ ,
$$
\begin{array}{l} T ^ {\pi} U ^ {k} - U ^ {k} = \sum_ {i = 1} ^ {k} \left[ \left(\beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i})\right) \left(\Pi_ {j = i + 1} ^ {k} (1 - \beta_ {j})\right) (\gamma \mathcal {P} ^ {\pi}) ^ {k - i + 1} (U ^ {0} - U ^ {\pi}) \right] \\ - \beta_ {k} (U ^ {0} - U ^ {\pi}) + \left(\Pi_ {j = 1} ^ {k} (1 - \beta_ {j})\right) (\gamma \mathcal {P} ^ {\pi}) ^ {k + 1} (U ^ {0} - U ^ {\pi}) \\ \end{array}
$$
where  $\left(\Pi_{j = k + 1}^{k}(1 - \beta_{j})\right) = 1$  and  $\beta_0 = 1$
Proof. If  $k = 0$ , we have
$$
\begin{array}{l} T ^ {\pi} U ^ {0} - U ^ {0} = T ^ {\pi} U ^ {0} - U ^ {\pi} - (U ^ {0} - U ^ {\pi}) \\ = T ^ {\pi} U ^ {0} - T ^ {\pi} U ^ {\pi} - \left(U ^ {0} - U ^ {\pi}\right) \\ = \gamma \mathcal {P} ^ {\pi} (U ^ {0} - U ^ {\pi}) - (U ^ {0} - U ^ {\pi}) \\ \end{array}
$$
If  $k = m$ , since  $T^{\pi}$  is a linear operator,
$$
\begin{array}{l} T ^ {\pi} U ^ {m} - U ^ {m} = T ^ {\pi} U ^ {m} - \left(1 - \beta_ {m}\right) T ^ {\pi} U ^ {m - 1} - \beta_ {m} U ^ {0} \\ = T ^ {\pi} U ^ {m} - \left(1 - \beta_ {m}\right) T ^ {\pi} U ^ {m - 1} - \beta_ {m} U ^ {\pi} - \beta_ {m} \left(U ^ {0} - U ^ {\pi}\right) \\ = T ^ {\pi} U ^ {m} - \left(1 - \beta_ {m}\right) T ^ {\pi} U ^ {m - 1} - \beta_ {m} T ^ {\pi} U ^ {\pi} - \beta_ {m} \left(U ^ {0} - U ^ {\pi}\right) \\ = \gamma \mathcal {P} ^ {\pi} \left(U ^ {m} - \left(1 - \beta_ {m}\right) U ^ {m - 1} - \beta_ {m} U ^ {\pi}\right) - \beta_ {m} \left(U ^ {0} - U ^ {\pi}\right) \\ = \gamma \mathcal {P} ^ {\pi} \left(\beta_ {m} \left(U ^ {0} - U ^ {\pi}\right) + \left(1 - \beta_ {m}\right) \left(T ^ {\pi} U ^ {m - 1} - U ^ {m - 1}\right)\right) - \beta_ {m} \left(U ^ {0} - U ^ {\pi}\right) \\ = (1 - \beta_ {m}) \gamma \mathcal {P} ^ {\pi} \sum_ {i = 1} ^ {m - 1} \left[ \left(\beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i})\right) \left(\Pi_ {j = i + 1} ^ {m - 1} (1 - \beta_ {j})\right) (\gamma \mathcal {P} ^ {\pi}) ^ {m - 1 - i + 1} (U ^ {0} - U ^ {\pi}) \right] \\ - \left(1 - \beta_ {m}\right) \gamma \mathcal {P} ^ {\pi} \beta_ {m - 1} \left(U ^ {0} - U ^ {\pi}\right) + \left(1 - \beta_ {m}\right) \gamma \mathcal {P} ^ {\pi} \left(\Pi_ {j = 1} ^ {m - 1} \left(1 - \beta_ {j}\right)\right) \left(\gamma \mathcal {P} ^ {\pi}\right) ^ {m} \left(U ^ {0} - U ^ {\pi}\right) \\ + \beta_ {m} \gamma \mathcal {P} ^ {\pi} (U ^ {0} - U ^ {\pi}) - \beta_ {m} (U ^ {0} - U ^ {\pi}) \\ = \sum_ {i = 1} ^ {m - 1} \left[ \left(\beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i})\right) \left(\Pi_ {j = i + 1} ^ {m} (1 - \beta_ {j})\right) (\gamma \mathcal {P} ^ {\pi}) ^ {m - i + 1} (U ^ {0} - U ^ {\pi}) \right] \\ - \beta_ {m - 1} \left(1 - \beta_ {m}\right) \gamma \mathcal {P} ^ {\pi} \left(U ^ {0} - U ^ {\pi}\right) + \beta_ {m} \gamma \mathcal {P} ^ {\pi} \left(U ^ {0} - U ^ {\pi}\right) \\ - \beta_ {m} \left(U ^ {0} - U ^ {\pi}\right) + \left(\Pi_ {j = 1} ^ {m} \left(1 - \beta_ {j}\right)\right) \left(\gamma \mathcal {P} ^ {\pi}\right) ^ {m + 1} \left(U ^ {0} - U ^ {\pi}\right) \\ \end{array}
$$
$$
\begin{array}{l} = \sum_ {i = 1} ^ {m} \left[ \left(\beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i})\right) \left(\Pi_ {j = i + 1} ^ {m} (1 - \beta_ {j})\right) (\gamma \mathcal {P} ^ {\pi}) ^ {m - i + 1} (U ^ {0} - U ^ {\pi}) \right] \\ - \beta_ {m} \left(U ^ {0} - U ^ {\pi}\right) + \left(\Pi_ {j = 1} ^ {m} \left(1 - \beta_ {j}\right)\right) \left(\gamma \mathcal {P} ^ {\pi}\right) ^ {m + 1} \left(U ^ {0} - U ^ {\pi}\right) \\ \end{array}
$$
Now, we prove the first rate of Theorem 1.
Proof of first rate in Theorem 1. Taking  $\| \cdot \|_{\infty}$ -norm both sides of equality in Lemma 4, we get
$$
\begin{array}{l} \left\| T ^ {\pi} U ^ {k} - U ^ {k} \right\| _ {\infty} \leq \sum_ {i = 1} ^ {k} \left| \beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i}) \right| \left(\Pi_ {j = i + 1} ^ {k} (1 - \beta_ {j})\right) \left\| (\gamma \mathcal {P} ^ {\pi}) ^ {k - i + 1} \left(U ^ {0} - U ^ {\pi}\right) \right\| _ {\infty} \\ + \beta_ {k} \left\| U ^ {0} - U ^ {\pi} \right\| _ {\infty} + \left(\Pi_ {i = 1} ^ {k} (1 - \beta_ {i})\right) \left\| (\gamma \mathcal {P} ^ {\pi}) ^ {k + 1} (U ^ {0} - U ^ {\pi}) \right\| _ {\infty} \\ \leq \left(\sum_ {i = 1} ^ {k} \gamma^ {k - i + 1} \left| \beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i}) \right| \left(\Pi_ {j = i + 1} ^ {k} (1 - \beta_ {j})\right) + \beta_ {k} + \gamma^ {k + 1} \Pi_ {j = 1} ^ {k} (1 - \beta_ {j})\right) \\ \left\| U ^ {0} - U ^ {\star} \right\| _ {\infty} \\ = \left(\sum_ {i = 1} ^ {k} \gamma^ {k + i - 1} \frac {\left(1 - \gamma^ {2}\right) ^ {2}}{1 - \gamma^ {(2 k + 2)}} + \gamma^ {2 k} \frac {1 - \gamma^ {2}}{1 - \gamma^ {2 k + 2}} + \gamma^ {k + 1} \frac {1 - \gamma^ {2}}{1 - \gamma^ {2 k + 2}}\right) \| U ^ {0} - U ^ {\pi} \| _ {\infty} \\ = \frac {\left(\gamma^ {- 1} - \gamma\right) \left(1 + 2 \gamma - \gamma^ {k + 1}\right)}{\left(\gamma^ {k + 1}\right) ^ {- 1} - \gamma^ {k + 1}} \| U ^ {0} - U ^ {\pi} \| _ {\infty}, \\ \end{array}
$$
where the first inequality comes from triangular inequality, second inequality is from Lemma 2, and equality come from calculations.
For the second rate of Theorem 1, we introduce following lemma.
Lemma 5. Let  $0 < \gamma < 1$ . Let  $T$  be Bellman consistency or optimality operator. For the iterates  $\{U^k\}_{k = 0,1,\ldots}$  of Anc-VI, if  $U^0 \leq TU^0$ , then  $U_{k - 1} \leq U_k \leq TU_{k - 1} \leq TU_k \leq U^\star$  for  $1 \leq k$ . Also, if  $U^0 \geq TU^0$ , then  $U_{k - 1} \geq U_k \geq TU_{k - 1} \geq TU_k \geq U^\star$  for  $1 \leq k$ .
Proof. First, let  $U^0 \leq TU^0$ . If  $k = 1$ ,  $U^0 \leq \beta_1 U^0 + (1 - \beta_1) TU^0 = U_1 \leq TU^0$  by assumption. Since  $U^0 \leq U^1$ ,  $TU^0 \leq TU^1$  by monotonicity of Bellman consistency and optimality operators.
By induction,
$$
U ^ {k} = \beta_ {k} U ^ {0} + (1 - \beta_ {k}) T U ^ {k - 1} \leq T U ^ {k - 1},
$$
and since  $\beta_{k}\leq \beta_{k - 1}$
$$
\begin{array}{l} \beta_ {k} U ^ {0} + \left(1 - \beta_ {k}\right) T U ^ {k - 1} \geq \beta_ {k - 1} U ^ {0} + \left(1 - \beta_ {k - 1}\right) T U ^ {k - 1} \\ \geq \beta_ {k - 1} U ^ {0} + (1 - \beta_ {k - 1}) T U ^ {k - 2} \\ = U ^ {k - 1}. \\ \end{array}
$$
Also,  $U^{k - 1} \leq U^k$  implies  $TU^{k - 1} \leq TU^k$  by monotonicity of Bellman consistency and optimality operators, and  $U^k \leq TU^k$  implies that  $U^k \leq \lim_{m\to \infty}(T)^m U^k = U^\star$  for all  $k = 0, 1, \ldots$ .
Now, suppose  $U^0 \geq TU^0$ . If  $k = 1$ ,  $U^0 \geq \beta_1 U^0 + (1 - \beta_1) TU^0 = U_1 \geq TU^0$  by assumption. Since  $U^0 \geq U^1$ ,  $TU^0 \geq TU^1$  by monotonicity of Bellman consistency and optimality operators.
$$
U ^ {k} = \beta_ {k} U ^ {0} + (1 - \beta_ {k}) T U ^ {k - 1} \geq T U ^ {k - 1},
$$
$$
\begin{array}{l} \beta_ {k} U ^ {0} + (1 - \beta_ {k}) T U ^ {k - 1} \leq \beta_ {k - 1} U ^ {0} + (1 - \beta_ {k - 1}) T U ^ {k - 1} \\ \leq \beta_ {k - 1} U ^ {0} + (1 - \beta_ {k - 1}) T U ^ {k - 2} \\ = U ^ {k - 1}. \\ \end{array}
$$
Also,  $U^{k-1} \geq U^k$  implies  $TU^{k-1} \geq TU^k$  by monotonicity of Bellman consistency and optimality operators, and  $U_k \geq TU_k$  implies that  $U^k \geq \lim_{m \to \infty} (T)^m U^k = U^\star$  for all  $k = 0, 1, \ldots$ .
Now, we prove following key lemmas.
Lemma 6. Let  $0 < \gamma \leq 1$ , and assume a fixed point  $U^{\pi}$  exists if  $\gamma = 1$ . For the iterates  $\{U^k\}_{k=0,1,\ldots}$  of  $\text{Anc-VI}$ , if  $U^0 \leq U^\pi$ ,
$$
\begin{array}{l} T ^ {\pi} U ^ {k} - U ^ {k} \leq \sum_ {i = 1} ^ {k} \left[ \left(\beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i})\right) \left(\Pi_ {j = i + 1} ^ {k} (1 - \beta_ {j})\right) \left(\gamma \mathcal {P} ^ {\pi}\right) ^ {k - i + 1} \left(U ^ {0} - U ^ {\pi}\right) \right] \\ - \beta_ {k} \left(U ^ {0} - U ^ {\pi}\right), \\ \end{array}
$$
Lemma 7. Let  $0 < \gamma < 1$ . For the iterates  $\{U^k\}_{k = 0,1,\ldots}$  of Anc-VI, if  $U^0 \geq T^\pi U^0$
$$
\begin{array}{l} T ^ {\pi} U ^ {k} - U ^ {k} \geq \sum_ {i = 1} ^ {k} \left[ \left(\beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i})\right) \left(\Pi_ {j = i + 1} ^ {k} (1 - \beta_ {j})\right) \left(\gamma \mathcal {P} ^ {\pi}\right) ^ {k - i + 1} \left(U ^ {0} - U ^ {\pi}\right) \right] \\ - \beta_ {k} (U ^ {0} - U ^ {\pi}), \\ \end{array}
$$
Proof of Lemma 6. If  $U^0 \leq U^\pi$ , we get
$$
\begin{array}{l} T ^ {\pi} U ^ {k} - U ^ {k} = \sum_ {i = 1} ^ {k} \left[ \left(\beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i})\right) \left(\Pi_ {j = i + 1} ^ {k} (1 - \beta_ {j})\right) \left(\gamma \mathcal {P} ^ {\pi}\right) ^ {k - i + 1} \left(U ^ {0} - U ^ {\pi}\right) \right] \\ - \beta_ {k} \left(U ^ {0} - U ^ {\pi}\right) + \left(\Pi_ {j = 1} ^ {k} \left(1 - \beta_ {j}\right)\right) \left(\gamma \mathcal {P} ^ {\pi}\right) ^ {k + 1} \left(U ^ {0} - U ^ {\pi}\right) \\ \leq \sum_ {i = 1} ^ {k} \left[ (\beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i})) \left(\Pi_ {j = i + 1} ^ {k} (1 - \beta_ {j})\right) (\gamma \mathcal {P} ^ {\pi}) ^ {k - i + 1} (U ^ {0} - U ^ {\pi}) \right] - \beta_ {k} (U ^ {0} - U ^ {\pi}), \\ \end{array}
$$
by Lemma 4 and the fact that  $\left(\Pi_{j=1}^{k}(1 - \beta_{j})\right)\left(\gamma \mathcal{P}^{\pi}\right)^{k+1}(U^{0} - U^{\pi}) \leq 0$ .
Proof of Lemma 7. If  $U^0 \geq TU^0$ ,  $U^0 - U^\pi \geq 0$  by Lemma 5. Hence, by Lemma 4, we have
$$
T ^ {\pi} U ^ {k} - U ^ {k} \geq \sum_ {i = 1} ^ {k} \left[ \left(\beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i})\right) \left(\Pi_ {j = i + 1} ^ {k} (1 - \beta_ {j})\right) \left(\gamma \mathcal {P} ^ {\pi}\right) ^ {k - i + 1} \left(U ^ {0} - U ^ {\pi}\right) \right] - \beta_ {k} \left(U ^ {0} - U ^ {\pi}\right),
$$
since  $0 \leq \left(\Pi_{j=1}^{k}\left(1 - \beta_{j}\right)\right)\left(\gamma \mathcal{P}^{\pi}\right)^{k+1}\left(U^{0} - U^{\pi}\right)$ .
Now, we prove the second rates of Theorem 1.
Proof of second rates in Theorem 1. Let  $0 < \gamma < 1$ . By Lemma 5, if  $U^0 \leq T^\pi U^0$ , then  $U^0 \leq U^\pi$ . Hence,
$$
\begin{array}{l} 0 \leq T ^ {\pi} U ^ {k} - U ^ {k} \\ \leq \sum_ {i = 1} ^ {k} \left[ \left(\beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i})\right) \left(\Pi_ {j = i + 1} ^ {k} (1 - \beta_ {j})\right) (\gamma \mathcal {P} ^ {\pi}) ^ {k - i + 1} (U ^ {0} - U ^ {\pi}) \right] - \beta_ {k} (U ^ {0} - U ^ {\pi}), \\ \end{array}
$$
by Lemma 6. Taking  $\| \cdot \|_{\infty}$ -norm both sides, we have
$$
\left\| T ^ {\pi} U ^ {k} - U ^ {k} \right\| _ {\infty} \leq \frac {\left(\gamma^ {- 1} - \gamma\right) \left(1 + \gamma - \gamma^ {k + 1}\right)}{\left(\gamma^ {k + 1}\right) ^ {- 1} - \gamma^ {k + 1}} \left\| U ^ {0} - U ^ {\pi} \right\| _ {\infty}.
$$
Otherwise, if  $U^0 \geq TU^0$ ,  $U^k \geq TU^k$  by Lemma 5. Since
$$
\begin{array}{l} 0 \geq T ^ {\pi} U ^ {k} - U ^ {k} \\ \geq \sum_ {i = 1} ^ {k} \left[ \left(\beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i})\right) \left(\Pi_ {j = i + 1} ^ {k} (1 - \beta_ {j})\right) (\gamma \mathcal {P} ^ {\pi}) ^ {k - i + 1} (U ^ {0} - U ^ {\pi}) \right] - \beta_ {k} (U ^ {0} - U ^ {\pi}), \\ \end{array}
$$
by Lemma 7, taking  $\| \cdot \|_{\infty}$ -norm both sides, we obtain same rate as before.
Lastly, Taylor series expansion for both rates at  $\gamma = 1$  is
$$
\begin{array}{l} \frac {\left(\gamma^ {- 1} - \gamma\right) \left(1 + 2 \gamma - \gamma^ {k + 1}\right)}{\left(\gamma^ {k + 1}\right) ^ {- 1} - \gamma^ {k + 1}} = \frac {2}{k + 1} - \frac {k - 1}{k + 1} (\gamma - 1) + O ((\gamma - 1) ^ {2}), \\ \frac {\left(\gamma^ {- 1} - \gamma\right) \left(1 + \gamma - \gamma^ {k + 1}\right)}{\left(\gamma^ {k + 1}\right) ^ {- 1} - \gamma^ {k + 1}} = \frac {1}{k + 1} - \frac {k}{k + 1} (\gamma - 1) + O ((\gamma - 1) ^ {2}). \\ \end{array}
$$
For the analyses of Anc-VI for Bellman optimality operator, we first prove following two lemmas.
Lemma 8. Let  $0 < \gamma \leq 1$ . If  $\gamma = 1$ , assume a fixed point  $U^{\star}$  exists. Then, if  $0 \leq \alpha \leq 1$  and  $U - (1 - \alpha)\tilde{U} - \alpha U^{\star} \leq \tilde{U}$ , there exist nonexpansive linear operator  $\mathcal{P}_H$  such that
$$
T ^ {\star} U - (1 - \alpha) T ^ {\star} \tilde {U} - \alpha T ^ {\star} U ^ {\star} \leq \gamma \mathcal {P} _ {H} \bar {U}.
$$
Lemma 9. Let  $0 < \gamma < 1$ . If  $0 \leq \alpha \leq 1$  and  $\bar{U} \leq U - (1 - \alpha)\tilde{U} - \alpha\hat{U}^{\star}$ , then there exist nonexpansive linear operator  $\hat{\mathcal{P}}_H$  such that
$$
\gamma \hat {\mathcal {P}} _ {H} (\bar {U}) \leq T ^ {\star} U - \alpha T ^ {\star} \tilde {U} - (1 - \alpha) \hat {T} ^ {\star} \hat {U} ^ {\star}.
$$
Proof of Lemma 8. First, let  $U = V$ ,  $\tilde{U} = \tilde{V}$ ,  $U^{\star} = V^{\star}$ ,  $\bar{U} = \bar{V}$ , and  $V - (1 - \alpha)\tilde{V} - \alpha V^{\star} \leq \bar{V}$ . If action space is finite,
$$
\begin{array}{l} T ^ {\star} V - (1 - \alpha) T ^ {\star} \tilde {V} - \alpha T ^ {\star} V ^ {\star} \leq T ^ {\pi} V - (1 - \alpha) T ^ {\pi} \tilde {V} - \alpha T ^ {\pi} V ^ {\star} \\ = \gamma \mathcal {P} ^ {\pi} \left(V - (1 - \alpha) \tilde {V} - \alpha V ^ {\star}\right) \\ \leq \gamma P ^ {\pi} \bar {V} \\ \end{array}
$$
where  $\pi$  is the greedy policy satisfying  $T^{\pi}V = T^{\star}V$ , first inequality is from  $T^{\pi}\tilde{V}\leq T^{\star}\tilde{V}$  and  $T^{\pi}V^{\star}\leq T^{\star}V^{\star}$ , and second inequality comes from Lemma 1. Thus, we can conclude  $\mathcal{P}_H = \mathcal{P}^\pi$ .
Otherwise, if action space is infinite, define  $\mathcal{P}(c\bar{V}) = c\sup_{s\in \mathcal{S}}\bar{V} (s)$  for  $c\in \mathbb{R}$  and previously given  $\bar{V}$ . Let  $M$  be linear space spanned by  $\bar{V}$  with  $\| \cdot \|_{\infty}$ -norm. Then,  $\mathcal{P}$  is linear functional on  $M$  and  $\| \mathcal{P}\|_{\mathrm{op}}\leq 1$  since  $\frac{|c\sup_{s\in\mathcal{S}}\bar{V}(s)|}{\|c\bar{V}\|_{\infty}}\leq 1$ . Due to Hahn-Banach extension Theorem, there exist linear functional  $\mathcal{P}_h\colon \mathcal{F}(\mathcal{S})\to \mathbb{R}$  with  $\mathcal{P}_h(\bar{V}) = \sup_{s\in \mathcal{S}}\bar{V} (s)$  and  $\| \mathcal{P}_h\|_{\mathrm{op}}\leq 1$ . Furthermore, we can define  $\mathcal{P}_H\colon \mathcal{F}(\mathcal{S})\to \mathcal{F}(\mathcal{S})$  such that  $\mathcal{P}_HV(s) = \mathcal{P}_h(V)$  for all  $s\in S$ . Then, since  $\| \mathcal{P}_H(V)\|_{\infty} = |\mathcal{P}_h(V)|\leq \| \mathcal{P}_h\|_{\mathrm{op}}\leq 1$  for  $\| V\|_{\infty}\leq 1$ , we have  $\| \mathcal{P}_H\|_{\infty}\leq 1$ . Therefore,  $\mathcal{P}_H$  is
nonexpansive linear operator in  $\| \cdot \|_{\infty}$ -norm. Then,
$$
\begin{array}{l} T ^ {\star} V (s) - (1 - \alpha) T ^ {\star} \tilde {V} (s) - \alpha T ^ {\star} V ^ {\star} (s) \\ = \sup _ {a \in \mathcal {A}} \left\{r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} [ V (s ^ {\prime}) ] \right\} - \sup _ {a \in \mathcal {A}} \left\{(1 - \alpha) r (s, a) + (1 - \alpha) \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} [ \tilde {V} (s ^ {\prime}) ] \right\} \\ - \sup  _ {a \in \mathcal {A}} \left\{\alpha r (s, a) + \alpha \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ V ^ {\star} \left(s ^ {\prime}\right) \right] \right\} \\ \leq \sup  _ {a \in \mathcal {A}} \left\{r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ V \left(s ^ {\prime}\right) \right] - (1 - \alpha) r (s, a) - (1 - \alpha) \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ \tilde {V} \left(s ^ {\prime}\right) \right] \right\} \\ - \sup  _ {a \in \mathcal {A}} \left\{\alpha r (s, a) + \alpha \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ V ^ {\star} \left(s ^ {\prime}\right) \right] \right\} \\ \leq \gamma \sup  _ {a \in \mathcal {A}} \left\{\mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ V (s ^ {\prime}) - (1 - \alpha) \tilde {V} (s ^ {\prime}) - \alpha V ^ {\star} (s ^ {\prime}) \right] \right\} \\ \leq \gamma \sup  _ {s ^ {\prime} \in \mathcal {S}} \left\{V (s ^ {\prime}) - (1 - \alpha) \tilde {V} (s ^ {\prime}) - \alpha V ^ {\star} (s ^ {\prime}) \right\} \\ \leq \gamma \sup_{s^{\prime}\in \mathcal{S}}\bar{V} (s^{\prime}). \\ \end{array}
$$
for all  $s\in S$  . Therefore, we have
$$
T ^ {\star} V - (1 - \alpha) T ^ {\star} \tilde {V} - \alpha T ^ {\star} V ^ {\star} \leq \gamma \mathcal {P} _ {H} (\bar {V}).
$$
Similarly, let  $U = Q$ ,  $\tilde{U} = \tilde{Q}$ ,  $U^{\star} = Q^{\star}$ ,  $\bar{U} = \bar{Q}$ , and  $Q - (1 - \alpha)\tilde{Q} - \alpha Q^{\star} \leq \bar{Q}$ .
If action space is finite,
$$
\begin{array}{l} T ^ {\star} Q - (1 - \alpha) T ^ {\star} \tilde {Q} - \alpha T ^ {\star} Q ^ {\star} \leq \gamma \mathcal {P} ^ {\pi} \left(Q - (1 - \alpha) \tilde {Q} - \alpha Q ^ {\star}\right) \\ \leq \gamma P ^ {\pi} \bar {Q} \\ \end{array}
$$
where  $\pi$  is the greedy policy satisfying  $T^{\pi}Q = T^{\star}Q$ , first inequality is from  $T^{\pi}\tilde{Q}\leq T^{\star}\tilde{Q}$  and  $T^{\pi}Q^{\star}\leq T^{\star}Q^{\star}$ , and second inequality comes from Lemma 1. Then, we can conclude  $\mathcal{P}_H = \mathcal{P}^\pi$ .
Otherwise, if action space is infinite, define  $\mathcal{P}(c\bar{Q}) = c\sup_{(s',a')\in S\times \mathcal{A}}\bar{Q} (s',a')$  for  $c\in \mathbb{R}$  and previously given  $\bar{Q}$ . Let  $M$  be linear space spanned by  $\bar{Q}$  with  $\| \cdot \|_{\infty}$ -norm. Then,  $\mathcal{P}$  is linear functional on  $M$  and  $\| \mathcal{P}\|_{\mathrm{op}}\leq 1$ . Due to Hahn-Banach extension Theorem, there exist linear functional  $\mathcal{P}_h\colon \mathcal{F}(\mathcal{S}\times \mathcal{A})\to \mathbb{R}$  with  $\mathcal{P}_h(\bar{Q}) = \sup_{(s',a')\in S\times \mathcal{A}}\bar{Q} (s',a')$  and  $\| \mathcal{P}_h\|_{\mathrm{op}}\leq 1$ . Furthermore, we can define  $\mathcal{P}_H\colon \mathcal{F}(\mathcal{S}\times \mathcal{A})\to \mathcal{F}(\mathcal{S}\times \mathcal{A})$  such that  $\mathcal{P}_HQ(s,a) = \mathcal{P}_h(Q)$  for all  $(s,a)\in S\times \mathcal{A}$  and  $\| P_H\|_{\infty}\leq 1$ . Therefore,  $\mathcal{P}_H$  is nonexpansive linear operator in  $\| \cdot \|_{\infty}$ -norm. Then,
$$
\begin{array}{l} = r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ \sup  _ {a ^ {\prime} \in \mathcal {A}} Q \left(s ^ {\prime}, a ^ {\prime}\right) \right] - (1 - \alpha) r (s, a) - (1 - \alpha) \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ \sup  _ {a ^ {\prime} \in \mathcal {A}} \tilde {Q} \left(s ^ {\prime}, a ^ {\prime}\right) \right] \\ - \alpha r (s, a) - \alpha \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ \sup  _ {a ^ {\prime} \in \mathcal {A}} Q ^ {\star} \left(s ^ {\prime}, a ^ {\prime}\right) \right] \\ \end{array}
$$