SlowGuess/NeurIPS_2023_OCR huggingface dataset:
for  $i = 1,\dots ,n$ . Note that  $\hat{\mathcal{P}}_i$  is nonnegative matrix since  $\mathcal{P}^{\hat{\pi}_i}$  is nonnegative matrix. Then, we have
$$
\hat {\mathcal {P}} _ {i} (V - (1 - \alpha) \tilde {V} - \alpha \hat {V} ^ {\star}) \leq T _ {i} ^ {\star} V - (1 - \alpha) T _ {i} ^ {\star} \tilde {V} - \alpha T _ {i} ^ {\star} \hat {V} ^ {\star}.
$$
By induction, there exist a sequence of matrices  $\{\hat{P}_i\}_{i = 1,\dots,n}$  satisfying
$$
\hat {\mathcal {P}} _ {n} \dots \hat {\mathcal {P}} _ {1} (V - (1 - \alpha) \tilde {V} - \alpha \hat {V} ^ {\star}) \leq T _ {G S} ^ {\star} V - (1 - \alpha) T _ {G S} ^ {\star} \tilde {V} - \alpha \hat {T} _ {G S} ^ {\star} \hat {V} ^ {\star},
$$
and denote  $\hat{P}_{GS}$  as  $\hat{\mathcal{P}}_n\cdots\hat{\mathcal{P}}_1$ . With same argument in proof of Lemma 19,  $\hat{P}_{GS}$  is  $\gamma$ -contractive nonnegative matrix.
If  $U = Q$ , with similar argument, let  $\hat{\pi}_i(\cdot |s) = \mathrm{argmin}_{a\in \mathcal{A}}\{Q(s,a) - (1 - \alpha)\tilde{Q} (s,a) - \alpha \hat{Q}^\star (s,a)\}$  and define matrix  $\hat{\mathcal{P}}_i$  as
$$
\mathcal {P} _ {i} (Q) = \left(U _ {1}, \dots , Q _ {i - 1}, \left(\gamma \mathcal {P} ^ {\hat {\pi} _ {i}} (Q)\right) _ {i}, Q _ {i + 1}, \dots , Q _ {n}\right).
$$
Denote  $\hat{P}_{GS}$  as  $\hat{\mathcal{P}}_n\cdot \cdot \cdot \hat{\mathcal{P}}_1$ . Then, with same argument in proof of Lemma 19,  $\hat{P}_{GS}$  is  $\gamma$ -contractive nonnegative matrix satisfying
$$
\hat {\mathcal {P}} _ {G S} (Q - (1 - \alpha) \tilde {Q} - \alpha \hat {Q} ^ {\star}) \leq T _ {G S} ^ {\star} Q - (1 - \alpha) T _ {G S} ^ {\star} \tilde {Q} - \alpha \hat {T} _ {G S} ^ {\star} \hat {Q} ^ {\star}.
$$
Next, we prove following key lemma.
Lemma 21. Let  $0 < \gamma < 1$ . For the iterates  $\{U^k\}_{k = 0,1,\ldots}$  of (GS-Anc-VI), there exist  $\gamma$ -contractive nonnegative matrices  $\{\mathcal{P}_{GS}^l\}_{l = 0,1,\ldots,k}$  and  $\{\hat{\mathcal{P}}_{GS}^l\}_{l = 0,1,\ldots,k}$  such that
$$
\begin{array}{l} T _ {G S} ^ {\star} U ^ {k} - U ^ {k} \leq \sum_ {i = 1} ^ {k} \left[ \left(\beta_ {i} - \beta_ {i - 1} \left(1 - \beta_ {i}\right)\right) \left(\Pi_ {j = i + 1} ^ {k} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k} ^ {i} \mathcal {P} _ {G S} ^ {l}\right) \left(U ^ {0} - U ^ {\star}\right) \right] \\ - \beta_ {k} (U ^ {0} - U ^ {\star}) + \left(\Pi_ {j = 1} ^ {k} (1 - \beta_ {j})\right) \left(\Pi_ {l = k} ^ {0} \mathcal {P} _ {G S} ^ {l}\right) (U ^ {0} - U ^ {\star}), \\ \end{array}
$$
$$
\begin{array}{l} T _ {G S} ^ {\star} U ^ {k} - U ^ {k} \geq \sum_ {i = 1} ^ {k} \left[ \left(\beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i})\right) \left(\Pi_ {j = i + 1} ^ {k} (1 - \beta_ {j})\right) \left(\Pi_ {l = k} ^ {i} \hat {\mathcal {P}} _ {G S} ^ {l}\right) \left(U ^ {0} - \hat {U} ^ {\star}\right) \right] \\ - \beta_ {k} (U ^ {0} - \hat {U} ^ {\star}) + \left(\Pi_ {j = 1} ^ {k} (1 - \beta_ {j})\right) \left(\Pi_ {l = k} ^ {0} \hat {\mathcal {P}} _ {G S} ^ {l}\right) (U ^ {0} - \hat {U} ^ {\star}), \\ \end{array}
$$
Proof of Lemma 21. First, we prove first inequality in Lemma 21 by induction.
If  $k = 0$
$$
\begin{array}{l} T _ {G S} ^ {\star} U ^ {0} - U ^ {0} = T _ {G S} ^ {\star} U ^ {0} - U ^ {\star} - (U ^ {0} - U ^ {\star}) \\ = T _ {G S} ^ {\star} U ^ {0} - T _ {G S} ^ {\star} U ^ {\star} - \left(U ^ {0} - U ^ {\star}\right) \\ \leq \mathcal {P} _ {G S} ^ {0} (U ^ {0} - U ^ {\star}) - (U ^ {0} - U ^ {\star}). \\ \end{array}
$$
where inequality comes from Lemma 19 with  $\alpha = 1, U = U^0$ .
$$
\begin{array}{l} T _ {G S} ^ {\star} U ^ {k} - U ^ {k} \\ = T _ {G S} ^ {\star} U ^ {k} - \left(1 - \beta_ {k}\right) T _ {G S} ^ {\star} U ^ {k - 1} - \beta_ {k} T _ {G S} ^ {\star} U ^ {\star} - \beta_ {k} \left(U ^ {0} - U ^ {\star}\right) \\ \leq \mathcal {P} _ {G S} ^ {k} \left(U ^ {k} - \left(1 - \beta_ {k}\right) U ^ {k - 1} - \beta_ {k} U ^ {\star}\right) - \beta_ {k} \left(U ^ {0} - U ^ {\star}\right) \\ = \mathcal {P} _ {G S} ^ {k} \left(\beta_ {k} \left(U ^ {0} - U ^ {\star}\right) + \left(1 - \beta_ {k}\right) \left(T _ {G S} ^ {\star} U ^ {k - 1} - U ^ {k - 1}\right)\right) - \beta_ {k} \left(U ^ {0} - U ^ {\star}\right) \\ \leq \left(1 - \beta_ {k}\right) \mathcal {P} _ {G S} ^ {k} \Bigg (\sum_ {i = 1} ^ {k - 1} \left[ \left(\beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i})\right) \left(\Pi_ {j = i + 1} ^ {k - 1} (1 - \beta_ {j})\right) \left(\Pi_ {l = k - 1} ^ {i} \mathcal {P} _ {G S} ^ {l}\right) (U ^ {0} - U ^ {\star}) \right] \\ \left. - \beta_ {k - 1} \left(U ^ {0} - U ^ {\star}\right) + \left(\Pi_ {j = 1} ^ {k - 1} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k - 1} ^ {0} \mathcal {P} _ {G S} ^ {l}\right) \left(U ^ {0} - U ^ {\star}\right)\right) \\ + \beta_ {k} \mathcal {P} _ {G S} ^ {k} (U ^ {0} - U ^ {\star}) - \beta_ {k} (U ^ {0} - U ^ {\star}) \\ = \sum_ {i = 1} ^ {k} \left[ \left(\beta_ {i} - \beta_ {i - 1} \left(1 - \beta_ {i}\right)\right) \left(\Pi_ {j = i + 1} ^ {k} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k} ^ {i} \mathcal {P} _ {G S} ^ {l}\right) \left(U ^ {0} - U ^ {\star}\right) \right] \\ - \beta_ {k} \left(U ^ {0} - U ^ {\star}\right) + \left(\Pi_ {j = 1} ^ {k} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k} ^ {0} \mathcal {P} _ {G S} ^ {l}\right) \left(U ^ {0} - U ^ {\star}\right) \\ \end{array}
$$
where the first inequality comes from Lemma 19 with  $\alpha = \beta_{k}, U = U^{k}, \tilde{U} = U^{k - 1}$ , and second inequality comes from nonnegativeness of  $\mathcal{P}_{GS}^k$ .
First, we prove second inequality in Lemma 21 by induction.
$$
\begin{array}{l} T _ {G S} ^ {\star} U ^ {0} - U ^ {0} = T _ {G S} ^ {\star} U ^ {0} - \hat {U} ^ {\star} - (U ^ {0} - \hat {U} ^ {\star}) \\ = T _ {G S} ^ {\star} U ^ {0} - \hat {T} _ {G S} ^ {\star} \hat {U} ^ {\star} - (U ^ {0} - \hat {U} ^ {\star}) \\ \geq \hat {\mathcal {P}} _ {G S} ^ {0} (U ^ {0} - \hat {U} ^ {\star}) - (U ^ {0} - \hat {U} ^ {\star}), \\ \end{array}
$$
where inequality comes from Lemma 20 with  $\alpha = 1, U = U^0$ .
$$
\begin{array}{l} T _ {G S} ^ {\star} U ^ {k} - U ^ {k} \\ = T _ {G S} ^ {\star} U ^ {k} - \left(1 - \beta_ {k}\right) T _ {G S} ^ {\star} U ^ {k - 1} - \beta_ {k} \hat {T} _ {G S} ^ {\star} \hat {U} ^ {\star} - \beta_ {k} \left(U ^ {0} - \hat {U} ^ {\star}\right) \\ \geq \hat {\mathcal {P}} _ {G S} ^ {k} \left(U ^ {k} - \left(1 - \beta_ {k}\right) U ^ {k - 1} - \beta_ {k} \hat {U} ^ {\star}\right) - \beta_ {k} \left(U ^ {0} - \hat {U} ^ {\star}\right) \\ = \hat {\mathcal {P}} _ {G S} ^ {k} \left(\beta_ {k} \left(U ^ {0} - \hat {U} ^ {\star}\right) + \left(1 - \beta_ {k}\right) \left(T _ {G S} ^ {\star} U ^ {k - 1} - U ^ {k - 1}\right)\right) - \beta_ {k} \left(U ^ {0} - \hat {U} ^ {\star}\right) \\ \geq \left(1 - \beta_ {k}\right) \hat {\mathcal {P}} _ {G S} ^ {k} \left(\sum_ {i = 1} ^ {k - 1} \left[ \left(\beta_ {i} - \beta_ {i - 1} \left(1 - \beta_ {i}\right)\right) \left(\Pi_ {j = i + 1} ^ {k - 1} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k - 1} ^ {i} \hat {\mathcal {P}} _ {G S} ^ {l}\right) \left(U ^ {0} - \hat {U} ^ {\star}\right) \right] \right. \\ \left. - \beta_ {k - 1} \left(U ^ {0} - \hat {U} ^ {\star}\right) + \left(\Pi_ {j = 1} ^ {k - 1} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k - 1} ^ {0} \hat {\mathcal {P}} _ {G S} ^ {l}\right) \left(U ^ {0} - \hat {U} ^ {\star}\right)\right) \\ + \beta_ {k} \hat {\mathcal {P}} _ {G S} ^ {k} (U ^ {0} - \hat {U} ^ {\star}) - \beta_ {k} (U ^ {0} - \hat {U} ^ {\star}) \\ = \sum_ {i = 1} ^ {k} \left[ \left(\beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i})\right) \left(\Pi_ {j = i + 1} ^ {k} (1 - \beta_ {j})\right) \left(\Pi_ {l = k} ^ {i} \hat {\mathcal {P}} _ {G S} ^ {l}\right) \left(U ^ {0} - \hat {U} ^ {\star}\right) \right] \\ - \beta_ {k} (U ^ {0} - \hat {U} ^ {\star}) + \left(\Pi_ {j = 1} ^ {k} (1 - \beta_ {j})\right) \left(\Pi_ {l = k} ^ {0} \hat {\mathcal {P}} _ {G S} ^ {l}\right) (U ^ {0} - \hat {U} ^ {\star}) \\ \end{array}
$$
where the first inequality comes from Lemma 20 with  $\alpha = \beta_{k}, U = U^{k}, \tilde{U} = U^{k - 1}$ , and nonnegativeness of  $\hat{\mathcal{P}}_{GS}^{k}$ .
Now, we prove the first rate in Theorem 7.
Proof of first rate in Theorem 7. Since  $B_{1} \leq A \leq B_{2}$  implies  $\|A\|_{\infty} \leq \sup \{\|B_{1}\|_{\infty}, \|B_{2}\|_{\infty}\}$  for  $A, B \in \mathcal{F}(\mathcal{X})$ , if we take  $\|\cdot\|_{\infty}$  right side of first inequality in Lemma 21, we have
$$
\begin{array}{l} \sum_ {i = 1} ^ {k} \left| \beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i}) \right| \left(\Pi_ {j = i + 1} ^ {k} (1 - \beta_ {j})\right) \left\| \left(\Pi_ {l = k} ^ {i} \mathcal {P} _ {G S} ^ {l}\right) (U ^ {0} - U ^ {\star}) \right\| _ {\infty} \\ + \beta_ {k} \| U ^ {0} - U ^ {\pi} \| _ {\infty} + \left(\Pi_ {j = 1} ^ {k} (1 - \beta_ {j})\right) \| \left(\Pi_ {l = k} ^ {0} \mathcal {P} _ {G S} ^ {l}\right) (U ^ {0} - U ^ {\star}) \| _ {\infty} \\ \leq \left(\sum_ {i = 1} ^ {k} \gamma^ {k - i + 1} \left| \beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i}) \right| \left(\Pi_ {j = i + 1} ^ {k} (1 - \beta_ {j})\right) + \beta_ {k} + \gamma^ {k + 1} \Pi_ {j = 1} ^ {k} (1 - \beta_ {j})\right) \\ \left\| U ^ {0} - U ^ {\star} \right\| _ {\infty} \\ = \frac {\left(\gamma^ {- 1} - \gamma\right) \left(1 + 2 \gamma - \gamma^ {k + 1}\right)}{\left(\gamma^ {k + 1}\right) ^ {- 1} - \gamma^ {k + 1}} \left\| U ^ {0} - U ^ {\star} \right\| _ {\infty}, \\ \end{array}
$$
where the first inequality comes from triangular inequality, second inequality is from  $\gamma$ -contraction of  $\mathcal{P}_{GS}^l$ , and last equality comes from calculations. If we take  $\| \cdot \|_{\infty}$  right side of second inequality in Lemma 21, we have
$$
\begin{array}{l} \sum_ {i = 1} ^ {k} \left| \beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i}) \right| \left(\Pi_ {j = i + 1} ^ {k} (1 - \beta_ {j})\right) \left\| \left(\Pi_ {l = k} ^ {i} \hat {\mathcal {P}} _ {G S} ^ {l}\right) (U ^ {0} - \hat {U} ^ {\star}) \right\| _ {\infty} \\ + \beta_ {k} \left\| U ^ {0} - U ^ {\pi} \right\| _ {\infty} + \left(\Pi_ {j = 1} ^ {k} (1 - \beta_ {j})\right) \left\| \left(\Pi_ {l = k} ^ {0} \hat {\mathcal {P}} _ {G S} ^ {l}\right) (U ^ {0} - \hat {U} ^ {\star}) \right\| _ {\infty} \\ \leq \left(\sum_ {i = 1} ^ {k} \gamma^ {k - i + 1} \left| \beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i}) \right| \left(\Pi_ {j = i + 1} ^ {k} (1 - \beta_ {j})\right) + \beta_ {k} + \gamma^ {k + 1} \Pi_ {j = 1} ^ {k} (1 - \beta_ {j})\right) \\ = \frac {\left(\gamma^ {- 1} - \gamma\right) \left(1 + 2 \gamma - \gamma^ {k + 1}\right)}{\left(\gamma^ {k + 1}\right) ^ {- 1} - \gamma^ {k + 1}} \left\| U ^ {0} - \hat {U} ^ {\star} \right\| _ {\infty}, \\ \end{array}
$$
where the first inequality comes from triangular inequality, second inequality is from  $\gamma$ -contraction of  $\hat{\mathcal{P}}_{GS}^{l}$ , and last equality comes from calculations. Therefore, we conclude
$$
\left\| T _ {G S} ^ {\star} U ^ {k} - U ^ {k} \right\| _ {\infty} \leq \frac {\left(\gamma^ {- 1} - \gamma\right) \left(1 + 2 \gamma - \gamma^ {k + 1}\right)}{\left(\gamma^ {k + 1}\right) ^ {- 1} - \gamma^ {k + 1}} \max  \left\{\left\| U ^ {0} - U ^ {\star} \right\| _ {\infty}, \left\| U ^ {0} - \hat {U} ^ {\star} \right\| _ {\infty} \right\}.
$$
For the second rates of Theorem 7, we introduce following lemma.
Lemma 22. Let  $0 < \gamma < 1$ . For the iterates  $\{U^k\}_{k = 0,1,\ldots}$  of (GS-Anc-VI), if  $U^0 \leq T_{GS}^\star U^0$ , then  $U^{k - 1} \leq U^k \leq T_{GS}^\star U^{k - 1} \leq T_{GS}^\star U^k \leq U^\star$  for  $1 \leq k$ . Also, if  $U^0 \geq T_{GS}^\star U^0$ , then  $U^{k - 1} \geq U^k \geq T_{GS}^\star U^{k - 1} \geq T_{GS}^\star U^k \geq U^\star$  for  $1 \leq k$ .
Proof. By Fact 3,  $\lim_{m\to \infty}T_{GS}^{\star}U = U^{\star}$ . By definition, if  $U\leq \tilde{U}$ ,  $T_{i}^{\star}U\leq T_{i}^{\star}\tilde{U}$  for any  $1\leq i\leq n$  and this implies that if  $U\leq \tilde{U}$ , then  $T_{GS}^{\star}U\leq T_{GS}^{\star}\tilde{U}$ . Hence, with same argument in proof of Lemma 5, we can obtain desired results.
Now, we prove the second rates in Theorem 7.
Proof of second rates in Theorem 7. If  $U^0 \leq T_{GS}^\star U^0$ , then  $U^0 - U^\star \leq 0$  and  $U^k \leq T_{GS}^\star U^k$  by Lemma 22. Hence, by Lemma 21, we get
$$
\begin{array}{l} 0 \leq T _ {G S} ^ {\star} U ^ {k} - U ^ {k} \\ = \sum_ {i = 1} ^ {k} \left[ \left(\beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i})\right) \left(\Pi_ {j = i + 1} ^ {k} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k} ^ {i} \mathcal {P} _ {G S} ^ {l}\right) \left(U ^ {0} - U ^ {\star}\right) \right] \\ - \beta_ {k} (U ^ {0} - U ^ {\pi}) + \left(\Pi_ {j = 1} ^ {k} (1 - \beta_ {j})\right) \left(\Pi_ {l = k} ^ {0} \mathcal {P} _ {G S} ^ {l}\right) (U ^ {0} - U ^ {\star}) \\ \leq \sum_ {i = 1} ^ {k} \left[ \left(\beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i})\right) \left(\Pi_ {j = i + 1} ^ {k} (1 - \beta_ {j})\right) \left(\Pi_ {l = k} ^ {i} \mathcal {P} _ {G S} ^ {l}\right) (U ^ {0} - U ^ {\star}) \right] - \beta_ {k} (U ^ {0} - U ^ {\star}), \\ \end{array}
$$
where the second inequality follows from  $\left(\Pi_{j=1}^{k}(1 - \beta_{j})\right)\left(\Pi_{l=k}^{i}\mathcal{P}_{GS}^{l}\right)(U^{0} - U^{\star}) \leq 0$ . Taking  $\|\cdot\|_{\infty}$ -norm both sides, we have
$$
\left\| T _ {G S} ^ {\star} U ^ {k} - U ^ {k} \right\| _ {\infty} \leq \frac {\left(\gamma^ {- 1} - \gamma\right) \left(1 + \gamma - \gamma^ {k + 1}\right)}{\left(\gamma^ {k + 1}\right) ^ {- 1} - \gamma^ {k + 1}} \left\| U ^ {0} - U ^ {\star} \right\| _ {\infty}.
$$
Otherwise, if  $U^0 \geq T_{GS}^\star U^0$ ,  $U^k \geq T_{GS}^\star U^k$  and  $U^0 \geq U^\star \geq \hat{U}^\star$  by Lemma 22 and 3. Thus, by Lemma 21, we get
$$
\begin{array}{l} 0 \geq T _ {G S} ^ {\star} U ^ {k} - U ^ {k} \\ \geq \sum_ {i = 1} ^ {k} \left[ \left(\beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i})\right) \left(\Pi_ {j = i + 1} ^ {k} (1 - \beta_ {j})\right) \left(\Pi_ {l = k} ^ {i} \hat {\mathcal {P}} _ {G S} ^ {l}\right) (U ^ {0} - \hat {U} ^ {\star}) \right] - \beta_ {k} (U ^ {0} - \hat {U} ^ {\star}), \\ \end{array}
$$
where the second inequality follows from  $0 \leq \left(\Pi_{j=1}^{k}(1 - \beta_{j})\right)\left(\Pi_{l=k}^{0}\hat{\mathcal{P}}_{GS}^{l}\right)(U^{0} - \hat{U}^{\star})$ . Taking  $\|\cdot\|_{\infty}$ -norm both sides, we have
$$
\left\| T _ {G S} ^ {\star} U ^ {k} - U ^ {k} \right\| _ {\infty} \leq \frac {\left(\gamma^ {- 1} - \gamma\right) \left(1 + \gamma - \gamma^ {k + 1}\right)}{\left(\gamma^ {k + 1}\right) ^ {- 1} - \gamma^ {k + 1}} \left\| U ^ {0} - \hat {U} ^ {\star} \right\| _ {\infty}.
$$
G Broader Impacts
Our work focuses on the theoretical aspects of reinforcement learning. There are no negative social impacts that we anticipate from our theoretical results.
H Limitations
Our analysis concerns value iteration. While value iteration is of theoretical interest, the analysis of value iteration is not sufficient to understand modern deep reinforcement learning practices.