SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} T ^ {\star} U ^ {1} - U ^ {1} = T ^ {\star} U ^ {1} - \left(1 - \beta_ {1}\right) T ^ {\star} U ^ {0} - \beta_ {1} \hat {U} ^ {\star} - \beta_ {1} \left(U ^ {0} - \hat {U} ^ {\star}\right) - \left(1 - \beta_ {1}\right) \epsilon^ {0} \\ \geq \gamma \hat {\mathcal {P}} ^ {1} ((1 - \beta_ {1}) \epsilon^ {0} + (1 - \beta_ {1}) \gamma \hat {\mathcal {P}} ^ {0} (U ^ {0} - \hat {U} ^ {\star}) + (2 \beta_ {1} - 1) (U ^ {0} - \hat {U} ^ {\star})) - \beta_ {1} (U ^ {0} - \hat {U} ^ {\star}) \\ - (1 - \beta_ {1}) \epsilon^ {0} \\ = (1 - \beta_ {1}) \gamma \hat {\mathcal {P}} ^ {1} \gamma \hat {\mathcal {P}} ^ {0} (U ^ {0} - \hat {U} ^ {\star}) + \gamma \hat {\mathcal {P}} ^ {1} (2 \beta_ {1} - 1) (U ^ {0} - \hat {U} ^ {\star}) - \beta_ {1} (U ^ {0} - \hat {U} ^ {\star}) \\ - (I - \gamma \hat {\mathcal {P}} ^ {1}) (1 - \beta_ {1}) \epsilon^ {0}. \\ \end{array}
$$
where inequality comes from Lemma 9 with  $\alpha = \beta_{1}, U = U^{1}, \tilde{U} = U^{0}$ , and previously defined  $\bar{U}$ . By induction,
$$
\begin{array}{l} U ^ {k} - (1 - \beta_ {k}) U ^ {k - 1} - \beta_ {k} \hat {U} ^ {\star} \\ = \beta_ {k} \left(U ^ {0} - \hat {U} ^ {\star}\right) + (1 - \beta_ {k}) \left(T ^ {\star} U ^ {k - 1} - U ^ {k - 1}\right) + (1 - \beta_ {k}) \epsilon^ {k - 1} \\ \geq \left(1 - \beta_ {k}\right) \sum_ {i = 1} ^ {k - 1} \left[ \left(\beta_ {i} - \beta_ {i - 1} \left(1 - \beta_ {i}\right)\right) \left(\Pi_ {j = i + 1} ^ {k - 1} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k - 1} ^ {i} \gamma \hat {\mathcal {P}} ^ {l}\right) \left(U ^ {0} - \hat {U} ^ {\star}\right) \right] \\ - \left(1 - \beta_ {k}\right) \beta_ {k - 1} \left(U ^ {0} - \hat {U} ^ {\star}\right) + \left(1 - \beta_ {k}\right) \left(\Pi_ {j = 1} ^ {k - 1} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k - 1} ^ {0} \gamma \hat {\mathcal {P}} ^ {l}\right) \left(U ^ {0} - \hat {U} ^ {\star}\right) \\ + \beta_ {k} (U ^ {0} - \hat {U} ^ {\star}) - (1 - \beta_ {k}) \sum_ {i = 1} ^ {k - 1} \Pi_ {j = i} ^ {k - 1} (1 - \beta_ {j}) \Pi_ {l = k - 1} ^ {i + 1} \gamma \hat {\mathcal {P}} ^ {l} (I - \gamma \hat {\mathcal {P}} ^ {i}) \epsilon^ {i - 1} + (1 - \beta_ {k}) \epsilon^ {k - 1}, \\ \end{array}
$$
$$
\begin{array}{l} T ^ {\star} U ^ {k} - U ^ {k} \\ = T ^ {\star} U ^ {k} - \left(1 - \beta_ {k}\right) T ^ {\star} U ^ {k - 1} - \beta_ {k} U ^ {0} - \left(1 - \beta_ {k}\right) \epsilon^ {k - 1} \\ = T ^ {\star} U ^ {k} - \left(1 - \beta_ {k}\right) T ^ {\star} U ^ {k - 1} - \beta_ {k} T ^ {\star} \hat {U} ^ {\star} - \beta_ {k} \left(U ^ {0} - \hat {U} ^ {\star}\right) - \left(1 - \beta_ {k}\right) \epsilon^ {k - 1} \\ \geq \gamma \hat {\mathcal {P}} ^ {k} \left(\left(1 - \beta_ {k}\right) \sum_ {i = 1} ^ {k - 1} \left[ \left(\beta_ {i} - \beta_ {i - 1} \left(1 - \beta_ {i}\right)\right) \left(\Pi_ {j = i + 1} ^ {k - 1} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k - 1} ^ {i} \gamma \hat {\mathcal {P}} ^ {l}\right) \left(U ^ {0} - \hat {U} ^ {\star}\right) \right] \right. \\ - \left(1 - \beta_ {k}\right) \beta_ {k - 1} \left(U ^ {0} - \hat {U} ^ {\star}\right) + \left(1 - \beta_ {k}\right) \left(\Pi_ {j = 1} ^ {k - 1} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k - 1} ^ {0} \gamma \hat {\mathcal {P}} ^ {l}\right) \left(U ^ {0} - \hat {U} ^ {\star}\right) \\ + \beta_ {k} (U ^ {0} - \hat {U} ^ {\star}) - (1 - \beta_ {k}) \sum_ {i = 1} ^ {k - 1} \Pi_ {j = i} ^ {k - 1} (1 - \beta_ {j}) \Pi_ {l = k - 1} ^ {i + 1} \gamma \hat {\mathcal {P}} ^ {l} \left(I - \gamma \hat {\mathcal {P}} ^ {i}\right) \epsilon^ {i - 1} + (1 - \beta_ {k}) \epsilon^ {k - 1} \\ - \beta_ {k} (U ^ {0} - \hat {U} ^ {\star}) - (1 - \beta_ {k}) \epsilon^ {k - 1} \\ = \sum_ {i = 1} ^ {k} \left[ \left(\beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i})\right) \left(\Pi_ {j = i + 1} ^ {k} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k} ^ {i} \gamma \hat {\mathcal {P}} ^ {l}\right) \left(U ^ {0} - \hat {U} ^ {\star}\right) \right] - \beta_ {k} \left(U ^ {0} - \hat {U} ^ {\star}\right) \\ + \Pi_ {j = 1} ^ {k} (1 - \beta_ {j}) \Pi_ {l = k} ^ {0} \gamma \hat {\mathcal {P}} ^ {l} (U ^ {0} - \hat {U} ^ {\star}) - \sum_ {i = 1} ^ {k} \Pi_ {j = i} ^ {k} (1 - \beta_ {j}) \Pi_ {l = k} ^ {i + 1} \gamma \hat {\mathcal {P}} ^ {l} \left(I - \gamma \hat {\mathcal {P}} ^ {i}\right) \epsilon^ {i - 1}, \\ \end{array}
$$
where inequality comes from Lemma 9 with  $\alpha = \beta_{k}, U = U^{k}, \tilde{U} = U^{k - 1}$ , and previously defined  $\bar{U}$
Now, we prove the first rate in Theorem 6.
Proof of first rate in Theorem 6. Since  $B_{1} \leq A \leq B_{2}$  implies  $\|A\|_{\infty} \leq \sup \{\|B_{1}\|_{\infty}, \|B_{2}\|_{\infty}\}$  for  $A, B \in \mathcal{F}(\mathcal{X})$ , if we take  $\|\cdot\|_{\infty}$  right side of first inequality in Lemma 17, we have
$$
\begin{array}{l} \frac {\left(\gamma^ {- 1} - \gamma\right) \left(1 + 2 \gamma - \gamma^ {k + 1}\right)}{\left(\gamma^ {k + 1}\right) ^ {- 1} - \gamma^ {k + 1}} \left\| U ^ {0} - U ^ {\star} \right\| _ {\infty} + (1 + \gamma) \sum_ {i = 1} ^ {k} \left(\Pi_ {j = i} ^ {k} (1 - \beta_ {j})\right) \gamma^ {k - i} \left\| \epsilon^ {i - 1} \right\| _ {\infty} \\ \leq \frac {\left(\gamma^ {- 1} - \gamma\right) \left(1 + 2 \gamma - \gamma^ {k + 1}\right)}{\left(\gamma^ {k + 1}\right) ^ {- 1} - \gamma^ {k + 1}} \left\| U ^ {0} - U ^ {\star} \right\| _ {\infty} + \frac {1 + \gamma}{1 + \gamma^ {k + 1}} \frac {1 - \gamma^ {k}}{1 - \gamma} \max  _ {0 \leq i \leq k - 1} \left\| \epsilon^ {i} \right\| _ {\infty}. \\ \end{array}
$$
If we apply second inequality of Lemma 17 and take  $\| \cdot \|_{\infty}$ -norm right side, we have
$$
\frac {\left(\gamma^ {- 1} - \gamma\right) \left(1 + 2 \gamma - \gamma^ {k + 1}\right)}{\left(\gamma^ {k + 1}\right) ^ {- 1} - \gamma^ {k + 1}} \left\| U ^ {0} - \hat {U} ^ {\star} \right\| _ {\infty} + \frac {1 + \gamma}{1 + \gamma^ {k + 1}} \frac {1 - \gamma^ {k}}{1 - \gamma} \max  _ {0 \leq i \leq k - 1} \| \epsilon^ {i} \| _ {\infty}.
$$
$$
\begin{array}{l} \left\| T ^ {\star} U ^ {k} - U ^ {k} \right\| _ {\infty} \leq \frac {\left(\gamma^ {- 1} - \gamma\right) \left(1 + 2 \gamma - \gamma^ {k + 1}\right)}{\left(\gamma^ {k + 1}\right) ^ {- 1} - \gamma^ {k + 1}} \max  \left\{\left\| U ^ {0} - U ^ {\star} \right\| _ {\infty}, \left\| U ^ {0} - \hat {U} ^ {\star} \right\| _ {\infty} \right\} \\ + \frac {1 + \gamma}{1 + \gamma^ {k + 1}} \frac {1 - \gamma^ {k}}{1 - \gamma} \max  _ {0 \leq i \leq k - 1} \left\| \epsilon^ {i} \right\| _ {\infty}. \\ \end{array}
$$
Now, for the second rate in Theorem 6, we present following key lemma.
Lemma 18. Let  $0 < \gamma < 1$ . For the iterates  $\{U^k\}_{k = 0,1,\ldots}$  of Anc-VI, if  $U^0 \geq T^\star U^0$ , there exist nonexpansive linear operators  $\{\mathcal{P}^l\}_{l = 0,1,\ldots,k}$  and  $\{\hat{\mathcal{P}}^l\}_{l = 0,1,\ldots,k}$  such that
$$
\begin{array}{l} T ^ {\star} U ^ {k} - U ^ {k} \leq \Pi_ {j = 1} ^ {k} (1 - \beta_ {j}) \Pi_ {l = k} ^ {0} \gamma \mathcal {P} ^ {l} (U ^ {0} - U ^ {\star}) - \sum_ {i = 1} ^ {k} \Pi_ {j = i} ^ {k} (1 - \beta_ {j}) \Pi_ {l = k} ^ {i + 1} \gamma \mathcal {P} ^ {l} (I - \gamma \mathcal {P} ^ {i}) \epsilon^ {i - 1} \\ - \beta^ {k} (U ^ {0} - U ^ {\star}), \\ \end{array}
$$
$$
\begin{array}{l} T ^ {\star} U ^ {k} - U ^ {k} \geq \sum_ {i = 1} ^ {k} \left[ \left(\beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i})\right) \left(\Pi_ {j = i + 1} ^ {k} (1 - \beta_ {j})\right) \left(\Pi_ {l = k} ^ {i} \gamma \hat {\mathcal {P}} ^ {l}\right) \left(U ^ {0} - \hat {U} ^ {\star}\right) \right] - \beta_ {k} \left(U ^ {0} - \hat {U} ^ {\star}\right) \\ - \sum_ {i = 1} ^ {k} \Pi_ {j = i} ^ {k} (1 - \beta_ {j}) \Pi_ {l = k} ^ {i + 1} \gamma \hat {\mathcal {P}} ^ {l} \left(I - \gamma \hat {\mathcal {P}} ^ {i}\right) \epsilon^ {i - 1}, \\ \end{array}
$$
for  $1 \leq k$ , where  $\Pi_{j=k+1}^{k}(1-\beta_{j}) = 1$ ,  $\Pi_{l=k}^{k+1}\gamma \mathcal{P}^{l} = \Pi_{l=k}^{k+1}\gamma \hat{\mathcal{P}}^{l} = I$ , and  $\beta_{0} = 1$ .
Proof of Lemma 18. If  $U^0 \geq T^\star U^0$ ,  $U^0 \geq \lim_{m \to \infty} (T^\star)^m U^0 = U^\star$  by Lemma 1. By Lemma 3, this also implies  $U^0 \geq \hat{U}^\star$ .
First, we prove first inequality in Lemma 18 by induction. If  $k = 1$
$$
\begin{array}{l} U ^ {1} - (1 - \beta_ {1}) U ^ {0} - \beta_ {1} U ^ {\star} = (1 - \beta_ {1}) \epsilon^ {0} + \beta_ {1} (U ^ {0} - U ^ {\star}) + (1 - \beta_ {1}) \left(T ^ {\star} U ^ {0} - U ^ {0}\right) \\ \leq \left(1 - \beta_ {1}\right) \epsilon^ {0} + \left(1 - \beta_ {1}\right) \gamma \mathcal {P} ^ {0} \left(U ^ {0} - U ^ {\star}\right) + \left(2 \beta_ {1} - 1\right) \left(U ^ {0} - U ^ {\star}\right) \\ \leq \left(1 - \beta_ {1}\right) \epsilon^ {0} + \left(1 - \beta_ {1}\right) \gamma \mathcal {P} ^ {0} \left(U ^ {\circ} - U ^ {\star}\right), \\ \end{array}
$$
where the second inequality is from the  $(2\beta_{1} - 1)(U^{0} - U^{\star})\leq 0$  , and first inequality comes from Lemma 8 with  $\alpha = 1,U = U^0,\bar{U} = U^0 -U^\star$  , and let  $\bar{U}$  be the entire right hand side of inequality. Then, we have
$$
\begin{array}{l} T ^ {\star} U ^ {1} - U ^ {1} = T ^ {\star} U ^ {1} - \left(1 - \beta_ {1}\right) T ^ {\star} U ^ {0} - \beta_ {1} U ^ {\star} - \beta_ {1} \left(U ^ {0} - U ^ {\star}\right) - \left(1 - \beta_ {1}\right) \epsilon^ {0} \\ \leq \gamma \mathcal {P} ^ {1} ((1 - \beta_ {1}) \epsilon^ {0} + (1 - \beta_ {1}) \gamma \mathcal {P} ^ {0} (U ^ {0} - U ^ {\star})) - \beta_ {1} (U ^ {0} - U ^ {\star}) - (1 - \beta_ {1}) \epsilon^ {0} \\ = (1 - \beta_ {1}) \gamma \mathcal {P} ^ {1} \gamma \mathcal {P} ^ {0} (U ^ {0} - U ^ {\star}) - \beta_ {1} (U ^ {0} - U ^ {\star}) - (I - \gamma \mathcal {P} ^ {1}) (1 - \beta_ {1}) \epsilon^ {0}. \\ \end{array}
$$
$$
\begin{array}{l} U ^ {k} - (1 - \beta_ {k}) U ^ {k - 1} - \beta_ {k} U ^ {\star} \\ = \beta_ {k} \left(U ^ {0} - U ^ {\star}\right) + (1 - \beta_ {k}) \left(T ^ {\star} U ^ {k - 1} - U ^ {k - 1}\right) + (1 - \beta_ {k}) \epsilon^ {k - 1} \\ \leq \beta_ {k} \left(U ^ {0} - U ^ {\star}\right) - \left(1 - \beta_ {k}\right) \beta_ {k - 1} \left(U ^ {0} - U ^ {\star}\right) + \left(1 - \beta_ {k}\right) \left(\Pi_ {j = 1} ^ {k - 1} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k - 1} ^ {0} \gamma \mathcal {P} ^ {l}\right) \left(U ^ {0} - U ^ {\star}\right) \\ + (1 - \beta_ {k}) \epsilon^ {k - 1} - (1 - \beta_ {k}) \sum_ {i = 1} ^ {k - 1} \Pi_ {j = i} ^ {k - 1} (1 - \beta_ {j}) \Pi_ {l = k - 1} ^ {i + 1} \gamma \mathcal {P} ^ {l} (I - \gamma \mathcal {P} ^ {i}) \epsilon^ {i - 1} \\ \leq \left(1 - \beta_ {k}\right) \left(\Pi_ {j = 1} ^ {k - 1} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k - 1} ^ {0} \gamma \mathcal {P} ^ {l}\right) \left(U ^ {0} - U ^ {\star}\right) + \left(1 - \beta_ {k}\right) \epsilon^ {k - 1} \\ - \left(1 - \beta_ {k}\right) \sum_ {i = 1} ^ {k - 1} \Pi_ {j = i} ^ {k - 1} \left(1 - \beta_ {j}\right) \Pi_ {l = k - 1} ^ {i + 1} \gamma \mathcal {P} ^ {l} \left(I - \gamma \mathcal {P} ^ {i}\right) \epsilon^ {i - 1}, \\ \end{array}
$$
where the second inequality is from  $\beta_{k} - (1 - \beta_{k})\beta_{k - 1}\leq 0$  and let  $\bar{U}$  be the entire right hand side of inequality. Then, we have
$$
\begin{array}{l} T ^ {\star} U ^ {k} - U ^ {k} \\ = T ^ {\star} U ^ {k} - \left(1 - \beta_ {k}\right) T ^ {\star} U ^ {k - 1} - \beta_ {k} T ^ {\star} U ^ {\star} - \beta_ {k} \left(U ^ {0} - U ^ {\star}\right) - \left(1 - \beta_ {k}\right) \epsilon^ {k - 1} \\ \leq \gamma \mathcal {P} ^ {k} \left(\left(1 - \beta_ {k}\right) \left(\Pi_ {j = 1} ^ {k - 1} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k - 1} ^ {0} \gamma \mathcal {P} ^ {l}\right) \left(U ^ {0} - U ^ {\star}\right) + \left(1 - \beta_ {k}\right) \epsilon^ {k - 1} \right. \\ \left. - \left(1 - \beta_ {k}\right) \sum_ {i = 1} ^ {k - 1} \Pi_ {j = i} ^ {k - 1} \left(1 - \beta_ {j}\right) \Pi_ {l = k - 1} ^ {i + 1} \gamma \mathcal {P} ^ {l} \left(I - \gamma \mathcal {P} ^ {i}\right) \epsilon^ {i - 1}\right) - \beta_ {k} \left(U ^ {0} - U ^ {\star}\right) - \left(1 - \beta_ {k}\right) \epsilon^ {k - 1} \\ = \Pi_ {j = 1} ^ {k} \left(1 - \beta_ {j}\right) \Pi_ {l = k} ^ {0} \gamma \mathcal {P} ^ {l} \left(U ^ {0} - U ^ {\star}\right) - \sum_ {i = 1} ^ {k} \Pi_ {j = i} ^ {k} \left(1 - \beta_ {j}\right) \Pi_ {l = k} ^ {i + 1} \gamma \mathcal {P} ^ {l} \left(I - \gamma \mathcal {P} ^ {i}\right) \epsilon^ {i - 1} \\ - \beta^ {k} (U ^ {0} - U ^ {\star}), \\ \end{array}
$$
where the first inequality comes from Lemma 8 with  $\alpha = \beta_{k}, U = U^{k}, \tilde{U} = U^{k-1}$ , and previously defined  $\bar{U}$ .
For the second inequality in Lemma 18, if  $k = 1$
$$
\begin{array}{l} U ^ {1} - (1 - \beta_ {1}) U ^ {0} - \beta_ {1} \hat {U} ^ {\star} = (1 - \beta_ {1}) \epsilon^ {0} + \beta_ {1} (U ^ {0} - \hat {U} ^ {\star}) + (1 - \beta_ {1}) (T ^ {\star} U ^ {0} - U ^ {0}) \\ = \left(1 - \beta_ {1}\right) \epsilon^ {0} + \beta_ {1} \left(U ^ {0} - \hat {U} ^ {\star}\right) + \left(1 - \beta_ {1}\right) \left(T ^ {\star} U ^ {0} - \hat {U} ^ {\star} - \left(U ^ {0} - \hat {U} ^ {\star}\right)\right) \\ \geq \left(1 - \beta_ {1}\right) \epsilon^ {0} + \beta_ {1} \left(U ^ {0} - \hat {U} ^ {\star}\right) - \left(1 - \beta_ {1}\right) \left(U ^ {0} - \hat {U} ^ {\star}\right) \\ \end{array}
$$
where the second inequality is from  $U^0 \geq T^\star U^0 \geq \hat{U}^\star$ , and let  $\bar{U}$  be the entire right hand side of inequality. Then, we have
$$
\begin{array}{l} T ^ {\star} U ^ {1} - U ^ {1} = T ^ {\star} U ^ {1} - \left(1 - \beta_ {1}\right) T ^ {\star} U ^ {0} - \beta_ {1} U ^ {\star} - \beta_ {1} \left(U ^ {0} - \hat {U} ^ {\star}\right) - \left(1 - \beta_ {1}\right) \epsilon^ {0} \\ \geq \gamma \mathcal {P} ^ {1} \left(\left(1 - \beta_ {1}\right) \epsilon^ {0} + \beta_ {1} \left(U ^ {0} - \hat {U} ^ {\star}\right) - \left(1 - \beta_ {1}\right) \left(U ^ {0} - \hat {U} ^ {\star}\right)\right) - \beta_ {1} \left(U ^ {0} - \hat {U} ^ {\star}\right) - \left(1 - \beta_ {1}\right) \epsilon^ {0} \\ = (2 \beta_ {1} - 1) \gamma \mathcal {P} ^ {1} (U ^ {0} - \hat {U} ^ {\star}) - \beta_ {1} (U ^ {0} - \hat {U} ^ {\star}) - (I - \gamma \mathcal {P} ^ {1}) (1 - \beta_ {1}) \epsilon^ {0}. \\ \end{array}
$$
$$
\begin{array}{l} U ^ {k} - (1 - \beta_ {k}) U ^ {k - 1} - \beta_ {k} \hat {U} ^ {\star} \\ \geq \left(1 - \beta_ {k}\right) \sum_ {i = 1} ^ {k - 1} \left[ \left(\beta_ {i} - \beta_ {i - 1} \left(1 - \beta_ {i}\right)\right) \left(\Pi_ {j = i + 1} ^ {k - 1} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k - 1} ^ {i} \gamma \hat {\mathcal {P}} ^ {l}\right) \left(U ^ {0} - \hat {U} ^ {\star}\right) \right] \\ + \left(\beta_ {k} - \left(1 - \beta_ {k}\right) \beta_ {k - 1}\right) \left(U ^ {0} - \hat {U} ^ {\star}\right) + \left(1 - \beta_ {k}\right) \epsilon^ {k - 1} \\ - \left(1 - \beta_ {k}\right) \sum_ {i = 1} ^ {k - 1} \Pi_ {j = i} ^ {k - 1} \left(1 - \beta_ {j}\right) \Pi_ {l = k - 1} ^ {i + 1} \gamma \hat {\mathcal {P}} ^ {l} \left(I - \gamma \hat {\mathcal {P}} ^ {i}\right) \epsilon^ {i - 1}, \\ \end{array}
$$
$$
\begin{array}{l} T ^ {\star} U ^ {k} - U ^ {k} \\ = T ^ {\star} U ^ {k} - \left(1 - \beta_ {k}\right) T ^ {\star} U ^ {k - 1} - \beta_ {k} \hat {T} ^ {\star} \hat {U} ^ {\star} - \beta_ {k} \left(U ^ {0} - \hat {U} ^ {\star}\right) - \left(1 - \beta_ {k}\right) \epsilon^ {k - 1} \\ \geq \gamma \hat {\mathcal {P}} ^ {k} \left(\left(1 - \beta_ {k}\right) \sum_ {i = 1} ^ {k - 1} \left[ \left(\beta_ {i} - \beta_ {i - 1} \left(1 - \beta_ {i}\right)\right) \left(\Pi_ {j = i + 1} ^ {k - 1} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k - 1} ^ {i} \gamma \hat {\mathcal {P}} ^ {l}\right) \left(U ^ {0} - \hat {U} ^ {\star}\right) \right] \right. \\ + \left(\beta_ {k} - (1 - \beta_ {k}) \beta_ {k - 1}\right) \left(U ^ {0} - \hat {U} ^ {\star}\right) - \left(1 - \beta_ {k}\right) \sum_ {i = 1} ^ {k - 1} \Pi_ {j = i} ^ {k - 1} \left(1 - \beta_ {j}\right) \Pi_ {l = k - 1} ^ {i + 1} \gamma \hat {\mathcal {P}} ^ {l} \left(I - \gamma \hat {\mathcal {P}} ^ {i}\right) \epsilon^ {i - 1} \\ \left. + \left(1 - \beta_ {k}\right) \epsilon^ {k - 1}\right) - \left(1 - \beta_ {k}\right) \epsilon^ {k - 1} - \beta_ {k} \left(U ^ {0} - \hat {U} ^ {\star}\right) \\ = \sum_ {i = 1} ^ {k} \left[ \left(\beta_ {i} - \beta_ {i - 1} \left(1 - \beta_ {i}\right)\right) \left(\Pi_ {j = i + 1} ^ {k} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k} ^ {i} \gamma \hat {\mathcal {P}} ^ {l}\right) \left(U ^ {0} - \hat {U} ^ {\star}\right) \right] - \beta_ {k} \left(U ^ {0} - \hat {U} ^ {\star}\right) \\ - \sum_ {i = 1} ^ {k} \Pi_ {j = i} ^ {k} (1 - \beta_ {j}) \Pi_ {l = k} ^ {i + 1} \gamma \hat {\mathcal {P}} ^ {l} \left(I - \gamma \hat {\mathcal {P}} ^ {i}\right) \epsilon^ {i - 1}, \\ \end{array}
$$
where inequality comes from Lemma 9 with  $\alpha = \beta_{k}, U = U^{k}, \tilde{U} = U^{k - 1}$ , and previously defined  $\tilde{U}$ .
Now, we prove the second rate in Theorem 6.
Proof of second rate in Theorem 6. If we take  $\| \cdot \|_{\infty}$  right side of first inequality in Lemma 18, we have
$$
\frac {\left(\gamma^ {- 1} - \gamma\right) \gamma}{\left(\gamma^ {k + 1}\right) ^ {- 1} - \gamma^ {k + 1}} \left\| U ^ {0} - U ^ {\star} \right\| _ {\infty} + \frac {1 + \gamma}{1 + \gamma^ {k + 1}} \frac {1 - \gamma^ {k}}{1 - \gamma} \max  _ {0 \leq i \leq k - 1} \left\| \epsilon^ {i} \right\| _ {\infty}.
$$
If we apply second inequality of Lemma 18 and take  $\| \cdot \|_{\infty}$ -norm right side, we have
$$
\frac {\left(\gamma^ {- 1} - \gamma\right) \left(1 + \gamma - \gamma^ {k + 1}\right)}{\left(\gamma^ {k + 1}\right) ^ {- 1} - \gamma^ {k + 1}} \left\| U ^ {0} - \hat {U} ^ {\star} \right\| _ {\infty} + \frac {1 + \gamma}{1 + \gamma^ {k + 1}} \frac {1 - \gamma^ {k}}{1 - \gamma} \max  _ {0 \leq i \leq k - 1} \left\| \epsilon^ {i} \right\| _ {\infty}.
$$
$$
\left\| T ^ {\star} U ^ {k} - U ^ {k} \right\| _ {\infty} \leq \frac {\left(\gamma^ {- 1} - \gamma\right) \left(1 + \gamma - \gamma^ {k + 1}\right)}{\left(\gamma^ {k + 1}\right) ^ {- 1} - \gamma^ {k + 1}} \left\| U ^ {0} - \hat {U} ^ {\star} \right\| _ {\infty} + \frac {1 + \gamma}{1 + \gamma^ {k + 1}} \frac {1 - \gamma^ {k}}{1 - \gamma} \max _ {0 \leq i \leq k - 1} \left\| \epsilon^ {i} \right\| _ {\infty},
$$
since  $\hat{U}^{\star}\leq U^{\star}\leq U^{0}$  implies that
$$
\frac {(\gamma^ {- 1} - \gamma) \gamma}{(\gamma^ {k + 1}) ^ {- 1} - \gamma^ {k + 1}} \left\| U ^ {0} - U ^ {\star} \right\| _ {\infty} \leq \frac {(\gamma^ {- 1} - \gamma) (1 + \gamma - \gamma^ {k + 1})}{(\gamma^ {k + 1}) ^ {- 1} - \gamma^ {k + 1}} \left\| U ^ {0} - \hat {U} ^ {\star} \right\| _ {\infty}.
$$
F Omitted proofs in Section 6
For the analyses, we first define  $\hat{T}_{GS}^{\star}\colon \mathbb{R}^{n}\to \mathbb{R}^{n}$  as
$$
\hat {T} _ {G S} ^ {\star} = \hat {T} _ {n} ^ {\star} \dots \hat {T} _ {2} ^ {\star} \hat {T} _ {1} ^ {\star},
$$
where  $\hat{T}_j^\star : \mathbb{R}^n \to \mathbb{R}^n$  is defined as
$$
\hat {T} _ {j} ^ {\star} (U) = \left(U _ {1}, \dots , U _ {j - 1}, \left(\hat {T} ^ {\star} (U)\right) _ {j}, U _ {j + 1}, \dots , U _ {n}\right)
$$
for  $j = 1, \dots, n$ , where  $\hat{T}^{\star}$  is Bellman anti-optimality operator.
Fact 3. [Classical result, [10, Proposition 1.3.2]]  $\hat{T}_{GS}^{\star}$  is a  $\gamma$ -contractive operator and has the same fixed point as  $\hat{T}^{\star}$ .
Now, we introduce the following lemmas.
Lemma 19. Let  $0 < \gamma < 1$ . If  $0 \leq \alpha \leq 1$ , then there exist  $\gamma$ -contractive nonnegative matrix  $\mathcal{P}_{GS}$  such that
$$
T _ {G S} ^ {\star} U - (1 - \alpha) T _ {G S} ^ {\star} \tilde {U} - \alpha T _ {G S} ^ {\star} U ^ {\star} \leq \mathcal {P} _ {G S} (U - (1 - \alpha) \tilde {U} - \alpha U ^ {\star}).
$$
Lemma 20. Let  $0 < \gamma < 1$ . If  $0 \leq \alpha \leq 1$ , then there exist  $\gamma$ -contractive nonnegative matrix  $\hat{\mathcal{P}}_{GS}$  such that
$$
\hat {\mathcal {P}} _ {G S} (U - (1 - \alpha) \tilde {U} - \alpha \hat {U} ^ {\star}) \leq T _ {G S} ^ {\star} U - (1 - \alpha) T _ {G S} ^ {\star} \tilde {U} - \alpha \hat {T} _ {G S} ^ {\star} \hat {U} ^ {\star}.
$$
Proof of Lemma 19. First let  $U = V, \tilde{U} = \tilde{V}, U^{\star} = V^{\star}$ . For  $1 \leq i \leq n$ , we have
$$
\begin{array}{l} T _ {i} ^ {\star} V (s _ {i}) - (1 - \alpha) T _ {i} ^ {\star} \tilde {V} (s _ {i}) - \alpha T _ {i} ^ {\star} V ^ {\star} (s _ {i}) \leq T _ {i} ^ {\pi_ {i}} V (s _ {i}) - (1 - \alpha) T _ {i} ^ {\pi_ {i}} \tilde {V} (s _ {i}) - \alpha T _ {i} ^ {\pi_ {i}} V ^ {\star} (s _ {i}) \\ = \gamma \mathcal {P} ^ {\pi_ {i}} \left(V - (1 - \alpha) \tilde {V} - \alpha V ^ {\star}\right) (s _ {i}), \\ \end{array}
$$
where  $\pi_{i}$  is the greedy policy satisfying  $T^{\pi_i}V = T^\star V$  and first inequality is from  $T^{\pi_i}\tilde{V}\leq T^\star \tilde{V}$  and  $T^{\pi_i}V^{\star}\leq T^{\star}V^{\star}$ . Then, define matrix  $\mathcal{P}_i$  as
$$
\mathcal {P} _ {i} (V) = \left(V _ {1}, \dots , V _ {i - 1}, \left(\gamma \mathcal {P} ^ {\pi_ {i}} (V)\right) _ {i}, V _ {i + 1}, \dots , V _ {n}\right)
$$
for  $i = 1,\dots ,n$ . Note that  $\mathcal{P}_i$  is nonnegative matrix since  $\mathcal{P}^{\pi_i}$  is nonnegative matrix. Then, we have
$$
T _ {i} ^ {\star} V - (1 - \alpha) T _ {i} ^ {\star} \tilde {V} - \alpha T _ {i} ^ {\star} V ^ {\star} \leq \mathcal {P} _ {i} (V - (1 - \alpha) \tilde {V} - \alpha V ^ {\star}).
$$
By induction, there exist a sequence of matrices  $\{\mathcal{P}_i\}_{i = 1,\dots,n}$  satisfying
$$
T _ {G S} ^ {\star} V - (1 - \alpha) T _ {G S} ^ {\star} \tilde {V} - \alpha T _ {G S} ^ {\star} V ^ {\star} \leq \mathcal {P} _ {n} \dots \mathcal {P} _ {1} (V - (1 - \alpha) \tilde {V} - \alpha V ^ {\star})
$$
since  $T_{i}^{\star}V^{\star} = V^{\star}$  for all  $i$ . Denote  $P_{GS}$  as  $\mathcal{P}_n\cdots \mathcal{P}_1$ . Then,  $P_{GS}$  is  $\gamma$ -contractive nonnegative matrix since
$$
\sum_ {j = 1} ^ {n} (P _ {G S}) _ {i j} = \sum_ {j = 1} ^ {n} (\mathcal {P} _ {i} \dots \mathcal {P} _ {1}) _ {i j} \leq \sum_ {j = 1} ^ {n} (\mathcal {P} _ {i}) _ {i j} = \gamma
$$
for  $1 \leq i \leq n$ , where first equality is from definition of  $\mathcal{P}_l$  for  $i + 1 \leq l \leq n$ , inequality comes from definition of  $\mathcal{P}_l$  for  $1 \leq l \leq i - 1$ , and last equality is induced by definition of  $\mathcal{P}_i$ . Therefore, this implies that  $\| P_{GS} \|_{\infty} \leq \gamma$ .
If  $U = Q$ , with similar argument of case  $U = V$ , let  $\pi_i$  be the greedy policy, define matrix  $\mathcal{P}_i$  as
$$
\mathcal {P} _ {i} (Q) = \left(Q _ {1}, \dots , Q _ {i - 1}, (\gamma \mathcal {P} ^ {\pi_ {i}} (Q)) _ {i}, Q _ {i + 1}, \dots , Q _ {n}\right),
$$
and denote  $P_{GS}$  as  $\mathcal{P}_n\cdots \mathcal{P}_1$ . Then,  $P_{GS}$  is  $\gamma$ -contractive nonnegative matrix satisfying
$$
T _ {G S} ^ {\star} Q - (1 - \alpha) T _ {G S} ^ {\star} \tilde {Q} - \alpha T _ {G S} ^ {\star} Q ^ {\star} \leq \mathcal {P} _ {G S} (Q - (1 - \alpha) \tilde {Q} - \alpha Q ^ {\star}).
$$
Proof of Lemma 20. First let  $U = V, \tilde{U} = \tilde{V}, \hat{U}^{\star} = \hat{V}^{\star}$ . For  $1 \leq i \leq n$ , we have
$$
\begin{array}{l} T _ {i} ^ {\star} V (s _ {i}) - (1 - \alpha) T _ {i} ^ {\star} \tilde {V} (s _ {i}) - \alpha \hat {T} _ {i} ^ {\star} \hat {V} ^ {\star} (s _ {i}) \\ = \sup _ {a \in \mathcal {A}} \left\{r (s _ {i}, a) + \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s _ {i}, a)} [ V (s ^ {\prime}) ] \right\} - \sup _ {a \in \mathcal {A}} \left\{(1 - \alpha) r (s _ {i}, a) + (1 - \alpha) \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s _ {i}, a)} [ \tilde {V} (s ^ {\prime}) ] \right\} \\ - \inf  _ {a \in \mathcal {A}} \left\{\alpha r (s _ {i}, a) + \alpha \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s _ {i}, a)} \left[ \hat {V} ^ {\star} (s ^ {\prime}) \right] \right\} \\ \geq \gamma \inf  _ {a \in \mathcal {A}} \left\{\mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s _ {i}, a)} \left[ V (s ^ {\prime}) - (1 - \alpha) \tilde {V} (s ^ {\prime}) - \alpha \hat {V} ^ {\star} (s ^ {\prime}) \right] \right\}. \\ \end{array}
$$
Let  $\hat{\pi}_i(\cdot |s) = \operatorname *{argmin}_{a\in \mathcal{A}}\mathbb{E}_{s'\sim P(\cdot |s,a)}\left[V(s') - (1 - \alpha)\tilde{V} (s') - \alpha \hat{V}^\star (s')\right]$  and define matrix  $\hat{\mathcal{P}}_i$  as
$$
\hat {\mathcal {P}} _ {i} (V) = \left(V _ {1}, \dots , V _ {i - 1}, \left(\gamma \mathcal {P} ^ {\hat {\pi} _ {i}} (V)\right) _ {i}, V _ {i + 1}, \dots , V _ {n}\right)
$$