SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} T ^ {\star} Q (s, a) - (1 - \alpha) T ^ {\star} \tilde {Q} (s, a) - \alpha T ^ {\star} Q ^ {\star} (s, a) \\ = r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ \sup  _ {a ^ {\prime} \in \mathcal {A}} Q \left(s ^ {\prime}, a ^ {\prime}\right) \right] - (1 - \alpha) r (s, a) - (1 - \alpha) \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ \sup  _ {a ^ {\prime} \in \mathcal {A}} \tilde {Q} \left(s ^ {\prime}, a ^ {\prime}\right) \right] \\ - \alpha r (s, a) - \alpha \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ \sup  _ {a ^ {\prime} \in \mathcal {A}} Q ^ {\star} \left(s ^ {\prime}, a ^ {\prime}\right) \right] \\ \leq \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ \sup  _ {a ^ {\prime} \in \mathcal {A}} \left\{Q \left(s ^ {\prime}, a ^ {\prime}\right) - (1 - \alpha) \tilde {Q} \left(s ^ {\prime}, a ^ {\prime}\right) \right\} \right] - \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ \sup  _ {a ^ {\prime} \in \mathcal {A}} \alpha Q \left(s ^ {\prime}, a ^ {\prime}\right) \right] \\ \leq \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ \sup  _ {a ^ {\prime} \in \mathcal {A}} \left\{Q \left(s ^ {\prime}, a ^ {\prime}\right) - (1 - \alpha) \tilde {Q} \left(s ^ {\prime}, a ^ {\prime}\right) - \alpha Q ^ {\star} \left(s ^ {\prime}, a ^ {\prime}\right) \right\} \right] \\ \leq \gamma \sup  _ {(s ^ {\prime}, a ^ {\prime}) \in \mathcal {S} \times \mathcal {A}} \left\{Q (s ^ {\prime}, a ^ {\prime}) - (1 - \alpha) \tilde {Q} (s ^ {\prime}, a ^ {\prime}) - \alpha Q ^ {\star} (s ^ {\prime}, a ^ {\prime}) \right\}, \\ \leq \gamma \sup_{(s^{\prime},a^{\prime})\in \mathcal{S}\times \mathcal{A}}\bar{Q} (s^{\prime},a^{\prime}) \\ \end{array}
$$
for all  $(s,a)\in S\times \mathcal{A}$  . Therefore, we have
$$
T ^ {\star} Q - (1 - \alpha) T ^ {\star} \tilde {Q} - \alpha T ^ {\star} Q ^ {\star} \leq \gamma \mathcal {P} _ {H} (\bar {Q}).
$$
Proof of Lemma 9. Note that  $\hat{T}^{\star}$  is Bellman anti-optimality operators for  $V$  or  $Q$ , and  $\hat{U}^{\star}$  is the fixed point of  $\hat{T}^{\star}$ . First, let  $U = V$ ,  $\tilde{U} = \tilde{V}$ ,  $\hat{U}^{\star} = \hat{V}^{\star}$ ,  $\bar{U} = \bar{V}$ , and  $\bar{V} \leq V - (1 - \alpha)\tilde{V} - \alpha \hat{V}^{\star}$ . Then,
$$
\begin{array}{l} T ^ {\star} V (s) - (1 - \alpha) T ^ {\star} \tilde {V} (s) - \alpha \hat {T} ^ {\star} \hat {V} ^ {\star} (s) \\ = \sup  _ {a \in \mathcal {A}} \left\{r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} [ V (s ^ {\prime}) ] \right\} - \sup  _ {a \in \mathcal {A}} \left\{(1 - \alpha) r (s, a) + (1 - \alpha) \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} [ \tilde {V} (s ^ {\prime}) ] \right\} \\ - \inf  _ {a \in \mathcal {A}} \left\{\alpha r (s, a) + \alpha \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ \hat {V} ^ {\star} \left(s ^ {\prime}\right) \right] \right\} \\ \geq \inf  _ {a \in \mathcal {A}} \left\{r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} [ V (s ^ {\prime}) ] - (1 - \alpha) r (s, a) - (1 - \alpha) \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} [ \tilde {V} (s ^ {\prime}) ] \right\} \\ - \inf  _ {a \in \mathcal {A}} \left\{\alpha r (s, a) + \alpha \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ \hat {V} ^ {\star} \left(s ^ {\prime}\right) \right] \right\} \\ \geq \gamma \inf  _ {a \in \mathcal {A}} \left\{\mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ V (s ^ {\prime}) - (1 - \alpha) \tilde {V} (s ^ {\prime}) - \alpha \hat {V} ^ {\star} (s ^ {\prime}) \right] \right\}. \\ \end{array}
$$
Then, if action space is finite,
$$
\begin{array}{l} T ^ {\star} V - (1 - \alpha) T ^ {\star} \tilde {V} - \alpha T ^ {\star} V ^ {\star} \geq \gamma \mathcal {P} ^ {\hat {\pi}} \left(V - (1 - \alpha) \tilde {V} - \alpha \hat {V} ^ {\star}\right) \\ \geq \gamma \mathcal {P} ^ {\bar {\pi}} \bar {V} \\ \end{array}
$$
where  $\hat{\pi}$  is the policy satisfying  $\hat{\pi} (\cdot |s) = \operatorname *{argmin}_{a\in \mathcal{A}}\mathbb{E}_{s'\sim P(\cdot |s,a)}\left[V(s') - (1 - \alpha)\tilde{V} (s') - \alpha \hat{V}^{\star}(s')\right]$  and second inequality comes from Lemma 1. Thus, we can conclude  $\mathcal{P}_H = \mathcal{P}^\pi$
Otherwise, if action space is infinite, define  $\hat{\mathcal{P}}(c\bar{V}) = c\inf_{s \in \mathcal{S}}\bar{V}(s)$  for  $c \in \mathbb{R}$  and previously given  $\bar{V}$ . Let  $M$  be linear space spanned by  $\bar{V}$  with  $\|\cdot\|_{\infty}$ -norm. Then,  $\hat{\mathcal{P}}$  is linear functional on  $M$  and  $\|\hat{\mathcal{P}}\|_{\mathrm{op}} \leq 1$  since  $\frac{|c\inf_{s \in \mathcal{S}}\bar{V}(s)|}{\|c\bar{V}\|_{\infty}} \leq 1$ . Due to Hahn-Banach extension Theorem, there exist linear functional  $\hat{\mathcal{P}}_h \colon \mathcal{F}(\mathcal{S}) \to \mathbb{R}$  with  $\hat{\mathcal{P}}_h(\bar{V}) = \inf_{s \in \mathcal{S}}\bar{V}(s)$  and  $\|\hat{\mathcal{P}}_h\|_{\mathrm{op}} \leq 1$ . Furthermore, we can define  $\hat{\mathcal{P}}_H \colon \mathcal{F}(\mathcal{S}) \to \mathcal{F}(\mathcal{S})$  such that  $\hat{\mathcal{P}}_H V(s) = \hat{\mathcal{P}}_h(V)$  for all  $s \in \mathcal{S}$ . Then  $\|\hat{\mathcal{P}}_H\|_{\infty} \leq 1$  since  $\|\hat{\mathcal{P}}_H(V)\|_{\infty} = |\hat{\mathcal{P}}_h(V)| \leq \|\hat{\mathcal{P}}_h\|_{\mathrm{op}} \leq 1$  for  $\|V\|_{\infty} \leq 1$ . Thus,  $\hat{\mathcal{P}}_H$  is nonexpansive linear operator in  $\|\cdot\|_{\infty}$ -norm. Then, we have
$$
\begin{array}{l} T ^ {\star} V (s) - (1 - \alpha) T ^ {\star} \tilde {V} (s) - \alpha \hat {T} ^ {\star} \hat {V} ^ {\star} (s) \geq \gamma \inf  _ {a \in \mathcal {A}} \left\{\mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ V (s ^ {\prime}) - (1 - \alpha) \tilde {V} (s ^ {\prime}) - \alpha \hat {V} ^ {\star} (s ^ {\prime}) \right] \right\} \\ \geq \gamma \inf  _ {s ^ {\prime} \in \mathcal {S}} \{V (s ^ {\prime}) - (1 - \alpha) \tilde {V} (s ^ {\prime}) - \alpha \hat {V} ^ {\star} (s ^ {\prime}) \} \\ \geq \gamma \inf  _ {s ^ {\prime} \in \mathcal {S}} \{\bar {V} (s ^ {\prime}) \} \\ \end{array}
$$
$$
\gamma \hat {\mathcal {P}} _ {H} (\bar {V}) \leq T ^ {\star} V (s) - (1 - \alpha) T ^ {\star} \tilde {V} (s) - \alpha \hat {T} ^ {\star} \hat {V} ^ {\star} (s).
$$
Similarly, let  $U = Q$ ,  $\tilde{U} = \tilde{Q}$ ,  $\hat{U}^{\star} = \hat{Q}^{\star}$ ,  $\bar{U} = \bar{Q}$ , and  $\bar{Q} \leq Q - (1 - \alpha)\tilde{Q} - \alpha \hat{Q}^{\star}$ . Then,
$$
\begin{array}{l} T ^ {\star} Q (s, a) - \alpha T ^ {\star} \tilde {Q} (s, a) - (1 - \alpha) \hat {T} ^ {\star} \hat {Q} ^ {\star} (s, a) \\ = r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ \sup  _ {a ^ {\prime} \in \mathcal {A}} Q \left(s ^ {\prime}, a ^ {\prime}\right) \right] - (1 - \alpha) r (s, a) - (1 - \alpha) \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ \sup  _ {a ^ {\prime} \in \mathcal {A}} \tilde {Q} \left(s ^ {\prime}, a ^ {\prime}\right) \right] \\ - \alpha r (s, a) - \alpha \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ \inf  _ {a ^ {\prime} \in \mathcal {A}} \hat {Q} ^ {\star} \left(s ^ {\prime}, a ^ {\prime}\right) \right] \\ \geq \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ \inf  _ {a ^ {\prime} \in \mathcal {A}} \left\{Q \left(s ^ {\prime}, a ^ {\prime}\right) - (1 - \alpha) \tilde {Q} \left(s ^ {\prime}, a ^ {\prime}\right) \right\} \right] - \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ \inf  _ {a ^ {\prime} \in \mathcal {A}} \alpha \hat {Q} \left(s ^ {\prime}, a ^ {\prime}\right) \right] \\ \geq \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ \inf  _ {a ^ {\prime} \in \mathcal {A}} \left\{Q \left(s ^ {\prime}, a ^ {\prime}\right) - (1 - \alpha) \tilde {Q} \left(s ^ {\prime}, a ^ {\prime}\right) - \alpha \hat {Q} ^ {\star} \left(s ^ {\prime}, a ^ {\prime}\right) \right\} \right]. \\ \end{array}
$$
Hence, if action space is finite,
$$
\begin{array}{l} T ^ {\star} Q - (1 - \alpha) T ^ {\star} \tilde {Q} - \alpha T ^ {\star} Q ^ {\star} \geq \gamma \mathcal {P} ^ {\hat {\pi}} \left(Q - (1 - \alpha) \tilde {Q} - \alpha Q ^ {\star}\right), \\ \geq \gamma \mathcal {P} ^ {\bar {\pi}} \bar {Q}, \\ \end{array}
$$
where  $\hat{\pi}$  is the policy satisfying  $\hat{\pi} (\cdot |s) = \operatorname *{argmin}_{a\in \mathcal{A}}\mathbb{E}_{s'\sim P(\cdot |s,a)}\left[Q(s') - (1 - \alpha)\tilde{Q} (s') - \alpha Q^{\star}(s')\right]$  and second inequality comes from Lemma 1. Then, we can conclude  $\mathcal{P}_H = \mathcal{P}^\hat{\pi}$ .
Otherwise, if action space is infinite, define  $\hat{\mathcal{P}}(c\bar{Q}) = c\inf_{(s',a') \in S \times \mathcal{A}} \bar{Q}(s',a')$  for  $c \in \mathbb{R}^n$  and previously given  $\bar{Q}$ . Let  $M$  be linear space spanned by  $\bar{Q}$  with  $\|\cdot\|_{\infty}$ -norm. Then,  $\mathcal{P}$  is linear functional on  $M$  with  $\|\hat{\mathcal{P}}\|_{\mathrm{op}} \leq 1$ . Due to Hahn-Banach extension Theorem, there exist linear functional  $\hat{\mathcal{P}}_h \colon \mathcal{F}(\mathcal{S} \times \mathcal{A}) \to \mathbb{R}$  with  $\hat{\mathcal{P}}_h(\bar{Q}) = \inf_{(s',a') \in S \times \mathcal{A}} \bar{Q}(s',a')$  and  $\|\hat{\mathcal{P}}_h\|_{\mathrm{op}} \leq 1$ . Furthermore, we can define  $\hat{\mathcal{P}}_H \colon \mathcal{F}(\mathcal{S} \times \mathcal{A}) \to \mathcal{F}(\mathcal{S} \times \mathcal{A})$  such that  $\mathcal{P}_H Q(s,a) = \hat{\mathcal{P}}_h(Q)$  for all  $(s,a) \in S \times \mathcal{A}$  and  $\|\hat{\mathcal{P}}_H\|_{\infty} \leq 1$ . Thus  $\hat{\mathcal{P}}_H$  is nonexpansive linear operator in  $\|\cdot\|_{\infty}$ -norm. Then, we have
$$
\begin{array}{l} T ^ {\star} Q (s, a) - \alpha T ^ {\star} \tilde {Q} (s, a) - (1 - \alpha) \hat {T} ^ {\star} \hat {Q} ^ {\star} (s, a) \\ \geq \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ \inf _ {a ^ {\prime} \in \mathcal {A}} \left\{Q (s ^ {\prime}, a ^ {\prime}) - (1 - \alpha) \tilde {Q} (s ^ {\prime}, a ^ {\prime}) - \alpha \hat {Q} ^ {\star} (s ^ {\prime}, a ^ {\prime}) \right\} \right] \\ \geq \gamma \inf  _ {(s ^ {\prime}, a ^ {\prime}) \in \mathcal {S} \times \mathcal {A}} \left\{Q (s ^ {\prime}, a ^ {\prime}) - (1 - \alpha) \tilde {Q} (s ^ {\prime}, a ^ {\prime}) - \alpha \hat {Q} ^ {\star} (s ^ {\prime}, a ^ {\prime}) \right\} \\ \geq \gamma \inf_{(s^{\prime},a^{\prime})\in \mathcal{S}\times \mathcal{A}}\bar{Q} (s^{\prime},a^{\prime}), \\ \end{array}
$$
$$
\gamma \hat {\mathcal {P}} _ {H} (\bar {Q}) \leq T ^ {\star} Q - (1 - \alpha) T ^ {\star} \tilde {Q} - \alpha \hat {T} ^ {\star} \hat {Q} ^ {\star}.
$$
Now, we present our key lemmas for the first rate of Theorem 2.
Lemma 10. Let  $0 < \gamma \leq 1$ . If  $\gamma = 1$ , assume a fixed point  $U^{\star}$  exists. For the iterates  $\{U^{k}\}_{k=0,1,\ldots}$  of Anc-VI, there exist nonexpansive linear operators  $\{\mathcal{P}^{l}\}_{l=0,1,\ldots,k}$  such that
$$
\begin{array}{l} T ^ {\star} U ^ {k} - U ^ {k} \leq \sum_ {i = 1} ^ {k} \left[ \left(\beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i})\right) \left(\Pi_ {j = i + 1} ^ {k} (1 - \beta_ {j})\right) \left(\Pi_ {l = k} ^ {i} \gamma \mathcal {P} ^ {l}\right) \left(U ^ {0} - U ^ {\star}\right) \right] \\ - \beta_ {k} (U ^ {0} - U ^ {\star}) + \left(\Pi_ {j = 1} ^ {k} (1 - \beta_ {j})\right) \left(\Pi_ {l = k} ^ {0} \gamma \mathcal {P} ^ {l}\right) (U ^ {0} - U ^ {\star}) \\ \end{array}
$$
where  $\Pi_{j = k + 1}^{k}(1 - \beta_{j}) = 1$  and  $\beta_0 = 1$
Lemma 11. Let  $0 < \gamma < 1$ . For the iterates  $\{U^k\}_{k=0,1,\ldots}$  of Anc-VI, there exist nonexpansive linear operators  $\{\hat{\mathcal{P}}^l\}_{l=0,1,\ldots,k}$  such that
$$
\begin{array}{l} T ^ {\star} U ^ {k} - U ^ {k} \geq \sum_ {i = 1} ^ {k} \left[ \left(\beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i})\right) \left(\Pi_ {j = i + 1} ^ {k} (1 - \beta_ {j})\right) \left(\Pi_ {l = k} ^ {i} \gamma \hat {\mathcal {P}} ^ {l}\right) \left(U ^ {0} - \hat {U} ^ {\star}\right) \right] \\ - \beta_ {k} (U ^ {0} - \hat {U} ^ {\star}) + \left(\Pi_ {j = 1} ^ {k} (1 - \beta_ {j})\right) \left(\Pi_ {l = k} ^ {0} \gamma \hat {\mathcal {P}} ^ {l}\right) (U ^ {0} - \hat {U} ^ {\star}), \\ \end{array}
$$
We prove previous lemmas by induction.
Proof of Lemma 10. If  $k = 0$
$$
\begin{array}{l} T ^ {\star} U ^ {0} - U ^ {0} = T ^ {\star} U ^ {0} - U ^ {\star} - (U ^ {0} - U ^ {\star}) \\ = T ^ {\star} U ^ {0} - T ^ {\star} U ^ {\star} - \left(U ^ {0} - U ^ {\star}\right) \\ \leq \gamma \mathcal {P} ^ {0} (U ^ {0} - U ^ {\star}) - (U ^ {0} - U ^ {\star}). \\ \end{array}
$$
where inequality comes from first inequality in Lemma 8 with  $\alpha = 1, U = U^0, \bar{U} = U^0 - U^\star$ . By induction,
$$
\begin{array}{l} U ^ {k} - (1 - \beta_ {k}) U ^ {k - 1} - \beta_ {k} U ^ {\star} \\ = \beta_ {k} \left(U ^ {0} - U ^ {\star}\right) + \left(1 - \beta_ {k}\right) \left(T ^ {\star} U ^ {k - 1} - U ^ {k - 1}\right) \\ \leq \left(1 - \beta_ {k}\right) \sum_ {i = 1} ^ {k - 1} \left[ \left(\beta_ {i} - \beta_ {i - 1} \left(1 - \beta_ {i}\right)\right) \left(\Pi_ {j = i + 1} ^ {k - 1} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k - 1} ^ {i} \gamma \mathcal {P} ^ {l}\right) \left(U ^ {0} - U ^ {\star}\right) \right] \\ - \left(1 - \beta_ {k}\right) \beta_ {k - 1} \left(U ^ {0} - U ^ {\star}\right) + \left(1 - \beta_ {k}\right) \left(\Pi_ {j = 1} ^ {k - 1} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k - 1} ^ {0} \gamma \mathcal {P} ^ {l}\right) \left(U ^ {0} - U ^ {\star}\right) \\ + \beta_ {k} (U ^ {0} - U ^ {\star}), \\ \end{array}
$$
and let  $\bar{U}$  be the entire right hand side of inequality. Then, we have
$$
\begin{array}{l} T ^ {\star} U ^ {k} - U ^ {k} \\ = T ^ {\star} U ^ {k} - \left(1 - \beta_ {k}\right) T ^ {\star} U ^ {k - 1} - \beta_ {k} U ^ {0} \\ = T ^ {\star} U ^ {k} - \left(1 - \beta_ {k}\right) T ^ {\star} U ^ {k - 1} - \beta_ {k} U ^ {\star} - \beta_ {k} \left(U ^ {0} - U ^ {\star}\right) \\ = T ^ {\star} U ^ {k} - \left(1 - \beta_ {k}\right) T ^ {\star} U ^ {k - 1} - \beta_ {k} T ^ {\star} U ^ {\star} - \beta_ {k} \left(U ^ {0} - U ^ {\star}\right) \\ \leq \gamma \mathcal {P} ^ {k} \left(\left(1 - \beta_ {k}\right) \sum_ {i = 1} ^ {k - 1} \left[ \left(\beta_ {i} - \beta_ {i - 1} \left(1 - \beta_ {i}\right)\right) \left(\Pi_ {j = i + 1} ^ {k - 1} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k - 1} ^ {i} \gamma \mathcal {P} ^ {l}\right) \left(U ^ {0} - U ^ {\star}\right) \right] \right. \\ - \left(1 - \beta_ {k}\right) \beta_ {k - 1} \left(U ^ {0} - U ^ {\star}\right) + \left(1 - \beta_ {k}\right) \left(\Pi_ {j = 1} ^ {k - 1} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k - 1} ^ {0} \gamma \mathcal {P} ^ {l}\right) \left(U ^ {0} - U ^ {\star}\right) \\ \left. + \beta_ {k} \left(U ^ {0} - U ^ {\star}\right)\right) - \beta_ {k} \left(U ^ {0} - U ^ {\star}\right) \\ = \sum_ {i = 1} ^ {k - 1} \left[ \left(\beta_ {i} - \beta_ {i - 1} \left(1 - \beta_ {i}\right)\right) \left(\Pi_ {j = i + 1} ^ {k} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k} ^ {i} \gamma \mathcal {P} ^ {l}\right) \left(U ^ {0} - U ^ {\star}\right) \right] \\ - \beta_ {k - 1} \left(1 - \beta_ {k}\right) \gamma \mathcal {P} ^ {k} \left(U ^ {0} - U ^ {\star}\right) + \beta_ {k} \gamma \mathcal {P} ^ {k} \left(U ^ {0} - U ^ {\star}\right) \\ - \beta_ {k} \left(U ^ {0} - U ^ {\star}\right) + \left(\Pi_ {j = 1} ^ {k} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k} ^ {0} \gamma \mathcal {P} ^ {l}\right) \left(U ^ {0} - U ^ {\star}\right) \\ = \sum_ {i = 1} ^ {k} \left[ \left(\beta_ {i} - \beta_ {i - 1} \left(1 - \beta_ {i}\right)\right) \left(\Pi_ {j = i + 1} ^ {k} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k} ^ {i} \gamma \mathcal {P} ^ {l}\right) \left(U ^ {0} - U ^ {\star}\right) \right] \\ - \beta_ {k} (U ^ {0} - U ^ {\star}) + \left(\Pi_ {j = 1} ^ {k} (1 - \beta_ {j})\right) \left(\Pi_ {l = k} ^ {0} \gamma \mathcal {P} ^ {l}\right) (U ^ {0} - U ^ {\star}). \\ \end{array}
$$
where inequality comes from first inequality in Lemma 8 with  $\alpha = \beta_k, U = U^k, \tilde{U} = U^{k-1}$ , and previously defined  $\bar{U}$ .
Proof of Lemma 11. Note that  $\hat{T}^{\star}$  is Bellman anti-optimality operators for  $V$  or  $Q$ , and  $\hat{U}^{\star}$  is the fixed point of  $\hat{T}^{\star}$ . If  $k = 0$ ,
$$
\begin{array}{l} T ^ {\star} U ^ {0} - U ^ {0} = T ^ {\star} U ^ {0} - \hat {U} ^ {\star} - (U ^ {0} - \hat {U} ^ {\star}) \\ = T ^ {\star} U ^ {0} - \hat {T} ^ {\star} \hat {U} ^ {\star} - (U ^ {0} - \hat {U} ^ {\star}) \\ \geq \gamma \hat {\mathcal {P}} ^ {0} (U ^ {0} - \hat {U} ^ {\star}) - (U ^ {0} - \hat {U} ^ {\star}). \\ \end{array}
$$
where inequality comes from second inequality in Lemma 9 with  $\alpha = 1, U = U^0, \bar{U} = U^0 - \hat{U}^\star$ .
$$
\begin{array}{l} U ^ {k} - (1 - \beta_ {k}) U ^ {k - 1} - \beta_ {k} \hat {U} ^ {\star} \\ = \beta_ {k} \left(U ^ {0} - \hat {U} ^ {\star}\right) + \left(1 - \beta_ {k}\right) \left(T ^ {\star} U ^ {k - 1} - U ^ {k - 1}\right) \\ \geq \left(1 - \beta_ {k}\right) \sum_ {i = 1} ^ {k - 1} \left[ \left(\beta_ {i} - \beta_ {i - 1} \left(1 - \beta_ {i}\right)\right) \left(\Pi_ {j = i + 1} ^ {k - 1} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k - 1} ^ {i} \gamma \hat {\mathcal {P}} ^ {l}\right) \left(U ^ {0} - \hat {U} ^ {\star}\right) \right] \\ - (1 - \beta_ {k}) \beta_ {k - 1} (U ^ {0} - \hat {U} ^ {\star}) + (1 - \beta_ {k}) (\Pi_ {j = 1} ^ {k - 1} (1 - \beta_ {j})) (\Pi_ {l = k - 1} ^ {0} \gamma \hat {\mathcal {P}} ^ {l}) (U ^ {0} - \hat {U} ^ {\star}) \\ + \beta_ {k} (U ^ {0} - \hat {U} ^ {\star}), \\ \end{array}
$$
$$
\begin{array}{l} T ^ {\star} U ^ {k} - U ^ {k} \\ = T ^ {\star} U ^ {k} - \left(1 - \beta_ {k}\right) T ^ {\star} U ^ {k - 1} - \beta_ {k} U ^ {0} \\ = T ^ {\star} U ^ {k} - \left(1 - \beta_ {k}\right) T ^ {\star} U ^ {k - 1} - \beta_ {k} \hat {U} ^ {\star} - \beta_ {k} \left(U ^ {0} - \hat {U} ^ {\star}\right) \\ = T ^ {\star} U ^ {k} - \left(1 - \beta_ {k}\right) T ^ {\star} U ^ {k - 1} - \beta_ {k} \hat {T} ^ {\star} \hat {U} ^ {\star} - \beta_ {k} \left(U ^ {0} - \hat {U} ^ {\star}\right) \\ \geq \gamma \hat {\mathcal {P}} ^ {k} \left(\left(1 - \beta_ {k}\right) \sum_ {i = 1} ^ {k - 1} \left[ \left(\beta_ {i} - \beta_ {i - 1} \left(1 - \beta_ {i}\right)\right) \left(\Pi_ {j = i + 1} ^ {k - 1} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k - 1} ^ {i} \gamma \hat {\mathcal {P}} ^ {l}\right) \left(U ^ {0} - \hat {U} ^ {\star}\right) \right] \right. \\ - \left(1 - \beta_ {k}\right) \beta_ {k - 1} \left(U ^ {0} - \hat {U} ^ {\star}\right) + \left(1 - \beta_ {k}\right) \left(\Pi_ {j = 1} ^ {k - 1} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k - 1} ^ {0} \gamma \hat {\mathcal {P}} ^ {l}\right) \left(U ^ {0} - \hat {U} ^ {\star}\right) \\ \left. + \beta_ {k} \left(U ^ {0} - \hat {U} ^ {\star}\right)\right) - \beta_ {k} \left(U ^ {0} - \hat {U} ^ {\star}\right) \\ = \sum_ {i = 1} ^ {k - 1} \left[ \left(\beta_ {i} - \beta_ {i - 1} \left(1 - \beta_ {i}\right)\right) \left(\Pi_ {j = i + 1} ^ {k} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k} ^ {i} \gamma \hat {\mathcal {P}} ^ {l}\right) \left(U ^ {0} - \hat {U} ^ {\star}\right) \right] \\ - \beta_ {k - 1} (1 - \beta_ {k}) \gamma \hat {\mathcal {P}} ^ {k} (U ^ {0} - \hat {U} ^ {\star}) + \beta_ {k} \gamma \hat {\mathcal {P}} ^ {k} \left(U ^ {0} - \hat {U} ^ {\star}\right) \\ - \beta_ {k} (U ^ {0} - \hat {U} ^ {\star}) + \left(\Pi_ {j = 1} ^ {k} (1 - \beta_ {j})\right) \left(\Pi_ {l = k} ^ {0} \gamma \hat {\mathcal {P}} ^ {l}\right) (U ^ {0} - \hat {U} ^ {\star}) \\ = \sum_ {i = 1} ^ {k} \left[ \left(\beta_ {i} - \beta_ {i - 1} \left(1 - \beta_ {i}\right)\right) \left(\Pi_ {j = i + 1} ^ {k} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k} ^ {i} \gamma \hat {\mathcal {P}} ^ {l}\right) \left(U ^ {0} - \hat {U} ^ {\star}\right) \right] \\ - \beta_ {k} (U ^ {0} - \hat {U} ^ {\star}) + \left(\Pi_ {j = 1} ^ {k} (1 - \beta_ {j})\right) \left(\Pi_ {l = k} ^ {0} \gamma \hat {\mathcal {P}} ^ {l}\right) (U ^ {0} - \hat {U} ^ {\star}). \\ \end{array}
$$
where inequality comes from second inequality in Lemma 9 with  $\alpha = \beta_k, U = U^k, \tilde{U} = U^{k-1}$ , and previously defined  $\bar{U}$ .
Now, we prove the first rate of Theorem 2.
Proof of first rate in Theorem 2. Since  $B_{1} \leq A \leq B_{2}$  implies  $\|A\|_{\infty} \leq \sup \left\{\|B_{1}\|_{\infty}, \|B_{2}\|_{\infty}\right\}$  for  $A, B \in \mathcal{F}(\mathcal{X})$ , if we take  $\|\cdot\|_{\infty}$  right side first inequality of Lemma 10, we have
$$
\begin{array}{l} \sum_ {i = 1} ^ {k} \left| \beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i}) \right| \left(\Pi_ {j = i + 1} ^ {k} (1 - \beta_ {j})\right) \left\| \left(\Pi_ {l = k} ^ {i} \gamma \mathcal {P} ^ {l}\right) (U ^ {0} - U ^ {\star}) \right\| _ {\infty} \\ + \beta_ {k} \left\| U ^ {0} - U ^ {\pi} \right\| _ {\infty} + \left(\Pi_ {j = 1} ^ {k} (1 - \beta_ {j})\right) \left\| \left(\Pi_ {l = k} ^ {0} \gamma \mathcal {P} ^ {l}\right) (U ^ {0} - U ^ {\star}) \right\| _ {\infty} \\ \leq \left(\sum_ {i = 1} ^ {k} \gamma^ {k - i + 1} \left| \beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i}) \right| \left(\Pi_ {j = i + 1} ^ {k} (1 - \beta_ {j})\right) + \beta_ {k} + \gamma^ {k + 1} \Pi_ {j = 1} ^ {k} (1 - \beta_ {j})\right) \\ \left\| U ^ {0} - U ^ {\star} \right\| _ {\infty} \\ = \frac {\left(\gamma^ {- 1} - \gamma\right) \left(1 + 2 \gamma - \gamma^ {k + 1}\right)}{\left(\gamma^ {k + 1}\right) ^ {- 1} - \gamma^ {k + 1}} \left\| U ^ {0} - U ^ {\star} \right\| _ {\infty}, \\ \end{array}
$$
where the first inequality comes from triangular inequality, second inequality is from nonexpansiveness of  $\mathcal{P}^l$ , and last equality comes from calculations.
If we take  $\| \cdot \|_{\infty}$  right side of second inequality of Lemma 10, similarly, we have
$$
\begin{array}{l} \sum_ {i = 1} ^ {k} \left| \beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i}) \right| \left(\Pi_ {j = i + 1} ^ {k} (1 - \beta_ {j})\right) \left\| \left(\Pi_ {l = k} ^ {i} \gamma \hat {\mathcal {P}} ^ {l}\right) (U ^ {0} - \hat {U} ^ {\star}) \right\| _ {\infty} \\ + \beta_ {k} \left\| U ^ {0} - U ^ {\pi} \right\| _ {\infty} + \left(\Pi_ {j = 1} ^ {k} (1 - \beta_ {j})\right) \left\| \left(\Pi_ {l = k} ^ {0} \gamma \hat {\mathcal {P}} ^ {l}\right) (U ^ {0} - \hat {U} ^ {\star}) \right\| _ {\infty} \\ \leq \left(\sum_ {i = 1} ^ {k} \gamma^ {k - i + 1} \left| \beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i}) \right| \left(\Pi_ {j = i + 1} ^ {k} (1 - \beta_ {j})\right) + \beta_ {k} + \gamma^ {k + 1} \Pi_ {j = 1} ^ {k} (1 - \beta_ {j})\right) \\ = \frac {\left(\gamma^ {- 1} - \gamma\right) \left(1 + 2 \gamma - \gamma^ {k + 1}\right)}{\left(\gamma^ {k + 1}\right) ^ {- 1} - \gamma^ {k + 1}} \left\| U ^ {0} - \hat {U} ^ {\star} \right\| _ {\infty}, \\ \end{array}
$$
where the first inequality comes from triangular inequality, second inequality is from nonexpan-siveness of  $\hat{\mathcal{P}}^l$ , and last equality comes from calculations. Therefore, we conclude
$$
\left\| T ^ {\star} U ^ {k} - U ^ {k} \right\| _ {\infty} \leq \frac {\left(\gamma^ {- 1} - \gamma\right) \left(1 + 2 \gamma - \gamma^ {k + 1}\right)}{\left(\gamma^ {k + 1}\right) ^ {- 1} - \gamma^ {k + 1}} \max  \left\{\left\| U ^ {0} - U ^ {\star} \right\| _ {\infty}, \left\| U ^ {0} - \hat {U} ^ {\star} \right\| _ {\infty} \right\}.
$$
Next, for the second rate in Theorem 2, we prove following lemmas by induction.
Lemma 12. Let  $0 < \gamma \leq 1$ . If  $\gamma = 1$ , assume a fixed point  $U^{\star}$  exists. For the iterates  $\{U^k\}_{k=0,1,\ldots}$  of Anc-VI, if  $T^{\star}U^{0} \leq U^{\star}$ , there exist nonexpansive linear operators  $\{\mathcal{P}^l\}_{l=0,1,\ldots,k}$  such that
$$
T ^ {\star} U ^ {k} - U ^ {k} \leq \sum_ {i = 1} ^ {k} \left[ \left(\beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i})\right) \left(\Pi_ {j = i + 1} ^ {k} (1 - \beta_ {j})\right) \left(\Pi_ {l = k} ^ {i} \gamma \mathcal {P} ^ {l}\right) \left(U ^ {0} - U ^ {\star}\right) \right] - \beta_ {k} \left(U ^ {0} - U ^ {\star}\right)
$$
Lemma 13. Let  $0 < \gamma < 1$ . For the iterates  $\{U^k\}_{k = 0,1,\ldots}$  of Anc-VI, if  $U^0 \geq T^\star U^0$ , there exist nonexpansive linear operators  $\{\hat{\mathcal{P}}^l\}_{l = 0,1,\ldots,k}$  such that