SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
T ^ {\star} U ^ {k} - U ^ {k} \geq \sum_ {i = 1} ^ {k} \left[ (\beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i})) \left(\Pi_ {j = i + 1} ^ {k} (1 - \beta_ {j})\right) \left(\Pi_ {l = k} ^ {i} \gamma \hat {\mathcal {P}} ^ {l}\right) (U ^ {0} - \hat {U} ^ {\star}) \right] - \beta_ {k} (U ^ {0} - \hat {U} ^ {\star}),
$$
Proof of Lemma 12. If  $k = 0$
$$
\begin{array}{l} T ^ {\star} U ^ {0} - U ^ {0} = T ^ {\star} U ^ {0} - U ^ {\star} - \left(U ^ {0} - U ^ {\star}\right) \\ \leq - \left(U ^ {0} - U ^ {\star}\right) \\ \end{array}
$$
where the second inequality is from the condition.
$$
\begin{array}{l} U ^ {k} - (1 - \beta_ {k}) U ^ {k - 1} - \beta_ {k} U ^ {\star} \\ \leq \left(1 - \beta_ {k}\right) \sum_ {i = 1} ^ {k - 1} \left[ \left(\beta_ {i} - \beta_ {i - 1} \left(1 - \beta_ {i}\right)\right) \left(\Pi_ {j = i + 1} ^ {k - 1} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k - 1} ^ {i} \gamma \mathcal {P} ^ {l}\right) \left(U ^ {0} - U ^ {\star}\right) \right] \\ - \left(1 - \beta_ {k}\right) \beta_ {k - 1} \left(U ^ {0} - U ^ {\star}\right) + \beta_ {k} \left(U ^ {0} - U ^ {\star}\right), \\ \end{array}
$$
and let  $\tilde{U}$  be the entire right hand side of inequality. Then, we have
$$
\begin{array}{l} T ^ {\star} U ^ {k} - U ^ {k} \\ = T ^ {\star} U ^ {k} - \left(1 - \beta_ {k}\right) T ^ {\star} U ^ {k - 1} - \beta_ {k} T ^ {\star} U ^ {\star} - \beta_ {k} \left(U ^ {0} - U ^ {\star}\right) \\ \leq \gamma \mathcal {P} ^ {k} \left(\left(1 - \beta_ {k}\right) \sum_ {i = 1} ^ {k - 1} \left[ \left(\beta_ {i} - \beta_ {i - 1} \left(1 - \beta_ {i}\right)\right) \left(\Pi_ {j = i + 1} ^ {k - 1} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k - 1} ^ {i} \gamma \mathcal {P} ^ {l}\right) \left(U ^ {0} - U ^ {\star}\right) \right] \right. \\ \left. - \left(1 - \beta_ {k}\right) \beta_ {k - 1} \left(U ^ {0} - U ^ {\star}\right) + \beta_ {k} \left(U ^ {0} - U ^ {\star}\right)\right) - \beta_ {k} \left(U ^ {0} - U ^ {\star}\right) \\ = \sum_ {i = 1} ^ {k} \left[ \left(\beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i})\right) \left(\Pi_ {j = i + 1} ^ {k} (1 - \beta_ {j})\right) \left(\Pi_ {l = k} ^ {i} \gamma \mathcal {P} ^ {l}\right) \left(U ^ {0} - U ^ {\star}\right) \right] - \beta_ {k} \left(U ^ {0} - U ^ {\star}\right), \\ \end{array}
$$
Proof of Lemma 13. If  $k = 0$
$$
\begin{array}{l} T ^ {\star} U ^ {0} - U ^ {0} = T ^ {\star} U ^ {0} - \hat {U} ^ {\star} - (U ^ {0} - \hat {U} ^ {\star}) \\ \geq - \left(U ^ {0} - \hat {U} ^ {\star}\right). \\ \end{array}
$$
where the second inequality is from the fact that  $U^0 \geq T^\star U^0$  implies  $T^\star U^0 \geq U^\star$  by Lemma 5 and  $U^\star \geq \hat{U}^\star$  by Lemma 3.
$$
\begin{array}{l} U ^ {k} - (1 - \beta_ {k}) U ^ {k - 1} - \beta_ {k} \hat {U} ^ {\star} \\ \geq \left(1 - \beta_ {k}\right) \sum_ {i = 1} ^ {k - 1} \left[ \left(\beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i})\right) \left(\Pi_ {j = i + 1} ^ {k - 1} (1 - \beta_ {j})\right) \left(\Pi_ {l = k - 1} ^ {i} \gamma \hat {\mathcal {P}} ^ {l}\right) \left(U ^ {0} - \hat {U} ^ {\star}\right) \right] \\ - (1 - \beta_ {k}) \beta_ {k - 1} (U ^ {0} - \hat {U} ^ {\star}) + \beta_ {k} (U ^ {0} - \hat {U} ^ {\star}), \\ \end{array}
$$
$$
\begin{array}{l} T ^ {\star} U ^ {k} - U ^ {k} \\ = T ^ {\star} U ^ {k} - \left(1 - \beta_ {k}\right) T ^ {\star} U ^ {k - 1} - \beta_ {k} \hat {T} ^ {\star} \hat {U} ^ {\star} - \beta_ {k} \left(U ^ {0} - \hat {U} ^ {\star}\right) \\ \geq \gamma \hat {\mathcal {P}} ^ {k} \left(\left(1 - \beta_ {k}\right) \sum_ {i = 1} ^ {k - 1} \left[ \left(\beta_ {i} - \beta_ {i - 1} \left(1 - \beta_ {i}\right)\right) \left(\Pi_ {j = i + 1} ^ {k - 1} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k - 1} ^ {i} \gamma \hat {\mathcal {P}} ^ {l}\right) \left(U ^ {0} - \hat {U} ^ {\star}\right) \right] \right. \\ \left. - \left(1 - \beta_ {k}\right) \beta_ {k - 1} \left(U ^ {0} - \hat {U} ^ {\star}\right) + \beta_ {k} \left(U ^ {0} - \hat {U} ^ {\star}\right)\right) - \beta_ {k} \left(U ^ {0} - \hat {U} ^ {\star}\right) \\ = \sum_ {i = 1} ^ {k} \left[ \left(\beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i})\right) \left(\Pi_ {j = i + 1} ^ {k} (1 - \beta_ {j})\right) \left(\Pi_ {l = k} ^ {i} \gamma \hat {\mathcal {P}} ^ {l}\right) \left(U ^ {0} - \hat {U} ^ {\star}\right) \right] - \beta_ {k} \left(U ^ {0} - \hat {U} ^ {\star}\right), \\ \end{array}
$$
where inequality comes from second inequality in Lemma 9 with  $\alpha = \beta_{k}, U = U^{k}, \tilde{U} = U^{k - 1}$ , and previously defined  $\bar{U}$ .
Now, we prove the second rates of Theorem 2.
Proof of second rates in Theorem 2. Let  $0 < \gamma < 1$ . Then, if  $U^0 \leq T^\star U^0$ , then  $T^\star U^0 \leq U^\star$  and  $U^k \leq T^\star U^k$  by Lemma 5. Hence, taking  $\| \cdot \|_{\infty}$ -norm both sides of first inequality in Lemma 12, we have
$$
\left\| T ^ {\star} U ^ {k} - U ^ {k} \right\| _ {\infty} \leq \frac {\left(\gamma^ {- 1} - \gamma\right) \left(1 + \gamma - \gamma^ {k + 1}\right)}{\left(\gamma^ {k + 1}\right) ^ {- 1} - \gamma^ {k + 1}} \left\| U ^ {0} - U ^ {\star} \right\| _ {\infty}.
$$
Otherwise, if  $U^0 \geq TU^0$ ,  $U^k \geq TU^k$  by Lemma 5. taking  $\| \cdot \|_{\infty}$ -norm both sides of second inequality in Lemma 13, we have
$$
\left\| T ^ {\star} U ^ {k} - U ^ {k} \right\| _ {\infty} \leq \frac {\left(\gamma^ {- 1} - \gamma\right) \left(1 + \gamma - \gamma^ {k + 1}\right)}{\left(\gamma^ {k + 1}\right) ^ {- 1} - \gamma^ {k + 1}} \left\| U ^ {0} - \hat {U} ^ {\star} \right\| _ {\infty}.
$$
C Omitted proofs in Section 3
First, we present the following lemma.
Lemma 14. Let  $\gamma = 1$ . Assume a fixed point  $U^{\star}$  exists. For the iterates  $\{U^k\}_{k=0,1,\ldots}$  of  $\text{Anc-}VI$ ,  $\left\| U^k - U^{\star} \right\|_{\infty} \leq \left\| U^0 - U^{\star} \right\|_{\infty}$ .
Proof. If  $k = 0$ , it is obvious. By induction,
$$
\begin{array}{l} \left\| U ^ {k} - U ^ {\star} \right\| _ {\infty} = \left\| \beta_ {k} U ^ {0} + (1 - \beta_ {k}) T U ^ {k - 1} - U ^ {\star} \right\| _ {\infty} \\ = \left\| \left(1 - \beta_ {k}\right) \left(T U ^ {k - 1} - U ^ {\star}\right) + \beta_ {k} \left(U ^ {0} - U ^ {\star}\right) \right\| _ {\infty} \\ \leq \left(1 - \beta_ {k}\right) \left\| T U ^ {k - 1} - U ^ {\star} \right\| _ {\infty} + \beta_ {k} \left\| U ^ {0} - U ^ {\star} \right\| _ {\infty} \\ \leq \left(1 - \beta_ {k}\right) \left\| U ^ {k - 1} - U ^ {\star} \right\| _ {\infty} + \beta_ {k} \left\| U ^ {0} - U ^ {\star} \right\| _ {\infty} \\ = \left\| U ^ {0} - U ^ {\star} \right\| _ {\infty} \\ \end{array}
$$
where the second inequality comes from nonexpansiveness of  $T$ .
Now, we present the proof of Theorem 3.
Proof of Theorem 3. First, if  $U^0 \leq TU^0$ , with same argument in proof of Lemma 5, we can show that  $U^{k - 1} \leq U^k \leq TU^{k - 1} \leq TU^k$  for  $k = 1,2,\ldots$ .
Since fixed point  $U^{\star}$  exists by assumption, Lemma 4 and 10 hold. Note that  $\gamma = 1$  implies  $\beta_{k} = \frac{1}{k + 1}$  and if we take  $\| \cdot \|_{\infty}$ -norm both sides for those inequalities in lemmas, by simple calculation, we have
$$
\left\| T U ^ {k} - U ^ {k} \right\| _ {\infty} \leq \frac {2}{k + 1} \left\| U ^ {0} - U ^ {\star} \right\| _ {\infty}
$$
for any fixed point  $U^{\star}$  (since  $0 \leq T^{\star} U^{k} - U^{k}$ , we can get upper bound of  $\left\| T^{\star} U^{k} - U^{k} \right\|_{\infty}$  from Lemma 10).
Suppose that there exist  $\{k_j\}_{j = 0,1,\ldots}$  such that  $U^{k_j}$  converges to some  $\tilde{U}^{\star}$ . Then,  $\lim_{j\to \infty}(T - I)U^{k_j} = (T - I)\tilde{U}^{\star} = 0$  since  $T - I$  is continuous. This implies that  $\tilde{U}^{\star}$  is a fixed point. By Lemma 14 and previous argument,  $U^k$  is increasing and bounded sequence in  $\mathbb{R}^n$ . Thus,  $U^k$  has single limit point, some fixed point  $\tilde{U}^{\star}$ . Furthermore, the fact that  $U^0\leq TU^0\leq \tilde{U}^{\star}$  implies that Lemma 6 and 12 hold. Therefore, we have
$$
\left\| T U ^ {k} - U ^ {k} \right\| _ {\infty} \leq \frac {1}{k + 1} \left\| U ^ {0} - \tilde {U} ^ {\star} \right\| _ {\infty}.
$$
Next, we prove the Theorem 4.
Proof of Theorem 4. By same argument in the proof of Theorem 3, if  $U^0 \leq TU^0$ , we can show that  $U^{k-1} \leq U^k \leq TU^{k-1} \leq TU^k$  for  $k = 1, 2, \ldots$ , and
for any fixed point  $U^{\star}$ . Since  $U^{k}$  is increasing and bounded by Lemma 14 and previous argument,  $U^{k}$  converges pointwise to some  $\tilde{U}^{\star}$  in general action-state space. We now show that  $TU^{k}$  also converges pointwise to  $T\tilde{U}^{\star}$ . First, let  $T$  be Bellman consistency operator and  $U = V, \tilde{U}^{\star} = \tilde{V}^{\pi}$ . By monotone convergence theorem,
$$
\begin{array}{l} \lim  _ {k \rightarrow \infty} T ^ {\pi} V ^ {k} (s) = \lim  _ {k \rightarrow \infty} \mathbb {E} _ {a \sim \pi (\cdot | s)} \left[ \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ r (s, a) + \gamma V ^ {k} (s ^ {\prime}) \right]\right] \\ = \mathbb {E} _ {a \sim \pi (\cdot | s)} \left[ \lim  _ {k \rightarrow \infty} \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ r (s, a) + \gamma V ^ {k} (s ^ {\prime}) \right]\right] \\ = \mathbb {E} _ {a \sim \pi (\cdot | s)} \left[ \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ r (s, a) + \gamma \lim  _ {k \rightarrow \infty} V ^ {k} \left(s ^ {\prime}\right)\right]\right] \\ = T ^ {\pi} \tilde {V} ^ {\pi} (s) \\ \end{array}
$$
for any fixed  $s \in S$ . With same argument, case  $U = Q$  also holds. If  $T$  is Bellman optimality operator, we use following lemma.
Lemma 15. Let  $W, W^k \in \mathcal{F}(\mathcal{X})$  for  $k = 0, 1, \ldots$ . If  $W^k(x) \leq W^{k+1}(x)$  for all  $x \in \mathcal{X}$ , and  $\{W^k\}_{k=0,1,\ldots}$ , converge pointwise to  $W$ , then  $\lim_{k \to \infty} \{\sup_x W^k(x)\} = \sup_x W(x)$ .
Proof.  $W^{k}(x) \leq W(x)$  implies that  $\sup_{x} W^{k}(x) \leq \sup_{x} W(x)$ . If  $\sup_{x} W(x) = a$ , there exist  $x$  which satisfying  $a - W(x) < \frac{\epsilon}{2}$ , and by definition of  $W$ , there exist  $W^{k}$  such that  $a - W^{k}(x) < \epsilon$  for any  $\epsilon > 0$ .
If  $U = V$  and  $\tilde{U}^{\star} = \tilde{V}^{\star}$ , by previous lemma and monotone convergence theorem, we have
$$
\begin{array}{l} \lim  _ {k \rightarrow \infty} T ^ {\star} V ^ {k} (s) = \lim  _ {k \rightarrow \infty} \sup  _ {a} \left\{\mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ r (s, a) + \gamma V ^ {k} (s ^ {\prime}) \right]\right\} \\ = \sup  _ {a} \left\{\lim  _ {k \rightarrow \infty} \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ r (s, a) + \gamma V ^ {k} (s ^ {\prime}) \right]\right\} \\ = \sup  _ {a} \left\{\mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ r (s, a) + \gamma \lim  _ {k \rightarrow \infty} V ^ {k} (s ^ {\prime}) \right]\right\} \\ = T ^ {\star} \tilde {U} ^ {\star} (s) \\ \end{array}
$$
for any fixed  $s \in S$ . With similar argument, case  $U = Q$  also holds.
Since  $TU^k \to T\tilde{U}^\star$  and  $U^k \to \tilde{U}^\star$  pointwisely,  $TU^k - U^k$  converges pointwise to  $T\tilde{U}^\star - \tilde{U}^\star = 0$ . Thus,  $\tilde{U}^\star$  is indeed fixed point of  $T$ . Furthermore, the fact that  $U^0 \leq TU^0 \leq \tilde{U}^\star$  implies that Lemma 6 and 12 hold. Therefore, we have
D Omitted proofs in Section 4
We present the proof of Theorem 5.
Proof of Theorem 5. First, we prove the case  $U^0 = 0$  for  $n \geq k + 2$ . Consider the MDP  $(S, \mathcal{A}, P, r, \gamma)$  such that
$$
\mathcal {S} = \{s _ {1}, \dots , s _ {n} \}, \quad \mathcal {A} = \{a _ {1} \}, \quad P (s _ {i} \mid s _ {j}, a _ {1}) = \mathbb {1} _ {\{i = j = 1, j = i + 1 \}}, \quad r (s _ {i}, a _ {1}) = \mathbb {1} _ {\{i = 2 \}}.
$$
Then,  $T = \gamma \mathcal{P}^{\pi}U + [0,1,0,\dots ,0]^{\intercal}, U^{\star} = [0,1,\gamma ,\dots ,\gamma^{n - 2}]^{\intercal}$ , and  $\| U^0 - U^\star \|_\infty = 1$ . Under the span condition, we can show that  $\left(U^k\right)_1 = \left(U^k\right)_l = 0$  for  $k + 2 \leq l \leq n$  by following lemma.
Lemma 16. Let  $T \colon \mathbb{R}^n \to \mathbb{R}^n$  be defined as before. Then, under span condition,  $\left(U^i\right)_1 = 0$  for  $0 \leq i \leq k$ , and  $\left(U^i\right)_j = 0$  for  $0 \leq i \leq k$  and  $i + 2 \leq j \leq n$ .
Proof. Case  $k = 0$  is obvious. By induction,  $\left( U^l \right)_1 = 0$  for  $0 \leq l \leq i - 1$ . Then  $\left( TU^l \right)_1 = 0$  for  $0 \leq l \leq i - 1$ . This implies that  $\left( TU^l - U^l \right)_1 = 0$  for  $0 \leq l \leq i - 1$ . Hence  $\left( U^i \right)_1 = \left( U^0 \right)_1 = 0$ . Again, by induction,  $\left( U^l \right)_j = 0$  for  $0 \leq l \leq i - 1$ ,  $l + 2 \leq j \leq n$ . Then  $\left( TU^l \right)_j = 0$  for  $0 \leq l \leq i - 1$ ,  $l + 3 \leq j \leq n$  and this implies that  $\left( TU^l - U^l \right)_j = 0$  for  $0 \leq l \leq i - 1$ ,  $l + 3 \leq j \leq n$ . Therefore,  $\left( U^i \right)_j = 0$  for  $i + 2 \leq j \leq n$ .
Then, we get
$$
T U ^ {k} - U ^ {k} = \left(0, 1 - \left(U ^ {k}\right) _ {2}, \gamma \left(U ^ {k}\right) _ {2} - \left(U ^ {k}\right) _ {3}, \ldots , \gamma \left(U ^ {k}\right) _ {k} - \left(U ^ {k}\right) _ {k + 1}, \gamma \left(U ^ {k}\right) _ {k + 1}, \underbrace {0 , \ldots , 0} _ {n - k - 2}\right),
$$
$$
\left(T U ^ {k} - U ^ {k}\right) _ {2} + \gamma^ {- 1} \left(T U ^ {k} - U ^ {k}\right) _ {3} + \dots + \gamma^ {- k} \left(T U ^ {k} - U ^ {k}\right) _ {k + 2} = 1.
$$
$$
(1 + \dots +\gamma^ {- k})\max_{1\leq i\leq n}\left\{\left|T U^{k} - U^{k}\right|_{i}\right\} \geq 1.
$$
$$
\left\| T U ^ {k} - U ^ {k} \right\| _ {\infty} \geq \frac {\gamma^ {k}}{\sum_ {i = 0} ^ {k} \gamma^ {i}} \left\| U ^ {0} - U ^ {\star} \right\| _ {\infty}.
$$
Now, we show that for any initial point  $U^0 \in \mathbb{R}^n$ , there exists an MDP which exhibits same lower bound with the case  $U^0 = 0$ . Denote by  $\mathrm{MDP}(0)$  and  $T_0$  the worst-case MDP and Bellman consistency or optimality operator constructed for  $U^0 = 0$ . Define an  $\mathrm{MDP}(U^0)$  ( $S, \mathcal{A}, P, r, \gamma$ ) for  $U^0 \neq 0$  as
$$
\begin{array}{l} \mathcal {S} = \{s _ {1}, \ldots , s _ {n} \},   \mathcal {A} = \{a _ {1} \},   P (s _ {i} | s _ {j}, a _ {1}) = \mathbb {1} _ {\{i = j = 1, j = i + 1 \}},   r (s _ {i}, a _ {1}) = \left(U ^ {0} - \mathcal {P} ^ {\pi} U ^ {0}\right) _ {i} + \mathbb {1} _ {\{i = 2 \}}. \\ \text {T h e n , B e l l m a n c o n s t e n c i s y o r o p e r t i m a l i t y o p e r a t o r T s a t i s f i e s} \end{array}
$$
$$
T U = T _ {0} (U - U ^ {0}) + U ^ {0}.
$$
Let  $\tilde{U}^{\star}$  be fixed point of  $T_0$ . Then, if  $U^{\star} = \tilde{U}^{\star} + U^{0}$ ,  $U^{\star}$  is fixed point of  $T$ . Furthermore, if  $\{U^i\}_{i=0}^k$  satisfies span condition
$$
U ^ {i} \in U ^ {0} + s p a n \{T U ^ {0} - U ^ {0}, T U ^ {1} - U ^ {1}, \dots , T U ^ {i - 1} - U ^ {i - 1} \}, \qquad i = 1, \dots , k,
$$
$\tilde{U}^i = U^i - U^0$  is a sequence satisfying
$$
\tilde {U} ^ {i} \in \underbrace {\tilde {U} ^ {0}} _ {= 0} + s p a n \{T _ {0} \tilde {U} ^ {0} - \tilde {U} ^ {0}, T _ {0} \tilde {U} ^ {1} - \tilde {U} ^ {1}, \ldots , T _ {0} \tilde {U} ^ {i - 1} - \tilde {U} ^ {i - 1} \}, \qquad i = 1, \ldots , k,
$$
which is the same span condition in Theorem 5 with respect to  $T_0$ . This is because
$$
T U ^ {i} - U ^ {i} = T _ {0} (U ^ {i} - U ^ {0}) - (U ^ {i} - U ^ {0}) = T \tilde {U} ^ {i} - \tilde {U} ^ {i}
$$
for  $i = 0,\dots ,k$ . Thus,  $\{\tilde{U}^i\}_{i = 0}^k$  is a sequence starting from 0 and satisfy the span condition for  $T_{0}$ . This implies that
$$
\begin{array}{l} \left\| T U ^ {k} - U ^ {k} \right\| _ {\infty} = \left\| T \tilde {U} ^ {k} - \tilde {U} ^ {k} \right\| _ {\infty} \\ \geq \frac {\gamma^ {k}}{\sum_ {i = 0} ^ {k} \gamma^ {i}} \left\| \tilde {U} ^ {0} - \tilde {U} ^ {\star} \right\| _ {\infty} \\ = \frac {\gamma^ {k}}{\sum_ {i = 0} ^ {k} \gamma^ {i}} \left\| U ^ {0} - U ^ {\star} \right\| _ {\infty}. \\ \end{array}
$$
Hence,  $\mathrm{MDP}(U^0)$  is indeed our desired worst-case instance. Lastly, the fact that  $U^0 - U^\star = \tilde{U}^0 - \tilde{U}^\star = -(0, 1, \gamma, \dots, \gamma^{n-2})$  implies  $U^0 \leq U^\star$ .
E Omitted proofs in Section 5
First, we prove following key lemma.
Lemma 17. Let  $0 < \gamma < 1$ . For the iterates  $\{U^k\}_{k = 0,1,\ldots}$  of Anc-VI, there exist nonexpansive linear operators  $\{\mathcal{P}^l\}_{l = 0,1,\dots,k}$  and  $\{\hat{\mathcal{P}}^l\}_{l = 0,1,\dots,k}$  such that
$$
\begin{array}{l} T ^ {\star} U ^ {k} - U ^ {k} \leq \sum_ {i = 1} ^ {k} \left[ \left(\beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i})\right) \left(\Pi_ {j = i + 1} ^ {k} (1 - \beta_ {j})\right) \left(\Pi_ {l = k} ^ {i} \gamma \mathcal {P} ^ {l}\right) \left(U ^ {0} - U ^ {\star}\right) \right] - \beta_ {k} \left(U ^ {0} - U ^ {\star}\right) \\ + \Pi_ {j = 1} ^ {k} (1 - \beta_ {j}) \Pi_ {l = k} ^ {0} \gamma \mathcal {P} ^ {l} (U ^ {0} - U ^ {\star}) - \sum_ {i = 1} ^ {k} \Pi_ {j = i} ^ {k} (1 - \beta_ {j}) \Pi_ {l = k} ^ {i + 1} \gamma \mathcal {P} ^ {l} \left(I - \gamma \mathcal {P} ^ {i}\right) \epsilon^ {i - 1}, \\ \end{array}
$$
$$
\begin{array}{l} T ^ {\star} U ^ {k} - U ^ {k} \geq \sum_ {i = 1} ^ {k} \left[ \left(\beta_ {i} - \beta_ {i - 1} (1 - \beta_ {i})\right) \left(\Pi_ {j = i + 1} ^ {k} (1 - \beta_ {j})\right) \left(\Pi_ {l = k} ^ {i} \gamma \hat {\mathcal {P}} ^ {l}\right) (U ^ {0} - \hat {U} ^ {\star}) \right] - \beta_ {k} (U ^ {0} - \hat {U} ^ {\star}) \\ + \Pi_ {j = 1} ^ {k} (1 - \beta_ {j}) \Pi_ {l = k} ^ {0} \gamma \hat {\mathcal {P}} ^ {l} (U ^ {0} - \hat {U} ^ {\star}) - \sum_ {i = 1} ^ {k} \Pi_ {j = i} ^ {k} (1 - \beta_ {j}) \Pi_ {l = k} ^ {i + 1} \gamma \hat {\mathcal {P}} ^ {l} (I - \gamma \hat {\mathcal {P}} ^ {i}) \epsilon^ {i - 1}, \\ \end{array}
$$
for  $1 \leq k$ , where  $\Pi_{j=k+1}^{k}(1 - \beta_{j}) = 1$ ,  $\Pi_{l=k}^{k+1} \gamma \mathcal{P}^{l} = \Pi_{l=k}^{k+1} \gamma \hat{\mathcal{P}}^{l} = I$ , and  $\beta_{0} = 1$ .
Proof of Lemma 17. First, we prove the first inequality in Lemma 17 by induction.
If  $k = 1$
$$
\begin{array}{l} U ^ {1} - (1 - \beta_ {1}) U ^ {0} - \beta_ {1} U ^ {\star} = (1 - \beta_ {1}) \epsilon^ {0} + \beta_ {1} (U ^ {0} - U ^ {\star}) + (1 - \beta_ {1}) (T ^ {\star} U ^ {0} - U ^ {0}) \\ \leq \left(1 - \beta_ {1}\right) \epsilon^ {0} + \left(1 - \beta_ {1}\right) \gamma \mathcal {P} ^ {0} \left(U ^ {0} - U ^ {\star}\right) + \left(2 \beta_ {1} - 1\right) \left(U ^ {0} - U ^ {\star}\right), \\ \end{array}
$$
where inequality comes from Lemma 8 with  $\alpha = 1, U = U^0, \bar{U} = U^0 - U^\star$ , and let  $\bar{U}$  be the entire right hand side of inequality. Then, we have
$$
\begin{array}{l} T ^ {\star} U ^ {1} - U ^ {1} = T ^ {\star} U ^ {1} - \left(1 - \beta_ {1}\right) T ^ {\star} U ^ {0} - \beta_ {1} U ^ {\star} - \beta_ {1} \left(U ^ {0} - U ^ {\star}\right) - \left(1 - \beta_ {1}\right) \epsilon^ {0} \\ \leq \gamma \mathcal {P} ^ {1} \left(\left(1 - \beta_ {1}\right) \epsilon^ {0} + \left(1 - \beta_ {1}\right) \gamma \mathcal {P} ^ {0} \left(U ^ {0} - U ^ {\star}\right) + \left(2 \beta_ {1} - 1\right) \left(U ^ {0} - U ^ {\star}\right)\right) - \beta_ {1} \left(U ^ {0} - U ^ {\star}\right) \\ - (1 - \beta_ {1}) \epsilon^ {0} \\ = (1 - \beta_ {1}) \gamma \mathcal {P} ^ {1} \gamma \mathcal {P} ^ {0} (U ^ {0} - U ^ {\star}) + \gamma \mathcal {P} ^ {1} (2 \beta_ {1} - 1) (U ^ {0} - U ^ {\star}) - \beta_ {1} (U ^ {0} - U ^ {\star}) \\ - (I - \gamma \mathcal {P} ^ {1}) (1 - \beta_ {1}) \epsilon^ {0}. \\ \end{array}
$$
where inequality comes from Lemma 8 with  $\alpha = \beta_{1}, U = U^{1}, \tilde{U} = U^{0}$ , and previously defined  $\bar{U}$ . By induction,
$$
\begin{array}{l} U ^ {k} - (1 - \beta_ {k}) U ^ {k - 1} - \beta_ {k} U ^ {\star} \\ = \beta_ {k} \left(U ^ {0} - U ^ {\star}\right) + (1 - \beta_ {k}) \left(T ^ {\star} U ^ {k - 1} - U ^ {k - 1}\right) + (1 - \beta_ {k}) \epsilon^ {k - 1} \\ \leq \left(1 - \beta_ {k}\right) \sum_ {i = 1} ^ {k - 1} \left[ \left(\beta_ {i} - \beta_ {i - 1} \left(1 - \beta_ {i}\right)\right) \left(\Pi_ {j = i + 1} ^ {k - 1} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k - 1} ^ {i} \gamma \mathcal {P} ^ {l}\right) \left(U ^ {0} - U ^ {\star}\right) \right] \\ - \left(1 - \beta_ {k}\right) \beta_ {k - 1} \left(U ^ {0} - U ^ {\star}\right) + \left(1 - \beta_ {k}\right) \left(\Pi_ {j = 1} ^ {k - 1} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k - 1} ^ {0} \gamma \mathcal {P} ^ {l}\right) \left(U ^ {0} - U ^ {\star}\right) \\ + \beta_ {k} (U ^ {0} - U ^ {\star}) - (1 - \beta_ {k}) \sum_ {i = 1} ^ {k - 1} \Pi_ {j = i} ^ {k - 1} (1 - \beta_ {j}) \Pi_ {l = k - 1} ^ {i + 1} \gamma \mathcal {P} ^ {l} \left(I - \gamma \mathcal {P} ^ {i}\right) \epsilon^ {i - 1} + (1 - \beta_ {k}) \epsilon^ {k - 1}, \\ \end{array}
$$
$$
\begin{array}{l} T ^ {\star} U ^ {k} - U ^ {k} \\ = T ^ {\star} U ^ {k} - \left(1 - \beta_ {k}\right) T ^ {\star} U ^ {k - 1} - \beta_ {k} U ^ {0} - \left(1 - \beta_ {k}\right) \epsilon^ {k - 1} \\ = T ^ {\star} U ^ {k} - \left(1 - \beta_ {k}\right) T ^ {\star} U ^ {k - 1} - \beta_ {k} T ^ {\star} U ^ {\star} - \beta_ {k} \left(U ^ {0} - U ^ {\star}\right) - \left(1 - \beta_ {k}\right) \epsilon^ {k - 1} \\ \leq \gamma \mathcal {P} ^ {k} \left(\left(1 - \beta_ {k}\right) \sum_ {i = 1} ^ {k - 1} \left[ \left(\beta_ {i} - \beta_ {i - 1} \left(1 - \beta_ {i}\right)\right) \left(\Pi_ {j = i + 1} ^ {k - 1} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k - 1} ^ {i} \gamma \mathcal {P} ^ {l}\right) \left(U ^ {0} - U ^ {\star}\right) \right] \right. \\ - (1 - \beta_ {k}) \beta_ {k - 1} (U ^ {0} - U ^ {\star}) + (1 - \beta_ {k}) \left(\Pi_ {j = 1} ^ {k - 1} (1 - \beta_ {j})\right) \left(\Pi_ {l = k - 1} ^ {0} \gamma \mathcal {P} ^ {l}\right) (U ^ {0} - U ^ {\star}) \\ + \beta_ {k} (U ^ {0} - U ^ {\star}) - (1 - \beta_ {k}) \sum_ {j = 1} ^ {k - 1} \Pi_ {j = i} ^ {k - 1} (1 - \beta_ {j}) \Pi_ {l = k - 1} ^ {i + 1} \gamma \mathcal {P} ^ {l} (I - \gamma \mathcal {P} ^ {i}) \epsilon^ {i - 1} + (1 - \beta_ {k}) \epsilon^ {k - 1} \\ - \beta_ {k} (U ^ {0} - U ^ {\star}) - (1 - \beta_ {k}) \epsilon^ {k - 1} \\ = \sum_ {i = 1} ^ {k} \left[ \left(\beta_ {i} - \beta_ {i - 1} \left(1 - \beta_ {i}\right)\right) \left(\Pi_ {j = i + 1} ^ {k} \left(1 - \beta_ {j}\right)\right) \left(\Pi_ {l = k} ^ {i} \gamma \mathcal {P} ^ {l}\right) \left(U ^ {0} - U ^ {\star}\right) \right] - \beta_ {k} \left(U ^ {0} - U ^ {\star}\right) \\ + \Pi_ {j = 1} ^ {k} (1 - \beta_ {j}) \Pi_ {l = k} ^ {0} \gamma \mathcal {P} ^ {l} (U ^ {0} - U ^ {\star}) - \sum_ {i = 1} ^ {k} \Pi_ {j = i} ^ {k} (1 - \beta_ {j}) \Pi_ {l = k} ^ {i + 1} \gamma \mathcal {P} ^ {l} \left(I - \gamma \mathcal {P} ^ {i}\right) \epsilon^ {i - 1}, \\ \end{array}
$$
where inequality comes from Lemma 8 with  $\alpha = \beta_{k}, U = U^{k}, \tilde{U} = U^{k - 1}$ , and previously defined  $\tilde{U}$ .
Now, we prove second inequality in Lemma 17 by induction.
$$
\text {I f} k = 1,
$$
$$
\begin{array}{l} U ^ {1} - (1 - \beta_ {1}) U ^ {0} - \beta_ {1} \hat {U} ^ {\star} = (1 - \beta_ {1}) \epsilon^ {0} + \beta_ {1} (U ^ {0} - \hat {U} ^ {\star}) + (1 - \beta_ {1}) (T ^ {\star} U ^ {0} - U ^ {0}) \\ \geq \left(1 - \beta_ {1}\right) \epsilon^ {0} + \left(1 - \beta_ {1}\right) \gamma \hat {\mathcal {P}} ^ {0} \left(U ^ {0} - \hat {U} ^ {\star}\right) + \left(2 \beta_ {1} - 1\right) \left(U ^ {0} - \hat {U} ^ {\star}\right), \\ \end{array}
$$
where inequality comes from Lemma 9 with  $\alpha = 1, U = U^0, \bar{U} = U^0 - \hat{U}^\star$ , and let  $\bar{U}$  be the entire right hand side of inequality. Then, we have