SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \boldsymbol {I} - \boldsymbol {X} ^ {\dagger} \boldsymbol {X} + \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {t} \boldsymbol {X} ^ {\dagger} \boldsymbol {X} \\ = \boldsymbol {I} - \boldsymbol {X} ^ {\dagger} \boldsymbol {X} + \sum_ {k = 0} ^ {t} \binom {t} {k} \left(- \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {k} \boldsymbol {X} ^ {\dagger} \boldsymbol {X} \\ = \boldsymbol {I} - \boldsymbol {X} ^ {\dagger} \boldsymbol {X} + \boldsymbol {X} ^ {\dagger} \boldsymbol {X} + \sum_ {k = 1} ^ {t} \binom {t} {k} \left(- \frac {\lambda}{n}\right) ^ {k} \left(\boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {k - 1} \boldsymbol {X} ^ {\top} \boldsymbol {X} \boldsymbol {X} ^ {\dagger} \boldsymbol {X} \tag {20} \\ = \boldsymbol {I} + \sum_ {k = 1} ^ {t} \binom {t} {k} \left(- \frac {\lambda}{n}\right) ^ {k} (\boldsymbol {X} ^ {\top} \boldsymbol {X}) ^ {k} \\ = \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {t}. \\ \end{array}
$$
The second identity can be proved in a similar way:
$$
\begin{array}{l} \left[ \boldsymbol {I} - \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {t} \right] \boldsymbol {X} ^ {\dagger} \boldsymbol {X} \boldsymbol {X} ^ {\top} \\ = - \sum_ {k = 1} ^ {t} \binom {t} {k} \left(- \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {k} \boldsymbol {X} ^ {\dagger} \boldsymbol {X} \boldsymbol {X} ^ {\top} \\ = - \sum_ {k = 1} ^ {t} \binom {t} {k} \left(- \frac {\lambda}{n}\right) ^ {k} \left(\boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {k - 1} \boldsymbol {X} ^ {\top} \boldsymbol {X} \boldsymbol {X} ^ {\dagger} \boldsymbol {X} \boldsymbol {X} ^ {\top} \tag {21} \\ = - \sum_ {k = 1} ^ {t} \binom {t} {k} \left(- \frac {\lambda}{n}\right) ^ {k} (\boldsymbol {X} ^ {\top} \boldsymbol {X}) ^ {k - 1} \boldsymbol {X} ^ {\top} \boldsymbol {X} \boldsymbol {X} ^ {\top} \\ = - \sum_ {k = 1} ^ {t} \binom {t} {k} \left(- \frac {\lambda}{n}\right) ^ {k} \boldsymbol {X} ^ {\top} (\boldsymbol {X} \boldsymbol {X} ^ {\top}) ^ {k} \\ = \boldsymbol {X} ^ {\top} \left[ \boldsymbol {I} - \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} \boldsymbol {X} ^ {\top}\right) ^ {t} \right]. \\ \end{array}
$$
We are now ready to prove the main result of this section.
Lemma A.4. The excess risk at the t-th epoch can be upper bounded as
$$
R \left(\boldsymbol {\theta} _ {t}\right) \leq \boldsymbol {\theta} ^ {* \top} \boldsymbol {B} \boldsymbol {\theta} ^ {*} + \varepsilon^ {\top} \boldsymbol {C} \varepsilon , \tag {22}
$$
$$
\boldsymbol {B} = \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {t} \boldsymbol {\Sigma} \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {t}, \tag {23}
$$
$$
\boldsymbol {C} = \left(\boldsymbol {X} \boldsymbol {X} ^ {\top}\right) ^ {- 1} \left[ \boldsymbol {I} - \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} \boldsymbol {X} ^ {\top}\right) ^ {t} \right] \boldsymbol {X} \boldsymbol {\Sigma} \boldsymbol {X} ^ {\top} \left[ \boldsymbol {I} - \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} \boldsymbol {X} ^ {\top}\right) ^ {t} \right] \left(\boldsymbol {X} \boldsymbol {X} ^ {\top}\right) ^ {- 1}, \tag {24}
$$
which characterizes bias term and variance term in the excess risk. Furthermore, there exists constant  $c$  such that with probability at least  $1 - \delta$  over the randomness of  $\varepsilon$ , we have
$$
\varepsilon^ {\top} \boldsymbol {C} \varepsilon \leq c \sigma_ {y} ^ {2} \log \frac {1}{\delta} \operatorname {T r} [ \boldsymbol {C} ]. \tag {25}
$$
Proof. First note that  $\mathbf{X} \mathbf{X}^{\top}$  is invertible by Assumption 3. Express the excess risk as follows
$$
\begin{array}{l} R (\boldsymbol {\theta} _ {t}) = \frac {1}{2} \mathbb {E} [ (y - \boldsymbol {x} ^ {\top} \boldsymbol {\theta} _ {t}) ^ {2} - (y - \boldsymbol {x} ^ {\top} \boldsymbol {\theta} ^ {*}) ^ {2} ] \\ = \frac {1}{2} \mathbb {E} \left[ \left(y - \boldsymbol {x} ^ {\top} \boldsymbol {\theta} ^ {*} + \boldsymbol {x} ^ {\top} \boldsymbol {\theta} ^ {*} - \boldsymbol {x} ^ {\top} \boldsymbol {\theta} _ {t}\right) ^ {2} - \left(y - \boldsymbol {x} ^ {\top} \boldsymbol {\theta} ^ {*}\right) ^ {2} \right] \\ = \frac {1}{2} \mathbb {E} \left[ \left(\boldsymbol {x} ^ {\top} \left(\boldsymbol {\theta} _ {t} - \boldsymbol {\theta} ^ {*}\right)\right) ^ {2} + 2 \left(y - \boldsymbol {x} ^ {\top} \boldsymbol {\theta} ^ {*}\right) \left(\boldsymbol {x} ^ {\top} \boldsymbol {\theta} ^ {*} - \boldsymbol {x} ^ {\top} \boldsymbol {\theta} _ {t}\right) \right] \tag {26} \\ = \frac {1}{2} \mathbb {E} \left[ \boldsymbol {x} ^ {\top} \left(\boldsymbol {\theta} _ {t} - \boldsymbol {\theta} ^ {*}\right) \right] ^ {2}. \\ \end{array}
$$
Recall that  $\theta_0 = 0$  and  $Y = X\theta^{*} + \varepsilon$  and we can further simplify the formula for  $\theta_{t}$  in lemma A.2:
$$
\begin{array}{l} \boldsymbol {\theta} _ {t} = \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {t} \left(\boldsymbol {\theta} _ {0} - \boldsymbol {X} ^ {\dagger} \boldsymbol {Y}\right) + \boldsymbol {X} ^ {\dagger} \boldsymbol {Y} \tag {27} \\ = \left[ \boldsymbol {I} - \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {t} \right] \boldsymbol {X} ^ {\dagger} (\boldsymbol {X} \boldsymbol {\theta} ^ {*} + \varepsilon). \\ \end{array}
$$
Plug it into the above expression for  $R(\pmb{\theta}_t)$ , we have
$$
\begin{array}{l} R (\boldsymbol {\theta} _ {t}) = \frac {1}{2} \mathbb {E} \left[ \boldsymbol {x} ^ {\top} \left[ \boldsymbol {I} - \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {t} \right] \boldsymbol {X} ^ {\dagger} (\boldsymbol {X} \boldsymbol {\theta} ^ {*} + \varepsilon) - \boldsymbol {x} ^ {\top} \boldsymbol {\theta} ^ {*} \right] ^ {2} \\ = \frac {1}{2} \mathbb {E} \left[ \boldsymbol {x} ^ {\top} \left(\boldsymbol {X} ^ {\dagger} \boldsymbol {X} - \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {t} \boldsymbol {X} ^ {\dagger} \boldsymbol {X} - \boldsymbol {I}\right) \boldsymbol {\theta} ^ {*} \right. \tag {28} \\ + \boldsymbol {x} ^ {\top} \left[ \boldsymbol {I} - \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {t} \right] \boldsymbol {X} ^ {\dagger} \boldsymbol {\varepsilon} \biggr ] ^ {2}. \\ \end{array}
$$
Applying lemma A.3, we obtain
$$
\begin{array}{l} R (\boldsymbol {\theta} _ {t}) = \frac {1}{2} \mathbb {E} \left[ - \boldsymbol {x} ^ {\top} \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {t} \boldsymbol {\theta} ^ {*} + \boldsymbol {x} ^ {\top} \boldsymbol {X} ^ {\top} \left[ \boldsymbol {I} - \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} \boldsymbol {X} ^ {\top}\right) ^ {t} \right] (\boldsymbol {X} \boldsymbol {X} ^ {\top}) ^ {- 1} \boldsymbol {\varepsilon} \right] ^ {2} \\ \leq \mathbb {E} \left[ \boldsymbol {x} ^ {\top} \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {t} \boldsymbol {\theta} ^ {*} \right] ^ {2} + \mathbb {E} \left[ \boldsymbol {x} ^ {\top} \boldsymbol {X} ^ {\top} \left[ \boldsymbol {I} - \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} \boldsymbol {X} ^ {\top}\right) ^ {t} \right] (\boldsymbol {X} \boldsymbol {X} ^ {\top}) ^ {- 1} \boldsymbol {\varepsilon} \right] ^ {2} \\ := \boldsymbol {\theta} ^ {* ^ {\top}} \boldsymbol {B} \boldsymbol {\theta} ^ {*} + \varepsilon^ {\top} \boldsymbol {C} \varepsilon . \tag {29} \\ \end{array}
$$
which proves the first claim in the lemma. The second part of the theorem directly follows from lemma 18 in Bartlett et al. [8].
A.4 Proof for the Bias Upper Bound
The next lemma guarantees that the sample covariance matrix  $\frac{1}{n}\pmb{X}^{\top}\pmb{X}$  concentrates well around  $\pmb{\Sigma}$ .
Lemma A.5. (Lemma 35 in Bartlett et al. [8]) There exists constant  $c$  such that for any  $0 < \delta < 1$  with probability as least  $1 - \delta$ ,
$$
\left\| \boldsymbol {\Sigma} - \frac {1}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X} \right\| \leq c \| \boldsymbol {\Sigma} \| \max  \left\{\sqrt {\frac {r (\boldsymbol {\Sigma})}{n}}, \frac {r (\boldsymbol {\Sigma})}{n}, \sqrt {\frac {\log \left(\frac {1}{\delta}\right)}{n}}, \frac {\log \left(\frac {1}{\delta}\right)}{n} \right\}. \tag {30}
$$
The following inequality will be useful in our proof to characterize the decaying rate of the bias term with  $t$ .
Lemma A.6. For any positive semidefinite matrix  $\pmb{P}$  which satisfies  $\| \pmb{P} \| \leq 1$ , we have
$$
\left\| \boldsymbol {P} (1 - \boldsymbol {P}) ^ {t} \right\| \leq \frac {1}{t}. \tag {31}
$$
Proof. Assume without loss of generality that  $\pmb{P}$  is diagonal. Then it suffices to consider separately each eigenvalue  $\sigma$  of  $\pmb{P}$ , and show that  $\sigma (1 - \sigma)^t\leq \frac{1}{t}$ .
In fact, by AM-GM inequality we have
$$
\sigma (1 - \sigma) ^ {t} \leq \frac {1}{t} \left[ \frac {t \sigma + (1 - \sigma) t}{t + 1} \right] ^ {t + 1} \leq \frac {1}{t}, \tag {32}
$$
Next we prove the main result of this section.
Lemma A.7. There exists constant  $c$  such that if  $0 \leq \lambda \leq \frac{1}{c\sum_{i > 0}\lambda_i}$ , then for any  $0 < \delta < 1$ , with probability at least  $1 - \delta$  the following bound on the bias term holds for any  $t$
$$
\boldsymbol {\theta} ^ {* \top} \boldsymbol {B} \boldsymbol {\theta} ^ {*} \leq c \| \boldsymbol {\theta} ^ {*} \| ^ {2} \left(\frac {1}{\lambda t} + \| \boldsymbol {\Sigma} \| \max  \left\{\sqrt {\frac {r (\boldsymbol {\Sigma})}{n}}, \frac {r (\boldsymbol {\Sigma})}{n}, \sqrt {\frac {\log \left(\frac {1}{\delta}\right)}{n}}, \frac {\log \left(\frac {1}{\delta}\right)}{n} \right\}\right). \tag {33}
$$
Proof. The bias can be decomposed into the following two terms
$$
\begin{array}{l} \boldsymbol {\theta} ^ {* \top} \boldsymbol {B} \boldsymbol {\theta} ^ {*} = \boldsymbol {\theta} ^ {* \top} \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {t} \left(\boldsymbol {\Sigma} - \frac {1}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {t} \boldsymbol {\theta} ^ {*} \tag {34} \\ + \boldsymbol {\theta} ^ {* \top} \left(\frac {1}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {2 t} \boldsymbol {\theta} ^ {*}. \\ \end{array}
$$
For sufficiently small learning rate  $\lambda$  as given by corollary A.1, we know that with high probability
$$
\left\| \boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X} \right\| \leq 1, \tag {35}
$$
which together with lemma A.5 gives a high probability bound on the first term:
$$
\begin{array}{l} \boldsymbol {\theta} ^ {* \top} \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {t} \left(\boldsymbol {\Sigma} - \frac {1}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {t} \boldsymbol {\theta} ^ {*} \\ \leq c \| \boldsymbol {\Sigma} \| \| \boldsymbol {\theta} ^ {*} \| ^ {2} \max  \left\{\sqrt {\frac {r (\boldsymbol {\Sigma})}{n}}, \frac {r (\boldsymbol {\Sigma})}{n}, \sqrt {\frac {\log \left(\frac {1}{\delta}\right)}{n}}, \frac {\log \left(\frac {1}{\delta}\right)}{n} \right\}. \tag {36} \\ \end{array}
$$
For the second term, invoke lemma A.6 with  $P = \frac{\lambda}{n} X^{\top}X$  and we get
$$
\begin{array}{l} \boldsymbol {\theta} ^ {* \top} \left(\frac {1}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {2 t} \boldsymbol {\theta} ^ {*} \leq \frac {1}{\lambda} \| \boldsymbol {\theta} ^ {*} \| ^ {2} \left\| \left(\frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {2 t} \right\| \tag {37} \\ \leq \frac {1}{2 \lambda t} \| \boldsymbol {\theta} ^ {*} \| ^ {2}. \\ \end{array}
$$
Putting these two bounds together gives the proof for the main theorem.
A.5 Proof for the Variance Upper Bound
Recall that  $\pmb{X} = \pmb{U} \tilde{\Lambda}^{\frac{1}{2}} \pmb{W}^{\top}$  is the singular value decomposition of data matrix  $\pmb{X}$ , where  $\pmb{U} = (\pmb{u}_1, \dots, \pmb{u}_n)$ ,  $\pmb{W} = (\pmb{w}_1, \dots, \pmb{w}_n)$ ,  $\tilde{\Lambda} = \mathrm{diag}\{\mu_1, \dots, \mu_n\}$  with  $\mu_1 \geq \mu_2 \geq \dots \mu_n$ .
$$
k _ {0} = \min  \{l \geq 0: \lambda_ {l + 1} \leq \frac {c _ {0} \sum_ {i > l} \lambda_ {i}}{n} \},
$$
$$
k _ {1} = \min  \left\{l \geq 0: \lambda_ {l + 1} \leq \frac {c _ {1} \sum_ {i > 0} \lambda_ {i}}{n} \right\}, \tag {38}
$$
$$
k _ {2} = \min  \{l \geq 0: \sum_ {i > l} \lambda_ {i} + n \lambda_ {l + 1} \leq c _ {2} c (t, n) \sum_ {i > 0} \lambda_ {i} \} \},
$$
for some constant  $c_0, c_1, c_2$  and function  $c(t, n)$ .
We further define
$$
k _ {3} = \min  \left\{l \geq 0: \mu_ {l + 1} \leq c _ {3} c (t, n) \sum_ {i > 0} \lambda_ {i} \right\}, \tag {39}
$$
for some constant  $c_{3}$
The next lemma shows that we can appropriately choose constants to ensure that  $k_{3} \leq k_{2}$  holds with high probability, and in some specific cases we have  $k_{2} \leq k_{1}$ .
Lemma A.8. For any function  $c(t, n)$  and constant  $c_2$ , there exists constants  $c, c_3$ , such that  $k_3 \leq k_2$  with probability at least  $1 - e^{-\frac{n}{c}}$ . Furthermore, if  $c(t, n)$  is a positive constant function, for any  $c_1$ , there exists  $c_2$  such that  $k_2 \leq k_1$ .
Proof. According to lemma A.1, there exists a constant  $c$ , with probability at least  $1 - e^{-\frac{n}{c}}$  we have
$$
\mu_ {k _ {2} + 1} \leq c \left(\sum_ {i > k _ {2}} \lambda_ {i} + n \lambda_ {k _ {2} + 1}\right) \leq c c _ {2} c (t, n) \sum_ {i > 0} \lambda_ {i}. \tag {40}
$$
Therefore, we know that  $k_{3} \leq k_{2}$  for  $c_{3} = cc_{2}$ .
By the definition of  $k_{1}$ , we have
$$
\sum_ {i > k _ {1}} \lambda_ {i} + n \lambda_ {k _ {1} + 1} \leq \left(c _ {1} + 1\right) \sum_ {i > 0} \lambda_ {i}, \tag {41}
$$
which implies that  $k_{2} \leq k_{1}$  for  $c_{2} = \frac{c_{1} + 1}{c(t,n)}$ , if  $c(t,n)$  is a positive constant.
Next we bound  $\operatorname{Tr}[C]$ , which implies an upper bound on the variance term.
Theorem A.1. There exist constants  $c, c_0, c_1, c_2$  such that if  $k_0 \leq \frac{n}{c}$ , then with probability at least  $1 - e^{-\frac{n}{c}}$ , the trace of the variance matrix  $C$  has the following upper bound for any  $t$ :
$$
\operatorname {T r} [ \boldsymbol {C} ] \leq c \left(\frac {k _ {1}}{n} + \frac {k _ {2}}{c (t , n) n} + c (t, n) \left(\frac {\lambda t}{n} \sum_ {i > 0} \lambda_ {i}\right) ^ {2}\right). \tag {42}
$$
Proof. We divide the eigenvalues of  $\mathbf{X} \mathbf{X}^{\top}$  into two groups based on whether they are greater than  $c_3 c(t, n) \sum_{i > 0} \lambda_i$ . The first group consists of  $\mu_1 \cdots \mu_{k_3}$ , and the second group consists of  $\mu_{k_3 + 1} \cdots \mu_n$ . For  $1 \leq j \leq k_3$ , we have
$$
1 - \left(1 - \frac {\lambda}{n} \mu_ {j}\right) ^ {t} \leq 1. \tag {43}
$$
Therefore we have the following upper bound on  $\left[\boldsymbol{I} - \left(\boldsymbol{I} - \frac{\lambda}{n}\boldsymbol{X}\boldsymbol{X}^{\top}\right)^{t}\right]^{2}$ :
$$
\begin{array}{l} \left[ \boldsymbol {I} - \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} \boldsymbol {X} ^ {\top}\right) ^ {t} \right] ^ {2} \\ = \boldsymbol {U} \operatorname {d i a g} \left\{\left[ 1 - \left(1 - \frac {\lambda}{n} \mu_ {1}\right) ^ {t} \right] ^ {2} \dots \left[ 1 - \left(1 - \frac {\lambda}{n} \mu_ {n}\right) ^ {t} \right] ^ {2} \right\} \boldsymbol {U} ^ {\top} \\ \preceq \boldsymbol {U} \operatorname {d i a g} \left\{\overbrace {1 , \cdots 1} ^ {k _ {3} \text {t i m e s}}, \overbrace {\left[ 1 - \left(1 - \frac {\lambda}{n} \mu_ {k _ {3} + 1}\right) ^ {t} \right] ^ {2}, \cdots \left[ 1 - \left(1 - \frac {\lambda}{n} \mu_ {n}\right) ^ {t} \right] ^ {2}} ^ {n - k _ {3} \text {t i m e s}} \right\} \boldsymbol {U} ^ {\top} \tag {44} \\ = U \operatorname {d i a g} \left\{\overbrace {1 , \cdots 1} ^ {k _ {3} \text {t i m e s}}, \overbrace {0 , \cdots 0} ^ {n - k _ {3} \text {t i m e s}} \right\} U ^ {\top} \\ + \boldsymbol {U} \operatorname {d i a g} \left\{\overbrace {0 , \cdots 0} ^ {\text {k 3 t i m e s}}, \overbrace {\left[ 1 - \left(1 - \frac {\lambda}{n} \mu_ {k _ {3} + 1}\right) ^ {t} \right] ^ {2}, \cdots \left[ 1 - \left(1 - \frac {\lambda}{n} \mu_ {n}\right) ^ {t} \right] ^ {2}} ^ {n - k _ {3}} \right\} \boldsymbol {U} ^ {\top}. \\ \end{array}
$$
For positive semidefinite matrices  $P, Q, R$  which satisfies  $Q \preceq R$ , it holds that  $\operatorname{Tr}[PQ] \leq \operatorname{Tr}[PR]$ . It implies the following upperbound of  $\operatorname{Tr}[C]$ :
$$
\begin{array}{l} \operatorname {T r} [ C ] \\ = \mathrm {T r} \left[ \left[ \boldsymbol {I} - \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} \boldsymbol {X} ^ {\top}\right) ^ {t} \right] ^ {2} \left(\boldsymbol {X} \boldsymbol {X} ^ {\top}\right) ^ {- 2} \boldsymbol {X} \boldsymbol {\Sigma} \boldsymbol {X} ^ {\top} \right] \\ \leq \underbrace {\operatorname {T r} \left[ U \operatorname {d i a g} \left\{\overbrace {1 , \cdots 1} ^ {k _ {3} \text {t i m e s}}, \overbrace {0 , \cdots 0} ^ {n - k _ {3} \text {t i m e s}} \right\} U ^ {\top} \left(X X ^ {\top}\right) ^ {- 2} X \Sigma X ^ {\top} \right]} _ {①} \\ + \operatorname {T r} \underbrace {\left[ U \operatorname {d i a g} \left\{\overbrace {0 , \cdots 0} ^ {k _ {3} \text {t i m e s}}, \overbrace {\left[ 1 - \left(1 - \frac {\lambda}{n} \mu_ {k _ {3} + 1}\right) ^ {t} \right] ^ {2}} ^ {n - k _ {3} \text {t i m e s}} , \dots \left[ 1 - \left(1 - \frac {\lambda}{n} \mu_ {n}\right) ^ {t} \right] ^ {2} \right\} U ^ {\top} \left(\boldsymbol {X} \boldsymbol {X} ^ {\top}\right) ^ {- 2} \boldsymbol {X} \boldsymbol {\Sigma} \boldsymbol {X} ^ {\top}} \right\rbrack . \\ \end{array}
$$
Bounding ①
Noticing  $\pmb{X} = \pmb{U}\tilde{\Lambda}^{\frac{1}{2}}\pmb{W}^{\top}$  and  $\pmb {\Sigma} = \sum_{i\geq 1}\lambda_i\pmb {v}_i\pmb {v}_i^\top$  , we express the first term as sums of eigenvector products,
$$
\begin{array}{l} \mathbf {U} = \operatorname {T r} \left[ \boldsymbol {U} \mathrm {d i a g} \left\{\overbrace {1 , \cdots 1} ^ {k _ {3} \mathrm {t i m e s}}, \overbrace {0 , \cdots 0} ^ {n - k _ {3} \mathrm {t i m e s}} \right\} \boldsymbol {U} ^ {\top} \left(\boldsymbol {X} \boldsymbol {X} ^ {\top}\right) ^ {- 2} \boldsymbol {X} \boldsymbol {\Sigma} \boldsymbol {X} ^ {\top} \right] \\ = \operatorname {T r} \left[ \boldsymbol {U} \operatorname {d i a g} \left\{\overbrace {1 , \cdots 1} ^ {k _ {3} \text {t i m e s}}, \overbrace {0 , \cdots 0} ^ {n - k _ {3} \text {t i m e s}} \right\} \boldsymbol {U} ^ {\top} \boldsymbol {U} \tilde {\boldsymbol {\Lambda}} ^ {- 2} \boldsymbol {U} ^ {\top} \boldsymbol {U} \tilde {\boldsymbol {\Lambda}} ^ {\frac {1}{2}} \boldsymbol {W} ^ {\top} \boldsymbol {\Sigma} \boldsymbol {W} \tilde {\boldsymbol {\Lambda}} ^ {\frac {1}{2}} \boldsymbol {U} ^ {\top} \right] \\ = \operatorname {T r} \left[ \operatorname {d i a g} \left\{\overbrace {1 , \cdots 1} ^ {k _ {3} \text {t i m e s}}, \overbrace {0 , \cdots 0} ^ {n - k _ {3} \text {t i m e s}} \right\} \tilde {\boldsymbol {\Lambda}} ^ {- 1} \boldsymbol {W} ^ {\top} \boldsymbol {\Sigma} \boldsymbol {W} \right] \tag {46} \\ = \sum_ {i \geq 1} \lambda_ {i} \operatorname {T r} \left[ \operatorname {d i a g} \left\{\overbrace {1 , \cdots 1} ^ {k _ {3} \text {t i m e s}}, \overbrace {0 , \cdots 0} ^ {n - k _ {3} \text {t i m e s}} \right\} \tilde {\boldsymbol {\Lambda}} ^ {- 1} \boldsymbol {W} ^ {\top} \boldsymbol {v} _ {i} \boldsymbol {v} _ {i} ^ {\top} \boldsymbol {W} \right] \\ = \sum_ {i \geq 1} \sum_ {1 \leq j \leq k _ {3}} \frac {\lambda_ {i}}{\mu_ {j}} \left(\boldsymbol {v} _ {i} ^ {\top} \boldsymbol {w} _ {j}\right) ^ {2}. \\ \end{array}
$$
Next we divide the above summation into  $1 \leq i \leq k_{1}$  and  $i > k_{1}$ . For the first part, notice that
$$
\begin{array}{l} \sum_ {1 \leq j \leq k _ {3}} \frac {\lambda_ {i}}{\mu_ {j}} \left(\boldsymbol {v} _ {i} ^ {\top} \boldsymbol {w} _ {j}\right) ^ {2} \leq \sum_ {1 \leq j \leq n} \frac {\lambda_ {i}}{\mu_ {j}} \left(\boldsymbol {v} _ {i} ^ {\top} \boldsymbol {w} _ {j}\right) ^ {2} \\ = \lambda_ {i} \boldsymbol {v} _ {i} ^ {\top} \left(\sum_ {1 \leq j \leq n} \frac {1}{\mu_ {j}} \boldsymbol {w} _ {j} \boldsymbol {w} _ {j} ^ {\top}\right) \boldsymbol {v} _ {i} \tag {47} \\ = \lambda_ {i} \boldsymbol {v} _ {i} ^ {\top} \boldsymbol {W} \tilde {\Lambda} ^ {- 1} \boldsymbol {W} ^ {\top} \boldsymbol {v} _ {i} \\ = \lambda_ {i} \pmb {v} _ {i} ^ {\top} W \tilde {\Lambda} ^ {\frac {1}{2}} U ^ {\top} U \tilde {\Lambda} ^ {- 2} U ^ {\top} U \tilde {\Lambda} ^ {\frac {1}{2}} W ^ {\top} \pmb {v} _ {i} \\ = \lambda_ {i} ^ {2} \tilde {\boldsymbol {x}} _ {i} ^ {\top} \left(\boldsymbol {X} \boldsymbol {X} ^ {\top}\right) ^ {- 2} \tilde {\boldsymbol {x}} _ {i}, \\ \end{array}
$$
where  $\tilde{\pmb{x}}_i$  is defined as  $\tilde{\pmb{x}}_i = \frac{\pmb{X}\pmb{v}_i}{\sqrt{\lambda_i}} = \frac{\pmb{U}\tilde{\Lambda}^{\frac{1}{2}}\pmb{W}^\top\pmb{v}_i}{\sqrt{\lambda_i}}.$
From the proof of lemma 11 in Bartlett et al. [8], we know that for any  $\sigma_x$ , there exists a constant  $c_0$  and  $c$  such that if  $k_0 \leq \frac{n}{c}$ , with probability at least  $1 - e^{-\frac{n}{c}}$  the first part can be bounded as
$$
\sum_ {1 \leq i \leq k _ {1}} \sum_ {1 \leq j \leq k _ {3}} \frac {\lambda_ {i}}{\mu_ {j}} \left(\boldsymbol {v} _ {i} ^ {\top} \boldsymbol {w} _ {j}\right) ^ {2} \leq \sum_ {1 \leq i \leq k _ {1}} \lambda_ {i} ^ {2} \tilde {\boldsymbol {x}} _ {i} \left(\boldsymbol {X} \boldsymbol {X} ^ {\top}\right) ^ {- 2} \tilde {\boldsymbol {x}} _ {i} \leq c \frac {k _ {1}}{n}, \tag {48}
$$
which gives a bound for the first part.
For the second part we interchange the order of summation and have
$$
\begin{array}{l} \sum_ {i \geq k _ {1}} \sum_ {1 \leq j \leq k _ {3}} \frac {\lambda_ {i}}{\mu_ {j}} \left(\boldsymbol {v} _ {i} ^ {\top} \boldsymbol {w} _ {j}\right) ^ {2} = \sum_ {1 \leq j \leq k _ {3}} \sum_ {i \geq k _ {1}} \frac {\lambda_ {i}}{\mu_ {j}} \left(\boldsymbol {v} _ {i} ^ {\top} \boldsymbol {w} _ {j}\right) ^ {2} \\ \leq \frac {1}{c _ {3} c (t , n) \sum_ {i > 0} \lambda_ {i}} \sum_ {1 \leq j \leq k _ {3}} \sum_ {i \geq k _ {1}} \lambda_ {i} \left(\boldsymbol {v} _ {i} ^ {\top} \boldsymbol {w} _ {j}\right) ^ {2} \\ = \frac {\lambda_ {k _ {1} + 1}}{c _ {3} c (t , n) \sum_ {i > 0} \lambda_ {i}} \sum_ {1 \leq j \leq k _ {3}} \sum_ {i \geq k _ {1}} \left(\boldsymbol {v} _ {i} ^ {\top} \boldsymbol {w} _ {j}\right) ^ {2} \tag {49} \\ \leq \frac {\lambda_ {k _ {1} + 1}}{c _ {3} c (t , n) \sum_ {i > 0} \lambda_ {i}} \sum_ {1 \leq j \leq k _ {3}} 1 \\ = \frac {\lambda_ {k _ {1} + 1} k _ {3}}{c _ {3} c (t , n) \sum_ {i > 0} \lambda_ {i}} \\ \leq c \frac {k _ {3}}{c (t , n) n}. \\ \end{array}
$$
for  $c$  large enough.
Putting 48 and 49 together, and noting that  $k_{3} \leq k_{2}$  with high probability as given in lemma A.8, we know there exists a constant  $c$  that with probability at least  $1 - e^{-\frac{n}{c}}$ ,
$$
① \leq c \frac {k _ {1}}{n} + c \frac {k _ {2}}{c (t , n) n}. \tag {50}
$$
Bounding ②
Similar to the first step in bounding ①, we note that