SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We wish to compute the eigenfunctions  $\xi, \psi \in \mathcal{H}$ , as well as the eigenvalues  $\lambda_{i}$  of  $\hat{A}$ . The left eigenfunctions satisfy  $\hat{A}^{*}\xi_{i} = \bar{\lambda}_{i}\xi_{i}$  and the right eigenfunctions satisfy  $\hat{A}\psi_{i} = \lambda_{i}\psi_{i}$ . In the following we will use Mollenhauer et al. [43, Proposition 3] to manipulate the eigendecomposition of operators in  $\mathcal{H}$ .
Consider the decomposition  $W = U_{r}V_{r}^{*}$  with  $U_{r}, V_{r} \in \mathbb{C}^{m \times r}$ , which is available for all considered estimators with  $r \leq m$ . For example, in the Nyström RRR estimator of proposition C.4, we can simply take  $U_{r} = D_{r}$  and  $V_{r} = E_{r}$ . For the Nyström KRR estimator instead,  $r = m$  and we can take the whole of  $W$  as our  $U_{r}$  and  $V_{r} = I$ .
To compute the right eigenfunctions  $\psi_{i}$ , such that  $(\widetilde{\Phi}_{Y}U_{r}V_{r}^{*}\widetilde{\Phi}_{X}^{*})\psi_{i} = \lambda_{i}\psi_{i}$ , consider the following equivalent eigendecomposition
$$
V _ {r} ^ {*} \widetilde {\Phi} _ {X} ^ {*} \widetilde {\Phi} _ {Y} U _ {r} \tilde {g} _ {i} = \lambda_ {i} \tilde {g} _ {i}, \quad \text {w h e r e} \psi_ {i} = \widetilde {\Phi} _ {Y} U _ {r} \tilde {g} _ {i}.
$$
Note that  $\widetilde{\Phi}_X^*\widetilde{\Phi}_Y = K_{\tilde{X},\tilde{Y}}$  is a finite-dimensional object which can easily be computed. The eigenfunctions  $\psi_{i}$  must be normalized such that  $\psi_i^*\psi_i = 1$  for every  $i$ , so we must have
$$
\tilde {g} _ {i} ^ {*} U _ {r} ^ {*} \widetilde {\Phi} _ {Y} ^ {*} \widetilde {\Phi} _ {Y} U _ {r} \tilde {g} _ {i} = 1.
$$
A very similar process can be followed to obtain the left eigenfunctions  $\xi_{i}$ , such that  $\widetilde{\Phi}_X V_r U_r^* \widetilde{\Phi}_Y^* \xi_i = \bar{\lambda}_i \xi_i$ . Here we consider instead
$$
U _ {r} ^ {*} \widetilde {\Phi} _ {Y} ^ {*} \widetilde {\Phi} _ {X} V _ {r} \bar {h} _ {i} = \bar {\lambda} _ {i} \bar {h} _ {i}, \quad \text {w h e r e} \xi_ {i} = \widetilde {\Phi} _ {X} V _ {r} \bar {h} _ {i}.
$$
where once again,  $\widetilde{\Phi}_Y^*\widetilde{\Phi}_X = K_{\tilde{X},\tilde{Y}}^\top$  and the eigenfunctions must be normalized such that  $\tilde{h}_i^* V_r^*\widetilde{\Phi}_X^*\widetilde{\Phi}_X V_r\tilde{h}_i = 1$  for every  $i$ . Finally,  $\psi$  and  $\xi$  must be orthogonal to each other: we must have for  $i,j\in [r]$  that  $\langle \psi_i,\bar{\xi}_j\rangle_{\mathcal{H}} = \delta_{ij}$  (where  $\delta_{ij}$  is a Dirac delta equals to 1 when  $i = j$  and 0 otherwise). We can compute
$$
\langle \psi_ {i}, \bar {\xi} _ {j} \rangle_ {\mathcal {H}} = \tilde {h} _ {i} ^ {*} V _ {r} ^ {*} K _ {\tilde {X}, \tilde {Y}} U _ {r} \tilde {g} _ {i} = \lambda_ {j} \tilde {h} _ {i} ^ {*} \tilde {g} _ {j},
$$
and note that  $\tilde{h}_i^*\tilde{g}_j = \delta_{ij}$ , but we must normalize  $\xi$  such that
$$
\xi_ {i} = \widetilde {\Phi} _ {X} V _ {r} \tilde {h} _ {i} / \bar {\lambda} _ {i}.
$$
D.3 Koopman modes
Given the eigendecomposition of any estimator  $\hat{A}$  as  $\hat{A}_r = \sum_{i=1}^r \lambda_i \psi_i \otimes \bar{\xi}_i$ , for an observable  $g$  we have the following
$$
\hat {A} _ {r} ^ {*} g = \sum_ {i = 1} ^ {r} \lambda_ {i} \xi_ {i} \langle g, \bar {\psi} _ {i} \rangle_ {\mathcal {H}}
$$
where  $\langle g,\bar{\psi}_i\rangle_{\mathcal{H}} = \gamma_i^g$  are the Koopman modes. Expanding the definition of  $\psi_{i}$  we get
$$
\gamma_ {i} ^ {g} = \langle g, \bar {\psi} _ {i} \rangle_ {\mathcal {H}} = \tilde {g} _ {i} ^ {*} U _ {r} ^ {*} \widetilde {\Phi} _ {Y} ^ {*} g = \tilde {g} _ {i} ^ {*} U _ {r} ^ {*} g _ {m} \in \mathbb {C} ^ {m}
$$
which we can efficiently compute.
E Excess risk of the Nyström KRR estimator
E.1 Almost-sure decomposition of the KRR excess risk
Lemma E.1 (Excess risk decomposition in operator norm for KRR): Let Assumptions 4.1 to 4.3 and 4.5 hold. Then the Nyström KRR estimator (8) satisfies almost surely
$$
\begin{array}{l} \mathcal {E} \left(\hat {A} _ {m, \lambda} ^ {K R R}\right) ^ {1 / 2} \leq a \lambda^ {1 / 2} + a \theta_ {1} ^ {2} \| (\hat {C} _ {\lambda} - C _ {\lambda}) C _ {\lambda} ^ {- 1 / 2} \| _ {\mathcal {B} (\mathcal {H})} + \theta_ {1} ^ {2} \| \left(C _ {Y X} - \hat {C} _ {Y X}\right) C _ {\lambda} ^ {- 1 / 2} \| _ {\mathcal {B} (\mathcal {H})} \\ + a \theta_ {1} \theta_ {2} \theta_ {3} \| P _ {X} ^ {\perp} C _ {\lambda} ^ {1 / 2} \| _ {\mathcal {B} (\mathcal {H})} + \theta_ {1} ^ {2} \| P _ {Y} ^ {\perp} C _ {\lambda} ^ {1 / 2} \| _ {\mathcal {B} (\mathcal {H})} \\ \end{array}
$$
where  $\theta_{1} := \| \hat{C}_{\lambda}^{-1/2} C_{\lambda}^{1/2}\|$ ,  $\theta_{2} := \| \hat{C}_{\lambda}^{1/2} C_{\lambda}^{-1/2}\|$ ,  $\theta_{3} := \| \hat{C}_{\lambda}^{-1} C_{\lambda}\|$ , and  $a$  is the constant of Assumption 4.5.
Proof of Lemma E.1: Let  $\theta_{1} := \| \hat{C}_{\lambda}^{-1 / 2}C_{\lambda}^{1 / 2}\|$ ,  $\theta_{3} := \| \hat{C}_{\lambda}^{-1}C_{\lambda}\|$ . As in Lemma C.1 define  $g_{\mathrm{KRR}}(\hat{C}) := U(U^{*}\hat{C}U + \lambda I)^{-1}U^{*}$ . We have
$$
\begin{array}{l} \mathcal {E} \left(\hat {A} _ {m, \lambda} ^ {\mathrm {K R R}}\right) ^ {1 / 2} = \left\| \Phi_ {Y | X} - \hat {A} _ {m, \lambda} ^ {\mathrm {K R R}} \Phi_ {X} \right\| _ {\mathcal {B} \left(L _ {\pi} ^ {2}, \mathcal {H}\right)} \\ \leq \left\| \Phi_ {Y | X} - A _ {\lambda} \Phi_ {X} \right\| _ {\mathcal {B} \left(L _ {\pi} ^ {2}, \mathcal {H}\right)} + \left\| \left(A _ {\lambda} - C _ {Y X} g _ {\mathrm {K R R}} (\hat {C})\right) \Phi_ {X} \right\| _ {\mathcal {B} \left(L _ {\pi} ^ {2}, \mathcal {H}\right)} \\ + \left\| \left(C _ {Y X} g _ {\mathrm {K R R}} (\hat {C}) - \hat {A} _ {m, \lambda} ^ {\mathrm {K R R}}\right) \Phi_ {X} \right\| _ {\mathcal {B} \left(L _ {\pi} ^ {2}, \mathcal {H}\right)} \\ \leq \underbrace {\| \Phi_ {Y | X} - A _ {\lambda} \Phi_ {X} \| _ {\mathcal {B} (L _ {\pi} ^ {2} , \mathcal {H})}} _ {A} + \underbrace {\| (A _ {\lambda} - C _ {Y X} g _ {\mathrm {K R R}} (\hat {C})) C ^ {1 / 2} \|} _ {B} \\ + \underbrace {\| \left(C _ {Y X} g _ {\mathrm {K R R}} (\hat {C}) - \hat {A} _ {m , \lambda} ^ {\mathrm {K R R}}\right) C ^ {1 / 2} \|} _ {C} \tag {32} \\ \end{array}
$$
where we used the polar decomposition  $\Phi_X^* = WC^{1/2}$  for some partial isometry  $W: \mathcal{H} \to L_{\pi}^{2}$ . The first term is
$$
\begin{array}{l} \left\| \Phi_ {Y | X} - A _ {\lambda} \Phi_ {X} \right\| = \left\| \Phi_ {X} \mathcal {K} _ {\pi} ^ {*} - C _ {Y X} C _ {\lambda} ^ {- 1} \Phi_ {X} \right\| \\ \leq a \lambda^ {1 / 2} + \left\| \left(I - P _ {\mathcal {H}}\right) \Phi_ {Y | X} \right\| \\ \end{array}
$$
where we used the definition of  $\Phi_{Y|X}$  and applied Lemma H.1.
The second term of our decomposition (32) can be bounded as follows: It holds
$$
B = \left\| C _ {Y X} \left(C _ {\lambda} ^ {- 1} - g _ {\mathrm {K R R}} (\hat {C})\right) C ^ {1 / 2} \right\|
$$
(by Lemma H.2:)  $\leq a\| C(C_{\lambda}^{-1} - g_{\mathrm{KRR}}(\hat{C}))C^{1 / 2}\|$
$$
\leq a \left(\underbrace {\| C \left(C _ {\lambda} ^ {- 1} - \hat {C} _ {\lambda} ^ {- 1}\right) C ^ {1 / 2} \|} _ {B _ {1}} + \underbrace {\| C \left(\hat {C} _ {\lambda} ^ {- 1} - g _ {\mathrm {K R R}} (\hat {C})\right) C ^ {1 / 2} \|} _ {B _ {2}}\right)
$$
We now bound the terms  $B_{1}$  and  $B_{2}$  separately.
$$
\begin{array}{l} B _ {1} = \left\| C \left(C _ {\lambda} ^ {- 1} - \hat {C} _ {\lambda} ^ {- 1}\right) C ^ {1 / 2} \right\| \\ = \left\| C C _ {\lambda} ^ {- 1} \left(\hat {C} _ {\lambda} - C _ {\lambda}\right) \hat {C} _ {\lambda} ^ {- 1} C ^ {1 / 2} \right\| \\ \leq \left\| C C _ {\lambda} ^ {- 1} \right\| \left\| \left(\hat {C} _ {\lambda} - C _ {\lambda}\right) C _ {\lambda} ^ {- 1 / 2} \right\| \left\| C _ {\lambda} ^ {1 / 2} \hat {C} _ {\lambda} ^ {- 1 / 2} \right\| \left\| \hat {C} _ {\lambda} ^ {- 1 / 2} C ^ {1 / 2} \right\| \\ \leq \theta_ {1} ^ {2} \| (\hat {C} _ {\lambda} - C _ {\lambda}) C _ {\lambda} ^ {- 1 / 2} \| \\ \end{array}
$$
Let  $\hat{P}_{\lambda} \coloneqq \hat{C}_{\lambda}^{1/2} g_{\mathrm{KRR}}(\hat{C}) \hat{C}_{\lambda}^{1/2}$ . We recall that  $g_{\mathrm{KRR}}(\hat{C}) = P_X g_{\mathrm{KRR}}(\hat{C})$ , so that
$$
\begin{array}{l} \hat {P} _ {\lambda} ^ {2} = \hat {C} _ {\lambda} ^ {1 / 2} (g _ {\mathrm {K R R}} (\hat {C}) \hat {C} _ {\lambda} P _ {X}) g _ {\mathrm {K R R}} (\hat {C}) \hat {C} _ {\lambda} ^ {1 / 2} \\ = \hat {C} _ {\lambda} ^ {1 / 2} P _ {X} g _ {\mathrm {K R R}} (\hat {C}) \hat {C} _ {\lambda} ^ {1 / 2} \\ = \hat {P} _ {\lambda}. \\ \end{array}
$$
This implies  $\hat{P}_{\lambda}^{2} = \hat{P}_{\lambda} = \hat{P}_{\lambda}^{*}$ . Hence  $\hat{P}_{\lambda}$  is an orthogonal projection, and defining  $\hat{P}_{\lambda}^{\perp} = I - \hat{P}_{\lambda}$  it holds  $\| \hat{P}_{\lambda}^{\perp}\|_{\mathcal{B}(\mathcal{H})}\leq 1$ . We can thus bound  $B_{2}$  as follows:
$$
\begin{array}{l} B _ {2} = \left\| C \left(\hat {C} _ {\lambda} ^ {- 1} - g _ {\mathrm {K R R}} (\hat {C})\right) C ^ {1 / 2} \right\| \\ = \left\| C \hat {C} _ {\lambda} ^ {- 1 / 2} \left(I - \hat {P} _ {\lambda}\right) \hat {C} _ {\lambda} ^ {- 1 / 2} C ^ {1 / 2} \right\| \\ \end{array}
$$
(by Lemma I.1)
$$
\begin{array}{l} = \| C \hat {C} _ {\lambda} ^ {- 1} \| _ {\mathcal {B} (\mathcal {H})} \| P _ {X} ^ {\perp} C _ {\lambda} ^ {1 / 2} \| _ {\mathcal {B} (\mathcal {H})} \| C _ {\lambda} ^ {- 1 / 2} \hat {C} _ {\lambda} ^ {1 / 2} \| _ {\mathcal {B} (\mathcal {H})} \| I - \hat {P} _ {\lambda} \| \| \hat {C} _ {\lambda} ^ {- 1 / 2} C ^ {1 / 2} \| _ {\mathcal {B} (\mathcal {H})} \\ \leq \theta_ {1} \theta_ {2} \theta_ {3} \| P _ {\bar {X}} ^ {\perp} C _ {\lambda} ^ {1 / 2} \| \\ \end{array}
$$
For the third term, due to Lemma C.1:
$$
\begin{array}{l} C = \left\| \left(C _ {Y X} - P _ {Y} \hat {C} _ {Y X}\right) g _ {\mathrm {K R R}} (\hat {C}) C ^ {1 / 2} \right\| (33) \\ \leq \left\| \left(C _ {Y X} - P _ {Y} \hat {C} _ {Y X}\right) C _ {\lambda} ^ {- 1 / 2} \right\| \left\| C _ {\lambda} ^ {1 / 2} \hat {C} _ {\lambda} ^ {- 1 / 2} \right\| _ {\mathcal {B} (\mathcal {H})} \left\| \hat {P} _ {\lambda} \right\| _ {\mathcal {B} (\mathcal {H})} \left\| \hat {C} _ {\lambda} ^ {- 1 / 2} C ^ {1 / 2} \right\| _ {\mathcal {B} (\mathcal {H})} (34) \\ \leq \theta_ {1} ^ {2} \| \left(C _ {Y X} - P _ {Y} \hat {C} _ {Y X}\right) C _ {\lambda} ^ {- 1 / 2} \| _ {\mathcal {B} (\mathcal {H})} (35) \\ \leq \theta_ {1} ^ {2} \left(\| P _ {Y} ^ {\perp} C _ {\lambda} ^ {1 / 2} \| _ {\mathcal {B} (\mathcal {H})} + \| \left(C _ {Y X} - \hat {C} _ {Y X}\right) C _ {\lambda} ^ {- 1 / 2} \| _ {\mathcal {B} (\mathcal {H})}\right) (36) \\ \end{array}
$$
where we used Lemma J.7 for the last inequality.
Starting again from (32) and putting everything together, we get
$$
\begin{array}{l} \mathcal {E} (\hat {A} _ {m, \lambda} ^ {\mathrm {K R R}}) ^ {1 / 2} \leq a \lambda^ {1 / 2} \\ + a \theta_ {1} ^ {2} \| (\hat {C} _ {\lambda} - C _ {\lambda}) C _ {\lambda} ^ {- 1 / 2} \| _ {\mathcal {B} (\mathcal {H})} \\ + \theta_ {1} ^ {2} \| \left(C _ {Y X} - \hat {C} _ {Y X}\right) C _ {\lambda} ^ {- 1 / 2} \| _ {\mathcal {B} (\mathcal {H})} \\ + a \theta_ {1} \theta_ {2} \theta_ {3} \| P _ {X} ^ {\perp} C _ {\lambda} ^ {1 / 2} \| _ {\mathcal {B} (\mathcal {H})} \\ + \theta_ {1} ^ {2} \| P _ {Y} ^ {\perp} C _ {\lambda} ^ {1 / 2} \| _ {\mathcal {B} (\mathcal {H})}. \\ \end{array}
$$
E.2 Excess risk rates for KRR
In order to control the terms appearing in our decomposition, we recall that Assumption 4.4 implies
$$
d _ {\text {e f f}} (\lambda) \leq C _ {\beta} \lambda^ {- \beta} \text {w h e r e} C _ {\beta} := \left\{ \begin{array}{c c} \frac {c}{1 - \beta} & , \beta <   1 \\ K ^ {2} & , \beta = 1 \end{array} , \right. \tag {37}
$$
where  $c$  is the constant of Assumption 4.4, see [9, Proposition 3 with  $b \to 1 / \beta$  and  $\beta \to c$ ] and [18, Lemma 11] which shows that the existence of a constant  $C_{\beta}$  such that the first part of (37) holds implies in return  $\lambda_i(C) \lesssim i^{-1 / \beta}$ .
Proof of Theorem 4.6: By Lemma E.1 taking  $P_{X} = P_{Y}$ , it holds almost surely
$$
\begin{array}{l} \mathcal {E} \left(\hat {A} _ {m, \lambda} ^ {\mathrm {K R R}}\right) ^ {1 / 2} \leq a \lambda^ {1 / 2} + a \theta_ {1} ^ {2} \| (\hat {C} _ {\lambda} - C _ {\lambda}) C _ {\lambda} ^ {- 1 / 2} \| _ {\mathcal {B} (\mathcal {H})} + \theta_ {1} ^ {2} \| \left(C _ {Y X} - \hat {C} _ {Y X}\right) C _ {\lambda} ^ {- 1 / 2} \| _ {\mathcal {B} (\mathcal {H})} \\ + a \theta_ {1} \theta_ {2} \theta_ {3} \| P _ {X} ^ {\perp} C _ {\lambda} ^ {1 / 2} \| _ {\mathcal {B} (\mathcal {H})} + \theta_ {1} ^ {2} \| P _ {Y} ^ {\perp} C _ {\lambda} ^ {1 / 2} \| _ {\mathcal {B} (\mathcal {H})} \\ \end{array}
$$
and we recall that  $\theta_{1} := \| \hat{C}_{\lambda}^{-1/2} C_{\lambda}^{1/2}\|$  and  $\theta_{3} := \| \hat{C}_{\lambda}^{-1} C_{\lambda}\|$ . We bound separately the terms appearing in this expression.
Bound of  $\theta_{1}$  and  $\theta_{2}$ . We control these term by bounding  $\| C_{\lambda}^{-1 / 2}(\hat{C} -C)C_{\lambda}^{-1 / 2}\|$ . By Lemma J.3 it holds for any  $\delta^{\prime}\in ]0,1[$  and any  $\lambda \in ]0,\| C\|_{\mathcal{B}(\mathcal{H})}]$  with probability  $1 - \delta^{\prime}$
$$
\left\| C _ {\lambda} ^ {- 1 / 2} (\hat {C} - C) C _ {\lambda} ^ {- 1 / 2} \right\| \leq \frac {4 c _ {\tau} \beta}{3 n \lambda^ {\tau}} + \sqrt {\frac {2 c _ {\tau} \beta}{n \lambda^ {\tau}}} \quad \text {w h e r e} \quad \beta = \log \left(\frac {8 K ^ {2}}{\delta^ {\prime} \lambda}\right) \tag {38}
$$
A sufficient condition to bound the right hand side of the previous expression by  $1/4$  is to have  $n\lambda^{\tau} > 32c_{\tau}\beta$  (in which cases both terms are bounded by  $1/8$ ). Assuming this holds,  $I - \|C_{\lambda}^{-1/2}(\hat{C} -$
$C) C_{\lambda}^{-1/2} \parallel$  is invertible and we also have
$$
\begin{array}{l} \theta_ {2} ^ {2} = \left\| \hat {C} _ {\lambda} ^ {1 / 2} C _ {\lambda} ^ {- 1 / 2} \right\| ^ {2} = \left\| C _ {\lambda} ^ {- 1 / 2} \hat {C} _ {\lambda} C _ {\lambda} ^ {- 1 / 2} \right\| = \left\| I - C _ {\lambda} ^ {- 1 / 2} \left(C - \hat {C}\right) C _ {\lambda} ^ {- 1 / 2} \right\| \\ \leq 1 + \left\| C _ {\lambda} ^ {- 1 / 2} \left(C - \hat {C}\right) C _ {\lambda} ^ {- 1 / 2} \right\| \\ \leq 1. 2 5 \\ \end{array}
$$
and thus  $\theta_{2}\leq 1.12$
while  $\theta_1^2 = \| \hat{C}_\lambda^{-1 / 2}C_\lambda^{1 / 2}\| ^2 = \| (C_\lambda^{-1 / 2}\hat{C}_\lambda C_\lambda^{-1 / 2})^{-1}\|$
$$
\stackrel {(i)} {\leq} (1 - \left\| C _ {\lambda} ^ {- 1 / 2} (\hat {C} - C) C _ {\lambda} ^ {- 1 / 2} \right\|) ^ {- 1}
$$
$$
\leq 1. 3 4
$$
and thus  $\theta_{1}\leq 1.16$
where  $(i)$  can be obtained by taking the Neumann expansion of  $I - \| C_{\lambda}^{-1 / 2}(\hat{C} -C)C_{\lambda}^{-1 / 2}\|$ .
Bound for  $\theta_{3}$ . By Lemma J.5 it holds with probability  $1 - \delta'$
$$
\left\| \left(C - \hat {C}\right) C _ {\lambda} ^ {- 1} \right\| _ {\mathrm {o p}} \leq \frac {2 K \sqrt {c _ {\tau}} \log \left(^ {2} / \delta^ {\prime}\right)}{\lambda^ {(\tau + 1) / 2} n} + \sqrt {\frac {2 K ^ {2} \operatorname {t r} \left(C _ {\lambda} ^ {- 2} C\right) \log \left(^ {2} / \delta^ {\prime}\right)}{n}} \tag {39}
$$
Both terms in the above rhs are bounded by  $1 / 4$  provided
$$
\lambda^ {(\tau + 1) / 2} n \geq 8 K \sqrt {c _ {\tau}} \log (2 / \delta^ {\prime})
$$
$$
n \geq 3 2 K ^ {2} \lambda^ {- (1 + \beta)} \log \left(2 / \delta^ {\prime}\right)
$$
where we used  $\mathrm{tr}(C_{\lambda}^{-2}C) = \sum \lambda_i(C)(\lambda_i(C) + \lambda)^{-2} \leq \lambda^{-1}\mathrm{tr}(C_{\lambda}^{-1}C) \leq C_\beta \lambda^{-(1 + \beta)}$ . When this is the case, we have  $\| (C - \hat{C})C_{\lambda}^{-1}\|_{\mathrm{op}} \leq 1 / 2 < 1$  and the operator  $I - (\hat{C}_{\lambda} - C_{\lambda})C_{\lambda}^{-1}$  is invertible.
$$
\begin{array}{l} \theta_ {3} = \left\| \left(\hat {C} _ {\lambda} C _ {\lambda} ^ {- 1}\right) ^ {- 1} \right\| = \left\| \left(I - \left(\hat {C} _ {\lambda} - C _ {\lambda}\right) C _ {\lambda} ^ {- 1}\right) ^ {- 1} \right\| \\ \stackrel {(i)} {\leq} \left(1 - \left\| \left(\hat {C} _ {\lambda} - C _ {\lambda}\right) C _ {\lambda} ^ {- 1} \right\| _ {\mathcal {B} (\mathcal {H})}\right) ^ {- 1} \\ \leq 2. \\ \end{array}
$$
where  $(i)$  can be obtained by considering the Neumann expansion of  $I - (\hat{C}_{\lambda} - C_{\lambda})C_{\lambda}^{-1}$ .
Bound for  $\| P_X^\perp C_\lambda^{1/2}\|$ . By Lemma J.6, provided  $\lambda \in ]0, \|C\|_{\mathcal{B}(\mathcal{H})}]$  it holds with probability  $1 - \delta'$
$$
\left\| P _ {X} ^ {\perp} C _ {\lambda} ^ {1 / 2} \right\| _ {\mathcal {B} (\mathcal {H})} \leq \sqrt {3 \lambda}
$$
provided  $m \geq \max(67, 5\operatorname{ess}\sup_{x \sim \pi} \|C_{\lambda}^{-1/2}\phi(x)\|^2) \log \frac{4K^2}{\lambda\delta'}$ , which by Lemma H.3 is ensured if  $m \geq \max(67, 5\frac{c_{\tau}}{\lambda^{\tau}}) \log \frac{4K^2}{\lambda\delta'}$ .
Bound for  $\| (C - \hat{C})C_{\lambda}^{-1 / 2}\|$  and  $\| (C_{YX} - \hat{C}_{YX})C_{\lambda}^{-1 / 2}\|$ . By Lemma J.4, for any  $\delta^{\prime}\in ]0,1[$ , each of the following events holds with probability  $1 - 2\delta^{\prime}$ :
$$
\max  \left(\| (C - \hat {C}) C _ {\lambda} ^ {- 1 / 2} \|, \| \left(C _ {Y X} - \hat {C} _ {Y X}\right) C _ {\lambda} ^ {- 1 / 2} \|\right) \leq \frac {2 K \sqrt {c _ {\tau}} \log \left(2 / \delta^ {\prime}\right)}{\lambda^ {\tau / 2} n} + \sqrt {\frac {2 K ^ {2} d _ {\text {e f f}} (\lambda) \log \left(2 / \delta^ {\prime}\right)}{n}} \tag {40}
$$
By Eq. (37) we have  $d_{\mathrm{eff}}(\lambda)\leq C_{\beta}\lambda^{-\beta}$ .
Choosing  $\delta' = \delta / 5$ , we get via a union bound with probability  $1 - \delta$  that  $\theta_1 \theta_2 \theta_3 \leq 2.6$ ,  $\theta_1^2 \leq 1.34$  and
$$
\begin{array}{l} \mathcal {E} \left(\hat {A} _ {m, \lambda} ^ {\mathrm {K R R}}\right) ^ {1 / 2} \leq a \lambda^ {1 / 2} + 1. 3 4 (a + 1) \left(\frac {2 K \sqrt {c _ {\tau}} \log \left(2 / \delta^ {\prime}\right)}{\lambda^ {\tau / 2} n} + \sqrt {\frac {2 K ^ {2} C _ {\beta} \log \left(2 / \delta^ {\prime}\right)}{n \lambda^ {\beta}}}\right) + (2. 6 a + 1. 3 4) \sqrt {3} \lambda^ {1 / 2} \\ \leq c _ {1} \lambda^ {1 / 2} + c _ {2} \lambda^ {- \tau / 2} n ^ {- 1} + c _ {3} \lambda^ {- \beta / 2} n ^ {- 1 / 2} \\ \end{array}
$$
where:  $c_{1} \coloneqq (5.5a + 2.33)$
$$
c _ {2} := 1. 3 4 (a + 1) 2 K \sqrt {c _ {\tau}} \log \left(2 / \delta^ {\prime}\right)
$$
$$
c _ {3} := 1. 3 4 (a + 1) \sqrt {2 K ^ {2} C _ {\beta} \log \left(2 / \delta^ {\prime}\right)}
$$
for any  $\lambda$  and  $m$  satisfying the constraints
$$
\left\{ \begin{array}{l} \lambda > n ^ {- 1 / \tau} \left(3 2 c _ {\tau}\right) ^ {1 / \tau} \log \left(\frac {8 K ^ {2}}{\delta^ {\prime} \lambda}\right) ^ {1 / \tau} \\ \lambda \geq n ^ {- 2 / (\tau + 1)} \left(8 K \sqrt {c _ {\tau}} \log \left(2 / \delta^ {\prime}\right)\right) ^ {2 / (\tau + 1)} \\ \lambda \geq n ^ {- 1 / (1 + \beta)} \left(3 2 K ^ {2} \log \left(2 / \delta^ {\prime}\right)\right) ^ {1 / (1 + \beta)} \\ \lambda \in ] 0, K ^ {2} ]. \\ m \geq \max  \left(6 7, 5 \frac {c _ {\tau}}{\lambda^ {\tau}}\right) \log \frac {4 K ^ {2}}{\lambda \delta^ {\prime}} \quad (\text {u n i f o r m s a m p l i n g}) \end{array} \right. \tag {41}
$$
We pick  $\lambda := c_{\lambda} n^{-1/(1+\beta)}$  which is asymptotically the saturating constraint (given that  $1/(1+\beta) < 1 < 2/(\tau+1) \leq 1/\tau$ ), where  $c_{\lambda}$  is a constant chosen to enforce the following equations (which are sufficient conditions for eq. (41) to hold):
$$
\left\{ \begin{array}{c} c _ {\lambda} ^ {\tau} n ^ {1 - \tau / (1 + \beta)} > \left(3 2 c _ {\tau}\right) \log \left(\frac {8 K ^ {2} n ^ {1 / (1 + \beta)}}{\delta^ {\prime} c _ {\lambda}}\right) \\ c _ {\lambda} ^ {(\tau + 1) / 2} n ^ {1 - (\tau + 1) / (2 (1 + \beta))} \geq 8 K \sqrt {c _ {\tau}} \log \left(^ {2} / \delta^ {\prime}\right) \\ c _ {\lambda} \geq \left(3 2 K ^ {2} \log \left(^ {2} / \delta^ {\prime}\right)\right) ^ {1 / (1 + \beta)} \\ c _ {\lambda} n ^ {- 1 / (1 + \beta)} \leq K ^ {2} \end{array} \right. \tag {42}
$$
As  $1 - (\tau + 1) / (2(1 + \beta)) > 0$ , a sufficient condition for the second equation is
$$
c _ {\lambda} \geq \left(8 K \sqrt {c _ {\tau}} \log \left(2 / \delta^ {\prime}\right)\right) ^ {2 / (\tau + 1)}.
$$
Assuming  $c_{\tau} \geq 8K^2$ , a sufficient condition to satisfy the first constraint is
$$
c _ {\lambda} ^ {\tau} n ^ {1 - \tau / (1 + \beta)} > (3 2 c _ {\tau}) 2 \max  \left(\log \left(n ^ {1 / (1 + \beta)}\right), \log \left(\left(\delta^ {\prime}\right) ^ {- 1}\right)\right)
$$
which is in particular ensured (noting that  $\log (n) / n^{\nu}\leq 1 / (\nu e)$  for any  $n,\nu >0$ ) whenever
$$
c _ {\lambda} > \left(6 4 c _ {\tau} \max  \left(\left(e (1 + \beta - \tau)\right) ^ {- 1}, \log \left(1 / \delta^ {\prime}\right)\right)\right) ^ {1 / \tau}
$$
Noting that  $1 + \beta -\tau \leq 1$  , we get that
$$
c _ {\lambda} > \left(6 4 c _ {\tau} \left(e (1 + \beta - \tau)\right) ^ {- 1} \log \left(1 / \delta^ {\prime}\right)\right) ^ {1 / \tau}
$$
is also sufficient. We recall that  $1 / (1 + \beta) < 1 < 2 / (\tau +1)\leq 1 / \tau$ , so that we can choose
$$
\boxed {c _ {\lambda} := \log (2 / \delta^ {\prime}) ^ {1 / \tau} \max ((3 2 K ^ {2}) ^ {1 / (1 + \beta)}, (8 K \sqrt {c _ {\tau}}) ^ {2 / (\tau + 1)}, (6 4 c _ {\tau} (e (1 + \beta - \tau)) ^ {- 1}) ^ {1 / \tau}, 8 K ^ {2})}
$$
while the last constraint  $n \geq (c_{\lambda} / K^2)^{1 + \beta}$  is satisfied by assumption.
$$
\begin{array}{l} \mathcal {E} \left(\hat {A} _ {m, \lambda} ^ {\mathrm {K R R}}\right) ^ {1 / 2} \leq c _ {1} \lambda^ {1 / 2} + c _ {2} \lambda^ {- \tau / 2} n ^ {- 1} + c _ {3} \lambda^ {- \beta / 2} n ^ {- 1 / 2} \\ \leq c _ {1} c _ {\lambda} ^ {1 / 2} n ^ {- 1 / (2 (1 + \beta))} + c _ {2} c _ {\lambda} ^ {- \tau / 2} n ^ {\tau / (2 (1 + \beta)) - 1} + c _ {3} c _ {\lambda} ^ {- \beta / 2} n ^ {\beta / (2 (1 + \beta)) - 1 / 2} \\ \leq c _ {1} c _ {\lambda} ^ {1 / 2} n ^ {- 1 / (2 (1 + \beta))} + c _ {2} c _ {\lambda} ^ {- \tau / 2} n ^ {- (1 + 2 \beta + (1 - \tau)) / (2 (1 + \beta))} + c _ {3} c _ {\lambda} ^ {- \beta / 2} n ^ {- 1 / (2 (1 + \beta))} \\ \leq \left(c _ {1} c _ {\lambda} ^ {1 / 2} + c _ {2} c _ {\lambda} ^ {- \tau / 2} + c _ {3} c _ {\lambda} ^ {- \beta / 2}\right) n ^ {- 1 / (2 (1 + \beta))}. \\ \end{array}
$$
which gives the claimed result. The last constraint (on  $m$ ) is satisfied by the assumptions of the lemma.
F Excess risk of the Nyström RRR estimator
Recalling (30), NyströmRRR estimator is of the form  $\hat{A}_{m,\lambda}^{\mathrm{RRR}} = [[\tilde{B}]]_r(\tilde{C}_\lambda)^{-1/2}$ , where  $\tilde{B} \coloneqq \tilde{C}_{YX}(\tilde{C}_\lambda)^{-1/2}$  for  $\tilde{C}_{YX} \coloneqq P_Y\hat{C}_{YX}P_X$  and  $\tilde{C}_\lambda \coloneqq P_X\hat{C}P_X + \lambda I$ . While the population version is  $A_\lambda^{\mathrm{RRR}} \coloneqq [[B]]_rC_\lambda^{-1/2}$  where  $B \coloneqq C_{YX}(C_\lambda)^{-1/2}$ .
In this section we follow the approach in [30] and decompose the operator norm excess risk in the following way:
$$
\mathcal {E} \left(\hat {A} _ {m, \lambda} ^ {\mathrm {R R R}}\right) ^ {1 / 2} = \| \Phi_ {Y | X} - A _ {\lambda} \Phi_ {X} \| _ {\mathcal {B} \left(L _ {\pi} ^ {2}, \mathcal {H}\right)} + \| \left(A _ {\lambda} - A _ {\lambda} ^ {\mathrm {R R R}}\right) \Phi_ {X} \| _ {\mathcal {B} \left(L _ {\pi} ^ {2}, \mathcal {H}\right)} + \| \left(A _ {\lambda} ^ {\mathrm {R R R}} - \hat {A} _ {m, \lambda} ^ {\mathrm {R R R}}\right) \Phi_ {X} \| _ {\mathcal {B} \left(L _ {\pi} ^ {2}, \mathcal {H}\right)}
$$
Then, recalling that  $A_{\lambda} = C_{YX}C_{\lambda}^{-1}$  and  $\hat{A}_{m,\lambda}^{\mathrm{RRR}} = \tilde{B}\tilde{C}_{\lambda}^{-1 / 2}$ , we also have  $A_{\lambda}^{\mathrm{RRR}} = P_B A_{\lambda}$  and  $\hat{A}_{m,\lambda}^{\mathrm{RRR}} = P_{\tilde{B}}\hat{A}_{m,\lambda}^{\mathrm{RRR}}$ , where  $P_B$  and  $P_{\tilde{B}}$  are orthogonal projectors onto leading  $r$  left singular vectors of  $B$  and  $\tilde{B}$ , respectively.