SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Assumption 4.2 (Universal kernel): The kernel  $k$  is universal, i.e.  $\operatorname{cl}(\operatorname{ran}(\Phi_X^*)) = L_\pi^2$ .
We refer the reader to [60, Definition 4.52] for a definition of a universal kernel. The third assumption on the RKHS is related to the embedding property from Fischer and Steinwart [18], connected to the embedding of interpolation spaces. For a detailed discussion see Section A.3.
Assumption 4.3 (Embedding property): There exists  $\tau \in ]0,1]$  and  $c_{\tau} > 0$  such that  $\operatorname{ess}\sup_{\boldsymbol{x} \sim \pi} \| C_{\lambda}^{-1/2} \phi(\boldsymbol{x}) \|^2 \leq c_{\tau} \lambda^{-\tau}$ .
Next, we make an assumption on the decay of the spectrum of the covariance operator that is of paramount importance for derivation of optimal learning bounds. In the following,  $\lambda_{i}(A)$  and  $\sigma_{i}(A)$  always denote the eigenvalues and singular values of an operator  $A$  (in decreasing order).
Assumption 4.4 (Spectral decay): There exists  $\beta \in ]0,\tau ]$  and  $c > 0$  such that  $\lambda_{i}(C)\leq ci^{-1 / \beta}$
This assumption is common in the literature, and we will see that the optimal learning rates depend on  $\beta$ . It implies the bound  $d_{\mathrm{eff}}(\lambda) \coloneqq \operatorname{tr}(C_{\lambda}^{-1}C) \lesssim \lambda^{-\beta}$  on the effective dimension, which is a key quantity in the analysis (both statements are actually equivalent, see Section E.2). Note that  $d_{\mathrm{eff}}(\lambda) = \mathbf{E}_{\boldsymbol{x} \sim \pi} \| C_{\lambda}^{-1/2} \phi(\boldsymbol{x}) \| \leq \operatorname{ess} \sup_{\boldsymbol{x} \sim \pi} \| C_{\lambda}^{-1/2} \phi(\boldsymbol{x}) \|$ , and thus it necessarily holds  $\beta \leq \tau$ . For a Gaussian kernel, both  $\beta$  and  $\tau$  can be chosen arbitrarily close to zero.
Finally, we make an assumption about the regularity of the problem itself. A common assumption occurring in the literature is that  $\mathbf{E}[f(X_1) \mid \bar{X}_0 = \cdot] \in \mathcal{H}$  for every  $f \in \mathcal{H}$ , meaning that one can define the Koopman operator directly on the space  $\mathcal{H}$ , i.e. the learning problem is well-specified. However, this assumption is often too strong. Following [30, D.1] we make a different assumption on the cross-covariance remarking that, irrespectively of the choice of RKHS, it holds true whenever the Koopman operator is self-adjoint (i.e. the dynamics is time-reversible).
Assumption 4.5 (Regularity of  $\kappa_{\pi}$ ): There exists  $a > 0$  such that  $C_{XY}C_{XY}^{*} \preccurlyeq a^{2}C^{2}$ .
Rates The risk can be decomposed as  $\mathcal{R}_{\mathrm{HS}}(A) = \mathcal{E}_{\mathrm{HS}}(A) + \mathcal{R}_{\mathrm{HS},0}$  where  $\mathcal{R}_{\mathrm{HS},0}$  is a constant and  $\mathcal{E}_{\mathrm{HS}}(A) \coloneqq \|\mathcal{K}_{\pi} \Phi_X^* - \Phi_X^* A^*\|_{\mathrm{HS}}^2$  corresponds to the excess risk (more details in Section B). Optimal learning bounds for the KRR estimator in the context of CME (i.e. in Hilbert-Schmidt norm) have been developed in [33] under Assumptions 4.1 to 4.4 in well-specified and misspecified settings. On the other hand, in the context of dynamical systems, Kostic et al. [29, Theorem 1] report the importance of reduced rank estimators that have a small excess risk in operator norm
$$
\mathcal {E} (A) := \left\| \mathcal {K} _ {\pi} \Phi_ {X} ^ {*} - \Phi_ {X} ^ {*} A ^ {*} \right\| _ {\mathcal {H} \rightarrow L _ {\pi} ^ {2}} ^ {2}. \tag {15}
$$
The rationale behind considering the operator norm is that it allows to control the error of the eigenvalues approximation and thus of the KMD (3), (4) as discussed below. Optimal learning bounds in operator norm for KRR, PCR and RRR are established in [30]. In this work we show that the same optimal rates remain valid for the Nyström KRR, PCR and RRR estimators. According to [29] and [30] these operator norm bounds lead to reliable approximation of the Koompan mode decomposition of Eq. (4).
We now provide our main result.
Theorem 4.6 (Operator norm error for KRR, i.i.d. data): Let assumptions 4.1 to 4.5 hold. Let  $(\pmb{x}_i,\pmb{y}_i)_{1\leq i\leq n}$  be i.i.d. samples, and let  $P_{Y} = P_{X}$  be the projection induced by m Nyström landmarks drawn uniformly from  $(\pmb{x}_i)_{1\leq i\leq n}$  without replacement. Let  $\lambda = c_{\lambda}n^{-1 / (1 + \beta)}$  where  $c_{\lambda}$  is a constant
given in the proof, and assume  $n \geq (c_{\lambda} / K^2)^{1 + \beta}$ . Then it holds with probability at least  $1 - \delta$
$$
\mathcal {E} (\hat {A} _ {m, \lambda} ^ {K R R}) ^ {1 / 2} \lesssim n ^ {- \frac {1}{2 (1 + \beta)}} \quad p r o v i d e d \quad m \gtrsim \max  (1, n ^ {\tau / (1 + \beta)}) \log (n / \delta).
$$
The proof is provided in Section E.2, but essentially relies on a decomposition involving the terms  $\| C_{\lambda}^{-1/2}(C_{YX} - \hat{C}_{YX}) \|$ ,  $\| C_{\lambda}^{-1/2}(C - \hat{C}) \|$ ,  $\| C_{\lambda}^{-1/2}(C - \hat{C}) C_{\lambda}^{-1/2} \|$ , as well as bounding the quantity  $\| P_X^\perp C^{1/2} \|$  where  $P_X^\perp$  denotes the projection on the orthogonal of  $\mathrm{ran}(P_X)$ . All these terms are bounded using two variants of the Bernstein inequality. Note that our results can easily be extended to leverage score sampling of the landmarks by bounding term  $\| P_X^\perp C^{1/2} \|$  by means of [53, Lemma 7]; the same rate could then be obtained using a smaller number  $m$  of Nyström points.
The rate  $n^{-1/(2(1+\beta))}$  is known to be optimal (up to the log factor) in this setting by assuming an additional lower bound on the decay of the covariance's eigenvalues of the kind  $\lambda_i(C) \gtrsim i^{-1/\beta}$ , see [30, Theorem 7 in D.4]. One can see that without particular assumptions  $(\beta = \tau = 1)$ , we only need the number  $m$  of inducing points to be of the order of  $\Omega(\sqrt{n})$  in order to get an optimal rates. For  $\tau$  fixed, this number increases when  $\beta$  decreases (faster decay of the covariance's spectrum), however note that the optimal rate depends on  $\beta$  and also improves in this case. The dependence in  $\tau$  is particularly interesting, as for instance with a Gaussian kernel it is known that  $\tau$  can be chosen arbitrarily closed to zero [33, 18]. In that case, the number  $m$  of inducing points can be taken on the order of  $\Omega(\log n)$ .
Note that a bound for the Nyström KRR estimator has been derived in Hilbert-Schmidt norm by Ahmad et al. [3]. Using the operator norm however allows to derive bounds on the eigenvalues (see discussion below), which is of paramount importance for practical applications. Moreover, we now provide a bound on the error of PCR and RRR estimators, which are not covered in [3].
Lemma 4.7 (Operator norm error for PCR and RRR, i.i.d. data): Under the assumptions of Theorem 4.6, taking  $\lambda = c_{\lambda}n^{-1 / (1 + \beta)}$  with  $c_{\lambda}$  as in Theorem 4.6,  $n\geq (c_{\lambda} / K^{2})^{1 + \beta}$ , and provided
$$
m \gtrsim \max  (1, n ^ {\tau / (1 + \beta)}) \log (n / \delta),
$$
it holds with probability at least  $1 - \delta$
$$
\mathcal {E} (\hat {A} _ {m, \lambda} ^ {R R R}) ^ {1 / 2} \lesssim c _ {\mathrm {R R R}} n ^ {- \frac {1}{2 (1 + \beta)}}, f o r r s. t. \sigma_ {r + 1} (\Phi_ {X} \mathcal {K} _ {\pi} ^ {*}) <   \min (\sigma_ {r} (\Phi_ {X} \mathcal {K} _ {\pi} ^ {*}), n ^ {- \frac {1}{2 (1 + \beta)}})
$$
and  $\mathcal{E}(\hat{A}_m^{PCR})^{1/2} \lesssim c_{\mathrm{PCR}} n^{-\frac{1}{2(1 + \beta)}},$  for  $r > n^{\frac{1}{\beta(1 + \beta)}}$ ,
where  $c_{\mathrm{RRR}} = (\sigma_r^2(\Phi_X \mathcal{K}_\pi^*) - \sigma_{r+1}^2(\Phi_X \mathcal{K}_\pi^*))^{-1}$  and  $c_{\mathrm{PCR}} = (\sigma_r(\Phi_X) - \sigma_{r+1}(\Phi_X))^{-1}$  are the problem dependant constants.
Note that when rank of  $\mathcal{K}_{\pi}$  is  $r$ , then there is no restriction on  $r$  for the RRR estimator, while for PCR the choice of  $r$  depends on the spectral decay property of the kernel. In general, if  $r > n^{\frac{1}{\beta(1 + \beta)}}$ , then  $\sigma_{r+1}(\Phi_X \mathcal{K}_{\pi}^*) \leq \sigma_{r+1}(\Phi_X) \lesssim n^{-1/(2(1 + \beta))}$ , which implies that RRR estimator can achieve the same rate of PCR but with smaller rank. Again the rate is sharp (up to the log factor) in this setting [30].
Koopman mode decomposition According to [29, Theorem 1], working in operator norm allows us to bound the error of our estimators for dynamic mode decomposition, as well as to quantify how close the eigenpairs  $(\hat{\lambda}_i,\hat{\varphi}_i)$  of an estimator  $\hat{A}^*$  are to being eigenpairs of the Koopman operator. Namely, recalling that for function  $\hat{\varphi}_i$ , the corresponding candidate for Koopman eigenfunction in  $L_{\pi}^{2}$  space is  $\Phi_X^*\hat{\varphi}_i$ , one has  $\| \mathcal{K}_{\pi}(\Phi_X^*\hat{\varphi}_i) - \hat{\lambda}_i(\Phi_X^*\hat{\varphi}_i)\| /\| \Phi_X^*\hat{\varphi}_i\| \leq \mathcal{E}(\hat{A})^{1 / 2}\| \hat{\varphi}_i\| /\| \Phi_X^*\hat{\varphi}_i\|$ . While eigenvalue and eigenfunction learning rates were studied, under additional assumptions, in [30], where the operator norm error rates were determinant, here, in Section 5, we empirically show that the proposed estimators accurately learn the Koopman spectrum. We refer the reader to Section D for the details on computation of eigenvalues, eigenfunctions and KMD of an estimator in practice.
Dealing with non-i.i.d. data The previous results hold for i.i.d. data, which is not a very realistic assumption when learning from sampled trajectories. Our results can however easily be extended to  $\beta$ -mixing processes by considering random variables  $Z_{i} = \sum_{j=1}^{k} X_{i+j}$  (thus representing portions of the trajectory) sufficiently separated in time to be nearly independent. We now consider a trajectory
$\pmb{x}_1, \ldots, \pmb{x}_{n+1}$  with  $\pmb{x}_1 \sim \pi$  and  $\pmb{x}_{t+1} \sim p(\pmb{x}_t, \cdot)$  for  $t \in [1, n]$ , and use Lemma J.8 (re-stated from [29]) which allows to translate concentration results on the  $Z_i$  to concentration on the  $X_i$  by means of the  $\beta$ -mixing coefficients defined as  $\beta_X(k) := \sup_{B \in \mathcal{B} \otimes \mathcal{B}} |\rho_k(B) - (\pi \times \pi)(B)|$  where  $\rho_k$  denotes the joint probability of  $(X_t, X_{t+k})$ . Using this result the concentration results provided in appendix can thus be generalized to the  $\beta$ -mixing setting, and apart from logarithmic dependencies we essentially obtain similar results to the i.i.d. setting except that the sample size  $n$  is replaced by  $p \approx n / (2k)$ .
5 Experimental validation
In this section we show how the estimators proposed in section 3 perform in various scenarios, ranging from synthetic low dimensional ODEs to large-scale molecular dynamics simulations. The code for reproducing all experiments is available online. Our initial aim is to demonstrate the speed of NysPCR and NysRRR, compared to the recently proposed alternative Streaming KAF (sKAF) [20]. Then we show that their favorable scaling properties make it possible to train on large molecular dynamics datasets without any subsampling. In particular we run a metastability analysis of the alanine dipeptide and the Trp-cage protein, showcasing the accuracy of our models' eigenvalue and eigenfunction estimates, as well as their efficiency on massive datasets ( $>500000$  points)
Efficiency Benchmarks on Lorenz '63 The chaotic Lorenz '63 system [36] consists of 3 ODEs with no measurement noise. With this toy dynamical system we can easily compare the Nyström estimators to two alternatives: 1. the corresponding exact estimators and 2. the sKAF algorithm which also uses randomized linear algebra to improve the efficiency of PCR. In this setting we sample long trajectories from the system, keeping the first points for training (the number of training points varies for the first experiment, and is fixed to 10 000 for the second, see fig. 2), and the subsequent ones for testing. In fig. 1 we compare the run-time and accuracy with of NysPCR and NysRRR versus their full counterparts. To demonstrate the different scaling regimes we fix the number of inducing points to 250 and increase the number of data points  $n$ . The accuracy of the two solvers (as measured with the normalized RMSE metric (nRMSE) [20] on the first variable) is identical for PCR and close for RRR, but the running time of the approximate solvers increases much slower with  $n$  than that of the exact solvers. Each experiment is repeated 20 times to display error bars over the choice of Nyström centers. In the second experiment, shown in fig. 2, we reproduce the setting of [20] by training at increasingly long forecast horizons. Plotting the nRMSE we verify that sKAF and NysPCR converge to very similar accuracy values, although NysPCR is approximately 10 times faster. NysRRR instead offers slightly better accuracy, at the expense of a higher running time compared to NysPCR. Error bars are the standard deviation of nRMSE over 5 successive test sets with 10 000 points each. In section K we show additional experiments including a comparison to the Nyström KRR estimator.
Molecular dynamics datasets An important application of Koopman operator theory is in the analysis of molecular dynamics (MD) datasets, where the evolution of a molecule's atomic positions as they evolve over time is modelled. Interesting systems are very high dimensional, with hundreds or thousands of atoms. Furthermore, trajectories are generated at very short time intervals  $(< 1$  ns) but
interesting events (e.g. protein folding/unfolding) occur at timescales on the order of at least  $10\mu \mathrm{s}$ , so that huge datasets are needed to have a few samples of the rare events. The top eigenfunctions of the Koopman operator learned on such trajectories can be used to project the high-dimensional state space onto low-dimensional coordinates which capture the long term, slow dynamics.
We take three 250 ns long simulations sampled at 1 ps of the alanine dipeptide [64], which is often taken as a model system for molecular dynamics [47, 46]. We use the pairwise distances between heavy atoms as features, yielding a 45-dimensional space. We train a NysRRR model with 10 000 centers on top of the full dataset (449 940 points are used for training, the rest for validation and testing) with lag time 100 ps, and recover a 2-dimensional representation which correlates well with the  $\phi, \psi$  backbone dihedral angles of the molecule, known to capture all relevant long-term dynamics. Figure 3a shows the top two eigenfunctions overlaid onto  $\phi, \psi$ , the first separates the slowest transition between low and high  $\phi$ ; the second separates low and high  $\psi$ . The implied time-scales from the first two non-trivial eigenvalues are 1262 ps and 69 ps, which are close to the values reported by Nuske et al. [47] (1400 ps and 70 ps) who used a more complex post-processing procedure to identify time-scales. We then train a PCCA+ [16] model on the first three eigenfunctions to obtain three states, as shown in fig. 3b. PCCA+ acts on top of a fine clustering (in our case obtained with k-means,  $k = 50$ ), to find the set of maximally stable states by analyzing transitions between the fine clusters. The coarse clusters clearly correspond to the two transitions described above.
Finally we take a  $208\mu s$  long simulation of the fast-folding Trp-cage protein [35], sampled every  $0.2\mathrm{ns}$ . Again, the states are the pairwise distances between non-hydrogen atoms belonging to the protein, in 10 296 dimensions. A NysRRR model is trained on 626 370 points, using 5000 centers in approximately 10 minutes. Note that without sketching this would be a completely intractable problem. Using a lag-time of  $10\mathrm{ns}$  we observe a spectral gap between the third and fourth eigenvalues, hence we train a  $\mathrm{PCCA + }$  model on the first 3 eigenfunctions to obtain the states shown in fig. 4. The first non-trivial Koopman eigenvector effectively distinguishes between the folded (state 1) and unfolded states as is evident from the first row of fig. 4. The second one instead can be used to identify a partially folded state of the protein (state 0), as can be seen from the insets in fig. 4.
We introduced three efficient kernel-based estimators of the Koopman operator relying on random projections, and provided a bound on their excess risk in operator norm - which is of paramount importance to control the accuracy of Koopman mode decomposition. Random projections allow to process efficiently even the longest trajectories, and these gains come for free as our estimators still enjoy optimal theoretical learning rates. We leave for future work the refinement our analysis under e.g. an additional source condition assumption or in the misspecified setting. Another future research direction shall be to devise ways to further reduce the computational complexity of the estimators.
This paper is part of a project that has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No. 819789). L. R. acknowledges the financial support of the European Research Council (grant SLING 819789), the AFOSR projects FA9550-18-1-7009, FA9550-17-1-0390 and BAA-AFRL-AFOSR-2016-0007 (European Office of Aerospace Research and Development), the EU H2020-MSCA-RISE project NoMADS - DLV-777826, and the Center for Brains, Minds and Machines (CBMM), funded by NSF STC award CCF-1231216. M. P., V. K. and P. N. acknowledge financial support from PNRR MUR project PE0000013-FAIR and the European Union (Projects 951847 and 101070617).
[1] Amirhesam Abedsoltan, Mikhail Belkin, and Parthe Pandit. Toward large kernel models, 2023. arXiv:2302.02605 [cs.LG].
[2] Thomas N. E. Greville Adi Ben-Israel. Generalized Inverses: Theory and Applications. CMS Books in Mathematics. Springer, 2nd edition, 2003.
[3] Tamim El Ahmad, Luc Brogat-Motte, Pierre Laforgue, and d'Alché-Buc Florence. Sketch In, Sketch Out: Accelerating both Learning and Inference for Structured Prediction with Kernels, 2023. arxiv:2302.10128.
[4] Romeo Alexander and Dimitrios Giannakis. Operator-theoretic framework for forecasting nonlinear time series with kernel analog techniques. Physica D: Nonlinear Phenomena, 409, 2020. doi: https://doi.org/10.1016/j.physd.2020.132520.
[5] Francis Bach. Sharp analysis of low-rank kernel matrix approximations. Journal of Machine Learning Research, 30, 2013.
[6] Peter J. Baddoo, Benjamin Herrmann, Beverley J. McKeon, and Steven L. Brunton. Kernel learning for robust dynamic mode decomposition: linear and nonlinear disambiguation optimization. Proceedings of the Royal Society A, 2022. doi: http://doi.org/10.1098/rspa.2021.0830.
[59] Le Song, Jonathan Huang, Alex Smola, and Kenji Fukumizu. Hilbert space embeddings of conditional distributions with applications to dynamical systems. In Proceedings of the 26th Annual International Conference on Machine Learning, page 961–968, 2009. doi: 10.1145/1553374.1553497.
[60] Ingo Steinwart and Andreas Christmann. Support Vector Machines. Springer Science & Business Media, 2008. URL https://link.springer.com/book/10.1007/978-0-387-77242-4.
[61] Ingo Steinwart and Clint Scovel. Mercer's theorem on general domains: On the interaction between measures, kernels, and RKHSs. Constructive Approximation, 35(3):363-417, 2012.
[62] Naoya Takeishi, Yoshinobu Kawahara, and Takehisa Yairi. Learning koopman invariant subspaces for dynamic mode decomposition. In Advances in Neural Information Processing Systems, page 1130-1140, 2017.
[63] Jonathan H. Tu, Clarence W. Rowley, Dirk M. Luchtenburg, Steven L. Brunton, and J. Nathan Kutz. On dynamic mode decomposition: Theory and applications. Journal of Computational Dynamics, 1(2):391-421, 2014. doi: 10.3934/jcd.2014.1.391.
[64] Christoph Wehmeyer and Frank Noé. Time-lagged autoencoders: Deep learning of slow collective variables for molecular kinetics. Journal of Chemical Physics, 148(24), 2018. doi: 10.1063/1.5011399.
[65] Christopher K. I. Williams and Matthias Seeger. Using the Nyström method to speed up kernel machines. In Advances in Neural Information Processing Systems 13, 2001.
[66] Matthew O. Williams, Ioannis G. Kevrekidis, and Clarence W. Rowley. A data-driven approximation of the Koopman operator: Extending dynamic mode decomposition. Journal of Nonlinear Science, 25(6):1307 - 1346, 2015. doi: 10.1007/s00332-015-9258-5.
[67] Matthew O. Williams, Clarence W. Rowley, and Ioannis G. Kevrekidis. A kernel-based method for data-driven Koopman spectral analysis. Journal of Computational Dynamics, 2(2):247-265, 2015. ISSN 2158-2491. doi: 10.3934/jcd.2015005.
[68] Tianbao Yang, Yu-Feng Li, Mehrdad Mahdavi, Rong Jin, and Zhi-Hua Zhou. Nyström method vs Random Fourier Features: A theoretical and empirical comparison. In Advances in Neural Information Processing Systems 24, 2012.
[69] Enoch Yeung, Soumya Kundu, and Nathan Hodas. Learning deep neural network representations for koopman operators of nonlinear dynamical systems. In 2019 American Control Conference (ACC), 2019. doi: 10.23919/ACC.2019.8815339.
[70] Vadim Yurinsky. Sums and Gaussian Vectors. Lecture Notes in Mathematics 1617. Springer, 1st edition, 1995.
Appendices for: "Estimating Koopman operators with sketching to provably learn large scale dynamical systems"
A Setting and notations
A.1 Operators and notations
We define the following operators:
-  $\Phi_X: L_\pi^2 \to \mathcal{H}$ , defined by  $\Phi_X f = \int_{\mathcal{X}} f(x) \phi(x) \, \mathrm{d}\pi(x)$  for any  $f \in L_\pi^2$ .
-  $\Phi_X^*:\mathcal{H}\to L_\pi^2$ , defined by  $\Phi_X^* h = \langle h,\phi (\cdot)\rangle_{\mathcal{H}}$  for any  $h\in \mathcal{H}$  (i.e. the embedding operator mapping a function to its  $\pi$ -equivalence class in  $L_{\pi}^{2}$ ).
-  $\Phi_{Y|X}: L_{\pi}^{2} \to \mathcal{H}$ , defined by  $\Phi_{Y|X} = \Phi_X \mathcal{K}_{\pi}^*$ .
-  $\Phi_{Y|X}^{*}:\mathcal{H}\to L_{\pi}^{2}$ , defined by  $\Phi_{Y|X}^{*} = \kappa_{\pi}\Phi_{X}^{*}$ .
-  $C: \mathcal{H} \to \mathcal{H}$  defined as  $C = \mathbf{E}_{x \sim \pi} \phi(x) \otimes \phi(x) = \Phi_X \Phi_X^*$ , satisfying  $\operatorname{tr}(C) \leq K^2$ . Note that under our assumptions, this also corresponds to the covariance of  $Y$ .
$C_{XY}\coloneqq \mathbf{E}_{(x,y)\sim \rho}\phi (x)\otimes \phi (y) = \Phi_X\Phi_{Y|X}^*$
As well as the following discretized variants:
-  $\hat{\Phi}_X: \mathbb{R}^n \to \mathcal{H}$ , defined by  $\hat{\Phi}_X v = \sum_{i=1}^{n} v_i \phi(x_i)$  for any  $v = [v_1, \ldots, v_n] \in \mathbb{R}^n$
-  $\hat{\Phi}_X^*:\mathcal{H}\to \mathbb{R}^n$  , defined by  $\hat{\Phi}_X^* h = [\langle \phi (x_1),h\rangle_{\mathcal{H}},\ldots ,\langle \phi (x_n),h\rangle_{\mathcal{H}}]^T$  for any  $h\in \mathcal{H}$
-  $\hat{\Phi}_{Y|X}:\mathbb{R}^n\to \mathcal{H}$ , defined by  $\hat{\Phi}_{Y|X}v = \sum_{i = 1}^{n}v_{i}\phi (y_{i})$  for any  $v = [v_{1},\ldots ,v_{n}]\in \mathbb{R}^{n}$ .
-  $\hat{\Phi}_{Y|X}^{*}:\mathcal{H}\to \mathbb{R}^{n}$ , defined by  $\hat{\Phi}_{Y|X}^{*}h = [\langle \phi (y_1),h\rangle_{\mathcal{H}},\ldots ,\langle \phi (y_n),h\rangle_{\mathcal{H}}]^T$  for any  $h\in \mathcal{H}$ .
-  $\hat{C} = \frac{1}{n}\hat{\Phi}_X\hat{\Phi}_X^* = \frac{1}{n}\sum_{i=1}^{n}\phi(x_i)\otimes\phi(x_i)\in\mathcal{L}(\mathcal{H})$  is the empirical covariance.
The Nystrom discretized operators are obtained by applying the kernel map to  $m \ll n$  inducing points  $\{\tilde{\pmb{x}}_j\}_{j=1}^m \subset \{\pmb{x}_j\}_{j=1}^n$  and  $\{\tilde{\pmb{y}}_j\}_{j=1}^m \subset \{\pmb{y}_j\}_{j=1}^n$ :
-  $\widetilde{\Phi}_X: \mathbb{R}^m \to \mathcal{H}$  such that  $\widetilde{\Phi}_X w = \sum_{j=1}^{m} w_j \phi(\tilde{x}_j)$ .
-  $\widetilde{\Phi}_Y: \mathbb{R}^m \to \mathcal{H}$  such that  $\widetilde{\Phi}_Y w = \sum_{j=1}^{m} w_j \phi(\widetilde{y}_j)$ .
Furthermore denote by  $P_{X}$  and  $P_{Y}$  the orthogonal projections onto span  $\tilde{\Phi}_X$  and span  $\tilde{\Phi}_Y$  respectively.
One important quantity to derive the rates is the so-called effective dimension, defined as
$$
d _ {\text {e f f}} (\lambda) := \operatorname {t r} \left(C _ {\lambda} ^ {- 1} C\right).
$$
where  $C_{\lambda}\coloneqq C + \lambda I$
A.2 Conditional mean embedding