SlowGuess/NeurIPS_2023_OCR huggingface dataset:
and  $P\left[\| (C_{YX} - \hat{C}_{YX})C_{\lambda}^{-1 / 2}\|_{\mathrm{op}}\leq \epsilon (\lambda ,\delta)\right]\geq 1 - \delta$  (52)
$$
w h e r e \quad \epsilon (\lambda , \delta) := \frac {2 K \sqrt {c _ {\tau}} \log (2 / \delta)}{\lambda^ {\tau / 2} n} + \sqrt {\frac {2 K ^ {2} d _ {\text {e f f}} (\lambda) \log (2 / \delta)}{n}} \tag {53}
$$
Proof of Lemma J.4: We first write the proof for the eq. (51). For this result, we use the fact that  $\| (C - \hat{C})C_{\lambda}^{-1 / 2}\|_{\mathcal{B}(\mathcal{H})}\leq \| (C - \hat{C})C_{\lambda}^{-1 / 2}\|_{\mathrm{HS}}$  and bound the Hilbert-Schmidt norm. As  $(\mathrm{HS}(\mathcal{H}),\| \cdot \|_{\mathrm{HS}(\mathcal{H})})$  is a Hilbert space, we apply Lemma J.1 on the random variables  $A_{i} = \phi (x_{i})\otimes \xi (x_{i})$  where  $\xi (x) = C_{\lambda}^{-1 / 2}\phi (x)$
$$
\begin{array}{l} \operatorname {e s s} \sup  \| A \| _ {\mathrm {H S}} = \operatorname {e s s} \sup  \| \phi (x) \| \| \xi (x) \| \\ \leq \frac {K \sqrt {c _ {\tau}}}{\lambda^ {\tau / 2}} \quad (\text {b y}) \\ \mathbf {E} [ \| A \| _ {\mathrm {H S}} ^ {2} ] = \mathbf {E} [ \| \phi (x) \| ^ {2} \| \xi (x) \| ^ {2} ] \\ \le K ^ {2} d _ {\text {e f f}} (\lambda) \\ \end{array}
$$
Thus applying Lemma J.1 with  $L = \frac{K\sqrt{c_{\tau}}}{\lambda^{\tau / 2}}$  and  $\sigma^2 = K^2 d_{\mathrm{eff}}(\lambda)$  gives
$$
P \left[ \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} A _ {i} - \mu \right\| _ {\mathrm {H S}} \leq \frac {2 K \sqrt {c _ {\tau}} \log (2 / \delta)}{\lambda^ {\tau / 2} n} + \sqrt {\frac {2 K ^ {2} d _ {\mathrm {e f f}} (\lambda) \log (2 / \delta)}{n}} \right] \geq 1 - \delta .
$$
This yields the desired result via the inequality between operator and Hilbert-Schmidt norms.
For the bound eq. (52) on the cross-covariance, we take  $A_{i} = \phi(y_{i}) \otimes \xi(x_{i})$  but the rest of the proof is unchanged.
Lemma J.5: Let Assumptions 4.1 and 4.3 hold. Let  $\delta \in ]0,1[$ . Then for i.i.d. samples  $(x_{i},y_{i})_{1\leq i\leq n}$  we get
$$
P \left[ \| \left(C - \hat {C}\right) C _ {\lambda} ^ {- 1} \| _ {\mathrm {o p}} \leq \frac {2 K \sqrt {c _ {\tau}} \log (2 / \delta)}{\lambda^ {(\tau + 1) / 2} n} + \sqrt {\frac {2 K ^ {2} \operatorname {t r} \left(C _ {\lambda} ^ {- 2} C\right) \log (2 / \delta)}{n}} \right] \geq 1 - \delta . \tag {54}
$$
Proof of Lemma J.5: For this result, we use the fact that  $\| (C - \hat{C})C_{\lambda}^{-1}\|_{\mathcal{B}(\mathcal{H})}\leq \| (C - \hat{C})C_{\lambda}^{-1}\|_{\mathrm{HS}}$  and bound the Hilbert-Schmidt norm. As  $(\mathrm{HS}(\mathcal{H}),\| \cdot \|_{\mathrm{HS}(\mathcal{H})})$  is a Hilbert space, we apply Lemma J.1 on the random variables  $A_{i} = \phi (x_{i})\otimes \omega (x_{i})$  where  $\omega (x) = C_{\lambda}^{-1}\phi (x)$ .
$$
\begin{array}{l} \operatorname {e s s} \sup  \| A \| _ {\mathrm {H S}} = \operatorname {e s s} \sup  \| \phi (x) \| \| \omega (x) \| \\ \leq \frac {K \sqrt {c _ {\tau}}}{\lambda^ {(\tau + 1) / 2}} \quad (\text {b y}) \\ \end{array}
$$
$$
\begin{array}{l} \mathbf {E} [ \| A \| _ {\mathrm {H S}} ^ {2} ] = \mathbf {E} [ \| \phi (x) \| ^ {2} \| \omega (x) \| ^ {2} ] \\ \leq K ^ {2} \mathbf {E} \left[ \operatorname {t r} \left(C _ {\lambda} ^ {- 2} \phi (x) \phi (x) ^ {*}\right) \right] \\ = K ^ {- 2} \operatorname {t r} \left(C _ {\lambda} ^ {- 2} C\right) \\ \end{array}
$$
Thus applying Lemma J.1 with  $L = \frac{K\sqrt{c_{\tau}}}{\lambda^{(\tau + 1) / 2}}$  and  $\sigma^2 = K^2\mathrm{tr}(C_\lambda^{-2}C)$  gives
$$
P \left[ \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} A _ {i} - \mu \right\| _ {\mathrm {H S}} \leq \frac {2 K \sqrt {c _ {\tau}} \log (2 / \delta)}{\lambda^ {(\tau + 1) / 2} n} + \sqrt {\frac {2 K ^ {2} \operatorname {t r} (C _ {\lambda} ^ {- 2} C) \log (2 / \delta)}{n}} \right] \geq 1 - \delta .
$$
J.3 Probabilistic inequalities
The following lemma is a restatement from [53, Lemma 6].
Lemma J.6 (Uniform Nyström approximation): Let Assumption 4.1 hold. Let  $P: \mathcal{H} \to \mathcal{H}$  denote the orthogonal projection on span  $\{\phi(\tilde{\boldsymbol{x}}_j) \mid 1 \leq j \leq m\}$ , where the landmarks  $(\tilde{\boldsymbol{x}}_j)_{1 \leq j \leq m}$  are drawn uniformly without replacement from the empirical data. Then for any  $\lambda \in ]0, \|C_\lambda\|_{\mathcal{B}(\mathcal{H})}]$  we have
$$
\| P ^ {\perp} C _ {\lambda} ^ {1 / 2} \| _ {\mathcal {B} (\mathcal {H})} ^ {2} \leq 3 \lambda
$$
with probability at least  $1 - \delta$  provided
$$
m \geq \max  (6 7, 5 \operatorname {e s s} \sup  \| C _ {\lambda} ^ {- 1 / 2} \phi (x) \| ^ {2}) \log \frac {4 K ^ {2}}{\lambda \delta}.
$$
J.4 Concentration lemmas for the sketched operators
Lemma J.7: It holds almost surely
$$
\| (C _ {Y X} - P _ {Y} \hat {C} _ {Y X}) C _ {\lambda} ^ {- 1 / 2} \| \leq \| P _ {Y} ^ {\perp} C _ {\lambda} ^ {1 / 2} \| + \| (C _ {Y X} - \hat {C} _ {Y X}) C _ {\lambda} ^ {- 1 / 2} \|
$$
Proof of Lemma J.7: It holds
$$
\begin{array}{l} C _ {Y X} - P _ {Y} \hat {C} _ {Y X} = C _ {Y X} - P _ {Y} C _ {Y X} + P _ {Y} C _ {Y X} - P _ {Y} \hat {C} _ {Y X} \\ = P _ {Y} ^ {\perp} C _ {Y X} + P _ {Y} (C _ {Y X} - \hat {C} _ {Y X}) \\ \end{array}
$$
$$
\begin{array}{l} \left\| \left(C _ {Y X} - P _ {Y} \hat {C} _ {Y X}\right) C _ {\lambda} ^ {- 1 / 2} \right\| = \left\| \left(P _ {Y} ^ {\perp} C _ {Y X} + P _ {Y} \left(C _ {Y X} - \hat {C} _ {Y X}\right)\right) C _ {\lambda} ^ {- 1 / 2} \right\| \\ \leq \left\| P _ {Y} ^ {\perp} C _ {Y X} C _ {\lambda} ^ {- 1 / 2} \right\| + \left\| P _ {Y} \left(C _ {Y X} - \hat {C} _ {Y X}\right) C _ {\lambda} ^ {- 1 / 2} \right\| \\ \leq \left\| P _ {Y} ^ {\perp} C _ {\lambda} ^ {1 / 2} \right\| \left\| C _ {\lambda} ^ {- 1 / 2} C _ {Y X} C _ {\lambda} ^ {- 1 / 2} \right\| + \left\| \left(C _ {Y X} - \hat {C} _ {Y X}\right) C _ {\lambda} ^ {- 1 / 2} \right\| \\ \end{array}
$$
Eventually it holds  $\| C_{\lambda}^{-1 / 2}C_{YX}C_{\lambda}^{-1 / 2}\| \leq 1$ . Indeed, as  $\pi$  is invariant, it holds that
$$
\| \mathcal {K} _ {\pi} \| = \sup  _ {f \in L _ {\pi} ^ {2}: \| f \| _ {L _ {\pi} ^ {2}} \leq 1} \int_ {x} \left| \int f (y) p (x, \mathrm {d} y) \right| ^ {2} \mathrm {d} \pi (x) \leq 1.
$$
and denoting  $\Phi_X = C^{1/2}U$  the polar decomposition of  $\Phi_X$  for some partial isometry  $U: L_{\pi}^2 \to \mathcal{H}$ , and using  $\Phi_{Y|X}^* = \mathcal{K}_{\pi} \Phi_X^*$ , we get
$$
\begin{array}{l} \left\| C _ {\lambda} ^ {- 1 / 2} C _ {Y X} C _ {\lambda} ^ {- 1 / 2} \right\| = \left\| C _ {\lambda} ^ {- 1 / 2} \Phi_ {Y | X} \Phi_ {X} ^ {*} C _ {\lambda} ^ {- 1 / 2} \right\| \\ \leq \| C _ {\lambda} ^ {- 1 / 2} C ^ {1 / 2} \| \| U \mathcal {K} _ {\pi} ^ {*} U ^ {*} \| \| C ^ {1 / 2} C _ {\lambda} ^ {- 1 / 2} \| \\ \leq 1. \\ \end{array}
$$
J.5 Concentration for mixing processes
Lemma J.8 (Kostic et al. [29, Lemma 1]): Let  $X$  be strictly stationary with values in a normed space  $(\mathcal{X},\| \cdot \|)$  and assume  $n = 2pk$  with  $p,k\in \mathbb{N}$ . Let  $Z_{1},\ldots ,Z_{p}$  be  $p$  independent copies of  $Z_{1} = \sum_{i = 1}^{k}X_{i}$ . Then for  $s > 0$ :
$$
P \left[ \left\| \sum_ {i = 1} ^ {n} X _ {i} \right\| > s \right] \leq 2 P \left[ \left\| \sum_ {j = 1} ^ {p} Z _ {j} \right\| > s / 2 \right] + 2 (p - 1) \beta_ {X} (k).
$$
K Additional experiments
In this section we extend the experiment of Figure 2 (on the Lorenz'63 dataset) to include the Nyström KRR setting. Since the KRR estimator does not use rank truncation as a regularizer, it will always output the full rank estimator. In Figure 5 we use the same settings as in the main text ( $m = 250$  and  $r = 50$ ), which lead to KRR being very close to RRR. No particular instability is encountered with these settings, and KRR is about  $2x$  faster than RRR. In Figure 6 we increase the number of centers, without changing the amount of regularization. Now KRR is very unstable, since it needs to estimate a larger number of eigenvalues which have a very small magnitude. RRR on the other hand maintains the rank regularization and has no instability, obtaining a small performance boost thanks to the increased number of centers  $m$ . To improve the stability of KRR, in Figure 7 we increase the regularization from  $\lambda = 10^{-4}$  to  $\lambda = 5 \times 10^{-3}$ . However, notice that a) the instability is not completely fixed and b) accuracy is noticeably reduced to the point where it's nearly the same as that of PCR with fewer centers.
Connected to the instability problems, KRR is not suited for Koopman operator learning since without rank regularization, a large number of spurious eigenpairs pop up during estimation.