Title: Estimating Koopman operators with sketching to provably learn large scale dynamical systems
Venue: NeurIPS 2023
Accepted: yes
Keywords: dynamical systems

kernel methods

koopman operator

sketching

molecular dynamics

efficient machine learning
Abstract: The theory of Koopman operators allows to deploy non-parametric machine learning algorithms to predict and analyze complex dynamical systems.
Estimators such as principal component regression (PCR) or reduced rank regression (RRR) in kernel spaces can be shown to provably learn Koopman operators from finite empirical observations of the system's time evolution. 
Scaling these approaches to very long trajectories is a challenge and requires introducing suitable approximations to make computations feasible. 
In this paper, we boost the efficiency of 
different kernel-based Koopman operator estimators using random projections (sketching).
We derive, implement and test the new ``sketched'' estimators with extensive experiments on synthetic and large-scale molecular dynamics datasets. 
Further, we establish non asymptotic error bounds giving a sharp characterization of the trade-offs between statistical learning rates and computational efficiency.
Our empirical and theoretical analysis shows that the proposed estimators provide a sound and efficient way to learn large scale dynamical systems.
In particular our experiments indicate that the proposed estimators retain the same accuracy of PCR or RRR, while being much faster.

Estimating Koopman operators with sketching to provably learn large scale dynamical systems

Abstract

The theory of Koopman operators allows to deploy non-parametric machine learning algorithms to predict and analyze complex dynamical systems. Estimators such as principal component regression (PCR) or reduced rank regression (RRR) in kernel spaces can be shown to provably learn Koopman operators from finite empirical observations of the system's time evolution. Scaling these approaches to very long trajectories is a challenge and requires introducing suitable approximations to make computations feasible. In this paper, we boost the efficiency of different kernelbased Koopman operator estimators using random projections (sketching). We derive, implement and test the new 'sketched' estimators with extensive experiments on synthetic and large-scale molecular dynamics datasets. Further, we establish non asymptotic error bounds giving a sharp characterization of the trade-offs between statistical learning rates and computational efficiency. Our empirical and theoretical analysis shows that the proposed estimators provide a sound and efficient way to learn large scale dynamical systems. In particular our experiments indicate that the proposed estimators retain the same accuracy of PCR or RRR, while being much faster. Code is available at https://github.com/Giodiro/NystromKoopman .

1 Introduction

In the physical world, temporally varying phenomena are everywhere, from biological processes in the cell to fluid dynamics to electrical fields. Correspondingly, they generate large amounts of data both through experiments and simulations. This data is often analyzed in the framework of dynamical systems, where the state of a system x is observed at a certain time t , and the dynamics is described by a function f which captures its evolution in time

The function f must capture the whole dynamics, and as such it may be non-linear and even stochastic for instance when modeling stochastic differential equations, or simply noisy processes. Applications of this general formulation arise in fields ranging from robotics, atomistic simulations, epidemiology, and many more. Along with a recent increase in the availability of simulated data, data-driven techniques for learning the dynamics underlying physical systems have become commonplace. The typical approach of such techniques is to acquire a dataset of training pairs ( x t , y t = x t +1 ) sampled in time, and use them to learn a model for f which minimizes a forecasting error. Since dynamical systems stem from real physical processes, forecasting is not the only goal and the ability to interpret the dynamics is paramount. One particularly important dimension for interpretation is the separation of dynamics into multiple temporal scales: fast fluctuations can e.g. be due to thermodynamical noise or electrical components in the system, while slow dynamics describe important conformational changes in molecules or mechanical effects.

Koopman operator theory [27, 28] provides an elegant framework in which the potentially non-linear dynamics of the system can be studied via the Koopman operator

which has the main advantage of being linear but is defined on a typically infinite-dimensional set of observable functions. The expectation in (1) is taken with respect to the potential stochasticity of f . Thanks to its linearity, the operator K can e.g. be applied twice to get two-steps-ahead forecasts, and one can compute its spectrum (beware however that K is not self-adjoint, unless the dynamical process is time-reversible). Accurately approximating the Koopman operator and its spectral properties is of high interest for the practical analysis of dynamical systems. However doing so efficiently for long temporal trajectories remains challenging. In this paper we are interested in designing estimators which are both theoretically accurate and computationally efficient.

Related works Learning the spectral properties of the Koopman operator directly from data has been considered for at least three decades [39, 40], resulting in a large body of previous work. Among the different approaches proposed over time (see Mezi´ c [41] for a recent review) it is most common to search for finite dimensional approximations to the operator, from which part of the spectrum and the Koopman modes [8] can be obtained. Dynamic mode decomposition (DMD) [56, 63], time-lagged independent component analysis (tICA) [42, 49] and many subsequent extensions [31] for example can be seen as minimizers of the forecasting error when ψ is restricted to be a linear function of the states [52]. Extended DMD (eDMD) [66, 25] and the variational approach for conformation dynamics (VAC) [45, 46] instead allow for a (potentially learnable, as in recent deep learning algorithms [32, 37, 69, 62]) dictionary of non-linear functions ψ . Kernel DMD [67, 26] and kernel tICA [57] are further generalizations which again approximate the Koopman operator but using an infinite dimensional space of features ψ , encoded by the feature map of a reproducing kernel. While often slow from a computational point of view, kernel methods are highly expressive and can be analyzed theoretically, to prove convergence and derive learning rates of the resulting estimators [29, 30]. Approximate kernel methods which are much faster to run have been recently used for Koopman operator learning by Baddoo et al. [6] where an iterative procedure is used to identify the best approximation to the full kernel, but no formal learning rates are demonstrated, and by Ahmad et al. [3] who derive learning rates in Hilbert-Schmidt norm (while we consider operator norm) for the Nyström KRR estimator (one of the three considered in this paper).

Contributions In this paper we adopt the kernel learning approach. Starting from the problem of approximating the Koopman operator in a reproducing kernel Hilbert space, we derive three different estimators based on different inductive biases: kernel ridge regression (KRR) which comes from Tikhonov regularization, principal component regression (PCR) which is equivalent to DMD and its extensions, and reduced rank regression (RRR) which comes from a constraint on the maximum rank of the estimator [24]. We show how to overcome the computational scalability problems inherent in full kernel methods using an approximation based on random projections which is known as the Nyström method [58, 65]. The approximate learning algorithms scale very easily to the largest datasets, with a computational complexity which goes from O ( n 3 ) for the exact algorithm to O ( n 2 ) for the approximate one. We can further show that the Nyström KRR, PCR and RRR estimators have the same convergence rates as their exact, slow counterparts - which are known to be optimal under our assumptions. We provide learning bounds in operator norm, which are known to translate to bounds for dynamic mode decomposition and are thus of paramount importance for applications. Finally, we thoroughly validate the approximate PCR and RRR estimators on synthetic dynamical systems, comparing efficiency and accuracy against their exact counterparts [29], as well as recently

proposed fast Koopman estimator streaming KAF [20]. To showcase a realistic scenario, we train on a molecular dynamics simulation of the fast-folding Trp-cage protein [35].

Structure of the paper We introduce the setting in Section 2, and define our three estimators in Section 3. In Section 4 we provide bounds on the excess risk of our estimators, and extensive experiments on synthetic as well as large-scale molecular dynamics datasets in Section 5.

2 Background and related work

Notation We consider a measurable space ( X , B ) where X corresponds to the state space, and denote L 2 π := L 2 ( X , B , π ) the L 2 space of functions on X w.r.t. to a probability measure π , and L ∞ π the space of measurable functions bounded almost everywhere. We denote HS ( H ) the space of Hilbert-Schmidt operators on a space H .

Setting The setting we will consider is that of Markovian, time-homogeneous stochastic process { X t } t ∈ N on X . By definition of a Markov process, X t only depends on X t -1 and not on any previous states. Time-homogeneity ensures that the transition probability P [ X t +1 ∈ B | X t = x ] for any measurable set B does not depend on t , and can be denoted with p ( x , B ) . This implies in particular that the distribution of ( X t , X t +1 ) does not depend on t , and we denote it ρ in the following. We further assume the existence of the invariant density π which satisfies π ( B ) = ∫ X π ( x ) p ( x , B ) d x . This classical assumption allows one to study a large class of stochastic dynamical systems, but also deterministic systems on the attractor, see e.g. [14]. The Koopman operator K π : L 2 π ( X ) → L 2 π ( X ) is a bounded linear operator, defined by

We are in particular interested in the eigenpairs ( λ i , φ i ) ∈ C × L 2 π , that satisfy

Through this decomposition it is possible to interpret the system by separating fast and slow processes, or projecting the states onto fewer dimensions [15, 19, 7]. In particular, the Koopman mode decomposition (KMD) allows to propagate the system state in time. Given an observable g : X → R d such that g ∈ span { φ i | i ∈ N } , the modes allow to reconstruct g ( x ) with a Koopman eigenfunction basis. The modes η g i ∈ C d are the coefficients of this basis expansion:

This decomposition describes the system's dynamics in terms of a stationary component (the Koopman modes), a temporal component (the eigenvalues λ i ) and a spatial component (eigenfunctions φ i ).

Kernel-based learning In this paper we approximate K π with kernel-based algorithms, using operators in reproducing kernel Hilbert spaces (RKHS) H associated with kernel k : X × X → R and feature map ϕ : X → H . We wish to find an operator A : H → H which minimizes the risk

The adjoint of A , denoted by A ∗ , should thus be understood as an estimator of the Koopman operator K π in H as will be clarified in (15). In practice π and ρ are unknown, and one typically has access to a dataset { ( x i , y i ) } n i =1 sampled from ρ , where each pair ( x i , y i = f ( x i )) may equivalently come from a single long trajectory or multiple shorter ones concatenated together. We thus use the empirical risk

as a proxy for (5). Since minimizing eq. (6) may require finding the solution to a very badly conditioned linear system, different regularization methods (such as Tikhonov or truncated SVD) can be applied on top of the empirical risk.

Remark 2.1 (Connections to other learning problems): The problem of minimizing eqs. (5) and (6) has strong connections to learning conditional mean embeddings [59, 22, 44, 33] where the predictors and targets are embedded in different RKHSs, and to structured prediction [12, 13] which is an even more general framework. On the other hand, the most substantial difference from the usual kernel regression setting [9] is the embedding of both targets and predictors into a RKHS, instead of just targets.

We denote the input and cross covariance C = E π [ ϕ ( x ) ⊗ ϕ ( x )] and C Y X = E ρ [ ϕ ( y ) ⊗ ϕ ( x )] , and their empirical counterparts as ˆ C = 1 n ∑ n i =1 [ ϕ ( x i ) ⊗ ϕ ( x i )] and ˆ C Y X = 1 n ∑ n i =1 ϕ ( y i ) ⊗ ϕ ( x i )] . We also use the abbreviation C λ := C + λI . Minimizing the empirical risk (6) with Tikhonov regularization [9] yields the following KRR estimator

Eq. (7) can be computed by transforming its expression with the kernel trick [23], to arrive at a form where one must invert the kernel matrix - a n × n matrix whose i, j -th entry is k ( x i , x j ) . This operation requires O ( n 3 ) time and O ( n 2 ) memory, severely limiting the scalability of KRR to n ≲ 100 000 points. Improving the scalability of kernel methods is a well-researched topic, with the most important solutions being random features [50, 51, 68, 21] and random projections [58, 65, 21]. In this paper we use the latter approach, whereby the kernel matrix is assumed to be approximately low-rank and is sketched to a lower dimensionality. In particular we will use the Nyström method to approximate the kernel matrix projecting it onto a small set of inducing points, chosen among the training set. The sketched estimators are much more efficient than the exact ones, increasingly so as the training trajectories become longer. For example, the state of the art complexity for solving (non vector valued) approximate kernel ridge regression is O ( n √ n ) time instead of O ( n 3 ) [38, 1, 10]. Furthermore, when enough inducing points are used (typically on the order of √ n ), the learning rates of the exact and approximate estimators are the same, and optimal [5, 53]. Hence it is possible - and in this paper we show it for learning the Koopman operator - to obtain large efficiency gains, without losing anything in terms of theoretical guarantees of convergence.

3 Nyström estimators for Koopman operator regression

In this section, we introduce three efficient approximations of the KRR, PCR and RRR estimators of the Koopman operator. Our estimators rely on the Nyström approximation, i.e. on random projections onto low-dimensional subspaces of H spanned by the feature-embeddings of subsets of the data. We thus consider two sets of m ≪ n inducing points { ˜ x j } j m =1 ⊂ { x t } n t =1 and { ˜ y j } j m =1 ⊂ { y t } n t =1 sampled respectively from the input and output data. The choice of these inducing points (also sometimes called Nyström centers) is important to obtain a good approximation. Common choices include uniform sampling, leverage score sampling [17, 55], and iterative procedures such as the one used in [6] to identify the most relevant centers. In this paper we focus on uniform sampling for simplicity, but we stress that our theoretical results in Section 4 can easily be extended to leverage scores sampling by means of [53, Lemma 7]. To formalize the Nyström estimators, we define operators ˜ Φ X , ˜ Φ Y : R m →H as ˜ Φ X w = ∑ m j =1 w j ϕ (˜ x j ) and ˜ Φ Y w = ∑ m j =1 w j ϕ (˜ y j ) , and denote P X and P Y the orthogonal projections onto span ˜ Φ X and span ˜ Φ Y respectively.

In the following paragraphs we apply the projection operators to three estimators corresponding to different choices of regularization. For each of them a specific proposition (proven in Section C) states an efficient way of computing it based on the kernel trick. For this purpose we introduce the kernel matrices K ˜ X,X , K ˜ Y ,Y ∈ R m × n between training set and inducing points with entries ( K ˜ X,X ) ji = k (˜ x j , x i ) , ( K ˜ Y ,Y ) ji = k (˜ y j , y i ) , and the kernel matrices of the inducing points K ˜ X, ˜ X , K ˜ Y , ˜ Y ∈ R m × m with entries ( K ˜ X,X ) jk = k (˜ x j , ˜ x k ) and ( K ˜ X,X ) jk = k (˜ y j , ˜ y k ) .

Kernel Ridge Regression (KRR) The cost of computing ˆ A λ defined in Eq. (7) is O ( n 3 ) [29] which is prohibitive for datasets containing long trajectories. However, applying the projection operators to each side of the empirical covariance operators, we obtain an estimator which additionally depends on the m inducing points:

If H is infinite dimensional, Eq. (8) cannot be computed directly. Proposition 3.1 (proven in Section C) provides a computable version of the estimator.

Proposition 3.1 (Nyström KRR): The Nyström KRR estimator (8) can be expressed as

The computational bottlenecks are the inversion of an m × m matrix and a large matrix multiplication, which overall need O (2 m 3 + 2 m 2 n ) operations. In particular, in Section 4 we will show that m ≍ √ n is sufficient to guarantee optimal rates even with minimal assumptions, leading to a final cost of O ( n 2 ) . Note that a similar estimator was derived in [3].

Please note that the O ( n 2 ) cost is for a straightforward implementation, and can indeed be reduced via iterative linear solvers (possibly preconditioned, to further reduce the practical running time), and randomized linear algebra techniques. In particular, we could leverage results from Rudi et al. [54] to reduce the computational cost to O ( n √ n ) .

Principal Component Regression (PCR) Typical settings in which Koopman operator theory is used focus on the decomposition of a dynamical system into a small set of components, obtained from the eigendecomposition of the operator itself. For this reason, a good prior on the Koopman estimator is for it to be low rank. The kernel PCR estimator ˆ A PCR = ˆ C Y X J ˆ C K † r formalizes this concept [29, 67], where here J · K r denotes the truncation to the first r components of the spectrum. Again this is expensive to compute when n is large, but the estimator can be sketched as follows:

The next proposition provides an efficiently implementable version of this estimator.

Proposition 3.2 (Nyström PCR): The sketched PCR estimator (10) satisfies

requiring O (2 m 3 +2 m 2 n ) operations, i.e. optimal rates can again be obtained at a cost of at most O ( n 2 ) operations.

Note that with m = n , ˆ A PCR m is equivalent to the kernel DMD estimator [67], also known as kernel analog forecasting (KAF) [4]. The sketched estimator of Proposition 3.2 was also recently derived in [6], albeit without providing theoretical guarantees.

Reduced Rank Regression (RRR) Another way to promote low-rank estimators is to add an explicit rank constraint when minimizing the empirical risk. Combining such a constraint with Tikhonov regularization corresponds to the reduced rank regression [24, 29] estimator:

Minimizing Eq. (12) requires solving a n × n generalized eigenvalue problem. The following proposition introduces the sketched version of this estimator, along with a procedure to compute it which instead requires the solution of a m × m eigenvalue problem. For m ≍ √ n , which is enough to guarantee optimal learning rates with minimal assumptions (see Section 4), this represents a reduction from O ( n 3 ) to O ( n √ n ) time.

Proposition 3.3 (Nyström RRR): The Nyström RRR estimator can be written as

To compute it, solve the m × m eigenvalue problem

for the first r eigenvectors W r = [ w 1 , . . . , w r ] , appropriately normalized. Then denoting D r := K † ˜ Y , ˜ Y K ˜ Y ,Y K X, ˜ X W r and E r := ( K ˜ X,X K X, ˜ X + nλK ˜ X, ˜ X ) † K ˜ X,X K Y, ˜ Y D r it holds

4 Learning bounds in operator norm for the sketched estimators

In this section, we state the main theoretical results showing that optimal rates for operator learning with KRR, PCR and RRR can be reached with Nyström estimators.

Assumptions We first make two assumptions on the space H used for the approximation, via its reproducing kernel k .

Assumption 4.1 (Bounded kernel): There exists K < ∞ such that ess sup x ∼ π ∥ ϕ ( x ) ∥ ≤ K .

Assumption 4.1 ensures that H is compactly embedded in L 2 π [61, Lemma 2.3], and we denote Φ ∗ X : H → L 2 π the embedding operator which maps any function in H to its equivalence class π -almost everywhere in L 2 π .

Assumption 4.2 (Universal kernel): The kernel k is universal, i.e. cl(ran(Φ ∗ X )) = L 2 π .

We refer the reader to [60, Definition 4.52] for a definition of a universal kernel. The third assumption on the RKHS is related to the embedding property from Fischer and Steinwart [18], connected to the embedding of interpolation spaces. For a detailed discussion see Section A.3.

Assumption 4.3 (Embedding property): There exists τ ∈ ]0 , 1] and c τ > 0 such that ess sup x ∼ π ∥ C -1 / 2 λ ϕ ( x ) ∥ 2 ≤ c τ λ -τ .

Next, we make an assumption on the decay of the spectrum of the covariance operator that is of paramount importance for derivation of optimal learning bounds. In the following, λ i ( A ) and σ i ( A ) always denote the eigenvalues and singular values of an operator A (in decreasing order).

Assumption 4.4 (Spectral decay): There exists β ∈ ]0 , τ ] and c > 0 such that λ i ( C ) ≤ ci -1 /β .

This assumption is common in the literature, and we will see that the optimal learning rates depend on β . It implies the bound d eff ( λ ) := tr( C -1 λ C ) ≲ λ -β on the effective dimension, which is a key quantity in the analysis (both statements are actually equivalent, see Section E.2). Note that d eff ( λ ) = E x ∼ π ∥ C -1 / 2 λ ϕ ( x ) ∥ ≤ ess sup x ∼ π ∥ C -1 / 2 λ ϕ ( x ) ∥ , and thus it necessarily holds β ≤ τ . For a Gaussian kernel, both β and τ can be chosen arbitrarily close to zero.

Finally, we make an assumption about the regularity of the problem itself. A common assumption occurring in the literature is that E [ f ( X 1 ) | X 0 = · ] ∈ H for every f ∈ H , meaning that one can define the Koopman operator directly on the space H , i.e. the learning problem is well-specified . However, this assumption is often too strong. Following [30, D.1] we make a different assumption on the cross-covariance remarking that, irrespectively of the choice of RKHS, it holds true whenever the Koopman operator is self-adjoint (i.e. the dynamics is time-reversible).

Assumption 4.5 (Regularity of K π ): There exists a > 0 such that C XY C ∗ XY ≼ a 2 C 2 .

Rates The risk can be decomposed as R HS ( A ) = E HS ( A ) + R HS , 0 where R HS , 0 is a constant and E HS ( A ) := ∥K π Φ ∗ X -Φ ∗ X A ∗ ∥ 2 HS corresponds to the excess risk (more details in Section B). Optimal learning bounds for the KRR estimator in the context of CME (i.e. in Hilbert-Schmidt norm) have been developed in [33] under Assumptions 4.1 to 4.4 in well-specified and misspecified settings. On the other hand, in the context of dynamical systems, Kostic et al. [29, Theorem 1] report the importance of reduced rank estimators that have a small excess risk in operator norm

The rationale behind considering the operator norm is that it allows to control the error of the eigenvalues approximation and thus of the KMD (3), (4) as discussed below. Optimal learning bounds in operator norm for KRR, PCR and RRR are established in [30]. In this work we show that the same optimal rates remain valid for the Nyström KRR, PCR and RRR estimators. According to [29] and [30] these operator norm bounds lead to reliable approximation of the Koompan mode decomposition of Eq. (4).

We now provide our main result.

Theorem 4.6 (Operator norm error for KRR, i.i.d. data): Let assumptions 4.1 to 4.5 hold. Let ( x i , y i ) 1 ≤ i ≤ n be i.i.d. samples, and let P Y = P X be the projection induced by m Nyström landmarks drawn uniformly from ( x i ) 1 ≤ i ≤ n without replacement. Let λ = c λ n -1 / (1+ β ) where c λ is a constant

given in the proof, and assume n ≥ ( c λ /K 2 ) 1+ β . Then it holds with probability at least 1 -δ

The proof is provided in Section E.2, but essentially relies on a decomposition involving the terms ∥ C -1 / 2 λ ( C Y X -ˆ C Y X ) ∥ , ∥ C -1 / 2 λ ( C -ˆ C ) ∥ , ∥ C -1 / 2 λ ( C -ˆ C ) C -1 / 2 λ ∥ , as well as bounding the quantity ∥ P ⊥ X C 1 / 2 ∥ where P ⊥ X denotes the projection on the orthogonal of ran( P X ) . All these terms are bounded using two variants of the Bernstein inequality. Note that our results can easily be extended to leverage score sampling of the landmarks by bounding term ∥ P ⊥ X C 1 / 2 ∥ by means of [53, Lemma 7]; the same rate could then be obtained using a smaller number m of Nyström points.

The rate n -1 / (2(1+ β )) is known to be optimal (up to the log factor) in this setting by assuming an additional lower bound on the decay of the covariance's eigenvalues of the kind λ i ( C ) ≳ i -1 /β , see [30, Theorem 7 in D.4]. One can see that without particular assumptions ( β = τ = 1 ), we only need the number m of inducing points to be of the order of Ω( √ n ) in order to get an optimal rates. For τ fixed, this number increases when β decreases (faster decay of the covariance's spectrum), however note that the optimal rate depends on β and also improves in this case. The dependence in τ is particularly interesting, as for instance with a Gaussian kernel it is known that τ can be chosen arbitrarily closed to zero [33, 18]. In that case, the number m of inducing points can be taken on the order of Ω(log n ) .

Note that a bound for the Nyström KRR estimator has been derived in Hilbert-Schmidt norm by Ahmad et al. [3]. Using the operator norm however allows to derive bounds on the eigenvalues (see discussion below), which is of paramount importance for practical applications. Moreover, we now provide a bound on the error of PCR and RRR estimators, which are not covered in [3].

Lemma 4.7 (Operator norm error for PCR and RRR, i.i.d. data): Under the assumptions of Theorem 4.6, taking λ = c λ n -1 / (1+ β ) with c λ as in Theorem 4.6, n ≥ ( c λ /K 2 ) 1+ β , and provided

it holds with probability at least 1 -δ

where c RRR = ( σ 2 r (Φ X K ∗ π ) -σ 2 r +1 (Φ X K ∗ π )) -1 and c PCR = ( σ r (Φ X ) -σ r +1 (Φ X )) -1 are the problem dependant constants.

Note that when rank of K π is r , then there is no restriction on r for the RRR estimator, while for PCR the choice of r depends on the spectral decay property of the kernel. In general, if r > n 1 β (1+ β ) , then σ r +1 (Φ X K ∗ π ) ≤ σ r +1 (Φ X ) ≲ n -1 / (2(1+ β )) , which implies that RRR estimator can achieve the same rate of PCR but with smaller rank. Again the rate is sharp (up to the log factor) in this setting [30].

Koopman mode decomposition According to [29, Theorem 1], working in operator norm allows us to bound the error of our estimators for dynamic mode decomposition, as well as to quantify how close the eigenpairs ( ˆ λ i , ˆ φ i ) of an estimator ˆ A ∗ are to being eigenpairs of the Koopman operator. Namely, recalling that for function ˆ φ i , the corresponding candidate for Koopman eigenfunction in L 2 π space is Φ ∗ X ˆ φ i , one has ∥K π (Φ ∗ X ˆ φ i ) -ˆ λ i (Φ ∗ X ˆ φ i ) ∥ / ∥ Φ ∗ X ˆ φ i ∥ ≤ E ( ˆ A ) 1 / 2 ∥ ˆ φ i ∥ / ∥ Φ ∗ X ˆ φ i ∥ . While eigenvalue and eigenfunction learning rates were studied, under additional assumptions, in [30], where the operator norm error rates were determinant, here, in Section 5, we empirically show that the proposed estimators accurately learn the Koopman spectrum. We refer the reader to Section D for the details on computation of eigenvalues, eigenfunctions and KMD of an estimator in practice.

Dealing with non-i.i.d. data The previous results hold for i.i.d. data, which is not a very realistic assumption when learning from sampled trajectories. Our results can however easily be extended to β -mixing processes by considering random variables Z i = ∑ k j =1 X i + j (thus representing portions of the trajectory) sufficiently separated in time to be nearly independent. We now consider a trajectory

FulIRRR

NysRRR

FulIPCR

nRMSE

1.2

1.0

Time

102

101

100

0.8

10-1

0.6

10-2

102

103

104

102

103

104

Number of samples

Number of samples

0.75

1.0

S

nRMSE

0.5

sKAF

NysRRR

Elapsed

time

0.50

0.25

0.0

0

2

4

Forecast time (s)

sKAF

ysRRR

ysPCR

x 1 , . . . , x n +1 with x 1 ∼ π and x t +1 ∼ p ( x t , · ) for t ∈ [1 , n ] , and use Lemma J.8 (re-stated from [29]) which allows to translate concentration results on the Z i to concentration on the X i by means of the β -mixing coefficients defined as β X ( k ) := sup B ∈B⊗B ∣ ∣ ρ k ( B ) -( π × π )( B ) ∣ ∣ where ρ k denotes the joint probability of ( X t , X t + k ) . Using this result the concentration results provided in appendix can thus be generalized to the β -mixing setting, and apart from logarithmic dependencies we essentially obtain similar results to the i.i.d. setting except that the sample size n is replaced by p ≈ n/ (2 k ) .

5 Experimental validation

In this section we show how the estimators proposed in section 3 perform in various scenarios, ranging from synthetic low dimensional ODEs to large-scale molecular dynamics simulations. The code for reproducing all experiments is available online. Our initial aim is to demonstrate the speed of NysPCR and NysRRR, compared to the recently proposed alternative Streaming KAF (sKAF) [20]. Then we show that their favorable scaling properties make it possible to train on large molecular dynamics datasets without any subsampling. In particular we run a metastability analysis of the alanine dipeptide and the Trp-cage protein, showcasing the accuracy of our models' eigenvalue and eigenfunction estimates, as well as their efficiency on massive datasets ( > 500 000 points)

Efficiency Benchmarks on Lorenz '63 The chaotic Lorenz '63 system [36] consists of 3 ODEs with no measurement noise. With this toy dynamical system we can easily compare the Nyström estimators to two alternatives: 1. the corresponding exact estimators and 2. the sKAF algorithm which also uses randomized linear algebra to improve the efficiency of PCR. In this setting we sample long trajectories from the system, keeping the first points for training (the number of training points varies for the first experiment, and is fixed to 10 000 for the second, see fig. 2), and the subsequent ones for testing. In fig. 1 we compare the run-time and accuracy with of NysPCR and NysRRR versus their full counterparts. To demonstrate the different scaling regimes we fix the number of inducing points to 250 and increase the number of data points n . The accuracy of the two solvers (as measured with the normalized RMSE metric (nRMSE) [20] on the first variable) is identical for PCR and close for RRR, but the running time of the approximate solvers increases much slower with n than that of the exact solvers. Each experiment is repeated 20 times to display error bars over the choice of Nyström centers. In the second experiment, shown in fig. 2, we reproduce the setting of [20] by training at increasingly long forecast horizons. Plotting the nRMSE we verify that sKAF and NysPCR converge to very similar accuracy values, although NysPCR is approximately 10 times faster. NysRRR instead offers slightly better accuracy, at the expense of a higher running time compared to NysPCR. Error bars are the standard deviation of nRMSE over 5 successive test sets with 10 000 points each. In section K we show additional experiments including a comparison to the Nyström KRR estimator.

Molecular dynamics datasets An important application of Koopman operator theory is in the analysis of molecular dynamics (MD) datasets, where the evolution of a molecule's atomic positions as they evolve over time is modelled. Interesting systems are very high dimensional, with hundreds or thousands of atoms. Furthermore, trajectories are generated at very short time intervals ( < 1 ns ) but

a

入1=0.924

入2=0.236

b

2

π

业

metastablestate

1

0

π

一π

T

一π

TT

interesting events (e.g. protein folding/unfolding) occur at timescales on the order of at least 10 µ s , so that huge datasets are needed to have a few samples of the rare events. The top eigenfunctions of the Koopman operator learned on such trajectories can be used to project the high-dimensional state space onto low-dimensional coordinates which capture the long term, slow dynamics.

We take three 250 ns long simulations sampled at 1 ps of the alanine dipeptide [64], which is often taken as a model system for molecular dynamics [47, 46]. We use the pairwise distances between heavy atoms as features, yielding a 45-dimensional space. We train a NysRRR model with 10 000 centers on top of the full dataset ( 449 940 points are used for training, the rest for validation and testing) with lag time 100 ps , and recover a 2-dimensional representation which correlates well with the ϕ, ψ backbone dihedral angles of the molecule, known to capture all relevant long-term dynamics. Figure 3a shows the top two eigenfunctions overlaid onto ϕ, ψ , the first separates the slowest transition between low and high ϕ ; the second separates low and high ψ . The implied time-scales from the first two non-trivial eigenvalues are 1262 ps and 69 ps , which are close to the values reported by Nüske et al. [47] ( 1400 ps and 70 ps ) who used a more complex post-processing procedure to identify time-scales. We then train a PCCA+ [16] model on the first three eigenfunctions to obtain three states, as shown in fig. 3b. PCCA+ acts on top of a fine clustering (in our case obtained with k-means, k = 50 ), to find the set of maximally stable states by analyzing transitions between the fine clusters. The coarse clusters clearly correspond to the two transitions described above.

Finally we take a 208 µ s long simulation of the fast-folding Trp-cage protein [35], sampled every 0 . 2 ns . Again, the states are the pairwise distances between non-hydrogen atoms belonging to the protein, in 10 296 dimensions. A NysRRR model is trained on 626 370 points, using 5000 centers in approximately 10 minutes. Note that without sketching this would be a completely intractable problem. Using a lag-time of 10 ns we observe a spectral gap between the third and fourth eigenvalues, hence we train a PCCA+ model on the first 3 eigenfunctions to obtain the states shown in fig. 4. The first non-trivial Koopman eigenvector effectively distinguishes between the folded (state 1) and unfolded states as is evident from the first row of fig. 4. The second one instead can be used to identify a partially folded state of the protein (state 0), as can be seen from the insets in fig. 4.

6 Conclusions

We introduced three efficient kernel-based estimators of the Koopman operator relying on random projections, and provided a bound on their excess risk in operator norm - which is of paramount importance to control the accuracy of Koopman mode decomposition. Random projections allow to process efficiently even the longest trajectories, and these gains come for free as our estimators still enjoy optimal theoretical learning rates. We leave for future work the refinement our analysis under e.g. an additional source condition assumption or in the misspecified setting. Another future research direction shall be to devise ways to further reduce the computational complexity of the estimators.

State 0

State 1

State 2

Eigenfunction

Eigenfunction 2

References

Amirhesam Abedsoltan, Mikhail Belkin, and Parthe Pandit. Toward large kernel models, 2023. arXiv:2302.02605 [cs.LG] .

Thomas N. E. Greville Adi Ben-Israel. Generalized Inverses: Theory and Applications . CMS Books in Mathematics. Springer, 2nd edition, 2003.

Tamim El Ahmad, Luc Brogat-Motte, Pierre Laforgue, and d'Alché-Buc Florence. Sketch In, Sketch Out: Accelerating both Learning and Inference for Structured Prediction with Kernels, 2023. arxiv:2302.10128 .

Romeo Alexander and Dimitrios Giannakis. Operator-theoretic framework for forecasting nonlinear time series with kernel analog techniques. Physica D: Nonlinear Phenomena , 409, 2020. doi: https://doi.org/10.1016/j.physd.2020.132520.

Francis Bach. Sharp analysis of low-rank kernel matrix approximations. Journal of Machine Learning Research , 30, 2013.

Peter J. Baddoo, Benjamin Herrmann, Beverley J. McKeon, and Steven L. Brunton. Kernel learning for robust dynamic mode decomposition: linear and nonlinear disambiguation optimization. Proceedings of the Royal Society A , 2022. doi: http://doi.org/10.1098/rspa.2021.0830.

Steven L. Brunton, Marko Budiši´ c, Eurika Kaiser, and J. Nathan Kutz. Modern koopman theory for dynamical systems. SIAM Review , 64(2):229-340, 2022. doi: 10.1137/21M1401243.

Marko Budiši` c, Ryan Mohr, and Igor Mezi` c. Applied koopmanism. Chaos , 22(4), 2012. doi: 10.1063/1.4772195.

Andrea Caponnetto and Ernesto De Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics , 7(3):331 - 368, 2007. doi: 10.1007/ s10208-006-0196-8.

Luigi Carratino, Stefano Vigogna, Daniele Calandriello, and Lorenzo Rosasco. ParK: Sound and efficient kernel ridge regression by featurespace partitions. In Advances in Neural Information Processing Systems 34 , pages 6430-6441, 2021.

Antoine Chatalic, Nicolas Schreuder, Lorenzo Rosasco, and Alessandro Rudi. Nyström Kernel Mean Embeddings. In Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 3006-3024. PMLR, 2022.

Carlo Ciliberto, Lorenzo Rosasco, and Alessandro Rudi. A Consistent Regularization Approach for Structured Prediction. In Advances in Neural Information Processing Systems , volume 29. Curran Associates, Inc., 2016.

Carlo Ciliberto, Lorenzo Rosasco, and Alessandro Rudi. A general framework for consistent structured prediction with implicit loss embeddings. Journal of Machine Learning Research , 21 (1), 2022.

G. Da Prato and J. Zabczyk. Ergodicity for Infinite Dimensional Systems . London Mathematical Society Lecture Note Series. Cambridge University Press, 1996. doi: 10.1017/ CBO9780511662829.

Michael Dellnitz and Oliver Junge. On the approximation of complicated dynamical behavior. SIAM Journal on Numerical Analysis , 36(2):491-515, 1999. doi: 10.1137/S0036142996313002.

Peter Deuflhard and Marcus Weber. Robust perron cluster analysis in conformation dynamics. Linear Algebra and its Applications , 398:161-184, 2005. doi: https://doi.org/10.1016/j.laa. 2004.10.026.

Petros Drineas, Malik Magdon-Ismail, Michael W. Mahoney, and David P. Woodruff. Fast approximation of matrix coherence and statistical leverage. Journal of Machine Learning Research , 13(1), 2012.

Simon Fischer and Ingo Steinwart. Sobolev norm learning rates for regularized least-squares algorithms. Journal of Machine Learning Research , 21(1):8464-8501, 2020.

Gary Froyland, Georg A. Gottwald, and Andy Hammerlindl. A computational method to extract macroscopic variables and their dynamics in multiscale systems. SIAM Journal on Applied Dynamical Systems , 13(4):1816-1846, 2014. doi: 10.1137/130943637.

Dimitrios Giannakis, Amelia Henriksen, Joel A. Tropp, and Rachel Ward. Learning to forecast dynamical systems from streaming data. SIAM Journal on Applied Dynamical Systems , 22(2): 527-558, 2023. doi: 10.1137/21M144983X.

Alex Gittens and Michael W. Mahoney. Revisiting the nyström method for improved large-scale machine learning. Journal of Machine Learning Research , 17:3977-4041, 2016.

Steffen Grünewälder, Guy Lever, Luca Baldassarre, Sam Patterson, Arthur Gretton, and Massimilano Pontil. Conditional mean embeddings as regressors. In Proceedings of the 29th International Conference on Machine Learning , pages 1803--1810, 2012.

Thomas Hofmann, Bernhard Schölkopf, and Alexander J. Smola. Kernel methods in machine learning. The Annals of Statistics , 36(3):1171 - 1220, 2008. doi: 10.1214/ 009053607000000677.

Alan Julian Izenman. Reduced-rank regression for the multivariate linear model. Journal of Multivariate Analysis , 5(2):248-264, 1975. doi: https://doi.org/10.1016/0047-259X(75) 90042-1.

Stefan Klus, Péter Koltai, and Christof Schütte. On the numerical approximation of the PerronFrobenius and Koopman operator. Journal of Computational Dynamics , 3(1):51-79, 2016. ISSN 2158-2491. doi: 10.3934/jcd.2016003.

Stefan Klus, Ingmar Schuster, and Krikamol Muandet. Eigendecompositions of transfer operators in reproducing kernel hilbert spaces. Journal of Nonlinear Science , 30(1):283-315, 2020. doi: 10.1007/s00332-019-09574-z.

B. O. Koopman. Hamiltonian systems and transformation in hilbert space. Proceedings of the National Academy of Sciences , 17(5):315-318, 1931. doi: 10.1073/pnas.17.5.315.

B. O. Koopman and J. v. Neumann. Dynamical systems of continuous spectra. Proceedings of the National Academy of Sciences , 18(3):255-263, 1932. doi: 10.1073/pnas.18.3.255.

Vladimir Kostic, Pietro Novelli, Andreas Maurer, Carlo Ciliberto, Lorenzo Rosasco, and Massimiliano Pontil. Learning dynamical systems via Koopman operator regression in Reproducing Kernel Hilbert Spaces. In Advances in Neural Information Processing Systems 35 , 2022.

Vladimir Kostic, Karim Lounici, Pietro Novelli, and Massimiliano Pontil. Koopman operator learning: Sharp spectral rates and spurious eigenvalues, 2023. arXiv:2302.02004 [cs.LG] .

J. Nathan Kutz, Steven L. Brunton, Binghi W. Brunton, and Joshua Proctor. Dynamic Mode Decomposition: Data-Driven Modeling of Complex Systems . SIAM, 2016.

Qianxiao Li, Felix Dietrich, Erik M. Bollt, and Ioannis G. Kevrekidis. Extended dynamic mode decomposition with dictionary learning: A data-driven adaptive spectral decomposition of the koopman operator. Chaos: An Interdisciplinary Journal of Nonlinear Science , 27(10), 2017. doi: 10.1063/1.4993854.

Zhu Li, Dimitri Meunier, Mattes Mollenhauer, and Arthur Gretton. Optimal Rates for Regularized Conditional Mean Embedding Learning. In Advances in Neural Information Processing Systems 35 , 2022.

Junhong Lin and Volkan Cevher. Optimal convergence for distributed learning with stochastic gradient methods and spectral algorithms. Journal of Machine Learning Research , 21(147): 1-63, 2020.

Kresten Lindorff-Larsen, Stefano Piana, Ron O. Dror, and David E. Shaw. How fast-folding proteins fold. Science , 334(6055):517-520, 2011. doi: 10.1126/science.1208351.

Edward N. Lorenz. Deterministic nonperiodic flow. Journal of Atmospheric Sciences , 20(2): 130 - 141, 1963. doi: https://doi.org/10.1175/1520-0469(1963)020<0130:DNF>2.0.CO;2.

Bethany Lusch, J. Nathan Kutz, and Steven L. Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature Communications , 9(1), 2018. doi: 10.1038/ s41467-018-07210-0.

Giacomo Meanti, Luigi Carratino, Lorenzo Rosasco, and Alessandro Rudi. Kernel methods through the roof: handling billions of points efficiently. In Advances in Neural Information Processing Systems 32 , 2020.

Igor Mezi` c. On the geometrical and statistical properties of dynamical systems: Theory and applications . PhD thesis, ProQuest LLC, California Institute of Technology, 1994.

Igor Mezi` c. Spectral properties of dynamical systems, model reduction and decompositions. Nonlinear Dynamics , 41:309-325, 2005. doi: 10.1007/s11071-005-2824-x.

Igor Mezi´ c. Koopman operator, geometry, and learning of dynamical systems. Notices of the American Mathematical Society , 68(7):1087-1105, 2021.

L. Molgedey and H. G. Schuster. Separation of a mixture of independent signals using time delayed correlations. Phys. Rev. Lett. , 72:3634-3637, 1994. doi: 10.1103/PhysRevLett.72.3634.

Mattes Mollenhauer, Ingmar Schuster, Stefan Klus, and Christof Schütte. Singular value decomposition of operators on reproducing kernel hilbert spaces. In Advances in Dynamics, Optimization and Computation , pages 109-131, 2020.

Krikamol Muandet, Kenji Fukumizu, Bharath Sriperumbudur, and Bernhard Schölkopf. Kernel mean embedding of distributions: A review and beyond. Foundations and Trends in Machine Learning , 10(1-2):1-141, 2017. doi: 10.1561/2200000060.

Frank Noé and Feliks Nüske. A variational approach to modeling slow processes in stochastic dynamical systems. Multiscale Modeling & Simulation , 11(2):635-655, 2013. doi: 10.1137/ 110858616.

Feliks Nüske, Bettina G. Keller, Guillermo Pérez-Hernández, Antonia S. J. S. Mey, and Frank Noé. Variational approach to molecular kinetics. Journal of Chemical Theory and Computation , 10(4):1739 - 1752, 2014. doi: 10.1021/ct4009156.

Feliks Nüske, Hao Wu, Jan-Hendrik Prinz, Christoph Wehmeyer, Cecilia Clementi, and Frank Noé. Markov state models from short non-equilibrium simulations - analysis and correction of estimation bias. Journal of Chemical Physics , 146(9), 2017. doi: 10.1063/1.4976518.

I. F. Pinelis and A. I. Sakhanenko. Remarks on inequalities for large deviation probabilities. Theory of Probability & Its Applications , 30(1):143-148, 1986. doi: 10.1137/1130013.

Guillermo Pérez-Hernández, Fabian Paul, Toni Giorgino, Gianni De Fabritiis, and Frank Noé. Identification of slow molecular order parameters for markov model construction. Journal of Chemical Physics , 139(1), 07 2013. doi: 10.1063/1.4811489.

Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems 20 , 2008.

Ali Rahimi and Benjamin Recht. Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. In Advances in Neural Information Processing Systems 21 , 2009.

Clarence W. Rowley, Igor Mezi` c, Shervin Bagheri, Philipp Schlatter, and Dan S. Henningson. Spectral analysis of nonlinear flows. Journal of Fluid Mechanics , pages 115-127, 2009. doi: 10.1017/S0022112009992059.

Alessandro Rudi, Raffaello Camoriano, and Lorenzo Rosasco. Less is more: Nyström computational regularization. In Advances in Neural Information Processing Systems 15 , pages 1657-1665, 2015.

Alessandro Rudi, Luigi Carratino, and Lorenzo Rosasco. FALKON: An optimal large scale kernel method. In Advances in Neural Information Processing Systems 30 , 2017.

Alessandro Rudi, Daniele Calandriello, Luigi Carratino, and Lorenzo Rosasco. On fast leverage score sampling and optimal learning. In Advances in Neural Information Processing Systems 31 , 2018.

Peter J. Schmid. Dynamic mode decomposition of numerical and experimental data. Journal of Fluid Mechanics , 656:5-28, 2010. doi: 10.1017/S0022112010001217.

Christian R. Schwantes and Vijay S. Pande. Modeling molecular kinetics with tICA and the kernel trick. Journal of Chemical Theory and Computation , 11(2):600-608, 2015. doi: 10.1021/ct5007357.

Alex J. Smola and Bernhard Schölkopf. Sparse greedy matrix approximation for machine learning. In ICML 17 , 2000.

Le Song, Jonathan Huang, Alex Smola, and Kenji Fukumizu. Hilbert space embeddings of conditional distributions with applications to dynamical systems. In Proceedings of the 26th Annual International Conference on Machine Learning , page 961-968, 2009. doi: 10.1145/ 1553374.1553497.

Ingo Steinwart and Andreas Christmann. Support Vector Machines . Springer Science & Business Media, 2008. URL https://link.springer.com/book/10.1007/978-0-387-77242-4 .

Ingo Steinwart and Clint Scovel. Mercer's theorem on general domains: On the interaction between measures, kernels, and RKHSs. Constructive Approximation , 35(3):363-417, 2012.

Naoya Takeishi, Yoshinobu Kawahara, and Takehisa Yairi. Learning koopman invariant subspaces for dynamic mode decomposition. In Advances in Neural Information Processing Systems , page 1130-1140, 2017.

Jonathan H. Tu, Clarence W. Rowley, Dirk M. Luchtenburg, Steven L. Brunton, and J. Nathan Kutz. On dynamic mode decomposition: Theory and applications. Journal of Computational Dynamics , 1(2):391-421, 2014. doi: 10.3934/jcd.2014.1.391.

Christoph Wehmeyer and Frank Noé. Time-lagged autoencoders: Deep learning of slow collective variables for molecular kinetics. Journal of Chemical Physics , 148(24), 2018. doi: 10.1063/1.5011399.

Christopher K. I. Williams and Matthias Seeger. Using the Nyström method to speed up kernel machines. In Advances in Neural Information Processing Systems 13 , 2001.

Mattew O. Williams, Ioannis G. Kevrekidis, and Clarence W. Rowley. A data-driven approximation of the Koopman operator: Extending dynamic mode decomposition. Journal of Nonlinear Science , 25(6):1307 - 1346, 2015. doi: 10.1007/s00332-015-9258-5.

Matthew O. Williams, Clarence W. Rowley, and Ioannis G. Kevrekidis. A kernel-based method for data-driven Koopman spectral analysis. Journal of Computational Dynamics , 2(2):247-265, 2015. ISSN 2158-2491. doi: 10.3934/jcd.2015005.

Tianbao Yang, Yu-Feng Li, Mehrdad Mahdavi, Rong Jin, and Zhi-Hua Zhou. Nyström method vs Random Fourier Features: A theoretical and empirical comparison. In Advances in Neural Information Processing Systems 24 , 2012.

Enoch Yeung, Soumya Kundu, and Nathan Hodas. Learning deep neural network representations for koopman operators of nonlinear dynamical systems. In 2019 American Control Conference (ACC) , 2019. doi: 10.23919/ACC.2019.8815339.

Vadim Yurinsky. Sums and Gaussian Vectors . Lecture Notes in Mathematics 1617. Springer, 1st edition, 1995.

Appendices for: 'Estimating Koopman operators with sketching to provably learn large scale dynamical systems'

A Setting and notations

A.1 Operators and notations

We define the following operators:

Φ X : L 2 π →H , defined by Φ X f = ∫ X f ( x ) ϕ ( x ) d π ( x ) for any f ∈ L 2 π .

Φ ∗ X : H → L 2 π , defined by Φ ∗ X h = ⟨ h, ϕ ( · ) ⟩ H for any h ∈ H (i.e. the embedding operator mapping a function to its π -equivalence class in L 2 π ).

Φ Y | X : L 2 π →H , defined by Φ Y | X = Φ X K ∗ π .

Φ ∗ Y | X : H → L 2 π , defined by Φ ∗ Y | X = K π Φ ∗ X .

C : H → H defined as C = E x ∼ π ϕ ( x ) ⊗ ϕ ( x ) = Φ X Φ ∗ X , satisfying tr( C ) ≤ K 2 . Note that under our assumptions, this also corresponds to the covariance of Y .

C XY := E ( x,y ) ∼ ρ ϕ ( x ) ⊗ ϕ ( y ) = Φ X Φ ∗ Y | X .

As well as the following discretized variants:

ˆ Φ X : R n →H , defined by ˆ Φ X v = ∑ n i =1 v i ϕ ( x i ) for any v = [ v 1 , . . . , v n ] ∈ R n

ˆ Φ ∗ X : H → R n , defined by ˆ Φ ∗ X h = [ ⟨ ϕ ( x 1 ) , h ⟩ H , . . . , ⟨ ϕ ( x n ) , h ⟩ H ] T for any h ∈ H

ˆ Φ Y | X : R n →H , defined by ˆ Φ Y | X v = ∑ n i =1 v i ϕ ( y i ) for any v = [ v 1 , . . . , v n ] ∈ R n .

ˆ Φ ∗ Y | X : H → R n , defined by ˆ Φ ∗ Y | X h = [ ⟨ ϕ ( y 1 ) , h ⟩ H , . . . , ⟨ ϕ ( y n ) , h ⟩ H ] T for any h ∈ H .

ˆ C = 1 n ˆ Φ X ˆ Φ ∗ X = 1 n ∑ n i =1 ϕ ( x i ) ⊗ ϕ ( x i ) ∈ L ( H ) is the empirical covariance.

The Nyström discretized operators are obtained by applying the kernel map to m ≪ n inducing points { ˜ x j } j m =1 ⊂ { x j } n j =1 and { ˜ y j } j m =1 ⊂ { y j } n j =1 :

˜ Φ Y : R m →H such that ˜ Φ Y w = ∑ m j =1 w j ϕ (˜ y j ) .

Furthermore denote by P X and P Y the orthogonal projections onto span ˜ Φ X and span ˜ Φ Y respectively.

One important quantity to derive the rates is the so-called effective dimension, defined as

where C λ := C + λI .

A.2 Conditional mean embedding

For any x ∈ X , we denote µ p ( x ) the conditional mean embedding associated to the transition kernel defined as

The following lemma provides a characterization of Φ ∗ Y | X in terms of the conditional mean embedding.

Lemma A.1: We have the following relations:

Proof of Lemma A.1: For the first property:

where we used that f and Φ ∗ X f coincide π -almost everywhere. The second property is a direct consequence of the definition of the adjoint. For (18), we simply use (17) and the definition of Φ Y | X to get

A.3 Power spaces

We now define the α -power space [ H ] α π in order to provide some intuition regarding Assumption 4.3. By Assumption 4.1, tr( C ) = ∫ tr( ϕ ( x ) ⊗ ϕ ( x )) d π ( x ) ≤ K 2 and thus C is trace-class (and compact). By [18], there exists a non-increasing summable sequence ( µ i ) i ∈ I for an at most countable index set I , a family ( e i ) i ∈ I ∈ H s.t. (Φ ∗ X e i ) i ∈ I is an orthonormal basis of span Φ ∗ X ⊆ L 2 π and ( µ 1 / 2 i e i ) i ∈ I is an orthonormal basis of (ker Φ ∗ X ) ⊥ ⊆ H such that

For α ≥ 0 , we now define the α -power space as

equipped with norm

We can now make the following assumption regarding the embedding of the power spaces into L ∞ π .

Assumption A.2 (Embedding): There exists τ ∈ [ β, 1] such that c τ := ∥ [ H ] τ π ↪ → L ∞ π ∥ 2 < ∞ .

We stress that Assumption A.2 implies in particular Assumption 4.3, and is a common assumption in the literature, see for instance [18].

B Expression of the risk

We have the following risk decomposition.

Lemma B.1: The risk can alternatively be written

where inf A ∈ HS ( H ) E HS ( A ) = 0 , and thus we interpret E HS as the excess risk.

Proof of Lemma B.1: Let ( h i ) i ∈ N be an orthonormal basis of H . Then

It holds

where we used the invariance property of π in ( i ) and Lemma A.1 for the last inequality. Then one can easily check that the sum of both corresponds to the full risk defined in (5):

□

C Expression of the estimators

In this section we give proofs of propositions 3.1 to 3.3 on how to efficiently compute the Nyström estimators.

For all three - KRR, PCR and RRR - estimators, the starting point is their respective full estimator which can be derived by following the first-order optimality criterion for the following minimization problems

where Π r is the orthogonal projection onto the top-r eigenvectors of ˆ C .

To derive the Nyström estimators, we project the embedded data ˆ Φ X , ˆ Φ Y | X onto the span of the embedded inducing points P X ˆ Φ X , P Y ˆ Φ Y | X - and then express the resulting estimators as ˜ Φ Y W ˜ Φ ∗ X with W ∈ R m × m . This form is particularly useful for later computing forecasts, eigenfunctions and Koopman modes with the estimator. In particular the following equalities for the projection (shown here for P X but equivalently exist for P Y )

and the characterization of P X through the SVD of ˜ Φ X = U Σ V ∗ , such that P X = UU ∗ .

C.1 Nyström KRR

We begin with the Nyström KRR estimator, providing an alternative but equivalent description in lemma C.1.

Lemma C.1 (Expression of the KRR regularization): Let U be such that P X = UU ∗ , U ∗ U = I . Then it holds

Proof of Lemma C.1: Using U ∗ U = I , it holds ( U ∗ ˆ CU + λI ) U ∗ = U ∗ ( UU ∗ ˆ CUU ∗ + λI ) and thus U ∗ ( UU ∗ ˆ CUU ∗ + λI ) -1 = ( U ∗ ˆ CU + λI ) -1 U ∗ . As a consequence,

Then we can provide the computatable formulas for Nyström KRR

Proposition C.2 (Nyström KRR): The Nyström KRR estimator, obtained by projection of eq. (23) is

Proof of Proposition C.2: Using the definition in eq. (26), and lemma C.1, we have

Now using the fact that Σ , V, V ∗ and U ∗ ˆ CU + λI are full-rank, it holds [2, eq. (20)]

Finally, by definition of P Y , ˆ C Y X and ˆ C ,

Remark C.1 (Alternative derivation of the Nyström KRR estimator): Note that the Nyström KRR estimator can equivalently be derived as the solution to a variational problem similar to eq. (23) , where the operator A is restricted to operate between spaces H ˜ X := span ˜ Φ X and H ˜ Y := span ˜ Φ Y .

C.2 Nyström PCR

Define the following filter on the spectrum of P X ˆ CP X : g PCR ( ˆ C ) = J P X ˆ CP X K † r , which truncates it to the first r components before taking the pseudo-inverse. The Nyström PCR estimator, obtained by projection of eq. (24) is

The next proposition provides an efficiently implementable version of the PCR estimator.

Proposition C.3 (Nyström PCR): The sketched PCR estimator eq. (27) satisfies

□

Proof of Proposition C.3: Webegin by computing the decomposition of P X ˆ CP X which is necessary to obtain g PCR ( ˆ C ) . The following expressions are equivalent [43, Proposition 3] for determining its eigenvectors ˜ h and eigenvalues λ :

Let the truncated eigenvalues be Λ r = diag[ λ 1 , . . . , λ r ] and the eigenvectors be H r = [ h 1 , . . . , h r ] . Then ˜ H r = UH r must be normalized such that ˜ H ∗ r ˜ H r = H ∗ r U ∗ UH r = I . The rank-r truncation J P X ˆ CP X K r is a projection onto ˜ H r ˜ H ∗ r :

where we used that U ∗ ˆ CU = H Λ H ∗ . Now substitute U = ˜ Φ X V Σ -1 to simplify the eigendecomposition of U ∗ ˆ CU :

where V Σ -2 V ∗ = K † ˜ X, ˜ X . Denote by D r = [ d 1 , . . . , d r ] the truncated eigenvectors such that H r = Σ -1 V ∗ K ˜ X,X K X, ˜ X D r , normalized such that H ∗ H = D ∗ K ˜ X,X K X, ˜ X K † ˜ X, ˜ X K ˜ X,X K X, ˜ X D = I ,

Finally, we can plug the pieces together to get

Remark C.2 (Variational problem for Nyström PCR): Note that, unlike the NysKRR estimator, the variational problem for NysPCR where the operator is restricted to A : H ˜ X → H ˜ Y is not equivalent to the one obtained in proposition C.3 by projecting the covariance operator. In fact, the former does not take the full covariance into account when computing the low-rank projection, but just the Nyström points.

C.3 Nyström RRR

The Nyström RRR estimator does not correspond to a specific spectral filter. We can nonetheless compute it starting from the expression of the exact empirical estimator [29], projecting the covariance operators, and rearranging the expression to result in a finite-dimensional procedure.

Proposition C.4 (Nyström RRR): The sketched RRR estimator can be written as

To compute it, solve the m × m eigenvalue problem

for the first r eigenvectors W r = [ w 1 , . . . , w r ] , normalized such that W ∗ r K ˜ X,X K Y, ˜ Y K † ˜ Y , ˜ Y K ˜ Y ,Y K X, ˜ X W r = I. Then let D r := K † ˜ Y , ˜ Y K ˜ Y ,Y K X, ˜ X W r and E r := ( K ˜ X,X K X, ˜ X + nλK ˜ X, ˜ X ) † K ˜ X,X K Y, ˜ Y U r , such that the following holds

Proof of Proposition C.4: Let B := n 1 / 2 P Y ˆ C Y X P X ( P X ˆ CP X + λI ) -1 / 2 . The computationally intensive part for this estimator is in evaluating the rank-r truncation J B K r . Its singular values and left singular vectors can be obtained by solving the symmetric eigenvalue problem BB ∗ q i = σ 2 i q i . We rewrite BB ∗

where the second and fourth equalities are applications of the push-through identity, the third by definition of projections and kernel matrices, and the last by collecting K ˜ X, ˜ X . By construction, the non-trivial eigenfunctions of BB ∗ are in the range of P Y ˆ Φ Y | X K X, ˜ X , therefore we can set q i = P Y ˆ Φ Y | X K X, ˜ X w i for some w i ∈ R m , and solve the following eigenvalue problem instead

The eigenfunctions of BB are therefore q = P K ˜ w , which must be normalized as where we have simplified the left term of both sides of the equation. ∗ i Y ˆ Φ Y | X X, X i

Thanks to this normalization, the projector onto the r leading left singular vectors of B is Q r Q ∗ r , where Q r = [ q 1 , . . . , q r ] . Then the NysRRR estimator can be written as

where

with the same techniques we used for rewriting BB ∗ . Finally, let D r and E r as in the statement. We can apply the projection to obtain

D Forecasting & Koopman Modes

The three estimators considered in Section C are all of the form

We will use this generic form to provide expressions for the following operations:

producing forecasts of the dynamical system at a future time,

computing the approximate eigenvalues and eigenfunctions of the Koopman operator,

computing the Koopman modes.

D.1 Forecasting

Given a new data-point x ∈ X and an observable function g ∈ H (note that this can simply be the identity function), we can approximate the one-step-ahead expectation E [ g ( X t +1 ) | X t = x ] =

( K π g )( x ) by using the obtained estimators ˆ A ∗ . Note that by the reproducing property ˜ Φ ∗ Y g = [ g ( y i ) , . . . , g ( y m )] ⊤ =: g m , then

D.2 Eigenfunctions and eigenvalues

We wish to compute the eigenfunctions ξ, ψ ∈ H , as well as the eigenvalues λ i of ˆ A . The left eigenfunctions satisfy ˆ A ∗ ξ i = ¯ λ i ξ i and the right eigenfunctions satisfy ˆ Aψ i = λ i ψ i . In the following we will use Mollenhauer et al. [43, Proposition 3] to manipulate the eigendecomposition of operators in H .

Consider the decomposition W = U r V ∗ r with U r , V r ∈ C m × r , which is available for all considered estimators with r ≤ m . For example, in the Nyström RRR estimator of proposition C.4, we can simply take U r = D r and V r = E r . For the Nyström KRR estimator instead, r = m and we can take the whole of W as our U r and V r = I .

To compute the right eigenfunctions ψ i , such that ( ˜ Φ Y U r V ∗ r ˜ Φ ∗ X ) ψ i = λ i ψ i , consider the following equivalent eigendecomposition

Note that ˜ Φ ∗ X ˜ Φ Y = K ˜ X, ˜ Y is a finite-dimensional object which can easily be computed. The eigenfunctions ψ i must be normalized such that ψ ∗ i ψ i = 1 for every i , so we must have

A very similar process can be followed to obtain the left eigenfunctions ξ i , such that ˜ Φ X V r U ∗ r ˜ Φ ∗ Y ξ i = ¯ λ i ξ i . Here we consider instead

where once again, ˜ Φ ∗ Y ˜ Φ X = K ⊤ ˜ X, ˜ Y and the eigenfunctions must be normalized such that ˜ h ∗ i V ∗ r ˜ Φ ∗ X ˜ Φ X V r ˜ h i = 1 for every i . Finally, ψ and ξ must be orthogonal to each other: we must have for i, j ∈ [ r ] that ⟨ ψ i , ¯ ξ j ⟩ H = δ ij (where δ ij is a Dirac delta equals to 1 when i = j and 0 otherwise). We can compute

and note that ˜ h ∗ i ˜ g j = δ ij , but we must normalize ξ such that

D.3 Koopman modes

Given the eigendecomposition of any estimator ˆ A as ˆ A r = ∑ r i =1 λ i ψ i ⊗ ¯ ξ i , for an observable g we have the following

where ⟨ g, ¯ ψ i ⟩ H = γ g i are the Koopman modes. Expanding the definition of ψ i we get

which we can efficiently compute.

E Excess risk of the Nyström KRR estimator

E.1 Almost-sure decomposition of the KRR excess risk

Lemma E.1 (Excess risk decomposition in operator norm for KRR): Let Assumptions 4.1 to 4.3 and 4.5 hold. Then the Nyström KRR estimator (8) satisfies almost surely

where θ 1 := ∥ ˆ C -1 / 2 λ C 1 / 2 λ ∥ , θ 2 := ∥ ˆ C 1 / 2 λ C -1 / 2 λ ∥ , θ 3 := ∥ ˆ C -1 λ C λ ∥ , and a is the constant of Assumption 4.5.

Proof of Lemma E.1: Let θ 1 := ∥ ˆ C -1 / 2 λ C 1 / 2 λ ∥ , θ 3 := ∥ ˆ C -1 λ C λ ∥ . As in Lemma C.1 define g KRR ( ˆ C ) := U ( U ∗ ˆ CU + λI ) -1 U ∗ . We have

where we used the polar decomposition Φ ∗ X = WC 1 / 2 for some partial isometry W : H → L 2 π . The first term is

where we used the definition of Φ Y | X and applied Lemma H.1.

The second term of our decomposition (32) can be bounded as follows: It holds

We now bound the terms B 1 and B 2 separately.

Let ˆ P λ := ˆ C 1 / 2 g KRR ( ˆ C ) ˆ C 1 / 2 . We recall that g KRR ( ˆ C ) = P X g KRR ( ˆ C ) , so that

This implies ˆ P 2 λ = ˆ P λ = ˆ P ∗ λ . Hence ˆ P λ is an orthogonal projection, and defining ˆ P ⊥ λ = I -ˆ P λ it holds ∥ ˆ P ⊥ λ ∥ B ( H ) ≤ 1 . We can thus bound B 2 as follows:

For the third term, due to Lemma C.1:

where we used Lemma J.7 for the last inequality.

Starting again from (32) and putting everything together, we get

E.2 Excess risk rates for KRR

In order to control the terms appearing in our decomposition, we recall that Assumption 4.4 implies

where c is the constant of Assumption 4.4, see [9, Proposition 3 with b → 1 /β and β → c ] and [18, Lemma 11] which shows that the existence of a constant C β such that the first part of (37) holds implies in return λ i ( C ) ≲ i -1 /β .

Proof of Theorem 4.6: By Lemma E.1 taking P X = P Y , it holds almost surely

and we recall that θ 1 := ∥ ˆ C -1 / 2 λ C 1 / 2 λ ∥ and θ 3 := ∥ ˆ C -1 λ C λ ∥ . We bound separately the terms appearing in this expression.

Bound of θ 1 and θ 2 . We control these term by bounding ∥ C -1 / 2 λ ( ˆ C -C ) C -1 / 2 λ ∥ . By Lemma J.3 it holds for any δ ′ ∈ ]0 , 1[ and any λ ∈ ]0 , ∥ C ∥ B ( H ) ] with probability 1 -δ ′

A sufficient condition to bound the right hand side of the previous expression by 1 / 4 is to have nλ τ > 32 c τ β (in which cases both terms are bounded by 1 / 8 ). Assuming this holds, I -∥ C -1 / 2 λ ( ˆ C -

C ) C -1 / 2 λ ∥ is invertible and we also have

where ( i ) can be obtained by taking the Neumann expansion of I -∥ C -1 / 2 λ ( ˆ C -C ) C -1 / 2 λ ∥ . Bound for θ 3 . By Lemma J.5 it holds with probability 1 -δ ′

Both terms in the above rhs are bounded by 1 / 4 provided

where we used tr( C -2 λ C ) = ∑ λ i ( C )( λ i ( C ) + λ ) -2 ≤ λ -1 tr( C -1 λ C ) ≤ C β λ -(1+ β ) . When this is the case, we have ∥ ( C -ˆ C ) C -1 λ ∥ op ≤ 1 / 2 < 1 and the operator I -( ˆ C λ -C λ ) C -1 λ is invertible.

where ( i ) can be obtained by considering the Neumann expansion of I -( ˆ C λ -C λ ) C -1 λ . Bound for ∥ P ⊥ X C 1 / 2 λ ∥ . By Lemma J.6, provided λ ∈ ]0 , ∥ C ∥ B ( H ) ] it holds with probability 1 -δ ′

provided m ≥ max(67 , 5 ess sup x ∼ π ∥ C -1 / 2 λ ϕ ( x ) ∥ 2 ) log 4 K 2 λδ ′ , which by Lemma H.3 is ensured if m ≥ max(67 , 5 c τ τ ) log 4 K 2 ′ .

λ λδ

Bound for ∥ ( C -ˆ C ) C -1 / 2 λ ∥ and ∥ ( C Y X -ˆ C Y X ) C -1 / 2 λ ∥ . By Lemma J.4, for any δ ′ ∈ ]0 , 1[ , each of the following events holds with probability 1 -2 δ ′ :

By Eq. (37) we have d eff ( λ ) ≤ C β λ -β .

Choosing δ ′ = δ/ 5 , we get via a union bound with probability 1 -δ that θ 1 θ 2 θ 3 ≤ 2 . 6 , θ 2 1 ≤ 1 . 34 and

for any λ and m satisfying the constraints

We pick λ := c λ n -1 / (1+ β ) which is asymptotically the saturating constraint (given that 1 / (1 + β ) < 1 < 2 / ( τ +1) ≤ 1 /τ ), where c λ is a constant choosen to enforce the following equations (which are sufficient conditions for eq. (41) to hold):

As 1 -( τ +1) / (2(1 + β )) > 0 , a sufficient condition for the second equation is

Assuming c τ ≥ 8 K 2 , a sufficient condition to satisfy the first constraint is

which is in particular ensured (noting that log( n ) /n ν ≤ 1 / ( νe ) for any n, ν > 0 ) whenever

Noting that 1 + β -τ ≤ 1 , we get that

is also sufficient. We recall that 1 / (1 + β ) < 1 < 2 / ( τ +1) ≤ 1 /τ , so that we can choose

while the last constraint n ≥ ( c λ /K 2 ) 1+ β is satisfied by assumption.

which gives the claimed result. The last constraint (on m ) is satisfied by the assumptions of the lemma. □

F Excess risk of the Nyström RRR estimator

Recalling (30), NyströmRRR estimator is of the form ˆ A RRR m,λ = J ˜ B K r ( ˜ C λ ) -1 / 2 , where ˜ B := ˜ C Y X ( ˜ C λ ) -1 / 2 for ˜ C Y X := P Y ˆ C Y X P X and ˜ C λ := P X ˆ CP X + λI . While the population version is A RRR λ := J B K r C -1 / 2 λ where B := C Y X ( C λ ) -1 / 2 .

In this section we follow the approach in [30] and decompose the operator norm excess risk in the following way:

Then, recalling that A λ = C Y X C -1 λ and ˆ A KRR m,λ = ˜ B ˜ C -1 / 2 λ , we also have A RRR λ = P B A λ and ˆ A RRR m,λ = P ˜ B ˆ A KRR m,λ , where P B and P ˜ B are orthogonal projectors onto leading r left singular vectors of B and ˜ B , respectively.

Thus,

where the last inequality is due to ∥ A λ ∥ ≤ a and [30, Proposition 4].

Recalling Lemma E.1, we observe that

Therefore, to prove Lemma 4.7 for the RRR estimator we just need to bound ∥ ˜ B ˜ B ∗ -BB ∗ ∥ B ( H ) . To that end, observe that, after some algebra, one obtains

and, consequently,

follows using that ∥ A λ ∥ B ( H ) ≤ a .

On the other hand,

which implies that

But, ε 1 can be bounded by Lemma J.6. Indeed, provided λ ∈ ]0 , ∥ C ∥ B ( H ) ] , it holds with probability 1 -δ ′ √

ε

/

2

1

ϕ

(

x

3

2

λ

) log provided

m

m

max(67

,

max(67

5

c

τ

λ

τ

) log

,

5 ess sup

4

K

2

λδ

′

.

x

∼

π

∥

C

-

λ

1

)

∥

4

K

λδ

2

′

, which by Lemma H.3 is ensured if

Additionally,

implies that

provided, as above, that nλ τ > 32 c τ β .

Therefore, setting ε 0 := max { a ∥ ˆ C Y X -C Y X ∥ B ( H ) , a 2 ∥ ˆ C -C ∥ B ( H ) } , for all i ∈ [ m ] we have

where ε 2 := ∥ ( A λ -ˆ A KRR m,λ ) C 1 / 2 λ ∥ is the variance of Nyström KRR estimator, and conclude that

Therefore, the proof of Lemma 4.7 for RRR estimator directly follows from the bound on aλ 1 / 2 + ε 2 given in the proof of Theorem 4.6, and the fact that, see e.g. [29], ε 0 ≲ n -1 / 2 ≲ λ 1 / 2 .

G Excess risk of the Nyström PCR estimator

for λ = 0 and with P ˜ C λ being the orthogonal projector onto leading r eigenspace of ˜ C λ . So, to prove Lemma 4.7 for PCR estimator, denote ˆ A PCR m,λ := ˆ A KRR m,λ P ˜ C λ for λ ≥ 0 , and let us define the population version A PCR λ = A λ P C λ , where P C λ being the orthogonal projector onto leading r eigenspace of C λ . As in the previous section we start with decomposition

The first and the second term are easily bounded by ∥ Φ Y | X -A λ Φ X ∥ B ( L 2 π , H ) ≤ aλ 1 / 2 , and

For the third term, start by observing that

where the second last inequality is due to [30, Proposition 4] and the last one uses Eq. (43). Moreover, we have that

where we have used Eq. (44) and the fact that P ˜ C λ is the spectral projector of ˜ C -1 / 2 λ . Therefore, due to

we obtain

provided that nλ > 32 c τ β .

Finally for the last term, observe that J ˜ C λ K † r and J ˜ C 0 K † r share the same eigenvectors, and hence J ˜ C 0 K † r -J ˜ C λ K † r = λ J ˜ C λ ˜ C 0 K † r . Hence, it holds that

Now, recalling Eq. (45), we can bound

and,

Thus, consequently, we obtain

To conclude, observe that r > n 1 β (1+ β ) due to Assumption 4.4 implies that σ r +1 (Φ X ) ≲ n -1 2(1+ β ) . Therefore, collecting all the terms, under the assumptions of Lemma 4.7 we obtain

where c PCR = ( σ 2 r (Φ X ) -σ 2 r +1 (Φ X )) -1 is the problem dependant constant.

H Auxiliary results

Lemma H.1 ([30, Proposition 2] with α = 1 ): Under Assumption 4.2 it holds

Lemma H.2: Let A be a bounded operator. Under Assumptions 4.1 and 4.5, it holds

Proof of Lemma H.2: Note that under Assumption 4.5, as C XY C Y X ≼ a 2 C 2 it also holds A ∗ C XY C Y X A ≼ a 2 AC 2 A and thus:

□

The next lemma is a consequence of Assumption 4.3 and will be used in our concentration inequalities.

Lemma H.3: Under Assumption 4.3, it holds π -almost surely for any ν :

The two following corollaries can be obtained picking ν = 0 and ν = -1 :

Proof of Lemma H.3: By [18, Theorem 9], it holds c τ := ∥ k τ π ∥ 2 ∞ = ess sup x ∼ π ∑ i ∈ I µ τ i | e i ( x ) | 2 (where ( e i ) is defined in Section A.3, and we recall that ( √ µ i e i ) i ∈ N is an orthonormal basis of H . Denoting µ i := λ i ( C ) , it holds

where we used sup | µ i | ≤ K 2 .

□

I Deterministic sketching results

Lemma I.1: Denoting R := I -ˆ C 1 / 2 g KRR ( ˆ C ) ˆ C 1 / 2 , it holds

Proof of Lemma I.1: This is a direct consequence of the fact that g KRR ( ˆ C ) ˆ C λ P X = P X :

J Concentration results

J.1 Generic concentration lemmas

All our concentration results derive from two versions of the Bernstein inequality. We first state an inequality for sums of random variables in a Hilbert space based on [70, Theorem 3.3.4], which itself derives from a result of [48].

Lemma J.1: Let ( A i ) 1 ≤ i ≤ n be i.i.d. copies of a random variable A in a separable Hilbert space ( H, ∥·∥ ) . Assume E A = µ and ∃ σ > 0 , ∃ L > 0 , ∀ p ≥ 2 , E ∥ A -µ ∥ p ≤ 1 2 p ! σ 2 L p -2 . Then for any δ ∈ ]0 , 1[ it holds:

The assumption on the moments holds in particular when ess sup ∥ A ∥ ≤ L/ 2 and E [ ∥ A ∥ 2 ] ≤ σ 2 .

Proof of Lemma J.1: See proof of [11, Lemma E.3] for a precise derivation based on [70, Theorem 3.3.4]. □

We now state a version of the Bernstein concentration inequality for self-ajoint operators in operator norm, which is a restatement of [34, Lemma 24]. In the following, we denote r eff ( A ) := tr( A ) / ∥ A ∥ the effective rank of a nonnegative definite operator A .

Lemma J.2 (Bernstein for self-ajoint operators acting on a Hilbert): Let H be a separable Hilbert space and A i be i.i.d. copies of a random variable A taking values in the space of self-adjoint Hilbert-Schmidt operators on H . Assume E A = 0 , ess sup ∥ A ∥ op ≤ c for some c > 0 (where ∥·∥ op denotes the operator norm) and that there exists a positive semi-definite trace class operator V such that E [ A 2 ] ≼ V . Then for any δ ∈ ]0 , 1[ and n ≥ 1 it holds

Proof of Lemma J.2: See [34, Appendix B.7, Lemma 24].

□

J.2 Applied concentration lemmas

Lemma J.3: Let Assumption 4.1 hold. Let δ ∈ ]0 , 1[ . Then for i.i.d. samples ( x i , y i ) 1 ≤ i ≤ n and any λ ∈ ]0 , ∥ C ∥ B ( H ) ] it holds

Proof of Lemma J.3: We apply Lemma J.2 on the random variables A i = ξ ( X i ) ⊗ ξ ( X i ) -C -1 / 2 λ CC -1 / 2 λ where ξ ( X i ) := C -1 / 2 λ ϕ ( X i ) . It holds

Thus applying Lemma J.2 with c = 2 c τ λ τ and V = c τ λ τ CC -1 λ , we get

where we used the fact that ∥ CC -1 λ ∥ B ( H ) ≤ 1 and controlled the effective rank using tr( CC -1 λ ) ≤ K 2 /λ and ∥ CC -1 λ ∥ B ( H ) = ∥ C ∥ B ( H ) / ( ∥ C ∥ B ( H ) +1) ≥ 1 / 2 because λ ≤ ∥ C ∥ B ( H ) by assumption. □

Lemma J.4: Let Assumptions 4.1 and 4.3 hold. Let δ ∈ ]0 , 1[ . Then for i.i.d. samples ( x i , y i ) 1 ≤ i ≤ n we get

Proof of Lemma J.4: We first write the proof for the eq. (51). For this result, we use the fact that ∥ ( C -ˆ C ) C -1 / 2 λ ∥ B ( H ) ≤ ∥ ( C -ˆ C ) C -1 / 2 λ ∥ HS and bound the Hilbert-Schmidt norm. As ( HS ( H ) , ∥·∥ HS ( H ) ) is a Hilbert space, we apply Lemma J.1 on the random variables A i = ϕ ( x i ) ⊗ ξ ( x i ) where ξ ( x ) = C -1 / 2 λ ϕ ( x ) .

Thus applying Lemma J.1 with L = K √ c τ λ τ/ 2 and σ 2 = K 2 d eff ( λ ) gives

This yields the desired result via the inequality between operator and Hilbert-Schmidt norms. For the bound eq. (52) on the cross-covariance, we take A i = ϕ ( y i ) ⊗ ξ ( x i ) but the rest of the proof is inchanged. □

Lemma J.5: Let Assumptions 4.1 and 4.3 hold. Let δ ∈ ]0 , 1[ . Then for i.i.d. samples ( x i , y i ) 1 ≤ i ≤ n we get

Proof of Lemma J.5: For this result, we use the fact that ∥ ( C -ˆ C ) C -1 λ ∥ B ( H ) ≤ ∥ ( C -ˆ C ) C -1 λ ∥ HS and bound the Hilbert-Schmidt norm. As ( HS ( H ) , ∥·∥ HS ( H ) ) is a Hilbert space, we apply Lemma J.1 on the random variables A i = ϕ ( x i ) ⊗ ω ( x i ) where ω ( x ) = C -1 λ ϕ ( x ) .

Thus applying Lemma J.1 with L = K √ c τ λ ( τ +1) / 2 and σ 2 = K 2 tr( C -2 λ C ) gives

This yields the desired result via the inequality between operator and Hilbert-Schmidt norms. □

J.3 Probabilistic inequalities

The following lemma is a restatement from [53, Lemma 6].

Lemma J.6 (Uniform Nyström approximation): Let Assumption 4.1 hold. Let P : H → H denote the orthogonal projection on span { ϕ (˜ x j ) ∣ ∣ 1 ≤ j ≤ m } , where the landmarks (˜ x j ) 1 ≤ j ≤ m are drawn uniformly without replacement from the empirical data. Then for any λ ∈ ]0 , ∥ C λ ∥ B ( H ) ] we have

with probability at least 1 -δ provided

J.4 Concentration lemmas for the sketched operators

Lemma J.7: It holds almost surely

Proof of Lemma J.7: It holds

Thus

Eventually it holds ∥ C -1 / 2 λ C Y X C -1 / 2 λ ∥ ≤ 1 . Indeed, as π is invariant, it holds that

and denoting Φ X = C 1 / 2 U the polar decomposition of Φ X for some partial isometry U : L 2 π →H , and using Φ ∗ Y | X = K π Φ ∗ X , we get

J.5 Concentration for mixing processes

Lemma J.8 (Kostic et al. [29, Lemma 1]): Let X be strictly stationary with values in a normed space ( X , ∥·∥ ) and assume n = 2 pk with p, k ∈ N . Let Z 1 , . . . , Z p be p independent copies of Z 1 = ∑ k i =1 X i . Then for s > 0 :

K Additional experiments

In this section we extend the experiment of Figure 2 (on the Lorenz'63 dataset) to include the Nyström KRR setting. Since the KRR estimator does not use rank truncation as a regularizer, it will always output the full rank estimator. In Figure 5 we use the same settings as in the main text ( m = 250 and r = 50 ), which lead to KRR being very close to RRR. No particular instability is encountered with these settings, and KRR is about 2x faster than RRR. In Figure 6 we increase the number of centers, without changing the amount of regularization. Now KRR is very unstable, since it needs to estimate a larger number of eigenvalues which have a very small magnitude. RRR on the other hand maintains the rank regularization and has no instability, obtaining a small performance boost thanks to the increased number of centers m . To improve the stability of KRR, in Figure 7 we increase the regularization from λ = 10 -4 to λ = 5 × 10 -3 . However, notice that a) the instability is not completely fixed and b) accuracy is noticeably reduced to the point where it's nearly the same as that of PCR with fewer centers.

1.2

1.0

0.8

sKAF

nRMSE

0.6

NysRRR

0.4

NysKRR

0.2

0.0

0

1

2

3

Forecast time (s)

1.2

1.0

0.8

sKAF

nRMSE

NysPCR(M=250)

0.6

NysRRR(M=1000)

NysKRR(M=1000)

0.4

0.2

0.0

0

1

2

3

4

Forecast time (s)

Connected to the instability problems, KRR is not suited for Koopman operator learning since without rank regularization, a large number of spurious eigenpairs pop up during estimation.

1.2

1.0

0.8

nRMSE

NysRRR(M=250)

0.6

NysRRR(M=1000)

NysKRR(M=1000)

0.4

0.2

0.0

0

1

2

3

4

Forecast time (s)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper utilizes theory developed around random projections to develop fast/efficient numerical methods for approximating the Koopman mode decomposition of data sets with long-time trajectories, that additionally come with learning rates that are optimal.\n\nSTRENGTHS:\n1. This paper developed new fast/efficient algorithms for computing the Koopman spectra, the effectiveness of which was demonstrated on very large data sets with long-time trajectories. These include a traditional benchmark (Lorenz '63), as well as molecular data sets, which are a growing area of interest to the machine learning and Koopman communities. Additionally, the molecular data sets were of a size that could not have been studied previously. \n\n2. This paper provided learning rates for their new algorithms, which they show to be optimal. \n\n3. This paper discusses how different methods for computing the Koopman mode decomposition (e.g., Nystrom RRR vs. PCR) differ in their learning rate dependencies. \n\n4. The paper was well-written.\n\nWEAKNESSES:\n1. Motivation for why very long-time trajectories of data would be needed for some systems was missing from the Introduction. This was partially discussed at the end in Sec. 5 \"Molecular dynamics datasets\", but developing this more is important. Additionally, the title and some places in the text describe the sketching approach as being useful for \"large scale dynamical systems\". As I understand it, the methods are beneficial for \"long-time trajectory dynamical systems\". Of course, the molecular examples shown have both large dimension and long-time trajectories, but the title and text should be corrected to accurately emphasize that it is long-time trajectory dynamical systems that this method is useful for. \n\n2. The discussion on how Nystrom PCR and RRR differ in their learning rates was very interesting. Including more on this (maybe in table form) would be nice. Additionally, it would be helpful to see all 3 methods compared, so KRR should be included in Figure 1. Additionally, the reason for using just RRR in Figs. 3 and 4 should be noted.\n\n3. I found it a little confusing that going from Eq. 8 to Eq. 9, the P_Y \\hat{C}_{YX} P_X  became K^\\dagger_{\\tilde{Y}, \\tilde{Y}} K_{\\tilde{Y}, Y} K_{X, \\tilde{X}}, but that going from Eq 10 to Eq 11 the P_Y \\hat{C}_{YX}  also led to K^\\dagger_{\\tilde{Y}, \\tilde{Y}} K_{\\tilde{Y}, Y} K_{X, \\tilde{X}}. I assume this has to do with the way the \\dagger was absorbed into the [ \\cdot ]_r?\n\n4. How the random $\\tilde{\\Phi}$ are generated should be explained in the text. \n\nMinor comments:\n\n1. Mezic, 2005 should be cited when discussing the Koopman mode decomposition and Budisic et al. 2012 should be cited when discussing how Koopman mode decomposition has been leveraged in the past (line 95). \n\n2. Abbreviations should be explained before using them (DMD, tICA, VAC, etc.). \n\n3. The box plot denoting the sKAF data in Fig. 2 has a strange shape in its lower box (an inverted U). Why is this? \n\n4. It would be helpful and appropriate to have citations when noting that \"An important application of Koopman operator theory is in the analysis of molecular dynamics\". \n\n5. The colors of Fig. 3 should be explained in more detail. \n\n6. It might be helpful to have a line across the 3 states in Fig. 4 to show how Eigenfunction 1 has linear separation between State 0 and the others, and Eigenfunction 2 has linear separation between State 1 and the others. \n\n7. There are a few typos: \ni. \"allows to deploy\" (line 1)\nii. \"as theirs exact\" (line 68)\niii. \"to study large class\" (line 90) \niv. \"]0, 1]\" (Assumption 4.3)\nv. \"]0, \\tau]\" (Assumption 4.4)\n\nQUESTIONS:\nTo summarize, the Weakness section above, \n\n1. Why would one want to use very long-time trajectory data when computing the Koopman mode decomposition? (You explain this some, but making it more explicit would be helpful)\n\n2. What are the differences between KRR, PCR, and RRR in their performance on the different data sets studied (at least the Lorenz '63)?\n\n3. How are the random $\\tilde{\\Phi}$ are generated?\n\nLIMITATIONS:\nThe authors do a sufficient job explaining their limitations (the assumptions they make) and what kinds of data sets their proposed methods work best on. The only exception to this is that it should be clarified that the proposed methods improve Koopman mode decomposition estimates of long-time trajectory data and not large dimension data.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes sketching algorithms for approximating three different kernel-based estimators of Koopman operators from large scale temporal data. New non-asymptotic error bounds are derived.\n\nSTRENGTHS:\n- The paper leverages the concept of sketching to speed up the estimation of Koopman operators without sacrificing accuracy. The newly proposed algorithms can potentially have a high impact in several scientific disciplines where the analysis of dynamical systems is of interest.  \n- The authors explain and contextualize very well their contributions and the related work is comprehensive. \n- The paper is very well written and organized. \n- New theoretical results are rigorously derived to provide robust guarantees for the proposed methodology.\n- The empirical analysis is sound and well-conducted. To showcase various aspects of the method, two distinct applications are considered. The first experiment effectively demonstrates the tradeoff between time and accuracy. The second experiment is particularly interesting as it provides a practical example where it is necessary to consider full, non-subsampled long trajectories. Furthermore, it focuses on the interpretation of dynamics and scientific discovery. It also highlights the scalability with respect to the state space dimension, which serves as motivation for the study of Koopman operator methods.\n\nWEAKNESSES:\n- While the idea of using sketching is not original, this contribution has the potential to significantly advance dynamical systems analysis by providing efficient learning algorithms for Koopman operators. Sketches for operators mapping between Hilbert spaces have recently been studied in [3]. However, the authors outline the unique aspects of their contribution and provide a clear motivation for deriving new results.\n\nQUESTIONS:\n- What is the complexity with respect to the dimension of the state space? Is it linear? \n- Does the Markovian assumption in line 84 present a limitation for real-world applications? Is there a way to adapt Koopman operator learning (not necessarily sketched) to the non-Markovian case?\n\nLIMITATIONS:\nLimitations are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper improves the computational efficiency of existing kernel method for Koopman operator approximation, including  KRR, PCR, RRR based on the Nystrom estimator. In this way, the authors reduce the original computational complexity from $\\mathcal{O}(n^3)$ to $\\mathcal{O}(n\\sqrt{n})$. They provide theoretical learning bounds and several numerical studies to demonstrate the efficacy of the proposed method.\n\nSTRENGTHS:\n[1] The theoretical estimation of the learning bounds guarantee the performance of the proposed methods in practical scenarios.\n\n[2] The experiments validate the efficacy of the proposed methods can scale to large dynamical systems.\n\nWEAKNESSES:\n[1] The idea of introducing random projections to the existing kernel methods is not novel enough, since the similar idea has been used in the  deep learning community, such as generative models [R1] and neural ODEs [R2].\n\n[2] The last two experiments seem to be interesting but the authors do not describe them clearly.\n\n[3] In line 62, the authors have typos about 'decomposition'.\n\n__References__\n\n[R1] Song, Y., Garg, S., Shi, J., & Ermon, S. (2020, August). Sliced score matching: A scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence (pp. 574-584). PMLR.\n\n[R2] Grathwohl, W., Chen, R. T., Bettencourt, J., Sutskever, I., & Duvenaud, D. (2018). Ffjord: Free-form continuous dynamics for scalable reversible generative models. arXiv preprint arXiv:1810.01367.\n\nQUESTIONS:\n[1] In the experiment of Molecular dynamics datasets, do the $\\phi,\\psi$ correspond to the first two eigenfunctions with largest eigenvalues of the approximated Koopman matrix?  And why not demonstrate the predictive performance of the proposed methods in the original 45-D state space in this task?\n\nLIMITATIONS:\nThe authors have not provided experiments to demonstrate the efficacy of the proposed methods with the non-i.i.d. data, but the current work is complete anyway.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper talks about estimating Koopman operators. Sketching (random projections), specifically the Nyström method is used as a computational tool to make things tractable. Furthermore, it is shown that these approximates have the same convergence rate as their slower counter parts. Further error bounds are provided in terms of the operator norm.\n\nSTRENGTHS:\nThe authors do a good job in positing the current work within the literature which is very wide otherwise.\n\nThe flow of the paper is very good and reads well. \n\nAssumptions in Section 4 are clearly mentioned before stating the main result.\n\nI appreciate authors commenting on the non iid case, apart from detailed analysis of the iid scenario.\n\nWEAKNESSES:\nI view the papers contributions as two fold. \n\nThe first part is the use of sketching for approximation of the three scenarios of KRR, PCR, RRR considered. In this case (as authors have acknowledged, references [3] and [6] therein), there are  other works who have pointed this out. \n\nAgain authors mention the special case of their sketched PCR as kernel DMD. These make the contributions of the current work from this point of view a bit restricted. Having said that, I greatly appreciate authors mentioning them in the paper!\n\nThe second is in terms of learning bounds via operator norm, where I think the contribution is solid. \n\nHowever, it is hard to have empirical evidence to support this and is understandable. It would be good to have some discussion around this, as Koopman operators are heavily used in practical settings and having some evidence would greatly improve the impact of the paper over it's current form.\n\nQUESTIONS:\n1) What is $A^\\ast$ is line 104? The equation 5 just above has only $A$, are they the same?\n2) In experimental evaluation, only PCR and RRR are considered, what about KRR ?\n\nLIMITATIONS:\nI don't see a direct potential negative social impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}