SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As a functional approximation problem, the solution of (11) is not parameterized by vector-valued coefficients, but rather functions of time. Although there are a few options to deal with function-valued solutions [53], we consider a vector-valued solution. A common drawback of such a discretization involves the loss of the inter-sample relations along the continuous signal. Crucially, this problem does not apply in our case, as the inter-sample relationships remain modeled for the discrete-time "Koopman kernel" due to its causal structure. Importantly, the vector-valued solution allows us to preserve all of the desirable properties derived in the continuous case.
Consider sampling  $[0, T]$  at  $H = T / \Delta t$  regular intervals to yield a discrete-time dataset from Assumption 1, discretized at points  $\mathbb{H} \equiv \{t_0 \cdots t_H\}$ . As a discretization of a function over time, with a slight abuse of notation, we denote the target vectors as  $y_{\mathrm{H}} = [y(t_0) \cdots y(t_{\mathrm{H}})]^{\top}$ . Thus, we are solving the time- and data-discretized version of the problem (5) that takes the form of a linear coregionalization model [58, 59].
Corollary 1 (Time-discrete Koopman kernel). Consider trajectory data  $\{\pmb{x}_{\mathrm{H}}^{(i)}\}_{i = 1}^{N}$  and let  $\mu_{j}\coloneqq \mathrm{e}^{\lambda_{j}\Delta t}$ ,  $\pmb{\mu}_{j}^{\top}:= [\mu_{j}^{0}\dots \mu_{j}^{H}]$ . Then, the scalar-induced matrix kernel  $K^{\mu_j}\colon \mathbb{X}_\mathrm{H}\times \mathbb{X}_\mathrm{H}\mapsto \mathcal{B}(\mathbb{Y}_\mathrm{H})$
$$
\boldsymbol {K} ^ {\mu_ {j}} \left(\boldsymbol {x} _ {\mathrm {H}}, \boldsymbol {x} _ {\mathrm {H}} ^ {\prime}\right) = \boldsymbol {\mu} _ {j} \boldsymbol {\mu} _ {j} ^ {* \top} \underbrace {\frac {1}{(H + 1) ^ {2}} \sum_ {m = 0} ^ {H} \sum_ {n = 0} ^ {H} \mu_ {j} ^ {- m} k ^ {j} \left(\boldsymbol {x} _ {\mathrm {H}} \left(t _ {m}\right) , \boldsymbol {x} _ {\mathrm {H}} ^ {\prime} \left(t _ {n}\right)\right) \mu_ {j} ^ {* - n}} _ {k ^ {\mu_ {j}} \left(\boldsymbol {x} _ {\mathrm {H}}, \boldsymbol {x} _ {\mathrm {H}} ^ {\prime}\right)}, \tag {16}
$$
satisfies the properties (i)-(iii) from Theorem 1 over  $\mathbb{H}$ , so that it defines an RKHS  $\mathcal{H}^{\mu_j}$ , is universal per Definition 1 over  $\mathbb{H}$  with  $\mathrm{span}\{\pmb{K}^{\mu}(\cdot, \pmb{x}_{\mathrm{H}}^{(1)}), \dots, \pmb{K}^{\mu}(\cdot, \pmb{x}_{\mathrm{H}}^{(N)})\}$  ( $\mathbf{KI}$ ) over  $\{\pmb{x}_{\mathrm{H}}^{(i)}\}_{i=1}^{N}$ . Given a collection of kernels  $\{\pmb{K}^{\mu_j}\}_{j=0}^{\bar{D}}$ , the matrix Koopman kernel  $\pmb{K}^{\mu_j} \colon \mathbb{X}_{\mathrm{H}} \times \mathbb{X}_{\mathrm{H}} \mapsto \mathcal{B}(\mathbb{Y}_{\mathrm{H}})$
$$
\boldsymbol {K} \left(\boldsymbol {x} _ {\mathrm {H}}, \boldsymbol {x} _ {\mathrm {H}} ^ {\prime}\right) = \sum_ {j = 1} ^ {\bar {D}} \boldsymbol {K} ^ {\mu_ {j}} \left(\boldsymbol {x} _ {\mathrm {H}}, \boldsymbol {x} _ {\mathrm {H}} ^ {\prime}\right), \tag {17}
$$
satisfies the properties (i)-(ii) from Proposition 1 over  $\mathbb{H}$ , defining  $RKHS\mathcal{H}^{\Delta t} \coloneqq S(\mathcal{H}^{\mu_1} \oplus \dots \oplus \mathcal{H}^{\mu_D})$ .
Now, we are fully equipped to obtain the time-discrete solution to our initial problem (5) provided a dataset of trajectories. Before presenting the solution to Koopman Kernel Regression, we introduce some helpful shorthand notation. We use the following kernel matrix abbreviations:  $k_{\boldsymbol{X}\boldsymbol{X}} = [k(\boldsymbol{x}^{(a)}, \boldsymbol{x}^{(b)})]_{a,b=1}^{N}, k(\boldsymbol{x}, \boldsymbol{X}) = [k(\boldsymbol{x}, \boldsymbol{x}^{(b)})]_{b=1}^{N}, K_{\boldsymbol{X}\boldsymbol{X}} = [K(\boldsymbol{x}^{(a)}, \boldsymbol{x}^{(b)})]_{a,b=1}^{N}$  and  $K(\boldsymbol{x}, \boldsymbol{X}) = [K(\boldsymbol{x}, \boldsymbol{x}^{(b)})]_{b=1}^{N}$ .
Proposition 2 (KKR). Consider a discrete-time dataset of Assumption 1,  $\mathbb{D}_N^{\Delta t} = \{\pmb{x}_{\mathrm{H}}^{(i)},y_{\mathrm{H}}^{(i)}\}_{i = 1}^{N}$ , and let  $\pmb{y}_{\mathrm{H}}^{\top} = [y_{\mathrm{H}}^{(1)\top}\dots y_{\mathrm{H}}^{(N)\top}]$  with  $\otimes$  the Kronecker product. Then,
$$
\boldsymbol {\alpha} _ {j} = k _ {\boldsymbol {X} _ {0} \boldsymbol {X} _ {0}} ^ {- 1} k _ {\boldsymbol {X} _ {\mathrm {H}} \boldsymbol {X} _ {\mathrm {H}}} ^ {\mu_ {j}} \left(\boldsymbol {I} _ {N} \otimes \boldsymbol {\mu} _ {j} ^ {* \top}\right) \underline {{\boldsymbol {\beta}}}, \quad \underline {{\boldsymbol {\beta}}} = \left(\boldsymbol {K} _ {\boldsymbol {X} _ {\mathrm {H}} \boldsymbol {X} _ {\mathrm {H}}} + \gamma \boldsymbol {I} _ {H + 1} \otimes \boldsymbol {I} _ {N}\right) ^ {- 1} \boldsymbol {y} _ {\mathrm {H}} \tag {18}
$$
defines a unique time-sampled solution to (11) in terms of eigenfunctions  $\hat{\phi}(\pmb{x}_0) = [k_{\pmb{x}_0 \pmb{X}_0}^j \pmb{\alpha}_j]_{j=1}^{\bar{D}}$ , determining an LTI predictor with  $\Lambda = \mathrm{diag}([\mu_1 \cdots \mu_D])$ ,
$$
z ^ {+} = \boldsymbol {\Lambda} z, \quad z _ {0} = \hat {\phi} \left(\boldsymbol {x} _ {0}\right), \tag {19a}
$$
$$
\hat {y} = \mathbf {1} ^ {\top} \mathbf {z}. \tag {19b}
$$
Notice how in (18), we re-scale the trajectory domain to that of the state-space. This enables us to write the forecast of (19), with a slight abuse of notation, using an extended observability matrix [60]
$$
\hat {y} _ {\mathrm {H}} = \boldsymbol {\Gamma} \hat {\phi} (\boldsymbol {x} _ {0}), \quad \boldsymbol {\Gamma} := \left[ \mathbf {1} ^ {\top} \mathbf {1} ^ {\top} \boldsymbol {\Lambda} \dots \mathbf {1} ^ {\top} \boldsymbol {\Lambda} ^ {H} \right] ^ {\top}. \tag {20}
$$
The confinement to a non-recurrent domain plays a crucial role in making the base kernel RKHSs isometric to "eigen-RKHSs"  $\mathcal{H}^{k^j} \cong \mathcal{H}^{k^{\mu_j}}$  via invariance transforms, guaranteeing a feasible return from the time-series domain  $\mathbb{X}_T$  to the state-space domain  $\mathbb{X}_0 \subseteq \mathbb{X}$  for evaluating the model over initial conditions.
Remark 1. The salient feature of our proposed KKR framework compared to existing methods is the fact that Koopman-invariance (KI) over data samples is independent from the outcome of an optimization algorithm, e.g. minimizing the forecasting risk to compute  $\beta$  in (18). Thus, we are able to directly optimize for a downstream task (forecasting) (OR) given a suitably rich set of eigenvalues.
For discrete-time predictors, we omit the time-step specification and denote the next state with “ $(\cdot)^{+}$ ”.
3.3 Selecting Eigenvalues
Until now, we have used the sufficient cardinality  $\bar{D} \in \mathbb{N}$  of an eigenvalue set that encloses [61] or is the true spectrum. However, we have provided no insight regarding the selection of  $\bar{D}$  spectral components or how they can be estimated. Here, we go beyond the learning-independent and nonconstructive existence result of [28] and provide a consistency guarantee and relate it to sampling eigenvalues without the knowledge of the true spectrum.
Proposition 3. Consider the oracle Koopman kernel  $\pmb{K}(\pmb{x}_{\mathrm{H}},\pmb{x}_{\mathrm{H}}^{\prime})$  and a dense set  $\{\mu_j\}_{j = 1}^{\infty}$  in  $\overline{\mathbb{B}_1(0)}$ . Then,  $\| K(\pmb{x}_{\mathrm{H}},\pmb{x}_{\mathrm{H}}^{\prime}) - \sum_{j = 1}^{D}\pmb{K}^{\mu_j}(\pmb{x}_{\mathrm{H}},\pmb{x}_{\mathrm{H}}^{\prime})\|_{\mathcal{B}(\mathbb{Y}_{\mathrm{H}})}\to 0,\forall \pmb{x}_{\mathrm{H}},\pmb{x}_{\mathrm{H}}^{\prime}\in \mathbb{X}_{\mathrm{H}}$  as  $D\to \infty$ .
As shown in Proposition 3, even if we do not know the oracle kernel, we can arbitrarily approximate it by sampling from a dense set supported on the closed complex unit disk  $\overline{\mathbb{B}_1(0)}$  [57, Theorem 3.0.2] with the error vanishing in the limit  $D\to \infty$ . There is no loss of generality when considering the unit disk as any finite radius disk can be scaled in the interval  $[0,T]$ . Furthermore, approximation of the oracle kernel by sampling a distribution over  $\overline{\mathbb{B}_1(0)}$  leads to an almost sure  $\mathcal{O}(1 / \sqrt{D})$  convergence rate. It is conceivable that faster rates can be obtained in practice by including prior knowledge to shape the spectral distribution, e.g. using well-known concepts such as leverage-scores or subspace orthogonality [62, 63]. Based on spectral priors one can include a more biased sampling technique by precomputing components of the operator spectrum, e.g. computing Fourier averages [64], to determine the phases  $\omega_{j}$  of complex-conjugate pairs  $\mu_{j,\pm} = |\mu_j| \mathrm{e}^{\pm \mathrm{i}\omega_j}$  and sample the modulus from another physics-informed distribution. However, rigorous considerations of optimized and efficient sampling are beyond the scope of this paper and rather a topic of future work.
3.4 Numerical Algorithm and Time-Complexity
Algorithm 1 Regression and LTI Forecasts using KKR
Data  $\mathbb{D} = \{\pmb{x}_{\mathrm{H}}^{(i)},y_{\mathrm{H}}^{(i)}\}_{i = 1}^{N}$  ,Eigenvalues  $\{\mu_j\}_{j = 1}^D$    
function REGRESS(D,  $\{\mu_j\}_{j = 1}^D$  ) form Gramians  $k_{X_0X_0},\{k_{X_HX_H}^{\mu_j}\}_{j = 1}^D,K_{X_HX_H}$  fit mode operator  $\hat{M} (\cdot):\mathbb{X}_{\mathrm{H}}\mapsto \mathbb{Y}_{\mathrm{H}}$  (18,right) recover eigenfunctions  $\hat{\phi} (\cdot):\mathbb{X}_0\mapsto \mathbb{Z}_0$  (18,left) construct  $\Gamma :\mathbb{Z}_0\mapsto \mathbb{Y}_\mathrm{H}$  (20,right) return LTI predictor  $\Gamma \hat{\phi} (\cdot):\mathbb{X}_0\mapsto \mathbb{Y}_\mathrm{H}$    
end function   
function FORECAST(xo) "lift"  $z_0 = \hat{\phi} (x_0)$  rollout  $\hat{y}_H = \Gamma z_0$  return trajectory  $\hat{y}_H$    
end function
For a better overview, the pseudocode for regression and forecasting of our method are shown in Algorithm 1. We also put the time-complexity of our algorithm into perspective w.r.t. Koopman operator regression of PCR/RRR [23] and ridge regression using state-of-the-art signature kernels [48] (RR-Sig-PDE) in Table 1. The training complexity of our KKR is comparable to that of RR-Sig-PDE regression and generally better than that of PCR/RRR. Given that accurate LTI forecasts require higher-rank predictors, the seemingly mild quadratic dependence makes  $D^{2} > NH$  and leads to a more costly matrix inversion. Furthermore, our LTI predictor also has a slightly better forecast complexity due to
not depending on trajectory length. Obviously, due to a mere matrix multiplication after an initial nonlinear map, LTI predictors have a significantly lower evaluation complexity than the nonlinear predictor of Sig-PDE's. Due to requiring updated observation sequences as inputs, Sig-PDE kernels introduce a raw evaluation complexity that is also quadratic in sequence length.
Method	Training	H-step forecast	N	# trajectories	Table 1:
KKR (ours)	O(N3H3+DN2H2d)	O(DH+DNd)	H	trajectory length	Time com-
PCR/RRR	O(D2N2H2+N2H2d)	O(DH+DNHd)	D	predictor rank	plexities.
RR-Sig-PDE	O(N3H3+N2H2l2d)	O(NH2l2d)	l	# time-delays
d	dim(input data)
4 Learning Guarantees
With a completely defined KKR estimator, we assess its essential learning-theoretic properties, i.e., the behavior of the learned functions w.r.t. to the ground truth with increasing dataset size.
4.1 Consistency
Although well-established in most function approximation settings [65-67], the setting of Koopman-based LTI predictor learning for nonlinear systems is void of consistency guarantees. Here we use a definition of universal consistency from [68] that describes the uniform convergence of the learned function to the target function as the sample size goes to infinity for any compact input space  $\mathbb{X}$  and every target function  $q\in C(\mathbb{X})$ . The existing convergence results for Koopman-based LTI predictors [69] are in the sense of strong operator topology — allowing the existence of empirical eigenvalues that are not guaranteed to be close to true ones even with increasing data [70]. This lack of spectral convergence has a cascaded effect in Koopman operator regression as, in turn, the convergence of eigenfunctions and mode coefficients is not guaranteed. Here, the convergence of modes is replaced by the convergence of eigenfunctions, and convergence of spectra is replaced by the convergence of (20) to the mode decomposition operator  $\hat{M}\equiv \Gamma \hat{\phi}\rightarrow M\equiv \Gamma \phi$  with the estimate denoted by  $(\hat{\cdot})$ .
Theorem 2 (Universal consistency). Consider a universal kernel  $\pmb{K}$  (17) and a data distribution supported on  $\mathbb{X}_{\mathrm{H}} \times \mathbb{Y}_{\mathrm{H}}$ . Then, as  $N \to \infty$ ,  $\| M - \hat{M} \|_{\mathbb{Y}_{\mathrm{H}}} \to 0$  and  $\| \phi_{\mu_j} - \hat{\phi}_{\mu_j} \|_{\mathbb{Y}_{\mathrm{H}}} \to 0, \forall j = 1, \ldots, D$ .
4.2 Generalization Gap: Uniform Bounds
Due to formulating the LTI predictor learning problem as a function regression problem in an RKHS, we can utilize well-established concepts from statistical learning to provide bounds on the generalization capabilities of KKR. Given a dataset of trajectories, the following empirical risk is minimized
$$
\hat {\mathcal {R}} _ {N} (\hat {M}) := \frac {1}{N} \sum_ {i \in [ N ]} \| y _ {\mathrm {H}} ^ {(i)} - \hat {M} (\boldsymbol {x} _ {\mathrm {H}} ^ {(i)}) \| _ {\mathbb {Y} _ {\mathrm {H}}} ^ {2}
$$
which is "in-sample" mean square error (MSE) w.r.t. a trajectory-data generating distribution  $\rho_{\mathcal{D}}$  of i.i.d. initial conditions. The true risk/generalization error of an estimator is the "out-of-sample" MSE of the model on the entire domain and denoted as  $\mathcal{R}(\cdot)$ . Those quantities are, in essence, the model's performance on test and training data, respectively. Allowing for statements on the test performance with an increasing amount of data by means of training performance is a desirable feature in data-driven learning. Hence, we analyze our model in terms of the generalization gap
$$
\left. \left| \mathcal {R} (\hat {M}) - \hat {\mathcal {R}} _ {N} (\hat {M}) \right| = \left| \mathbb {E} _ {\left(\boldsymbol {x} _ {\mathrm {H}}, y _ {\mathrm {H}}\right) \sim \rho_ {\mathcal {D}}} \left[ \| y _ {\mathrm {H}} - \hat {M} \left(\boldsymbol {x} _ {\mathrm {H}}\right) \| _ {\mathbb {Y} _ {\mathrm {H}}} ^ {2} \right] - \frac {1}{N} \sum_ {i = 1} ^ {N} \| y _ {\mathrm {H}} ^ {(i)} - \hat {M} \left(\boldsymbol {x} _ {\mathrm {H}} ^ {(i)}\right) \| _ {\mathbb {Y} _ {\mathrm {H}}} ^ {2} \right|. \right. \tag {21}
$$
To ensure a well-specified problem, we require models in the hypothesis to admit a bounded norm.
Assumption 2 (Bounded RKHS Norm). The unknown function  $M$  has a bounded norm in the RKHS  $\mathcal{H}^{\Delta t}$  attached to the Koopman kernel  $\pmb{K}(\cdot, \cdot)$ , i.e.,  $\| M\|_{\mathcal{H}^{\Delta t}} \leq B$  for some  $B \in \mathbb{R}_+$ .
The above smoothness assumption is mild, e.g., satisfied by band-limited continuous trajectories [71] and computable from data [72, 73]. In stark contrast, well-specified Koopman operator regression [23] requires the operator to map the RKHS onto itself, which is a very strong assumption [34, 35].
To derive the main result of this section, we utilize the framework of Rademacher random variables for measuring complexity of our model's hypothesis space, a concept generally explored in [74] and more particularly for classes of operator-valued kernels in [75]. Conveniently, the derivation is, in terms of the RKHS  $\mathcal{H}^{\Delta t}$ , similar to standard methods on RKHS-based complexity bounds [74]. We use well-known results based on concentration inequalities to provide high probability bounds on a model's generalization gap in terms of those complexities. Finally, we upper bound any constant with quantities specified in our assumptions and can state the following result.
Theorem 3 (Generalization Gap of KKR). Let  $\mathbb{D}_N^{\Delta t} = \{\pmb{x}_{\mathrm{H}}^{(i)},y_{\mathrm{H}}^{(i)}\}_{i = 1}^N$  be a dataset as in Assumption 1 consistent with a Lipschitz system on a non-recurrent domain. Then the generalization gap (21) of a model  $\hat{M}$  from Proposition 2 under Assumption 2 is, with probability  $1 - \delta$ , upper bounded by
$$
\left| \mathcal {R} (\hat {M}) - \hat {\mathcal {R}} _ {N} (\hat {M}) \right| \leq 4 R B \sqrt {\frac {\kappa H ^ {2}}{N}} + \sqrt {\frac {8 \log \frac {2}{\delta}}{N}} \in \mathcal {O} \left(\frac {H}{\sqrt {N}}\right), \tag {22}
$$
where  $R$  is an upper bound on the loss in the domain, and  $\kappa$  the supremum of the base kernel.
We observe an overall dependence of order  $\mathcal{O}(1 / \sqrt{N})$  w.r.t. data points, resembling the regular Monte Carlo rate to be expected when working with Rademacher complexities. Remarkably, an increase in the order of the predictor  $D$  cannot widen the generalization gap but will eventually decrease the
Figure 2: Forecasting performance (48 i.i.d. runs) for the bi-stable system for  $H = 14$  and  $N = 50$  for respectively optimal  $D_{\mathrm{KKR}} = 100$ ,  $D_{\mathrm{PCR}} = 10$  and 15 delays for Sig-PDEs. Left: Exemplary trajectories showing the advantage of learning with time-series kernels. Right: The generalization gap with an increasing forecast horizon, demonstrating generalization advantages of KKR.
empirical risk due to the consistency of eigenspaces (Proposition 3). Combined, our findings are a substantial improvement, both quantitatively and in terms of interpretability, over existing risk bounds on forecasting error [23, Theorem 1]. Additionally, our intuitive non-recurrence requirement is easily verifiable from data. In contrast, the Koopman operator regression in RKHS comes with various strong assumptions [35] that require commonly unavailable expert knowledge. Also, the generalization of existing Koopman-based statistical learning approaches depends on rank while ours is rank-independent. The significant implications of our results are demonstrated in the following.
In our experiments<sup>6</sup>, we report the squared error of the forecast vector for the length of data trajectories averaged over multiple repetitions with corresponding min-max intervals. We validate our theoretical guarantees and compare to state-of-the-art operator and time-series approaches in RKHS. For fairness, the same kernel and hyperparameters are chosen for our KKR, PCR (EDMD), RRR [23] and regression with signature kernels (Sig-PDE) [48]. Note, PCR and RRR are provided with the same trajectory data split into one-step data pairs while the time and observation time-delays are fed as data to the Sig-PDE regressor due to its recurrent structure. Along with code for reproduction of our experiments, we provide a JAX [76] reliant Python module implementing a sklearnn [77] compliant KKR estimator at https://github.com/TUM-ITR/koopcore.
Bi-stable system Consider an ODE  $\dot{x} = ax + bx^3$  that arises in modeling of nonlinear friction. The parameters are  $a = 4$ ,  $b = -4$ , making for a bi-stable system at fixed points  $\pm 1$ . The numerical results are depicted in Figure 2. Sample trajectories both on training and testing data indicate the utility of the forecast risk minimization of KKR. While EDMD correctly captures the initial trend of most trajectories it fails to match the accuracy of Sig-PDE or our KKR predictors that utilize time-series structure. Furthermore, the behavior of KKR's generalization gap for an increasing time horizon  $T = H\Delta t$ ,  $\Delta t = 1/14s$  closely matches our theoretical analysis.
Van der Pol oscillator Consider an ODE  $\ddot{x} = \dot{x}(2 - 10x^2) - 0.8x$  describing a dissipative system whose nonlinear damping induces a stable limit cycle — a phenomenon present in various dynamics.
In Figure 3 two fundamental effects are validated: the generalization gap with increasing data and consistency with test risk that does not deteriorate for increasing eigenspace cardinality. The performance of PCR/RRR is strongly tied to predictor rank while Sig-PDE's less so w.r.t. delay length.
Table 2: Average risk (20 runs)  $[ \times 10^{-2} ]$  for Van der Pol for various spectral sampling and lengthscales,  $N = 200$ ,  $H = 14$ .
ρ(μ)	uniform	boundary-biased	physics-informed
D	16	200	16	200	16	200
Rℓ=101	13.7	5.38	11.2	5.38	5.60	5.58
Rℓ=100	6.46	0.78	4.10	0.78	0.97	0.92
Rℓ=10-1	7.12	1.74	4.33	1.74	1.83	1.80
Crucially, our KKR approach does not require a careful choice of the eigenspace cardinality to perform for a specific amount of data. Although the eigenvalues that determine the eigenspaces are randomly chosen from a uniform distribution in the unit ball, KKR consistently outperforms PCR/RRR. In Table 2 we show the spectral sampling and hyperparameter effects. We employ the following strategies: uniform - uniform distribution on the complex unit disk, boundary-biased - a distribution on the complex unit disk skewed towards the unit circle, physics-informed - eigenvalues of various vector field Jacobians. As expected, physics-informed performs well with lower rank
$^{6}$ Additional details on the numerical experiments can be found in the supplementary material.
Figure 3: Forecasting risks (20 i.i.d. runs) for the Van der Pol system over a time-horizon  $H = 14$  ( $T = 1s$ ). Left: Generalization gap for the best  $D / l$  (ours 500, PCR 62, RRR 100, RR-Sig-PDE 10) is depicted with a growing number of data points. Right: Test risk behavior with an increasing amount of eigenspaces is shown for  $N = 200$ . Shaded areas depict min-max risk intervals.
Figure 4: Cumulative error and forecast risks (5 train-test splits) for flow past cylinder data and  $H = 99$ . Our KKR with orders-of-magnitude greater usable  $\ell$ -range and accuracy. Left: Cumulative absolute error for the best  $D / \ell$  (ours 200/70, PCR 200/35) is depicted over timesteps. Right: Forecast risk for 99 steps within a range of RBF lengthscales. Shaded areas depict min-max intervals.
compared to uninformed approaches. However, it is outperformed by unit-ball sampling approaches for higher rank due to a lack of coverage. Table 3 includes CPU timings for completeness.
#data = N × H = 200 × 14	KKR	PCR	RRR	Sig-PDE	Table 3: Computation times for Van der Pol.
Training [s]/Forecast [ms]	8.0/54	90/84	88/150	8.6/5900
Flow past a cylinder We consider high-dimensional data of velocity magnitudes in a Kármán vortex street under varying initial cylinder placement, as illustrated in Figure 5.
The cylinder position is varied on a  $7 \times 7$  grid in a  $50 \times 100$ -dimensional space and the flow is recorded over  $H = 99$ . The quantity of interest is a velocity magnitude sensor placed in the wake of the cylinder. In forecasting from an initial velocity field, KKR outperforms PCR by orders-of-magnitude as shown in Figure 4. We omit Sig-PDE regression due to persistent divergence after  $\approx 20$  steps. The latter is hardly surprising, given that Sig-PDE models iterate one step predictions based only on the shapes of time-delays while LTI models directly output time-series from initial conditions.
Figure 5: Flow illustration. Area of initial cylinder positions shaded.