SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In Takens' formulation, in the case of infinitely long, noise-free time series, there is no importance bias towards any specific lag  $s$ , and their number  $k$  is the only relevant factor. This is often determined using the False Nearest Neighbours algorithm (Kennel et al., 1992; Wallot & Mønster, 2018). However, when dealing with limited noisy observations, the choice of  $s$  becomes equally critical as it directly relates to the sampling frequency (Broomhead & King, 1986). In particular, using too high a value would force a modeling of uncorrelated points while too low would model noise. A correct choice of  $s$  instead allows for extracting seasonality or other relevant trends as well as the underlying dynamics. In practice, for noisy systems, the choice of both  $k$  and  $s$  is inherently a difficult problem due to the absence of clear theoretical guidelines (Tran & Hasegawa, 2019). Despite that, interpretability can be exploited to propose task-dependent heuristics (Small & Tse, 2004), often requiring a trade-off between redundancy and the irrelevance of concatenated features.
In this work, we propose two settings. The first is a rule of thumb and proceeds as follows. We first apply a trend-filtering method to the time series, such as an  $\ell_1$  trend (Kim et al., 2009) (this allows for a choice of  $s$  that captures the temporal variability of the system rather than the superposed noise). We then set the product  $k\cdot s$  as the average distance between peaks and hollows (dbp) (Kugiumtzis, 1996). Finally, we balance the contributions by adopting  $s = k = \sqrt{\mathrm{dbp}}$ . Alternatively, in a supervised setting, both parameters are optimized by CV.
Polynomial order  $(n)$  The polynomial order controls the amount of non-linearity in the embedding. As in most previous works, we found no need for further complexity than  $n = 2$ .
Embedding dimension  $(\bar{d}_r)$  In Takens' formulation, an embedding size double the dimensionality of the underlying manifold is sufficient for reconstructing the system dynamics. However, when dealing with real-world data, this is unknown, and a trade-off must be struck between the omission of relevant features and redundancy, which can increase collinearity and computation times. In line with the scope of our work, we choose to give priority to a fair comparison with previous RC
methods (Bianchi et al., 2020), which identify  $\bar{d}_r = 75$  as an optimal dimensionality before the readout. In Appendix C.1, we undertake a brief exploration to determine whether this value is also a reasonable choice for NVAR embeddings. While we did not observe any redundancy in univariate datasets, we did observe a performance drop after different  $\bar{d}_r$  for some multivariate datasets. The study suggests  $70 \lesssim \bar{d}_r \lesssim 100$  as a suitable range.
Linear readout regularization  $(\lambda_{ridge})$  For the readout, we adopt the optimal regularization (OCReP) proposed in Cancelliere et al. (2015), i.e., set  $\lambda_{ridge}$  to the product between the minimum and maximum singular value of  $\mathbf{R}_{NVAR}|_{\bar{d}_r}$ .
RBF lengthscale  $(\gamma_{rbf})$  We set the lengthscale in Eq. 6 to the median pairwise distance between all the representations  $\theta[i]$  in the training set.
3.4 Asymptotic complexity
NVARk is based on the individual representations of time series. Hence, the computationally intensive step is performed only once per time series and results in  $\mathcal{O}(N)$  complexity. This also implies that the computation can be parallelized along N. This provides a significant computational advantage over comparison-based kernels, which must perform expensive computations for all the relevant pairs of series, i.e.,  $\mathcal{O}(N^2)$ .
In terms of scalability with the length of the series  $(T)$ , the advantage of NVAR over any recursive RC approach is the critical difference in how the hidden states are created. Eq. 5 is not iterative and only performs a fixed number  $\bar{d}_r - d_x$  of concatenation operations, independent of the length of the time series. In contrast, reservoir-based methods are limited by the expensive recursive update of the reservoir state (Eq. 2), which is  $\mathcal{O}(Td_r^2)$ , where  $d_{r}$  is the size of the reservoir (which is usually quite large). As evidence, improving the scalability of the reservoir update is an active area of research (Dong et al., 2020). The complexity of NVARk is limited to the only ridge regression of the output layer. This corresponds to solving a linear system with  $T$  training examples and  $D$  features, where D is at most  $\bar{d}_r$ . Such a system can usually be solved efficiently, e.g. by using LU (or Cholesky) factorization in asymptotically  $\mathcal{O}(Td_r^2)$ .
As for the scalability with the dimensionality of the input series, all operations are bounded by  $\tilde{d}_r^2$ , which we treat as a constant and do not discuss complexity in connection to this parameter. Overall, NVARk exhibits an asymptotic complexity at most  $\mathcal{O}(NT\tilde{d}_r^2)$ .
In this section, we present an experimental demonstration of the performance of NVARk. In line with the established literature (Cuturi, 2011; Cuturi & Doucet, 2011; Baydogan & Runger, 2016; Paparrizos et al., 2020), our main evaluation consists of time series classification. In particular, we pair different pre-computed kernel matrices with a Support Vector Machine (SVM) classifier (Steinwart & Christmann, 2008), a popular kernel classification method that lies between simple linear models and more advanced deep learning architectures. The evaluation is performed over 130 UTS datasets from the UCR archive and 19 MTS datasets from the UEA archive (Bagnall A. & E.) (Appendix B.1) and focuses on accuracy, execution time and scalability. In Appendix C.5, we present a snapshot of kernel Principal Component Analysis (kPCA) for dimensionality reduction and visualization.
Performance of the proposed approach is investigated under two different hyperparameter settings. We indicate with NVARk the setting in which  $k$  and  $s$  are determined by the general proposed rule of thumb of Sec. 3.3. The maximum number of dimensions is set to  $\bar{d}_r = 75$ . As for the other hyperparameters, we follow the setting given in Sec. 3.3. We instead indicate with NVARk* the configuration in which  $k$  and  $s$  are optimized in a CV loop over a small grid. As the datasets cover a wide range of series lengths, the CV grid is slightly adapted consequently (Appendix. B.2).
The comparison is performed against previously proposed kernels: the reservoir-model Echo State Network (rmESN) (Bianchi et al., 2020) as the reservoir equivalent to our approach. We set  $D = 75$  as the dimensionality before the linear readout. This allows us to directly compare the quality of the extracted dynamic features with or without a reservoir; the Time Cluster Kernel (TCK) (Mikalsen et al., 2018) as representative of model-based kernels and SOTA for MTS; the Shift-INvariant
Kernel (SINK) (Paparrizos & Franklin, 2019) as representative of Fourier-based methods and SOTA for UTS. A comparison with the Global Alignment Kernel (GAK) (Cuturi, 2011) is presented in Appendix C.4. Guidelines in Appendix B.2 contain the hyperparameter settings and links to the code implementations for NVARK and all the aforementioned baselines.
Despite the fact that some datasets in our benchmark can be also investigated using Deep Learning (DL) methods (Ismail Fawaz et al., 2019), the use case for kernel methods is substantially different, usually focused on small datasets, where the application of DL is not trivial. In particular, kernels can compute the similarity between just two time series right away. Consequently, in the literature, kernels are usually not directly tested against DL. Nevertheless, in our work, we compare to ESN-based representations, which are known to be competitive with a variety of DL architectures (Bianchi et al., 2020; Shahi et al., 2022). Thus, ESN can be seen as a proxy for more complex architectures.
As preprocessing, we apply zero-padding to match all series to the maximum length in each dataset (to allow comparison with TCK) and follow the common practice of standardizing all series to zero mean and unit variance. For each dataset-method pair, we allow for a maximum execution time of 20 hours.
4.1 SVM classification of UTS
We show here SVM classification for univariate datasets. SVM requires the setting of the hyperplane regularization  $C$ , which we optimize in a CV loop from logarithmically spaced values in  $[10^{-3}, 10^{3}]$ . This is the only free parameter in all methods except NVARk*, in which it is jointly optimized with  $k$  and  $s$ . For the CV loop, we adopt a 10-fold CV with the size of the validation set corresponding to  $33\%$  of the train set. SVM accuracy for each dataset is obtained by averaging 10 repetitions with different seeds. Tab. 1 reports the average classification accuracy for all kernels within the presented framework, as well as the relative ranking (individual results are reported in Appendix D). Overall,
Table 1: SVM classification of UTS datasets, performance metrics across 130 datasets.
NVARK*	NVARK	RMESN	TCK	SINK
AVERAGE SCORE	0.803	0.779	0.757	0.726	0.771
AVERAGE RANK	/	2.14	2.62	2.85	2.20
N FIRST RANKED	/	52	26	23	43
MEDIAN TIME (S) / DATASET	14.27	0.59	40.87	44.24	23.73
NVARk obtains very competitive results, with the highest average accuracy and rank. The case of univariate datasets is a comfortable scenario for NVARk to operate in as, in most cases, all possible terms in  $\mathbf{R}_{NVAR}$  can be considered without exceeding  $\bar{d}_r$  and the effect of collinearity is minimized. NVARk and SINK appear indistinguishable under a Friedman  $^+$  Nemenyi statistical test, which places NVARk under the SOTA in terms of accuracy. Interestingly, the difference between NVARk and rmESN is significant at the  $95\%$  level with p-value  $\mathrm{NVARk - rmESN} = 0.013$  . Results for NVARk* show that the performance of NVARk can be very efficiently improved by a CV optimization of just the two integer embedding hyperparameters.
4.2 Execution time and scalability
In terms of execution time, the scalability properties of NVARk (Sec. 3.4) manifest in exceptional performance. In Fig. 2 (left) and Tab. 1 (last row), we compare running times to compute the traintrain and the train-test kernel matrices and observe that the improvement of NVARk is consistent across datasets and baselines, with speedup factors up to  $\times 10^{3}$  in the high- $T$  and high- $N$  regimes.
Results for NVARk* can be achieved with a slowdown factor of  $\sim 25$  (parallelized grid search + 1 iteration with optimal parameters) with respect to the heuristic-based NVARk. Interestingly, for 113/130 datasets, NVARk* execution time is faster than a single iteration for rmESN.
For a study of the scaling, in Fig. 2 (middle, right) we progressively vary the length of the Abnormal-Heartbeat dataset (by interpolation) and the number of training samples (by random sampling) in the Crop dataset. Results show that NVARk performs exceptionally well in all regimes, while rmESN
Figure 2: Left: For all univariate datasets, the plot shows the ratio between the execution time to compute the train-train and train-test kernels of different methods and NVARk. Middle: Scalability of kernels with the series length. Right: Scalability of kernels with the training size.
scales poorly with the length of the series and TCK and SINK with the size of the training sample. NVARk, instead, is advantaged by its non-recursive structure and linear complexity in  $N$ .
4.3 SVM classification of MTS
We present here SVM classification over the 19 MTS datasets. We do not include SINK as there is no extension or evaluation provided for MTS. Aside from that, the evaluation is conducted identically to the univariate case. For multivariate datasets, we expect to drift away from the ideal operating
Table 2: SVM classification of MTS datasets. Average across 10 seeds. Best accuracy is in bold.
DATASET	NVARK*	NVARK	RMESN	TCK
SPOKENARABICDIG.	0.980±0.004	0.980±0.003	0.948±0.004	0.985±0.002
JAPANESEVOWELS	0.981±0.008	0.975±0.007	0.968±0.004	0.961±0.004
PENDIGITS	0.986±0.000	0.983±0.000	0.976±0.001	0.954±0.001
RACKETSPORTS	0.889±0.019	0.847±0.007	0.859±0.008	0.872±0.006
LSST	0.566±0.009	0.543±0.006	0.597±0.009	0.474±0.008
LIBRAS	0.972±0.000	0.900±0.000	0.824±0.010	0.793±0.010
FINGERMOVEMENTS	0.572±0.034	0.581±0.032	0.576±0.022	0.505±0.030
NATOPS	0.896±0.020	0.895±0.019	0.828±0.015	0.806±0.016
CHARACTERTRAJ.	0.982±0.000	0.915±0.008	0.987±0.002	0.974±0.002
ERING	0.898±0.021	0.864±0.025	0.814±0.014	0.957±0.005
BASICMOTIONS	1.000±0.000	0.938±0.027	0.995±0.011	1.000±0.000
ARTICULARYWORDREC.	0.978±0.008	0.976±0.005	0.986±0.006	0.983±0.003
EPILEPSY	0.986±0.000	0.986±0.000	0.971±0.005	0.947±0.005
UWAVEGESTURELIB.	0.946±0.003	0.939±0.011	0.862±0.007	0.868±0.007
SELFREGULATIONSCP1	0.746±0.017	0.688±0.032	0.769±0.019	0.827±0.007
SELFREGULATIONSCP2	0.566±0.017	0.567±0.028	0.511±0.027	0.488±0.014
CRICKET	0.979±0.010	0.986±0.000	0.982±0.006	0.958±0.000
STANDWALKJUMP	0.433±0.047	0.607±0.049	0.373±0.056	0.433±0.079
EIGENWORMS	0.944±0.009	0.950±0.012	0.939±0.013	0.594±0.000
AVERAGE SCORE	0.858	0.848	0.830	0.809
AVERAGE RANK	/	1.68	2.05	2.26
N FIRST RANKED	/	11	3	5
MEDIAN TIME (S) / DATASET	19.67	3.87	26.47	184.08
scenario of NVARk. In particular, all the possible dynamic features may not be equally effective and, by randomly selecting a few, important correlations may be missed or spurious ones captured. Despite that, the experiments in Tab. 2 show surprisingly good results, even for datasets for which  $\bar{d}_r$  is just a fraction of all possible concatenation terms. We believe that this finding strengthens the connection to the generalized Takens' theorem. It is also interesting to notice good performance in diverse cases of time series that do not immediately arise from dynamical systems. Exploring this aspect constitutes interesting future works, e.g., time series representation learning and state space reconstruction as a possible underpinning.
Finally, we observe that the two hyperparameter configurations, NVARk* and NVARk, tend to be ranked next to each other. This supports the quality of the proposed general setting, which allows the kernel to extract meaningful features while preserving great computational efficiency.
In order to tease out which NVARk component is most important, we here present an ablation study. We separately consider fitting the linear readout directly on the input time series (no concatenation), sampling only linear terms from Eq. 5, and sampling only non-linear terms. This is an interesting ablation study that, to the best of our knowledge, has not been investigated in previous works. Results are presented in Tab. 3.
Table 3: Ablation study across 130 UCR univariate and 19 multivariate UEA datasets
no concat	linear only	non-linear only	all	balanced
SVM score for univariate datasets	mean	0.536	0.707	0.764	0.779	0.779
std	0.010	0.008	0.006	0.006	0.006
SVM score for multivariate datasets	mean	0.765	0.840	0.840	0.848	0.849
std	0.012	0.008	0.013	0.014	0.012
As a general trend, we observe that using all terms leads to the best results, followed by non-linear only, linear only, and fitting directly on the input series. Note also that results are in line with our interpretation using the generalized Taken's theorem. For univariate datasets, linear only tends to underperform, as the total number of concatenated dimensions is very small (equal to  $k$ ) and may be not sufficient in reconstructing the underlying manifold. In the multivariate case, the two variants perform very similarly, as the lags of all attributes may already constitute a considerable pool of terms.
To conclude, we consider an additional variant of Eq. 5 in which we first concatenate all the lag terms, and then fill the remaining dimensions, up to  $\bar{d_r}$ , by sampling nonlinear terms. Interestingly, for multivariate datasets, this leads to slightly better performance and a reduction in variance. We believe this might eventually compensate for the imbalance between all possible linear and non-linear terms, while also reducing the chance of adding noise from uncorrelated dimensions.
We have proposed a kernel for time series that integrates the NVAR framework into reservoir-based kernel architectures. NVARk compares time series based on the linear dynamics of NVAR embeddings, which are built from concatenating lags and nonlinear functionals to the original series. In terms of accuracy, NVARk outperforms the corresponding RC architecture. Computationally, it is exceptionally efficient and based on a few integer hyperparameters, which together allow for further improvement of the results with simple supervised grid-based optimization.
As for future work directions, we believe it would be effective to target specific weaknesses of NVARk. In particular, as the input dimensionality of the time series approaches the maximum embedding dimension  $(\bar{d}_r)$ , the proposed approach finds little margin for concatenating more lags and non-linear terms. Such cases would require substantially increasing  $\bar{d}_r$  and, consequently, place excessive strain on the linear readout, which inherently possesses limited expressive capacity. We then expect the effectiveness of NVARk to decrease. To overcome this, one can consider replacing the random sampling of dimensions with more refined strategies that prioritize the selection of meaningful terms. Alternatively, it would be interesting to explore how different linear layers, e.g. Lasso, would perform in this regime. Alongside these, different unsupervised strategies can be considered to learn optimal values for the embedding parameters. In particular, we note an unexplored affinity with signature transforms (Chevyrev & Kormilitzin, 2016), which we plan on deepening to infer optimal settings for the maximum dimensionality of the NVAR embedding.
G.D. acknowledges the Beckers Group for funding this research. V.V.G. thanks Leverhulme Trust for support via the Leverhulme Research Centre for Functional Materials Design. We would like to thank Chao Huang, Filippo Maria Bianchi, Davide Bacciu, and Claudio Gallicchio for their fruitful feedback and support for the work. Also, we would like to thank all the reviewers for their invaluable comments.
Abanda, A., Mori, U., and Lozano, J. A. A review on distance based time series classification. Data Mining and Knowledge Discovery, 33(2):378-412, 2019.
Bagnall A., Lines J., V. W. and E., K. The UEA and UCR time series classification repository. www. timeseriesclassification.com.
Baydogan, M. G. and Runger, G. Time series representation and similarity based on local autopatterns. Data Mining and Knowledge Discovery, 30(2):476-509, 2016.
Berndt, D. J. and Clifford, J. Using dynamic time warping to find patterns in time series. KDD workshop, 10 (16):359-370, 1994.
Bianchi, F. M., Scardapane, S., Løkse, S., and Jenssen, R. Reservoir computing approaches for representation and classification of multivariate time series. IEEE transactions on neural networks and learning systems, 32 (5):2169-2179, 2020.
Bollt, E. On explaining the surprising success of reservoir computing forecaster of chaos? The universal machine learning dynamical system with contrast to VAR and DMD. Chaos: An Interdisciplinary Journal of Nonlinear Science, 31(1):013108, 2021.
Broomhead, D. S. and King, G. P. Extracting qualitative dynamics from experimental data. Physica D: Nonlinear Phenomena, 20(2-3):217-236, 1986.
Cancelliere, R., Gai, M., Gallinari, P., and Rubini, L. OCReP: An optimally conditioned regularization for pseudoinversion based neural training. Neural Networks, 71:76-87, 2015.
Casdagli, M., Eubank, S., Farmer, J. D., and Gibson, J. State space reconstruction in the presence of noise. Physica D: Nonlinear Phenomena, 51(1-3):52-98, 1991.
Cha, S.-H. Comprehensive survey on distance/similarity measures between probability density functions. *City*, 1 (2):1, 2007.
Chatzis, S. P. and Demiris, Y. Echo state gaussian process. IEEE Transactions on Neural Networks, 22(9): 1435-1445, 2011.
Chen, H., Tang, F., Tino, P., and Yao, X. Model-based kernel for efficient time series analysis. Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 392-400, 2013.
Chevyrev, I. and Kormilitzin, A. A primer on the signature method in machine learning. arXiv preprint arXiv:1603.03788, 2016.
Cuturi, M. Fast global alignment kernels. Proceedings of the 28th international conference on machine learning (ICML-11), pp. 929-936, 2011.
Cuturei, M. and Doucet, A. Autoregressive kernels for time series. arXiv preprint arXiv:1101.0673, 2011.
Deyle, E. R. and Sugihara, G. Generalized theorems for nonlinear state space reconstruction. Plos one, 6(3): e18295, 2011.
Ding, H., Trajcevski, G., Scheuermann, P., Wang, X., and Keogh, E. Querying and mining of time series data: experimental comparison of representations and distance measures. Proceedings of the VLDB Endowment, 1 (2):1542-1552, 2008.
Dong, J., Ohana, R., Rafayelyan, M., and Krzakala, F. Reservoir computing meets recurrent kernels and structured transforms. Advances in Neural Information Processing Systems, 33:16785-16796, 2020.
Dong, J., Borve, E., Rafayelyan, M., and Unser, M. Asymptotic stability in reservoir computing. In 2022 International Joint Conference on Neural Networks (IJCNN), pp. 01-08. IEEE, 2022.
A Takens' delay embeddings
This section introduces the field of state space reconstruction in dynamical system theory (Strogatz, 2018) and deepens the theoretical interpretation of Sec. 3.2. As in Deyle & Sugihara (2011), we start by defining some key concepts that might be useful to introduce the reader to the subject:
- a manifold is, intuitively, a generalized surface  $M \subseteq \mathbb{R}^N$  that locally looks like an Euclidean space. More rigorously, locally diffeomorphic to some  $\mathbb{R}^k$ ;
- a dynamical system  $\phi : M \to M$  is a diffeomorphism defining, for discrete times, trajectories on a manifold  $M$ ;
- an embedding is a 1:1 smooth map  $f: M_1 \to M_2$  between two manifolds. Embeddings preserve topological properties, e.g. resolves all trajectories without crossings, and geometrical invariants, such as eigenvalues and fixed points;
- an observation function  $y: M \to \mathbb{R}^D$  is any function that assigns a vector to each point on  $\mathbf{M}$ .
It is generally true that we can imagine an observed MTS  $\mathbf{X} \in \mathbb{R}^{T \times D}$  as a realization of a more complex and unknown smooth dynamical system  $\phi^*$ , acting on states  $\mathbf{x}^*$  on an unknown manifold  $M^*$ :
$$
\mathbf {x} _ {t + 1} ^ {*} = \phi^ {*} \left(\mathbf {x} _ {t} ^ {*}\right) \tag {7}
$$
The observed and original values are related by an observation function  $y^{*}:M^{*}\to \mathbb{R}^{D}$  that projects the underlying states onto what we observe:
$$
\mathbf {x} _ {t} = y ^ {*} \left(\mathbf {x} _ {t} ^ {*}\right) \quad (+ \epsilon_ {t}) \tag {8}
$$
This projection can drastically reduce the available information, but one can wonder to which extent it is possible to recover the original system when only the observed vectors  $(\mathbf{x}_t)$  are accessible. We focus here on the state reconstruction problem, which is that of finding a valid embedding  $f:M^{*}\to \mathbb{R}^{n}$  of the original manifold  $M^{*}$ :
$$
\mathbf {s} _ {t} = f \left(\mathbf {x} _ {t} ^ {*}\right) \tag {9}
$$
This is, by using the observed states  $(\mathbf{x}_t)$ , creating richer states  $(\mathbf{s}_t)$  for which the dynamics is topologically equivalent to the unknown  $\phi^*$ . In fact, if  $f$  is an embedding, then smooth dynamics  $\Phi$  is induced on the space of reconstructed vectors:
$$
\Phi \left(\mathbf {s} _ {t}\right) = f \circ \phi^ {*} \circ f ^ {- 1} \left(\mathbf {s} _ {t}\right) \tag {10}
$$
and is equivalent to  $\phi^{*}$ . If we succeed, instead of the observed  $\mathbf{x}$ , we can use the reconstructed vectors  $\mathbf{s}$  as a better representation of the underlying evolving system  $\mathbf{x}^{*}$ .