SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Whitney's theorem (Whitney, 1936) states the general result that every compact m-dimensional manifold admits an embedding in  $\mathbb{R}^{2m + 1}$ . For dynamical systems this means that, a set of  $2m + 1$  observation functions, where  $y_{k}:M\to \mathbb{R}$ , is sufficient to construct a valid embedding:
$$
\mathbf {s} _ {t} = \left[ y _ {1} \left(\mathbf {x} _ {t} ^ {*}\right), y _ {2} \left(\mathbf {x} _ {t} ^ {*}\right), \dots , y _ {2 m + 1} \left(\mathbf {x} _ {t} ^ {*}\right) \right] \tag {11}
$$
In the case of scalar observations  $(\mathrm{D} = 1)$ , Takens' theorem (Takens, 1981) considers a single observation function  $(y)$ , i.e., the projection on the only observed component, but acting on multiple lagged versions of the system:
$$
\begin{array}{l} \mathbf {s} _ {t} = \left[ y \left(\mathbf {x} _ {t} ^ {*}\right), y \left(\phi \left(\mathbf {x} _ {t} ^ {*}\right)\right), y \left(\phi^ {2} \left(\mathbf {x} _ {t} ^ {*}\right)\right), \dots , y \left(\phi^ {2 m} \left(\mathbf {x} _ {t} ^ {*}\right)\right) \right] \tag {12} \\ = \left[ x _ {t}, x _ {t - \tau}, \dots , x _ {t - 2 m \tau} \right] \\ \end{array}
$$
The idea behind Eq. 12 is that the information about the unobserved state variables is contained in the relations between past and future values of the observed time series and a  $2m + 1$  dimensional embedding (often less) is sufficient to extract that.
Generalized versions of Takens' theorem have also been formulated. In particular, we find in Deyle & Sugihara (2011) an extension to cases where lags of multiple observation functions can be used to create the embedding. This is a useful result in cases where one cannot consider many lags, but can instead make use of fewer lags spread across multiple available dimensions. In more detail, the allowed set of observation functions is any  $\{y_k\}_{k=1}^{2m+1}$  containing terms  $\{y_r\}$  that cannot be written as a lagged version of any other element in the set and terms  $\{y_q\}$  that satisfy  $y_q = y_r(\phi^b(\mathbf{x}^*))$  for a choice of  $b$ . Both Takens' and Whitney's theorems can be recovered from this formulation: it reduces to the case of Takens' when only one  $\{y_r\} = y$  is chosen and all the others belong to the family of  $\{y_q\}$ ; it reduces to the case of Whitney's when  $\{y_r\} = \{y_k\}$  and  $\{y_q\} = \emptyset$ .
All terms in the NVAR representation are reminiscent of this formulation. This points to the direction of  $\mathbf{R}_{NVAR}$  being a closer approximation of the underlying state than the initial data  $\mathbf{X}$  and also suggests that the final number of dimensions plays a more important role than which are included and which are not. In practice, the dimensionality  $m$  of the original manifold  $M^{*}$  is unknown, making the selection of the embedding dimensionality a potentially rich field of study. On top of that, the observed data may be corrupted by multiple noise sources, which increases the importance of sampling the series correctly with a meticulous choice of the lag size  $(\tau \equiv s)$ .
B Experimental evaluation details
Details of the 130 Univariate and 19 Multivariate datasets are reported in Tab. 4. Both cover an extensive range of data points and series lengths. From the UEA archive, we have excluded 6 high-dimensional datasets for which the application of the NVAR kernel is not appropriate, and 5 datasets for which SVM accuracy was  $< 0.5$  for all approaches, which indicates that kernel methods are, in general, not a suitable solution.
Table 4: Attributes of UTS and MTS datasets. The splitting between the training and testing set is provided by the respective archive.
UTS DATASET	Ntrain	Ntest	T	D	Nclass
130 DATASETS	16÷8926	20÷16800	15÷18530	1	2÷60
MTS DATASET	Ntrain	Ntest	Tmin	Tmax	D	Nclass
SPOKENARABICDIG.	6599	2199	4	93	13	10
JAPANESEVOWELS	270	370	7	29	12	9
PENDIGITS	7494	3498	8	8	2	10
RACKETSPORTS	151	152	30	30	6	4
LSST	2459	2466	36	36	6	14
LIBRAS	180	180	45	45	2	15
FINGERMOV.	316	100	50	50	28	2
NATOPS	180	180	51	51	24	6
CHARACTERTRAJ.	1422	1436	60	182	3	20
ERING	30	270	65	65	4	6
BASICMOTIONS	40	40	100	100	6	4
ARTICULARYWORDREC.	275	300	144	144	9	25
EPILEPSY	137	138	206	206	3	4
UWAVEGESTURELIB.	120	320	315	315	3	8
SELFREGULATIONSCP1	268	293	896	896	6	2
SELFREGULATIONSCP2	200	180	1152	1152	7	2
CRICKET	108	72	1197	1197	6	12
STANDWALKJUMP	12	15	2500	2500	4	3
EIGENWORMS	128	131	17984	17984	6	5
B.2 Implementations and details for reproducibility
For reproducibility, we list here the parameter setting and the code implementation for each method. Experiments were run on a 32-core AMD Ryzen™ Threadripper PRO CPU.
NVAR kernel (NVARk)
- the allowed number of lags  $k = \sqrt{\text{average distance between peaks and hollows}}$ , after series have been  $\ell_{1}$  trend-filtered (Kim et al., 2009);
- lag size  $s = k$
- order of the polynomial functionals  $n = 2$
maximum dimensionality of the embedding  $\bar{d}_r = 75$
- readout regularization set by the OCReP method (Cancelliere et al., 2015);
- RBF lengthscale as the median pairwise distance between the representations.
An easy-to-use Python implementation of the NVAR kernel is made publicly available at https://github. com/gdefe/nvark-kernel.
Table 5: Grid values for the CV optimization of the NVARk embedding parameters.
DATASET	CV GRID
T<400	K=[1,2,3,4,5,K*], s=[1,2,3,4,5,s*]
T≥400	K=[1,2,3,4,5,10,20,K*], s=[1,5,10,20,s*]
SPOKENARABICDIGITS	K=[1,2,3,4,K*], s=[1,2,3,4,s*]
JAPANESEVOWELS	"
PENDIGITS	"
RACKETSPORTS	"
LSST	"
LIBRAS	"
FINGERMOVEMENTS	"
NATOPS	"
CHARACTERTRAJECTORIES	K=[1,2,3,4,10,20,K*], s=[1,5,20,s*]
ERING	"
BASICMOTIONS	"
ARTICULARYWORDRECOGNITION	"
EPILEPSY	"
UWAVEGESTURELIBRARY	"
SELFREGULATIONSCP1	"
SELFREGULATIONSCP2	"
CRICKET	"
STANDWALKJUMP	"
EIGENWORMS	"
optimized NVAR kernel (NVARk*) The lag size  $s$  and the allowed number of lags  $k$  are optimized in a CV loop scrolling a small grid. Values for the grid are reported in Tab. 5.  $k^*$  and  $s^*$  refer to the values obtained from the rule of thumb.
In general, one should allow for larger values for  $k$  and  $s$  in the case of longer series. A higher  $k$  allows capturing longer-term dynamics, while a larger  $s$  avoid oversampling the series. In the case of multivariate datasets, we observed that many lags are hardly required. This is possible because few lags spread across different dimensions can be sufficient to construct the embedding. Also, the pool of features ( $[\mathbf{R}_{lag}||\mathbf{R}_{nonlin}]$ ) largely increases with the dimensionality of the dataset and  $k$ , which affects the stability of the approach.
reservoir-model ESN (rmESN) For the rmESN, we follow the setting given in Bianchi et al. (2020):
number of internal reservoir units  $R = 800$
- spectral radius  $\rho = 0.99$ ;
- no leakage;
- percentage of non-zero connectivity within the reservoir states  $\beta = 0.25$
- input scaling  $\omega = 0.15$ ;
- noise level in the reservoir update  $\epsilon_{t} = 0.001$ ;
- transient dropped states  $n_{drop} = 5$
- number of dimensions after the dimensionality reduction module  $D = 75$  (via tenor-PCA, also proposed in the same paper);
- favoring comparability with NVARk, we set the readout regularization by the OCReP method;
- favoring comparability with NVARk, we set the RBF lengthscale to the median pairwise distance between the representations.
As implementation, we used the public Python code provided by the authors: https://github.com/FilippoMB/Time-series-classification-and-clustering-with-Reservoir-Computing.
Time Cluster Kernel (TCK) For TCK, we follow the guidelines given in the original paper and in the provided code. For a dataset of  $N$  samples and  $D$  attributes, these are:
- minimum percentage of subsample  $\min N = 0.8$
- min number of attributes for each GMM  $\min D = 1$  for univariate datasets,  $\min D = 2$  for multivariate datasets;
- max number of attributes for each GMM  $\max D = \min \left( {{ceil}\left( {{0.9} * D}\right) ,{15}}\right)$  ;
- min length of time segments for each GMM  $\min T = 6$
- max length of time segments for each GMM  $\max T = \min (floor(0.8 * T), 25)$ ;
- max number of mixture components for each GMM  $C = 10$  if  $N < 100$ ,  $C = 40$  otherwise;
number of randomizations for each number of components  $G = 30$
number of iterations  $I = 20$
The Matlab implementation of TCK that we use is part of the code available at https://github.com/FilippoMB/TCK_AE.
Global Alignment Kernel (GAK) We set the multiplicative factor for the bandwidth of the radial function to  $\sigma_{GAK} = 2$ . Python code is available from the `tslearn` package (Tavenard et al., 2020). As reported in the package documentation, the method can lead to numerical issues for long series.
Shift INvariant Kernel (SINK) For SINK, we adopted the Python implementation provided by the author at https://github.com/TheDatumOrg/grail-python. We use  $\gamma = 5$  and retain all the energy after Fourier-transforming the series.
C.1 Feature redundancy
For studying whether previously adopted values for the embedding dimensionality  $(\bar{d}_r)$  are also suitable for NVARk, we adopted an experimental approach and studied how many features can be included before experiencing redundancy. We do so by plotting, for a few univariate and multivariate datasets, the accuracy of an SVM classifier against the  $\bar{d}_r$  parameter (Fig. 3). While do not observe redundancy in univariate datasets,
Figure 3: For some univariate (left) and multivariate (right) time series datasets, the plots show the accuracy of an SVM classifier against the NVAR embedding dimensionality  $\bar{d}_r$ . Embedding parameters are indicated as  $(k / s)$  and determined via the proposed rule of thumb. Features redundancy manifests for some multivariate datasets (Cricket, UWaveGestureLibrary) as an accuracy drop.
we do observe a drop in performance after different  $\bar{d}_r$  for some multivariate datasets. We identify the region  $70\leq \bar{d}_r\leq 100$  as a good, though non-optimal, setting, which is in line with previous RC studies (Bianchi et al., 2020).
C.2 Sensitivity analysis
In this section, we address the sensitivity of NVARk to the different hyperparameters:
Figure 4: Sensitivity of NVARk to the hidden dimensionality (left), evaluated across 16 multivariate datasets; sensitivity to the regularization parameter (center) and RBF lengthscale (right), evaluated across 20 randomly sampled univariate datasets. Red arrows indicate our parameter's choice.
-  $\bar{d}_r$ : as we show in Fig. 4 (left), the average performance for multivariate datasets is not considerably impacted in the range  $\bar{d}_r \in [70, 150]$ . As expected, we observe a significant drop in performance for high values (redundancy).
-  $\lambda_{\mathrm{ridge}}$ : we show in Fig. 4 (center) little sensitivity in the region  $\lambda_{\mathrm{ridge}} \in [5, 50]$ . Results are averaged across a pool of 20 randomly sampled univariate datasets. The employed OCReP method (Cancelliere et al., 2015) outperforms the best choice by  $\sim 2\%$ , as it adapts the value to each individual dataset.
-  $\gamma_{\mathrm{rbf}}$ : in Fig. 4 (right), we range different multiplicative factors for the median pairwise distance between all representations and show little sensitivity in a neighborhood of our choice  $\gamma_{\mathrm{rbf}_{\mathrm{mul}}} = 1$ .
-  $k$  and  $s$ : The sensitivity to the embedding parameters strongly depends on the dataset, although it generally tends to be high. In response, we introduce our dataset-specific heuristic adapting ideas from a precedent study (Kugiumtzis, 1996), which we find works well in practice. A reasonable idea of the average sensitivity to these parameters can be obtained by comparing the performance of NVAR, where  $k$  and  $s$  are chosen by the heuristic, to the performance of NVARk*, where  $k$  and  $s$  are optimized. Unfortunately, understanding sensitivity by considering other settings is challenging, as the existing literature predominantly focuses on the noise-free case. Furthermore, there is no strong evidence for the wider applicability of such approaches outside dynamical systems.
C.3Uneven lengths
NVARk can process and compare time series of different lengths. We report in Tab. 6 the SVM accuracy scores for three datasets with uneven lengths.
Table 6: SVM classification accuracy for datasets with uneven lengths marked with the [dL] notation. For comparison, here are copied the respective results from Sec. 4.3 after zero padding preprocessing.
DATASET	NVARK*	NVARK
SPOKENARABICDIGITS [DL]	0.971±0.003	0.971±0.003
SPOKENARABICDIGITS	0.980±0.004	0.980±0.003
JAPANESEVOWELS [DL]	0.973±0.007	0.975±0.004
JAPANESEVOWELS	0.981±0.008	0.975±0.007
CHARACTERTRAJECTORIES [DL]	0.983±0.005	0.933±0.000
CHARACTERTRAJECTORIES	0.982±0.000	0.915±0.008
C.4 Comparison with Global Alignment Kernel
We compare here the performance of the NVAR kernel against the popular Global Alignment Kernel (GAK) baseline from Cuturi (2011). This has been moved from the main text as authors of SINK have already demonstrated superior performance to GAK, following a similar evaluation. As reported in the tslearn package documentation, the provided implementation can lead to numerical issues when used with relatively long time series. To face this, we evaluate GAK on time series linearly interpolated to  $T = 400$  when their length exceeds this threshold. We report in Tab. 7 the average results across 122 univariate datasets. We report in Tab. 8 results for the multivariate datasets. Datasets interpolated to  $T = 400$ , before being processed by GAK, are marked with the symbol (*).
Table 7: Average performances for NVARk and GAK across 130 univariate datasets.
NVARK*	NVARK	GAK
AVERAGE SCORE	0.803	0.779	0.728
N FIRST RANKED	/	77	56
Table 8: SVM classification of MTS datasets. Average across 10 seeds. Best accuracy is in bold.
DATASET	NVARK	GAK
SPOKENARABICDIG.	0.980±0.003	time limit
JAPANESEVOWELS	0.975±0.007	0.974±0.002
PENDIGITS	0.983±0.000	0.978±0.000
RACKETSPORTS	0.847±0.007	0.836±0.008
LSST	0.543±0.006	0.500±0.000
LIBRAS	0.900±0.000	0.783±0.007
FINGERMOVEMENTS	0.581±0.032	0.565±0.035
NATOPS	0.895±0.019	0.918±0.003
CHARACTERTRAJ.	0.915±0.008	0.988±0.000
ERING	0.864±0.025	0.931±0.002
BASICMOTIONS	0.938±0.027	0.995±0.011
ARTICULARYWORDREC.	0.976±0.005	0.980±0.000
EPILEPSY	0.986±0.000	0.845±0.004
UWAVEGESTURELIB.	0.939±0.011	0.865±0.002
SELFREGULATIONSCP1	0.688±0.032	0.902±0.008(*)
SELFREGULATIONSCP2	0.567±0.028	0.526±0.017(*)
CRICKET	0.986±0.000	0.958±0.000(*)
STANDWALKJUMP	0.607±0.049	0.333±0.172(*)
EIGENWORMS	0.950±0.012	0.482±0.005(*)
AVERAGE SCORE	0.848	0.756
N FIRST RANKED	13	6
C.5 Example kPCA visualization
Despite the underlying equivalence between NVAR and simple RC, better performance can be obtained by finding a better parameter setting, which is not trivial for RC. In our approach, a better setting is easier to find with fewer and more interpretable parameters. Here, we aim to visually highlight the possible extent of the differences between NVAR and rmESN, i.e., theoretically equivalent methods under different working points. To do so, we projected the data onto the first principal components using different kernel matrices (kPCA). When inspecting the projections for different datasets, we observed a few critical differences, predominantly for longer series. One example is illustrated in Fig. 5 (CinCECGTorso dataset). These appear in the form of a qualitatively
Figure 5: kPCA visualization of the CinCECGTorso dataset. Projection onto the first three principal components obtained with different kernels. Different colors correspond to different classes.
better grouping and class separation in the projected space. Tracing back the origin of this improvement, e.g., a
better working point or a better resilience to noise, would represent interesting future work. For an alternative visualization, we display here the two-dimensional projection of the CinCECGTorso dataset using different kernels.
D Individual results for SVM classification of UTS
We report here the individual accuracies obtained under the SVM classification framework of univariate time series of Sec. 4.1. Each accuracy is obtained as a mean of 10 repetitions with 10 different random seeds. For readability, the standard deviation over the runs is not reported. Datasets interpolated to  $\mathrm{T} = 400$ , before being processed by GAK, are marked with the symbol (*)