SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Giovanni De Felice John Y. Goulermas* Vladimir V. Gusev  
Department of Computer Science  
University of Liverpool  
gdefe@liverpool.ac.uk gusev@liverpool.ac.uk
Kernel design is a pivotal but challenging aspect of time series analysis, especially in the context of small datasets. In recent years, Reservoir Computing (RC) has emerged as a powerful tool to compare time series based on the underlying dynamics of the generating process rather than the observed data. However, the performance of RC highly depends on the hyperparameter setting, which is hard to interpret and costly to optimize because of the recurrent nature of RC. Here, we present a new kernel for time series based on the recently established equivalence between reservoir dynamics and Nonlinear Vector AutoRegressive (NVAR) processes. The kernel is non-recurrent and depends on a small set of meaningful hyperparameters, for which we suggest an effective heuristic. We demonstrate excellent performance on a wide range of real-world classification tasks, both in terms of accuracy and speed. This further advances the understanding of RC representation learning models and extends the typical use of the NVAR framework to kernel design and representation of real-world time series data.
Time series are arguably one of the most important types of data in the modern era (Hamilton, 2020) and ubiquitous both in scientific research (Strogatz, 2018) and practical applications (Zhang et al., 2018; Zeroual et al., 2020). A key element in the design of most machine learning protocols for time series is a quantification of similarity (Ding et al., 2008; Abanda et al., 2019; Echihabi et al., 2020). This is especially true for kernel methods (Schölkopf et al., 2001), which search for linear solutions after projecting the data into a higher (possibly infinite) dimensional space and represent an effective alternative to non-linear models. The performance of kernel methods is heavily impacted by the definition of a positive semi-definite (PSD) similarity function for the type of data at hand, i.e., a kernel. However, their design is challenging for structured data such as univariate (UTS) and multivariate time series (MTS), which exhibit temporal autocorrelation, inter-attributes (or dimensions) dependencies and possibly a variety of temporal distortions and misalignments (Paparrizos et al., 2020).
To overcome this, a promising approach is to identify and compare the underlying dynamics rather than the raw observed data. Notably, kernels based on Reservoir Computing (RC) stand out in this context (Chen et al., 2013; Bianchi et al., 2020). RC-based kernels use a randomized and untrained layer of recurrently connected nodes (the reservoir) to map each time series into a rich dynamical feature space. The similarity is then obtained by comparing individual readouts capturing dynamics. In practice, RC is highly sensitive to a large set of hyperparameters having little interpretability, the optimization of which is challenging for the comprehensive range of values.
*Deceased May 2022
Recently, a theoretical work by Bollt (2021) has demonstrated that a simple RC can be formally rewritten as a non-linear vector autoregressive (NVAR) model. In the NVAR framework, the reservoir is replaced by a simple concatenation of the input series with time-delayed copies and nonlinear functionals, such as products (Gauthier et al., 2021). This reduces the number of hyperparameters and has proven to be very effective in chaotic systems forecasting (Shahi et al., 2022), as much to earn the title of "Next-Generation Reservoir Computing". However, its applicability and performance for real-world data and outside of forecasting dynamical systems are largely unexplored.
In this work, we investigate whether an NVAR process can replace a reservoir for kernel design and if the extracted features are just as effective in comparing real-world time series data. We propose a very efficient and effective NVAR kernel for UTS and MTS data. The main idea is as follows (Fig. 1). First, we enrich each time series with lagged copies of itself and additional nonlinear terms, forming a high-dimensional NVAR embedding. Then, we extract a linear parameterization of the time evolution in the embedding space, representative of the underlying dynamics, analogously to RC representation learning methods (Bianchi et al., 2020). Finally, these fitted parameters are used to compute the similarity between the time series. Our main contributions branch out into different communities investigating time series, namely, kernel design, RC and NVAR, bringing specific benefits to each of them:
- From a kernel design perspective, we provide an NVAR kernel for UTS and MTS and a general parameter setting based on simple heuristics. Based on our experiments on a wide range of datasets, the proposed approach matches the state-of-the-art (SOTA) in terms of accuracy and is considerably faster than SOTA, representing the best compromise between accuracy and efficiency.
- From the perspective of RC, we advance its success for TS representation by incorporating NVAR-made dynamics. The resulting model is more accurate, interpretable, and non-recurrent, which overcomes difficulties in hyperparameter optimization and sheds light on the potential RC performance.
- From an NVAR perspective, we extend the framework to real-world time series representation and kernels, well beyond the original work on forecasting synthetic, noise-free, and chaotic dynamical systems (Gauthier et al., 2021).
- We establish a connection to Takens' theorem (Takens, 1981) and the field of state space reconstruction (Sauer et al., 1991), which theoretically underpin our approach, highlighting a compelling avenue for the use of dynamical systems theory in machine learning.
Through the paper, we indicate variables as lowercase  $(x)$ ; constants as uppercase  $(X)$ ; vectors and UTS as bold lowercase  $(\mathbf{x})$ ; matrices and MTS as bold uppercase  $\mathbf{X}$ . An index between square parenthesis  $x[n]$  indicates the  $n$ -th sample of a set. For an MTS  $\mathbf{X}$ , the corresponding lower-case  $\mathbf{x}_t$  indicates all dimensions at time stamp  $t$  while  $\mathbf{x}^d$  indicates dimension  $d$  at all timestamps.
2 Related methods
In this section, we provide a background on the previously proposed kernels for time series data, grouped by category. Despite the abundance of time series similarity measures (Yang & Shahabi, 2004; Paparrizos & Gravano, 2015; Janati et al., 2020), we limit the discussion to PSD metrics.
Lock-step Lock-step approaches view time series as static vectors and directly use common distance metrics (Cha, 2007). The PSD property is usually obtained by superposing a linear or radial kernel (Scholkopf et al., 2002). This is very efficient, but ignores any temporal structure within the series and is not applicable for different lengths.
Elastic Elastic measures account for time distortions (e.g. shifts) or different lengths between sequences. Lu et al. (2008) proposed to interpolate the series and treat the problem as a distance between curves. The Global Alignment Kernel (GAK) (Cuturi, 2011) and KDTW (Marteaux & Gibet, 2014) instead are built on the well-known Dynamic Time Warping (DTW) (Berndt & Clifford, 1994) and compute the similarity from the cost of one-to-many alignments between the series. The
Shift-invariant kernel (SINK) (Paparrizos & Franklin, 2019) computes a cross-correlation similarity in the Fourier domain. However, for the aforementioned methods, the computation of the pairwise similarity is computationally expensive and does not account for the relations between different attributes in the multivariate case.
Model-based Model-based kernels process the series with a probabilistic or deterministic model and base the similarity on the extracted information. Such a model can be a single generative model, as in the Fisher kernel (Jaakkola et al., 1999) or the probability product kernels (Jebara et al., 2004), or a parameterized family of probability distributions, as in the autoregressive kernel (Cuturi & Doucet, 2011). Similarly to the latter, the Time Cluster Kernel (TCK) (Mikalsen et al., 2018) is obtained from an ensemble of Gaussian mixture posteriors, sharing the same parametric form but trained on different subsets of the dataset. Among non-probabilistic approaches, the Learned Pattern Similarity (LPS) (Baydogan & Runger, 2016) uses an ensemble of regression trees to extract local patterns from each series. The downside of all such methods comes from the specific functional form, which may limit the generalizability of the extracted features. Ensemble methods are also unsuited for datasets with few training samples.
Reservoir-based Reservoir Computing (RC) is a class of recurrent neural networks that keep recurrent connections untrained in order to overcome the high cost of back-propagation through time (Werbos, 1990) and the vulnerability to exploding and vanishing gradient (Pascanu et al., 2013). Its simplest form, the Echo State Network (ESN) (Jaeger, 2001), is composed of three layers: an input layer, a hidden layer of connected neurons (reservoir) and a readout layer:
$$
\text {i n p u t :} \quad \mathbf {u} _ {t} = \mathbf {W} ^ {i n} \mathbf {x} _ {t} \tag {1}
$$
$$
\text {r e s e r v o i r :} \quad \mathbf {r} _ {t} = (1 - \alpha) \mathbf {r} _ {t - 1} + \alpha f (\mathbf {A} \mathbf {r} _ {t - 1} + \mathbf {u} _ {t} + \epsilon_ {t}) \tag {2}
$$
$$
\text {r e a d o u t :} \quad \mathbf {y} _ {t} = \mathbf {W} ^ {\text {o u t}} \mathbf {r} _ {t} + \mathbf {c} \tag {3}
$$
where  $\mathbf{x}_t\in \mathbb{R}^{d_x}$ ,  $\mathbf{u}_t\in \mathbb{R}^{d_r}$  and  $\mathbf{r}_t\in \mathbb{R}^{d_r}$  are, respectively, the input, its upscaling projection and the reservoir states at time  $t$  (with  $d_r\gg d_x$  and  $\mathbf{r}_0\coloneqq \mathbf{0}$ );  $0\leq \alpha \leq 1$  is the leak parameter;  $\epsilon_{t}$  is an additive noise and  $f$  is any non-linear activation function. The reservoir  $(\mathbf{A}\in \mathbb{R}^{d_r\times d_r})$  and the projection  $(\mathbf{W}^{in}\in \mathbb{R}^{d_r\times d_x})$  matrices are kept untrained, while the readout  $(\mathbf{W}^{out}\in \mathbb{R}^{d_y\times d_r})$  and the bias term (c) are learned by fitting the reservoir states to the output  $(\mathbf{y_t}\in \mathbb{R}^{d_y})$  via ridge regression.
Despite their randomness, the reservoir states constitute a rich pool of heterogeneous features that give complete knowledge of the underlying dynamical system generating the observed time series (Løkse et al., 2017; Bianchi et al., 2020). In fact, as opposed to fitting a specific functional form, a solid theoretical ground ensures the quality of the reservoir features (Hart et al., 2020; Gonon et al., 2023; Gonon & Ortega, 2021; Hart et al., 2021). Nevertheless, performance is highly sensitive to the choice of a large set of hyperparameters controlling the reservoir initialization and update (Lukoševicius, 2012). For instance, input scaling  $(||\mathbf{W}^{in}||)$  and spectral radius  $(\rho(\mathbf{A}))$  assure stable reservoir dynamics (Gallicchio, 2019) and control the degree of injected nonlinearity. This is not easy to judge and requires experienced insight into nonlinear dynamics. In general, the interpretation and setting of ESN parameters is a complex task and, to this day, an active area of research (Dong et al., 2022; Steiner et al., 2022; Zhang et al., 2022). Simple supervised optimization, such as cross-validation (CV), is often prohibitive, as the optimization space is large and the recursive nature of the reservoir forces the computation of Eq. 2 for the whole length of the series before assessing the performance of one hyperparameter setting.
Regarding RC-based kernels, Chatzis & Demiris (2011) originally built a radial kernel directly on top of the reservoir states. In Chen et al. (2013), all series are processed by a shared reservoir and compared based on the respective readouts, which are individually trained on the one-step-ahead prediction task  $(\mathbf{y}_t = \mathbf{x}_{t + 1})$ . Finally, Bianchi et al. (2020) observed that more information is retained by using  $\mathbf{y}_t = \mathbf{r}_{t + 1}$ , and an effective kernel can be obtained by applying a radial function on the vectorization of all the readout weights.
Figure 1: Building of each representation vector in the NVAR kernel. The input series  $\mathbf{X}$  is concatenated with  $\mathbf{R}_{lag}$ , which contains  $k$  lags of all input dimensions, and with  $\mathbf{R}_{nonlin}$ , containing products between dimensions of  $\mathbf{X}$  and  $\mathbf{R}_{lag}$ . Among all possible concatenation terms, only a random subset of dimensions is considered. The readout then extracts the linear dynamics ( $\mathbf{W}^{out}$ ) of the embedding states and the bias term (c) by performing a ridge regression fit along the temporal dimension. These fit parameters are vectorized to form the representation.
This section presents the main building blocks of NVAR models, followed by our proposal of an NVAR-based kernel and a general setting for the hyperparameters that govern it.
3.1 The Nonlinear Vector AutoRegressive model
Recently, the work of Boltt (2021) has established a connection between the ESN architecture and Nonlinear Vector AutoRegressive (NVAR) models. Under specific conditions, this takes the form of a formal equivalence, where the coefficients of an NVAR process can be expressed in terms of  $\mathbf{A}$ ,  $\mathbf{W}^{in}$  and  $\mathbf{W}^{out}$ . By building on such equivalence, in Gauthier et al. (2021), the input layer (Eq. 1) and the reservoir (Eq. 2) of an ordinary ESN are replaced with the deterministic operations that lead to a feature matrix  $\mathbf{R}_{NVAR}$ . In detail, this is the result of concatenating three terms: the input series  $(\mathbf{X})$ ,  $k$  lagged copies of the input, each of which is increasingly spaced by  $s$  time-stamps  $(\mathbf{R}_{lag})$ , and all possible products² between lagged and unlagged dimensions, up to a polynomial order  $n$  ( $\mathbf{R}_{nonlin}$ ):
$$
\mathbf {r} _ {l a g, t} = \left[ \left. \mathbf {x} _ {t - s} \right| | \mathbf {x} _ {t - 2 s} \left| \right| \dots | | \mathbf {x} _ {t - k s} \right]
$$
$$
\mathbf {r} _ {\text {n o n l i n}, t} = \left[ \right. \mathbf {x} _ {t} \left. \right| | \mathbf {r} _ {\text {l a g}, t} ] \left. \right. \left\lceil \otimes \right\rceil \left[ \right. \mathbf {x} _ {t} \left. \right| | \mathbf {r} _ {\text {l a g}, t} \left. \right]\left. \right. \left\lceil \otimes \right\rceil \dots \quad (n \text {t i m e s}) \tag {4}
$$
$$
\mathbf {R} _ {N V A R} = \mathbf {X} | | \mathbf {R} _ {l a g} | | \mathbf {R} _ {n o n l i n}
$$
where  $\cdot ||\cdot$  is the column-wise concatenation and  $\cdot \lceil \otimes \rceil \cdot$  performs the outer product and concatenates all unique monomials.
This procedure is very efficient as it is non-recurrent and the size of  $\mathbf{R}_{NVAR}$  is usually smaller than a typical reservoir. On top of that, it is determined by a small set of integer hyperparameters:  $k$ ,  $s$  and  $n$ . In principle, to sustain the equivalence to RC, an infinite amount of lags ( $k \to \infty$ ) should be considered. Nevertheless, it has been observed that a small  $k$  with  $n = 2$  can still perform exceptionally well in typical dynamical systems forecasting tasks (Gauthier et al., 2021; Shahi et al., 2022; Gauthier et al., 2022). However, applicability and performance of NVAR models beyond forecasting yet remain largely unexplored, particularly in the context of representing and comparing real-world time series data. In this regard, they present a promising solution to significantly enhance RC representations and kernels.
3.2 NVAR kernel
We present here the NVAR kernel (NVARk), which integrates the NVAR framework with RC-based kernel architectures. Given a pair of time series,  $\mathbf{X}[i]$ ,  $\mathbf{X}[j]$ , with dimensionality  $d_x$  of possibly different lengths, NVARk operates in three steps, which we present in the following paragraphs. Fig. 1 depicts a graphical reference, while Algorithm 1 explicits the algorithmic process.
$^2$ other forms of nonlinearity can be used depending on prior knowledge, but the polynomial functionals can be motivated as universal approximators of dynamical systems (Franz & Scholkopf, 2006)
Algorithm 1 NVAR kernel
input MTS list  $\mathbf{X}[1:N]$  , number of lags  $k$  lag size  $s$  embedding size  $\bar{d}_r$    
1:  $\mathbf{L} = \{(a,p)\mid a\in \{1,\dots,d_x\} ,p = \{1,\dots,k\} \}$  (dimension  $a$  lagged  $p$  times)   
2:  $\mathbf{P} = \{((a,p),(b,q))\mid a,b\in \{1,\dots,d_x\} ,p,q = \{0,\dots,k\} \}$  (all possible products)   
3: samples  $\leftarrow$  sample  $(\bar{d}_r - d_x)$  elements from  $\mathbf{L}\cup \mathbf{P}$  at random   
4: for  $i$  in [1:N] do   
5:  $\mathbf{R}[i]\gets \mathbf{X}[i]$    
6: for elem in samples do   
7: if elem  $= (a,p)$  is in L then   
8: concatenate lag term:  $\mathbf{R}[i]\gets [\mathbf{R}[i]\parallel \mathbf{x}_{t - p\times s}^a [i]]$    
9: end if   
10: if elem  $= ((a,p),(b,q))$  is in P then   
11: concatenate nonlinear term:  $\mathbf{R}[i]\gets [\mathbf{R}[i]\parallel \mathbf{x}_{t - p\times s}^a [i]*\mathbf{x}_{t - q\times s}^b [i]]$    
12: end if   
13: end for   
14: fit a linear model mapping  $\mathbf{r}_{t - 1}[i]$  to  $\mathbf{r}_t[i]$    
15: vectorize fit parameters to obtain representation  $\theta [i]$    
16: end for   
17: for all representation pairs:  $K_{ij}\gets \mathrm{RBF}(\theta [i],\theta [j])$    
output K
NVAR embeddings The general idea is to transform the time series using the NVAR map defined by Eq. 4, with  $n = 2$ . However, for a given choice of the number of lags  $k$ , the resulting number of additional dimensions scales with  $O(d_x^2 k^2)$ . For moderately high  $d_x$ , this can result in high collinearity and raise the curse of dimensionality in the following readout, making it slow and inaccurate. It also limits the possible choices of  $k$  and, with it, the ability to capture longer-term memory effects. The method, as it is, then only seems tractable in the context of dynamical systems, where  $d_x$  is usually very contained. To address this issue and make the method applicable to higher dimensional datasets, we propose to concatenate just a random subsample of all possible dimensions:
$$
\left. \mathbf {R} _ {N V A R} \right| _ {\bar {d} _ {r}} = \mathbf {X} \left\| \mathbf {r} _ {1} \right\| \left. \mathbf {r} _ {2} \right\| \dots \| \mathbf {r} _ {\bar {d} _ {r} - d _ {x}} \tag {5}
$$
with  $\{\mathbf{r}_a\}_{a = 1}^{\bar{d}_r - d_x}$  randomly sampled columns in  $[\mathbf{R}_{lag}||\mathbf{R}_{nonlin}]$  with uniform probability and  $\bar{d}_r$  a threshold hyperparameter (notation  $[i]$  is omitted for all terms and the same operations are applied to time series  $[j])$ . This converts the deterministic NVAR map into a random feature map. We support this choice in the following, by pointing out an unexplored connection of NVAR models to the theory of state space reconstruction (SSR) (Kantz & Schreiber, 2004; Strogatz, 2018).
It is generally true that we can interpret the observed data as a realization of more complex underlying dynamics  $(\phi^{*})$ , acting on some unknown states  $(\mathbf{x}^{*})$ :  $\mathbf{x}_{t + 1}^{*} = \phi^{*}(\mathbf{x}_{t}^{*})$ . SSR focuses on using the observed data  $(\mathbf{x})$  to construct embedded states  $(\mathbf{s}_t)$  for which the dynamics is topologically equivalent to  $\phi^{*}$ , and thus a better representation of the underlying system (Casdagli et al., 1991). In this regard, the celebrated Takens' theorem (Takens, 1981) claims that a valid embedding, called delay embedding (Packard et al., 1980), can be formed by concatenating the input with its lags and, most importantly, guarantees that not all lags are needed, but only a sufficient number  $\ell$ :  $\mathbf{s}_t = [x_t,x_{t - s},x_{t - 2s},\dots,x_{t - \ell s}]$ . More recently, the work of Deyle & Sugihara (2011) presented a generalization of Takens' theorem to multivariate inputs in which, to match  $\ell$ , lags can be spread over different observed dimensions and alternative functions of the input can also be considered. All terms in the NVAR representation, i.e., lags and nonlinear terms, are reminiscent of this formulation, hence our choice of not considering all terms, but posing instead an upper threshold on the number of concatenation terms (Eq. 5).
This also allows for a different perspective on our approach. We can interpret  $\mathbf{R}_{NVAR}\big|_{\bar{d}_r}$  as an attempt to create states for which the dynamics approximates the underlying one generating the data. As there is generally a loss of information in the generation process, a comparison based on such states would be more truthful than directly using the observed data. In practice, the presence of noise separates us from obtaining such exact states (Casdagli et al., 1991) and places our method under the field of **embedology** (Sauer et al., 1991), i.e., the building of delay-observation maps with special
features, overcoming the sensitivity to noise in the original Takens' formulation. This theoretical framework is more extensively discussed in Appendix A.
An alternative to Eq. 5 would have been to generate the full NVAR representation followed by a dimensionality reduction module as in Bianchi et al. (2020). In our preliminary evaluation, this proved to be computationally more expensive and led to poorer performance, possibly because of the dimensionality reduction not preserving the delay embedding structure.
Linear readout Individual linear readouts (Eq. 3) are trained by ridge regression to solve a one-step-ahead prediction task on the same embedding states, i.e., using  $\mathbf{y}_t = \mathbf{r}_{t + 1}$ . The parameters of each model are then extracted and vectorized to obtain the representations  $\pmb{\theta}[i]$  and  $\pmb{\theta}[j]$ . This step removes the dependence on the length of the TS, which means that the kernel can compare TS of different lengths.
As the readout operates on the input concatenated with its lags and quadratic products, these representation vectors can be interpreted as encapsulating statistics related to the auto-mutual and mutual information within the dimensions of the input. In contrast, representations from RC methods encompass hardly interpretable reservoir dynamics.
RBF aggregation Finally, a radial function is used to ensure the PSD property and form the kernel:
$$
K (\mathbf {X} [ i ], \mathbf {X} [ j ]) = \exp \left\{- \left| \left| \boldsymbol {\theta} [ i ] - \boldsymbol {\theta} [ j ] \right| \right| ^ {2} / 2 \gamma_ {r b f} ^ {2} \right\} \tag {6}
$$
3.3 Hyperparameter setting
We display here the hyperparameters to construct NVARk and propose heuristic expressions.
Number of lags  $(k)$  and lag size  $(s)$  The parameters  $k$  and  $s$  play pivotal roles in the NVAR framework (Eq. 4) and, consequently, in our approach. These key parameters are of clear interpretation:  $k$  controls the number of lags and extends memory capacity, i.e., capturing significant dependencies that extend further into the past;  $s$  controls the spacing between lags, which manages the temporal resolution at which these dependencies are probed. In contrast, key RC parameters are arguably less interpretable.