Title: Time Series Kernels based on Nonlinear Vector AutoRegressive Delay Embeddings
Venue: NeurIPS 2023
Accepted: yes
Keywords: Time Series

Kernel methods

NVAR processes

Dynamical systems

Reservoir Computing
Abstract: Kernel design is a pivotal but challenging aspect of time series analysis, especially in the context of small datasets. In recent years, Reservoir Computing (RC) has emerged as a powerful tool to compare time series based on the underlying dynamics of the generating process rather than the observed data. However, the performance of RC highly depends on the hyperparameter setting, which is hard to interpret and costly to optimize because of the recurrent nature of RC. Here, we present a new kernel for time series based on the recently established equivalence between reservoir dynamics and Nonlinear Vector AutoRegressive (NVAR) processes. The kernel is non-recurrent and depends on a small set of meaningful hyperparameters, for which we suggest an effective heuristic. We demonstrate excellent performance on a wide range of real-world classification tasks, both in terms of accuracy and speed. This further advances the understanding of RC representation learning models and extends the typical use of the NVAR framework to kernel design and representation of real-world time series data.

Time Series Kernels based on Nonlinear Vector AutoRegressive Delay Embeddings

Abstract

Kernel design is a pivotal but challenging aspect of time series analysis, especially in the context of small datasets. In recent years, Reservoir Computing (RC) has emerged as a powerful tool to compare time series based on the underlying dynamics of the generating process rather than the observed data. However, the performance of RC highly depends on the hyperparameter setting, which is hard to interpret and costly to optimize because of the recurrent nature of RC. Here, we present a new kernel for time series based on the recently established equivalence between reservoir dynamics and Nonlinear Vector AutoRegressive (NVAR) processes. The kernel is non-recurrent and depends on a small set of meaningful hyperparameters, for which we suggest an effective heuristic. We demonstrate excellent performance on a wide range of real-world classification tasks, both in terms of accuracy and speed. This further advances the understanding of RC representation learning models and extends the typical use of the NV AR framework to kernel design and representation of real-world time series data.

1 Introduction

Time series are arguably one of the most important types of data in the modern era (Hamilton, 2020) and ubiquitous both in scientific research (Strogatz, 2018) and practical applications (Zhang et al., 2018; Zeroual et al., 2020). A key element in the design of most machine learning protocols for time series is a quantification of similarity (Ding et al., 2008; Abanda et al., 2019; Echihabi et al., 2020). This is especially true for kernel methods (Schölkopf et al., 2001), which search for linear solutions after projecting the data into a higher (possibly infinite) dimensional space and represent an effective alternative to non-linear models. The performance of kernel methods is heavily impacted by the definition of a positive semi-definite (PSD) similarity function for the type of data at hand, i.e., a kernel . However, their design is challenging for structured data such as univariate (UTS) and multivariate time series (MTS), which exhibit temporal autocorrelation, inter-attributes (or dimensions) dependencies and possibly a variety of temporal distortions and misalignments (Paparrizos et al., 2020).

To overcome this, a promising approach is to identify and compare the underlying dynamics rather than the raw observed data. Notably, kernels based on Reservoir Computing (RC) stand out in this context (Chen et al., 2013; Bianchi et al., 2020). RC-based kernels use a randomized and untrained layer of recurrently connected nodes (the reservoir ) to map each time series into a rich dynamical feature space. The similarity is then obtained by comparing individual readouts capturing dynamics. In practice, RC is highly sensitive to a large set of hyperparameters having little interpretability, the optimization of which is challenging for the comprehensive range of values.

Recently, a theoretical work by Bollt (2021) has demonstrated that a simple RC can be formally rewritten as a non-linear vector autoregressive (NV AR) model. In the NV AR framework, the reservoir is replaced by a simple concatenation of the input series with time-delayed copies and nonlinear functionals, such as products (Gauthier et al., 2021). This reduces the number of hyperparameters and has proven to be very effective in chaotic systems forecasting (Shahi et al., 2022), as much to earn the title of "Next-Generation Reservoir Computing". However, its applicability and performance for real-world data and outside of forecasting dynamical systems are largely unexplored.

In this work, we investigate whether an NVAR process can replace a reservoir for kernel design and if the extracted features are just as effective in comparing real-world time series data. We propose a very efficient and effective NV AR kernel for UTS and MTS data. The main idea is as follows (Fig. 1). First, we enrich each time series with lagged copies of itself and additional nonlinear terms, forming a high-dimensional NVAR embedding. Then, we extract a linear parameterization of the time evolution in the embedding space, representative of the underlying dynamics, analogously to RC representation learning methods (Bianchi et al., 2020). Finally, these fitted parameters are used to compute the similarity between the time series. Our main contributions branch out into different communities investigating time series, namely, kernel design, RC and NV AR, bringing specific benefits to each of them:

From a kernel design perspective, we provide an NVAR kernel for UTS and MTS and a general parameter setting based on simple heuristics. Based on our experiments on a wide range of datasets, the proposed approach matches the state-of-the-art (SOTA) in terms of accuracy and is considerably faster than SOTA, representing the best compromise between accuracy and efficiency.

From the perspective of RC, we advance its success for TS representation by incorporating NVAR-made dynamics. The resulting model is more accurate, interpretable, and nonrecurrent, which overcomes difficulties in hyperparameter optimization and sheds light on the potential RC performance.

From an NVAR perspective, we extend the framework to real-world time series representation and kernels, well beyond the original work on forecasting synthetic, noise-free, and chaotic dynamical systems (Gauthier et al., 2021).

We establish a connection to Takens' theorem (Takens, 1981) and the field of state space reconstruction (Sauer et al., 1991), which theoretically underpin our approach, highlighting a compelling avenue for the use of dynamical systems theory in machine learning.

1.1 Notation

Through the paper, we indicate variables as lowercase ( x ); constants as uppercase ( X ) ; vectors and UTS as bold lowercase ( x ) ; matrices and MTS as bold uppercase X . An index between square parenthesis x [ n ] indicates the n -th sample of a set. For an MTS X , the corresponding lower-case x t indicates all dimensions at time stamp t while x d indicates dimension d at all timestamps.

2 Related methods

In this section, we provide a background on the previously proposed kernels for time series data, grouped by category. Despite the abundance of time series similarity measures (Yang & Shahabi, 2004; Paparrizos & Gravano, 2015; Janati et al., 2020), we limit the discussion to PSD metrics.

Lock-step Lock-step approaches view time series as static vectors and directly use common distance metrics (Cha, 2007). The PSD property is usually obtained by superposing a linear or radial kernel (Schölkopf et al., 2002). This is very efficient, but ignores any temporal structure within the series and is not applicable for different lengths.

Elastic Elastic measures account for time distortions (e.g. shifts) or different lengths between sequences. Lu et al. (2008) proposed to interpolate the series and treat the problem as a distance between curves. The Global Alignment Kernel (GAK) (Cuturi, 2011) and KDTW (Marteau & Gibet, 2014) instead are built on the well-known Dynamic Time Warping (DTW) (Berndt & Clifford, 1994) and compute the similarity from the cost of one-to-many alignments between the series. The

Shift-invariant kernel (SINK) (Paparrizos & Franklin, 2019) computes a cross-correlation similarity in the Fourier domain. However, for the aforementioned methods, the computation of the pairwise similarity is computationally expensive and does not account for the relations between different attributes in the multivariate case.

Model-based Model-based kernels process the series with a probabilistic or deterministic model and base the similarity on the extracted information. Such a model can be a single generative model, as in the Fisher kernel (Jaakkola et al., 1999) or the probability product kernels (Jebara et al., 2004), or a parameterized family of probability distributions, as in the autoregressive kernel (Cuturi & Doucet, 2011). Similarly to the latter, the Time Cluster Kernel (TCK) (Mikalsen et al., 2018) is obtained from an ensemble of Gaussian mixture posteriors, sharing the same parametric form but trained on different subsets of the dataset. Among non-probabilistic approaches, the Learned Pattern Similarity (LPS) (Baydogan & Runger, 2016) uses an ensemble of regression trees to extract local patterns from each series. The downside of all such methods comes from the specific functional form, which may limit the generalizability of the extracted features. Ensemble methods are also unsuited for datasets with few training samples.

Reservoir-based Reservoir Computing (RC) is a class of recurrent neural networks that keep recurrent connections untrained in order to overcome the high cost of back-propagation through time (Werbos, 1990) and the vulnerability to exploding and vanishing gradient (Pascanu et al., 2013). Its simplest form, the Echo State Network (ESN) (Jaeger, 2001), is composed of three layers: an input layer , a hidden layer of connected neurons ( reservoir ) and a readout layer :

where x t ∈ R d x , u t ∈ R d r and r t ∈ R d r are, respectively, the input, its upscaling projection and the reservoir states at time t (with d r ≫ d x and r 0 := 0 ); 0 ≤ α ≤ 1 is the leak parameter; ϵ t is an additive noise and f is any non-linear activation function. The reservoir ( A ∈ R d r × d r ) and the projection ( W in ∈ R d r × d x ) matrices are kept untrained, while the readout ( W out ∈ R d y × d r ) and the bias term ( c ) are learned by fitting the reservoir states to the output ( y t ∈ R d y ) via ridge regression.

Despite their randomness, the reservoir states constitute a rich pool of heterogeneous features that give complete knowledge of the underlying dynamical system generating the observed time series (Løkse et al., 2017; Bianchi et al., 2020). In fact, as opposed to fitting a specific functional form, a solid theoretical ground ensures the quality of the reservoir features (Hart et al., 2020; Gonon et al., 2023; Gonon & Ortega, 2021; Hart et al., 2021). Nevertheless, performance is highly sensitive to the choice of a large set of hyperparameters controlling the reservoir initialization and update (Lukoševiˇ cius, 2012). For instance, input scaling ( || W in || ) and spectral radius ( ρ ( A ) ) assure stable reservoir dynamics (Gallicchio, 2019) and control the degree of injected nonlinearity. This is not easy to judge and requires experienced insight into nonlinear dynamics. In general, the interpretation and setting of ESN parameters is a complex task and, to this day, an active area of research (Dong et al., 2022; Steiner et al., 2022; Zhang et al., 2022). Simple supervised optimization, such as cross-validation (CV), is often prohibitive, as the optimization space is large and the recursive nature of the reservoir forces the computation of Eq. 2 for the whole length of the series before assessing the performance of one hyperparameter setting.

Regarding RC-based kernels, Chatzis & Demiris (2011) originally built a radial kernel directly on top of the reservoir states. In Chen et al. (2013), all series are processed by a shared reservoir and compared based on the respective readouts, which are individually trained on the one-step-ahead prediction task ( y t = x t +1 ). Finally, Bianchi et al. (2020) observed that more information is retained by using y t = r t +1 , and an effective kernel can be obtained by applying a radial function on the vectorization of all the readout weights.

ime

rt+1

Wout

rt+c

θ = [vec(wout),c]

X

Rlag

Rnonlin

Linearreadout

Input time series: X

NVAR embedding: RNvAR|d.

3 Methodology

This section presents the main building blocks of NVAR models, followed by our proposal of an NVAR-based kernel and a general setting for the hyperparameters that govern it.

3.1 The Nonlinear Vector AutoRegressive model

Recently, the work of Bollt (2021) has established a connection between the ESN architecture and Nonlinear Vector AutoRegressive (NVAR) models. Under specific conditions, this takes the form of a formal equivalence, where the coefficients of an NV AR process can be expressed in terms of A , W in and W out . By building on such equivalence, in Gauthier et al. (2021), the input layer (Eq. 1) and the reservoir (Eq. 2) of an ordinary ESN are replaced with the deterministic operations that lead to a feature matrix R NVAR . In detail, this is the result of concatenating three terms: the input series ( X ), k lagged copies of the input, each of which is increasingly spaced by s time-stamps ( R lag ), and all possible products 2 between lagged and unlagged dimensions, up to a polynomial order n ( R nonlin ):

where · || · is the column-wise concatenation and · ⌈⊗⌉ · performs the outer product and concatenates all unique monomials.

This procedure is very efficient as it is non-recurrent and the size of R NVAR is usually smaller than a typical reservoir. On top of that, it is determined by a small set of integer hyperparameters: k , s and n . In principle, to sustain the equivalence to RC, an infinite amount of lags ( k → ∞ ) should be considered. Nevertheless, it has been observed that a small k with n = 2 can still perform exceptionally well in typical dynamical systems forecasting tasks (Gauthier et al., 2021; Shahi et al., 2022; Gauthier et al., 2022). However, applicability and performance of NVAR models beyond forecasting yet remain largely unexplored, particularly in the context of representing and comparing real-world time series data. In this regard, they present a promising solution to significantly enhance RC representations and kernels.

3.2 NVAR kernel

We present here the NVAR kernel (NVARk), which integrates the NVAR framework with RC-based kernel architectures. Given a pair of time series, X [ i ] , X [ j ] , with dimensionality d x of possibly different lengths, NVARk operates in three steps, which we present in the following paragraphs. Fig. 1 depicts a graphical reference, while Algorithm 1 explicits the algorithmic process.

Algorithm 1 NVAR kernel

NVAR embeddings The general idea is to transform the time series using the NV AR map defined by Eq. 4, with n = 2 . However, for a given choice of the number of lags k , the resulting number of additional dimensions scales with O ( d 2 x k 2 ) . For moderately high d x , this can result in high collinearity and raise the curse of dimensionality in the following readout, making it slow and inaccurate. It also limits the possible choices of k and, with it, the ability to capture longer-term memory effects. The method, as it is, then only seems tractable in the context of dynamical systems, where d x is usually very contained. To address this issue and make the method applicable to higher dimensional datasets, we propose to concatenate just a random subsample of all possible dimensions:

with { r a } ¯ d r -d x a =1 randomly sampled columns in [ R lag || R nonlin ] with uniform probability and ¯ d r a threshold hyperparameter (notation [ i ] is omitted for all terms and the same operations are applied to time series [ j ] ). This converts the deterministic NVAR map into a random feature map. We support this choice in the following, by pointing out an unexplored connection of NV AR models to the theory of state space reconstruction (SSR) (Kantz & Schreiber, 2004; Strogatz, 2018).

It is generally true that we can interpret the observed data as a realization of more complex underlying dynamics ( ϕ ∗ ), acting on some unknown states ( x ∗ ): x ∗ t +1 = ϕ ∗ ( x ∗ t ) . SSR focuses on using the observed data ( x ) to construct embedded states ( s t ) for which the dynamics is topologically equivalent to ϕ ∗ , and thus a better representation of the underlying system (Casdagli et al., 1991). In this regard, the celebrated Takens' theorem (Takens, 1981) claims that a valid embedding, called delay embedding (Packard et al., 1980), can be formed by concatenating the input with its lags and, most importantly, guarantees that not all lags are needed, but only a sufficient number ℓ : s t = [ x t , x t -s , x t -2 s , ..., x t -ℓs ] . More recently, the work of Deyle & Sugihara (2011) presented a generalization of Takens' theorem to multivariate inputs in which, to match ℓ , lags can be spread over different observed dimensions and alternative functions of the input can also be considered. All terms in the NVAR representation, i.e., lags and nonlinear terms, are reminiscent of this formulation, hence our choice of not considering all terms, but posing instead an upper threshold on the number of concatenation terms (Eq. 5).

This also allows for a different perspective on our approach. We can interpret R NVAR | ¯ d r as an attempt to create states for which the dynamics approximates the underlying one generating the data. As there is generally a loss of information in the generation process, a comparison based on such states would be more truthful than directly using the observed data. In practice, the presence of noise separates us from obtaining such exact states (Casdagli et al., 1991) and places our method under the field of embedology (Sauer et al., 1991), i.e., the building of delay-observation maps with special

features, overcoming the sensitivity to noise in the original Takens' formulation. This theoretical framework is more extensively discussed in Appendix A.

An alternative to Eq. 5 would have been to generate the full NVAR representation followed by a dimensionality reduction module as in Bianchi et al. (2020). In our preliminary evaluation, this proved to be computationally more expensive and led to poorer performance, possibly because of the dimensionality reduction not preserving the delay embedding structure.

Linear readout Individual linear readouts (Eq. 3) are trained by ridge regression to solve a onestep-ahead prediction task on the same embedding states, i.e., using y t = r t +1 . The parameters of each model are then extracted and vectorized to obtain the representations θ [ i ] and θ [ j ] . This step removes the dependence on the length of the TS, which means that the kernel can compare TS of different lengths.

As the readout operates on the input concatenated with its lags and quadratic products, these representation vectors can be interpreted as encapsulating statistics related to the auto-mutual and mutual information within the dimensions of the input. In contrast, representations from RC methods encompass hardly interpretable reservoir dynamics.

RBF aggregation Finally, a radial function is used to ensure the PSD property and form the kernel:

3.3 Hyperparameter setting

We display here the hyperparameters to construct NVARk and propose heuristic expressions.

Number of lags ( k ) and lag size ( s ) The parameters k and s play pivotal roles in the NV AR framework (Eq. 4) and, consequently, in our approach. These key parameters are of clear interpretation: k controls the number of lags and extends memory capacity, i.e., capturing significant dependencies that extend further into the past; s controls the spacing between lags, which manages the temporal resolution at which these dependencies are probed. In contrast, key RC parameters are arguably less interpretable.

In Takens' formulation, in the case of infinitely long, noise-free time series, there is no importance bias towards any specific lag s , and their number k is the only relevant factor. This is often determined using the False Nearest Neighbours algorithm (Kennel et al., 1992; Wallot & Mønster, 2018). However, when dealing with limited noisy observations, the choice of s becomes equally critical as it directly relates to the sampling frequency (Broomhead & King, 1986). In particular, using too high a value would force a modeling of uncorrelated points while too low would model noise. A correct choice of s instead allows for extracting seasonality or other relevant trends as well as the underlying dynamics. In practice, for noisy systems, the choice of both k and s is inherently a difficult problem due to the absence of clear theoretical guidelines (Tran & Hasegawa, 2019). Despite that, interpretability can be exploited to propose task-dependent heuristics (Small & Tse, 2004), often requiring a trade-off between redundancy and the irrelevance of concatenated features.

In this work, we propose two settings. The first is a rule of thumb and proceeds as follows. We first apply a trend-filtering method to the time series, such as an ℓ 1 trend (Kim et al., 2009) (this allows for a choice of s that captures the temporal variability of the system rather than the superposed noise). We then set the product k · s as the average distance between peaks and hollows (dbp) (Kugiumtzis, 1996). Finally, we balance the contributions by adopting s = k = √ dbp. Alternatively, in a supervised setting, both parameters are optimized by CV.

Polynomial order ( n ) The polynomial order controls the amount of non-linearity in the embedding. As in most previous works, we found no need for further complexity than n = 2 .

Embedding dimension ( ¯ d r ) In Takens' formulation, an embedding size double the dimensionality of the underlying manifold is sufficient for reconstructing the system dynamics. However, when dealing with real-world data, this is unknown, and a trade-off must be struck between the omission of relevant features and redundancy, which can increase collinearity and computation times. In line with the scope of our work, we choose to give priority to a fair comparison with previous RC

methods (Bianchi et al., 2020), which identify ¯ d r = 75 as an optimal dimensionality before the readout. In Appendix C.1, we undertake a brief exploration to determine whether this value is also a reasonable choice for NVAR embeddings. While we did not observe any redundancy in univariate datasets, we did observe a performance drop after different ¯ d r for some multivariate datasets. The study suggests 70 ≲ ¯ d r ≲ 100 as a suitable range.

Linear readout regularization ( λ ridge ) For the readout, we adopt the optimal regularization (OCReP) proposed in Cancelliere et al. (2015), i.e., set λ ridge to the product between the minimum and maximum singular value of R NVAR | ¯ d r .

RBF lengthscale ( γ rbf ) We set the lengthscale in Eq. 6 to the median pairwise distance between all the representations θ [ i ] in the training set.

3.4 Asymptotic complexity

NVARk is based on the individual representations of time series. Hence, the computationally intensive step is performed only once per time series and results in O ( N ) complexity. This also implies that the computation can be parallelized along N. This provides a significant computational advantage over comparison-based kernels, which must perform expensive computations for all the relevant pairs of series, i.e., O ( N 2 ) .

In terms of scalability with the length of the series ( T ), the advantage of NVAR over any recursive RC approach is the critical difference in how the hidden states are created. Eq. 5 is not iterative and only performs a fixed number ¯ d r -d x of concatenation operations, independent of the length of the time series. In contrast, reservoir-based methods are limited by the expensive recursive update of the reservoir state (Eq. 2), which is O ( Td 2 r ) , where d r is the size of the reservoir (which is usually quite large). As evidence, improving the scalability of the reservoir update is an active area of research (Dong et al., 2020). The complexity of NVARk is limited to the only ridge regression of the output layer. This corresponds to solving a linear system with T training examples and D features, where D is at most ¯ d r . Such a system can usually be solved efficiently, e.g. by using LU (or Cholesky) factorization in asymptotically O ( T ¯ d 2 r ) .

As for the scalability with the dimensionality of the input series, all operations are bounded by ¯ d 2 r , which we treat as a constant and do not discuss complexity in connection to this parameter. Overall, NVARk exhibits an asymptotic complexity at most O ( NT ¯ d 2 r ) .

4 Experimental evaluation

In this section, we present an experimental demonstration of the performance of NVARk. In line with the established literature (Cuturi, 2011; Cuturi & Doucet, 2011; Baydogan & Runger, 2016; Paparrizos et al., 2020), our main evaluation consists of time series classification. In particular, we pair different pre-computed kernel matrices with a Support Vector Machine (SVM) classifier (Steinwart & Christmann, 2008), a popular kernel classification method that lies between simple linear models and more advanced deep learning architectures. The evaluation is performed over 130 UTS datasets from the UCR archive and 19 MTS datasets from the UEA archive (Bagnall A. & E.) (Appendix B.1) and focuses on accuracy, execution time and scalability. In Appendix C.5, we present a snapshot of kernel Principal Component Analysis (kPCA) for dimensionality reduction and visualization.

Performance of the proposed approach is investigated under two different hyperparameter settings. We indicate with NVARk the setting in which k and s are determined by the general proposed rule of thumb of Sec. 3.3. The maximum number of dimensions is set to ¯ d r = 75 . As for the other hyperparameters, we follow the setting given in Sec. 3.3. We instead indicate with NVARk* the configuration in which k and s are optimized in a CV loop over a small grid. As the datasets cover a wide range of series lengths, the CV grid is slightly adapted consequently (Appendix. B.2).

The comparison is performed against previously proposed kernels: the reservoir-model Echo State Network (rmESN) (Bianchi et al., 2020) as the reservoir equivalent to our approach. We set D = 75 as the dimensionality before the linear readout. This allows us to directly compare the quality of the extracted dynamic features with or without a reservoir; the Time Cluster Kernel (TCK) (Mikalsen et al., 2018) as representative of model-based kernels and SOTA for MTS; the Shift-INvariant

Kernel (SINK) (Paparrizos & Franklin, 2019) as representative of Fourier-based methods and SOTA for UTS. A comparison with the Global Alignment Kernel (GAK) (Cuturi, 2011) is presented in Appendix C.4. Guidelines in Appendix B.2 contain the hyperparameter settings and links to the code implementations for NVARk and all the aforementioned baselines.

Despite the fact that some datasets in our benchmark can be also investigated using Deep Learning (DL) methods (Ismail Fawaz et al., 2019), the use case for kernel methods is substantially different, usually focused on small datasets, where the application of DL is not trivial. In particular, kernels can compute the similarity between just two time series right away. Consequently, in the literature, kernels are usually not directly tested against DL. Nevertheless, in our work, we compare to ESN-based representations, which are known to be competitive with a variety of DL architectures (Bianchi et al., 2020; Shahi et al., 2022). Thus, ESN can be seen as a proxy for more complex architectures.

As preprocessing, we apply zero-padding to match all series to the maximum length in each dataset (to allow comparison with TCK) and follow the common practice of standardizing all series to zero mean and unit variance. For each dataset-method pair, we allow for a maximum execution time of 20 hours.

4.1 SVM classification of UTS

We show here SVM classification for univariate datasets. SVM requires the setting of the hyperplane regularization C , which we optimize in a CV loop from logarithmically spaced values in [ 10 -3 , 10 3 ]. This is the only free parameter in all methods except NV ARk*, in which it is jointly optimized with k and s . For the CV loop, we adopt a 10-fold CV with the size of the validation set corresponding to 33% of the train set. SVM accuracy for each dataset is obtained by averaging 10 repetitions with different seeds. Tab. 1 reports the average classification accuracy for all kernels within the presented framework, as well as the relative ranking (individual results are reported in Appendix D). Overall,

NVARk obtains very competitive results, with the highest average accuracy and rank. The case of univariate datasets is a comfortable scenario for NV ARk to operate in as, in most cases, all possible terms in R NVAR can be considered without exceeding ¯ d r and the effect of collinearity is minimized. NVARk and SINK appear indistinguishable under a Friedman + Nemenyi statistical test, which places NVARk under the SOTA in terms of accuracy. Interestingly, the difference between NVARk and rmESN is significant at the 95% level with p-value NVARk -rmESN = 0 . 013 . Results for NVARk* show that the performance of NVARk can be very efficiently improved by a CV optimization of just the two integer embedding hyperparameters.

4.2 Execution time and scalability

In terms of execution time, the scalability properties of NV ARk (Sec. 3.4) manifest in exceptional performance. In Fig. 2 (left) and Tab. 1 (last row), we compare running times to compute the traintrain and the train-test kernel matrices and observe that the improvement of NVARk is consistent across datasets and baselines, with speedup factors up to × 10 3 in the highT and highN regimes.

Results for NVARk* can be achieved with a slowdown factor of ∼ 25 (parallelized grid search + 1 iteration with optimal parameters) with respect to the heuristic-based NV ARk. Interestingly, for 113/130 datasets, NVARk* execution time is faster than a single iteration for rmESN.

For a study of the scaling, in Fig. 2 (middle, right) we progressively vary the length of the AbnormalHeartbeat dataset (by interpolation) and the number of training samples (by random sampling) in the Crop dataset. Results show that NVARk performs exceptionally well in all regimes, while rmESN

5

2

5

Timelimitexceeded

·rmESN·TCK·SINK

1000

2

5

AbnormalHeartbeat

+NVARk+rmESN+TCK+SINK

1000

5

2

Crop

+NVARk+rmESN+TCK+SINK

2

2

5

100

100

2

e

10

2

5

S

e

10

2

100

10

5

2

5

e

1

2

1

2

215

0.1

5

2

0.1

2

5

0.1

2

0

20

40

60

80

100

120

10

2

5

5100

2

510002

5

10k2

10

2

5

100

2

5

510002

5

10k

UCR datasetindex

Serieslength

Training samples

scales poorly with the length of the series and TCK and SINK with the size of the training sample. NVARk, instead, is advantaged by its non-recursive structure and linear complexity in N .

4.3 SVM classification of MTS

We present here SVM classification over the 19 MTS datasets. We do not include SINK as there is no extension or evaluation provided for MTS. Aside from that, the evaluation is conducted identically to the univariate case. For multivariate datasets, we expect to drift away from the ideal operating

scenario of NVARk. In particular, all the possible dynamic features may not be equally effective and, by randomly selecting a few, important correlations may be missed or spurious ones captured. Despite that, the experiments in Tab. 2 show surprisingly good results, even for datasets for which ¯ d r is just a fraction of all possible concatenation terms. We believe that this finding strengthens the connection to the generalized Takens' theorem. It is also interesting to notice good performance in diverse cases of time series that do not immediately arise from dynamical systems. Exploring this aspect constitutes interesting future works, e.g., time series representation learning and state space reconstruction as a possible underpinning.

Finally, we observe that the two hyperparameter configurations, NVARk* and NVARk, tend to be ranked next to each other. This supports the quality of the proposed general setting, which allows the kernel to extract meaningful features while preserving great computational efficiency.

4.4 Ablation study

In order to tease out which NVARk component is most important, we here present an ablation study. We separately consider fitting the linear readout directly on the input time series (no concatenation), sampling only linear terms from Eq. 5, and sampling only non-linear terms. This is an interesting ablation study that, to the best of our knowledge, has not been investigated in previous works. Results are presented in Tab. 3.

As a general trend, we observe that using all terms leads to the best results, followed by non-linear only , linear only , and fitting directly on the input series. Note also that results are in line with our interpretation using the generalized Taken's theorem. For univariate datasets, linear only tends to underperform, as the total number of concatenated dimensions is very small (equal to k ) and may be not sufficient in reconstructing the underlying manifold. In the multivariate case, the two variants perform very similarly, as the lags of all attributes may already constitute a considerable pool of terms.

To conclude, we consider an additional variant of Eq. 5 in which we first concatenate all the lag terms, and then fill the remaining dimensions, up to ¯ d r , by sampling nonlinear terms. Interestingly, for multivariate datasets, this leads to slightly better performance and a reduction in variance. We believe this might eventually compensate for the imbalance between all possible linear and non-linear terms, while also reducing the chance of adding noise from uncorrelated dimensions.

5 Conclusions

We have proposed a kernel for time series that integrates the NVAR framework into reservoirbased kernel architectures. NVARk compares time series based on the linear dynamics of NVAR embeddings, which are built from concatenating lags and nonlinear functionals to the original series. In terms of accuracy, NVARk outperforms the corresponding RC architecture. Computationally, it is exceptionally efficient and based on a few integer hyperparameters, which together allow for further improvement of the results with simple supervised grid-based optimization.

As for future work directions, we believe it would be effective to target specific weaknesses of NV ARk. In particular, as the input dimensionality of the time series approaches the maximum embedding dimension ( ¯ d r ), the proposed approach finds little margin for concatenating more lags and non-linear terms. Such cases would require substantially increasing ¯ d r and, consequently, place excessive strain on the linear readout, which inherently possesses limited expressive capacity. We then expect the effectiveness of NVARk to decrease. To overcome this, one can consider replacing the random sampling of dimensions with more refined strategies that prioritize the selection of meaningful terms. Alternatively, it would be interesting to explore how different linear layers, e.g. Lasso, would perform in this regime. Alongside these, different unsupervised strategies can be considered to learn optimal values for the embedding parameters. In particular, we note an unexplored affinity with signature transforms (Chevyrev & Kormilitzin, 2016), which we plan on deepening to infer optimal settings for the maximum dimensionality of the NVAR embedding.

References

Abanda, A., Mori, U., and Lozano, J. A. A review on distance based time series classification. Data Mining and Knowledge Discovery , 33(2):378-412, 2019.

Bagnall A., Lines J., V. W. and E., K. The UEA and UCR time series classification repository. www. timeseriesclassification.com .

Baydogan, M. G. and Runger, G. Time series representation and similarity based on local autopatterns. Data Mining and Knowledge Discovery , 30(2):476-509, 2016.

Berndt, D. J. and Clifford, J. Using dynamic time warping to find patterns in time series. KDD workshop , 10 (16):359-370, 1994.

Bianchi, F. M., Scardapane, S., Løkse, S., and Jenssen, R. Reservoir computing approaches for representation and classification of multivariate time series. IEEE transactions on neural networks and learning systems , 32 (5):2169-2179, 2020.

Bollt, E. On explaining the surprising success of reservoir computing forecaster of chaos? The universal machine learning dynamical system with contrast to V AR and DMD. Chaos: An Interdisciplinary Journal of Nonlinear Science , 31(1):013108, 2021.

Broomhead, D. S. and King, G. P. Extracting qualitative dynamics from experimental data. Physica D: Nonlinear Phenomena , 20(2-3):217-236, 1986.

Cancelliere, R., Gai, M., Gallinari, P., and Rubini, L. OCReP: An optimally conditioned regularization for pseudoinversion based neural training. Neural Networks , 71:76-87, 2015.

Casdagli, M., Eubank, S., Farmer, J. D., and Gibson, J. State space reconstruction in the presence of noise. Physica D: Nonlinear Phenomena , 51(1-3):52-98, 1991.

Cha, S.-H. Comprehensive survey on distance/similarity measures between probability density functions. City , 1 (2):1, 2007.

Chatzis, S. P. and Demiris, Y. Echo state gaussian process. IEEE Transactions on Neural Networks , 22(9): 1435-1445, 2011.

Chen, H., Tang, F., Tino, P., and Yao, X. Model-based kernel for efficient time series analysis. Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining , pp. 392-400, 2013.

Chevyrev, I. and Kormilitzin, A. A primer on the signature method in machine learning. arXiv preprint arXiv:1603.03788 , 2016.

Cuturi, M. Fast global alignment kernels. Proceedings of the 28th international conference on machine learning (ICML-11) , pp. 929-936, 2011.

Cuturi, M. and Doucet, A. Autoregressive kernels for time series. arXiv preprint arXiv:1101.0673 , 2011.

Deyle, E. R. and Sugihara, G. Generalized theorems for nonlinear state space reconstruction. Plos one , 6(3): e18295, 2011.

Ding, H., Trajcevski, G., Scheuermann, P., Wang, X., and Keogh, E. Querying and mining of time series data: experimental comparison of representations and distance measures. Proceedings of the VLDB Endowment , 1 (2):1542-1552, 2008.

Dong, J., Ohana, R., Rafayelyan, M., and Krzakala, F. Reservoir computing meets recurrent kernels and structured transforms. Advances in Neural Information Processing Systems , 33:16785-16796, 2020.

Dong, J., Börve, E., Rafayelyan, M., and Unser, M. Asymptotic stability in reservoir computing. In 2022 International Joint Conference on Neural Networks (IJCNN) , pp. 01-08. IEEE, 2022.

Echihabi, K., Zoumpatianos, K., Palpanas, T., and Benbrahim, H. The lernaean hydra of data series similarity search: An experimental evaluation of the state of the art. arXiv preprint arXiv:2006.11454 , 2020.

Franz, M. O. and Schölkopf, B. A unifying view of wiener and volterra theory and polynomial kernel regression. Neural computation , 18(12):3097-3118, 2006.

Gallicchio, C. Chasing the echo state property. European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning , 2019.

Gauthier, D. J., Bollt, E., Griffith, A., and Barbosa, W. A. Next generation reservoir computing. Nature communications , 12(1):1-8, 2021.

Gauthier, D. J., Fischer, I., and Röhm, A. Learning unseen coexisting attractors. Chaos: An Interdisciplinary Journal of Nonlinear Science , 32(11):113107, 2022.

Gonon, L. and Ortega, J.-P. Fading memory echo state networks are universal. Neural Networks , 138:10-13, 2021.

Gonon, L., Grigoryeva, L., and Ortega, J.-P. Approximation bounds for random neural networks and reservoir systems. The Annals of Applied Probability , 33(1):28-69, 2023.

Hamilton, J. D. Time series analysis . Princeton university press, 2020.

Hart, A., Hook, J., and Dawes, J. Embedding and approximation theorems for echo state networks. Neural Networks , 128:234-247, 2020.

Hart, A. G., Hook, J. L., and Dawes, J. H. Echo state networks trained by Tikhonov least squares are L2 ( µ ) approximators of ergodic dynamical systems. Physica D: Nonlinear Phenomena , 421:132882, 2021.

Ismail Fawaz, H., Forestier, G., Weber, J., Idoumghar, L., and Muller, P.-A. Deep learning for time series classification: a review. Data mining and knowledge discovery , 33(4):917-963, 2019.

Jaakkola, T. S., Diekhans, M., Haussler, D., et al. Using the Fisher kernel method to detect remote protein homologies. ISMB , 99:149-158, 1999.

Jaeger, H. The 'echo state' approach to analysing and training recurrent neural networks-with an erratum note. Bonn, Germany: German National Research Center for Information Technology GMD Technical Report , 148 (34):13, 2001.

Janati, H., Cuturi, M., and Gramfort, A. Spatio-temporal alignments: Optimal transport through space and time. International Conference on Artificial Intelligence and Statistics , pp. 1695-1704, 2020.

Jebara, T., Kondor, R., and Howard, A. Probability product kernels. The Journal of Machine Learning Research , 5:819-844, 2004.

Kantz, H. and Schreiber, T. Nonlinear time series analysis , volume 7. Cambridge university press, 2004.

Kennel, M. B., Brown, R., and Abarbanel, H. D. Determining embedding dimension for phase-space reconstruction using a geometrical construction. Physical review A , 45(6):3403, 1992.

Kim, S.-J., Koh, K., Boyd, S., and Gorinevsky, D. ℓ 1 trend filtering. SIAM Review , 51(2):339-360, 2009.

Kugiumtzis, D. State space reconstruction parameters in the analysis of chaotic time series-the role of the time window length. Physica D: Nonlinear Phenomena , 95(1):13-28, 1996.

Løkse, S., Bianchi, F. M., and Jenssen, R. Training echo state networks with regularization through dimensionality reduction. Cognitive Computation , 9:364-378, 2017.

Lu, Z., Leen, T. K., Huang, Y., and Erdogmus, D. A reproducing kernel hilbert space framework for pairwise time series distances. In Proceedings of the 25th international conference on Machine learning , pp. 624-631, 2008.

Lukoševiˇ cius, M. A practical guide to applying echo state networks. In Neural networks: Tricks of the trade , pp. 659-686. Springer, 2012.

Marteau, P.-F. and Gibet, S. On recursive edit distance kernels with application to time series classification. IEEE transactions on neural networks and learning systems , 26(6):1121-1133, 2014.

Mikalsen, K. Ø., Bianchi, F. M., Soguero-Ruiz, C., and Jenssen, R. Time series cluster kernel for learning similarities between multivariate time series with missing data. Pattern Recognition , 76:569-581, 2018.

Packard, N. H., Crutchfield, J. P., Farmer, J. D., and Shaw, R. S. Geometry from a time series. Physical review letters , 45(9):712, 1980.

Paparrizos, J. and Franklin, M. J. Grail: efficient time-series representation learning. Proceedings of the VLDB Endowment , 12(11):1762-1777, 2019.

Paparrizos, J. and Gravano, L. k-shape: Efficient and accurate clustering of time series. Proceedings of the 2015 ACM SIGMOD international conference on management of data , pp. 1855-1870, 2015.

Paparrizos, J., Liu, C., Elmore, A. J., and Franklin, M. J. Debunking four long-standing misconceptions of time-series distance measures. Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data , pp. 1887-1905, 2020.

Pascanu, R., Mikolov, T., and Bengio, Y. On the difficulty of training recurrent neural networks. International conference on machine learning , pp. 1310-1318, 2013.

Sauer, T., Yorke, J. A., and Casdagli, M. Embedology. Journal of statistical Physics , 65(3):579-616, 1991.

Schölkopf, B., Herbrich, R., and Smola, A. J. A generalized representer theorem. International conference on computational learning theory , pp. 416-426, 2001.

Schölkopf, B., Smola, A. J., Bach, F., et al. Learning with kernels: support vector machines, regularization, optimization, and beyond . MIT press, 2002.

Shahi, S., Fenton, F. H., and Cherry, E. M. Prediction of chaotic time series using recurrent neural networks and reservoir computing techniques: A comparative study. Machine Learning with Applications , 8:100300, 2022.

Small, M. and Tse, C. K. Optimal embedding parameters: a modelling paradigm. Physica D: Nonlinear Phenomena , 194(3-4):283-296, 2004.

Steiner, P., Jalalvand, A., and Birkholz, P. Cluster-based input weight initialization for echo state networks. IEEE Transactions on Neural Networks and Learning Systems , 2022.

Steinwart, I. and Christmann, A. Support vector machines . Springer Science & Business Media, 2008.

Strogatz, S. H. Nonlinear dynamics and chaos: with applications to physics, biology, chemistry, and engineering . CRC press, 2018.

Takens, F. Detecting strange attractors in turbulence. Dynamical systems and turbulence, Warwick 1980 , pp. 366-381, 1981.

Tavenard, R., Faouzi, J., Vandewiele, G., Divo, F., Androz, G., Holtz, C., Payne, M., Yurchak, R., Rußwurm, M., Kolar, K., and Woods, E. Tslearn, a machine learning toolkit for time series data. Journal of Machine Learning Research , 21(118):1-6, 2020. URL http://jmlr.org/papers/v21/20-091.html .

Tran, Q. H. and Hasegawa, Y. Topological time-series analysis with delay-variant embedding. Physical Review E , 99(3):032209, 2019.

Wallot, S. and Mønster, D. Calculation of average mutual information (AMI) and false-nearest neighbors (FNN) for the estimation of embedding parameters of multidimensional time series in Matlab. Frontiers in psychology , 9:1679, 2018.

Werbos, P. J. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE , 78(10): 1550-1560, 1990.

Whitney, H. Differentiable manifolds. Annals of Mathematics , pp. 645-680, 1936.

Yang, K. and Shahabi, C. A PCA-based similarity measure for multivariate time series. Proceedings of the 2nd ACM international workshop on Multimedia databases , pp. 65-74, 2004.

Zeroual, A., Harrou, F., Dairi, A., and Sun, Y. Deep learning methods for forecasting COVID-19 time-series data: A comparative study. Chaos, Solitons & Fractals , 140:110121, 2020.

Zhang, Z., Zhang, K., and Khelifi, A. Multivariate time series analysis in climate and environmental research . Springer, 2018.

Zhang, Z., Zhu, Y., Wang, X., and Yu, W. Optimal echo state network parameters based on behavioural spaces. Neurocomputing , 503:299-313, 2022.

Appendix

A Takens' delay embeddings

This section introduces the field of state space reconstruction in dynamical system theory (Strogatz, 2018) and deepens the theoretical interpretation of Sec. 3.2. As in Deyle & Sugihara (2011), we start by defining some key concepts that might be useful to introduce the reader to the subject:

a manifold is, intuitively, a generalized surface M ⊆ R N that locally looks like an Euclidean space. More rigorously, locally diffeomorphic to some R k ;

a dynamical system ϕ : M → M is a diffeomorphism defining, for discrete times, trajectories on a manifold M;

an embedding is a 1:1 smooth map f : M 1 → M 2 between two manifolds. Embeddings preserve topological properties, e.g. resolves all trajectories without crossings, and geometrical invariants, such as eigenvalues and fixed points;

an observation function y : M → R D is any function that assigns a vector to each point on M.

It is generally true that we can imagine an observed MTS X ∈ R T × D as a realization of a more complex and unknown smooth dynamical system ϕ ∗ , acting on states x ∗ on an unknown manifold M ∗ :

The observed and original values are related by an observation function y ∗ : M ∗ → R D that projects the underlying states onto what we observe:

This projection can drastically reduce the available information, but one can wonder to which extent it is possible to recover the original system when only the observed vectors ( x t ) are accessible. We focus here on the state reconstruction problem , which is that of finding a valid embedding f : M ∗ → R n of the original manifold M ∗ :

This is, by using the observed states ( x t ) , creating richer states ( s t ) for which the dynamics is topologically equivalent to the unknown ϕ ∗ . In fact, if f is an embedding, then smooth dynamics Φ is induced on the space of reconstructed vectors:

and is equivalent to ϕ ∗ . If we succeed, instead of the observed x , we can use the reconstructed vectors s as a better representation of the underlying evolving system x ∗ .

Whitney's theorem (Whitney, 1936) states the general result that every compact m-dimensional manifold admits an embedding in R 2 m +1 . For dynamical systems this means that, a set of 2 m +1 observation functions, where y k : M → R , is sufficient to construct a valid embedding:

In the case of scalar observations (D=1), Takens' theorem (Takens, 1981) considers a single observation function ( y ), i.e., the projection on the only observed component, but acting on multiple lagged versions of the system:

The idea behind Eq. 12 is that the information about the unobserved state variables is contained in the relations between past and future values of the observed time series and a 2 m +1 dimensional embedding (often less) is sufficient to extract that.

Generalized versions of Takens' theorem have also been formulated. In particular, we find in Deyle & Sugihara (2011) an extension to cases where lags of multiple observation functions can be used to create the embedding. This is a useful result in cases where one cannot consider many lags, but can instead make use of fewer lags spread across multiple available dimensions. In more detail, the allowed set of observation functions is any { y k } 2 m +1 k =1 containing terms { y r } that cannot be written as a lagged version of any other element in the set and terms { y q } that satisfy y q = y r ( ϕ b ( x ∗ )) for a choice of b . Both Takens' and Whitney's theorems can be recovered from this formulation: it reduces to the case of Takens' when only one { y r } = y is chosen and all the others belong to the family of { y q } ; it reduces to the case of Whitney's when { y r } = { y k } and { y q } = ∅ .

All terms in the NVAR representation are reminiscent of this formulation. This points to the direction of R NVAR being a closer approximation of the underlying state than the initial data X and also suggests that the final number of dimensions plays a more important role than which are included and which are not. In practice, the dimensionality m of the original manifold M ∗ is unknown, making the selection of the embedding dimensionality a potentially rich field of study. On top of that, the observed data may be corrupted by multiple noise sources, which increases the importance of sampling the series correctly with a meticulous choice of the lag size ( τ ≡ s ) .

B Experimental evaluation details

B.1 Datasets

Details of the 130 Univariate and 19 Multivariate datasets are reported in Tab. 4. Both cover an extensive range of data points and series lengths. From the UEA archive, we have excluded 6 high-dimensional datasets for which the application of the NV AR kernel is not appropriate, and 5 datasets for which SVM accuracy was < 0 . 5 for all approaches, which indicates that kernel methods are, in general, not a suitable solution.

B.2 Implementations and details for reproducibility

For reproducibility, we list here the parameter setting and the code implementation for each method. Experiments were run on a 32-core AMD Ryzen™ Threadripper PRO CPU.

NVAR kernel (NVARk)

the allowed number of lags k = √ average distance between peaks and hollows, after series have been ℓ 1 trend-filtered (Kim et al., 2009);

lag size s = k ;

order of the polynomial functionals n = 2 ;

maximum dimensionality of the embedding ¯ d r = 75 ;

readout regularization set by the OCReP method (Cancelliere et al., 2015);

RBF lengthscale as the median pairwise distance between the representations.

An easy-to-use Python implementation of the NVAR kernel is made publicly available at https://github. com/gdefe/nvark-kernel .

optimized NVAR kernel (NVARk*) The lag size s and the allowed number of lags k are optimized in a CV loop scrolling a small grid. Values for the grid are reported in Tab. 5. k ∗ and s ∗ refer to the values obtained from the rule of thumb.

In general, one should allow for larger values for k and s in the case of longer series. A higher k allows capturing longer-term dynamics, while a larger s avoid oversampling the series. In the case of multivariate datasets, we observed that many lags are hardly required. This is possible because few lags spread across different dimensions can be sufficient to construct the embedding. Also, the pool of features ( [ R lag || R nonlin ] ) largely increases with the dimensionality of the dataset and k , which affects the stability of the approach.

reservoir-model ESN (rmESN) For the rmESN, we follow the setting given in Bianchi et al. (2020):

number of internal reservoir units R = 800 ;

spectral radius ρ = 0 . 99 ;

no leakage;

percentage of non-zero connectivity within the reservoir states β = 0 . 25 ;

input scaling ω = 0 . 15 ;

noise level in the reservoir update ϵ t = 0 . 001 ;

transient dropped states n drop = 5 ;

number of dimensions after the dimensionality reduction module D = 75 (via tenor-PCA, also proposed in the same paper);

favoring comparability with NVARk, we set the readout regularization by the OCReP method;

favoring comparability with NVARk, we set the RBF lengthscale to the median pairwise distance between the representations.

As implementation, we used the public Python code provided by the authors: https://github.com/ FilippoMB/Time-series-classification-and-clustering-with-Reservoir-Computing .

Time Cluster Kernel (TCK) For TCK, we follow the guidelines given in the original paper and in the provided code. For a dataset of N samples and D attributes, these are:

minimum percentage of subsample minN = 0 . 8 ;

min number of attributes for each GMM minD = 1 for univariate datasets, minD = 2 for multivariate datasets;

max number of attributes for each GMM maxD = min ( ceil (0 . 9 ∗ D ) , 15) ;

min length of time segments for each GMM minT = 6 ;

max length of time segments for each GMM maxT = min ( floor (0 . 8 ∗ T ) , 25) ;

max number of mixture components for each GMM C = 10 if N < 100 , C = 40 otherwise;

number of randomizations for each number of components G = 30 ;

number of iterations I = 20 .

The Matlab implementation of TCK that we use is part of the code available at https://github.com/ FilippoMB/TCK_AE .

Global Alignment Kernel (GAK) We set the multiplicative factor for the bandwidth of the radial function to σ GAK = 2 . Python code is available from the tslearn package (Tavenard et al., 2020). As reported in the package documentation, the method can lead to numerical issues for long series.

Shift INvariant Kernel (SINK) For SINK, we adopted the Python implementation provided by the author at https://github.com/TheDatumOrg/grail-python . We use γ = 5 and retain all the energy after Fourier-transforming the series.

C Additional experiments

C.1 Feature redundancy

For studying whether previously adopted values for the embedding dimensionality ( ¯ d r ) are also suitable for NVARk, we adopted an experimental approach and studied how many features can be included before experiencing redundancy. We do so by plotting, for a few univariate and multivariate datasets, the accuracy of an SVM classifier against the ¯ d r parameter (Fig. 3). While do not observe redundancy in univariate datasets,

—ArrowHead (6/6)

—HouseTwenty (6/6)

2

—SwedishLeaf (4/4)

—GunPoint (7/7)

3

456789

10

2

3

4567

NVARembeddingdimensionality

—Libras (4/4)

-Cricket (7/7)

—UWaveGestureLibrary (7/7)

FingerMovements (1/1)

-Epilepsy (2/2)

1

0.9

SVM accuracy

0.8

0.7

0.6

0.5

2

3

456789

10

2

3

456789

100

2

3

45

NVARembeddingdimensionality

we do observe a drop in performance after different ¯ d r for some multivariate datasets. We identify the region 70 ≤ ¯ d r ≤ 100 as a good, though non-optimal, setting, which is in line with previous RC studies (Bianchi et al., 2020).

SVM accuracy

0.9

0.8

0.7

0.6

0.5

0.4

1

C.2 Sensitivity analysis

In this section, we address the sensitivity of NV ARk to the different hyperparameters:

multiv. datasets)

(20 univ. datasets)

0.88

0.74

0.75

0.72

0.70

(16

0.87

0.70

0.65

NVARk average accuracy (

accuracy

0.86

0.68

0.60

NVARk average

0.66

with OCReP=0.772+-0.004

0.55

0.85

0.64

0.50

0.84

0.62

0.45

50

75100

hidden dimensionality

150

200

250

300

103

10-2

10-1

100

Aridge

101

102

103

10-1

100

YRsF multiplier

10

102

¯ d r : as we show in Fig. 4 (left), the average performance for multivariate datasets is not considerably impacted in the range ¯ d r ∈ [70 , 150] . As expected, we observe a significant drop in performance for high values (redundancy).

λ ridge: we show in Fig. 4 (center) little sensitivity in the region λ ridge ∈ [5 , 50] . Results are averaged across a pool of 20 randomly sampled univariate datasets. The employed OCReP method (Cancelliere et al., 2015) outperforms the best choice by ∼ 2% , as it adapts the value to each individual dataset.

γ rbf: in Fig. 4 (right), we range different multiplicative factors for the median pairwise distance between all representations and show little sensitivity in a neighborhood of our choice γ rbf mul = 1 .

k and s : The sensitivity to the embedding parameters strongly depends on the dataset, although it generally tends to be high. In response, we introduce our dataset-specific heuristic adapting ideas from a precedent study (Kugiumtzis, 1996), which we find works well in practice. A reasonable idea of the average sensitivity to these parameters can be obtained by comparing the performance of NVAR, where k and s are chosen by the heuristic, to the performance of NV ARk*, where k and s are optimized. Unfortunately, understanding sensitivity by considering other settings is challenging, as the existing literature predominantly focuses on the noise-free case. Furthermore, there is no strong evidence for the wider applicability of such approaches outside dynamical systems.

C.3 Uneven lengths

NVARk can process and compare time series of different lengths. We report in Tab. 6 the SVM accuracy scores for three datasets with uneven lengths.

C.4 Comparison with Global Alignment Kernel

We compare here the performance of the NVAR kernel against the popular Global Alignment Kernel (GAK) baseline from Cuturi (2011). This has been moved from the main text as authors of SINK have already demonstrated superior performance to GAK, following a similar evaluation. As reported in the tslearn package documentation, the provided implementation can lead to numerical issues when used with relatively long time series. To face this, we evaluate GAK on time series linearly interpolated to T = 400 when their length exceeds this threshold. We report in Tab. 7 the average results across 122 univariate datasets. We report in Tab. 8 results for the multivariate datasets. Datasets interpolated to T=400, before being processed by GAK, are marked with the symbol (*).

C.5 Example kPCA visualization

Despite the underlying equivalence between NVAR and simple RC, better performance can be obtained by finding a better parameter setting, which is not trivial for RC. In our approach, a better setting is easier to find with fewer and more interpretable parameters. Here, we aim to visually highlight the possible extent of the differences between NVAR and rmESN, i.e., theoretically equivalent methods under different working points. To do so, we projected the data onto the first principal components using different kernel matrices (kPCA). When inspecting the projections for different datasets, we observed a few critical differences, predominantly for longer series. One example is illustrated in Fig. 5 ( CinCECGTorso dataset). These appear in the form of a qualitatively

NVARk*

NVARK

rmESN

CinCECGTorso

0.2

0.2

0.2

0.2

Q

0.6

0.4

2°0

I°0-

~0.2

'0-

1

better grouping and class separation in the projected space. Tracing back the origin of this improvement, e.g., a

better working point or a better resilience to noise, would represent interesting future work. For an alternative visualization, we display here the two-dimensional projection of the CinCECGTorso dataset using different kernels.

D Individual results for SVM classification of UTS

We report here the individual accuracies obtained under the SVM classification framework of univariate time series of Sec. 4.1. Each accuracy is obtained as a mean of 10 repetitions with 10 different random seeds. For readability, the standard deviation over the runs is not reported. Datasets interpolated to T=400, before being processed by GAK, are marked with the symbol (*).

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis focus of this paper on a new type of kernel for reservoir computing (using nonlinear vector autoregressive delay embeddings) that is faster and more accurate than related kernels. \n\nThe three main contributions\n* Introduction of a new kernel for time-series modeling\n* State of the art results on uni- and multimodal problems on real-world datasets (related work has used synthetic datasets.)  \n* Theoretical connections to Taken's theorem and the field of state space reconstruction. \n\nThere are only a small number of (intuitive) hyper parameters for tuning the model. While this approach does not seem suitable for large datasets (where deep learning-based solutions reign), it appears to be a nice and efficient solution for smaller time-series problems.\n\nSTRENGTHS:\nOverall the paper is well written. I don't have a deep understanding of reservoir computing but I think they authors do a nice job of connecting different technical areas. If I was working on problems in this area I would be strongly interested in building a deeper understanding of this paper. \n\nThe authors do a nice job of contextualizing the work, especially given that this is deviates from many recent deep learning oriented time-series papers. \n\nThe core idea behind the kernel is elegant and clearly has a large impact on efficiency of the approach.  \n\nI appreciate the discussions on asymptotic complexity and the qualitative notes and visualizations.\n\nWEAKNESSES:\nIt looks like the authors only use 122 UCR unimodal datasets but my understanding is that the most recent UCR benchmark (since 2018) has been using 128 unimodal datasets. Is there a good reason for this discrepancy? Similarly, the UEA corpus should have 30 datasets whereas only 22 are used in this paper. \n\nMinor: it's challenging to interpret the graphs (especially Fig 2) when printed in black and white.\n\nQUESTIONS:\nAt the end of section 2, there is discussion on the lack of stability of techniques for reservoir computing. Perhaps I'm being naive, but this seems obvious given that many of the parameters (e.g., W_in and A) are randomly initialized and not optimized over -- this means that parameters that are updated (e.g., W_out and c) need to compensate for potentially poor choices of W_in and A. I know this isn't something you introduced, but generally speaking why is this a reasonable idea?\n\nThe approach is described as \"a simple concatenation of the input series with time-delayed copies and nonlinear functionals.\" Could this approach simply be implemented using convolutions (with dilations)? Would there be a more straightforward way of writing out this math in this case?\n\nLIMITATIONS:\nThis paper is more theoretical and does not touch on noteworthy societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a kernel that can be applied to time series - both\nunivariate and multivariate - that draws inspiration from reservoir\ncomputing.  Practically, it constructs an expanded time series by\nsampling lags and then computing polynomial combinations of the\noriginal and lagged data through time.  Experiments show that the\nkernel, when combined with an SVM, is more accurate and more efficient\nthan a number of relevant competitors.\n\nSTRENGTHS:\nThis paper pulls on an interesting thread of work in reservoir\ncomputing (RC) that makes connections between RC and nonlinear vector\nautoregressive models (NVARs).  RC can be computationally expensive,\nand the NVAR approach simply combines time series with lagged copies\nand nonlinear functionals, like products.  The paper further connects\nthe use of lags to Takens' theorem about reproducing noise-free\nnonlinear dynamics through state space reconstruction using lagged\nobservations.  The discussion about the various connections to this\nwork is interesting.\n\nThe paper is clear and relatively easy to read and understand.  RC and\nNVAR are summarized in a useful way without getting bogged down in\ndetails, but with enough information to follow the presentation.  The\nfigures are also very helpful in understanding the paper.\n\nOne potential weakness of the approach is the relatively large number\nof hyperparameters, including the number and size of the lags, the\norder of the polynomial regression, and the embedding dimension.  The\npaper, though, does a good job presenting and arguing for heuristic\nchoices for these parameters that seem to perform well in practice.\n\nFinally, empirical results suggest that the proposed approach is as\naccurate as the state of the art and more efficient.\n\nWEAKNESSES:\nThe primary contribution of the paper is a kernel that is accurate and\nfast to compute.  There are claims in the introduction where the\nauthors suggest that the kernel is more interpretable than competitors\nthough there is no discussion of that in the remainder of the paper.\n\nThe approach is conceptually simple, which is good - fit a model to an\naugmented time series (lags and products) and use the model parameters\nas a vector-based representation of the time series.  The\nhyperparameter choices - of which there are many - are made\nheuristically.  Some indication of the sensitivity of the approach to\nvariation in those choices would be good.  That is, are the heuristics\na result of just those values working empirically; or is the method\nhighly sensitive to those choices?\n\nThe complexity analysis would benefit from doing a direct comparison.\nThat section claims that the proposed methods does one expensive\ncomputation (fitting a model) per, compared to other methods that do\nan expensive computation per pair of instances.  What is the total\ncomplexity of the approaches?  If competing approaches have less\nexpensive per-pair cost they may still be more efficient overall.\n\nIt's unclear what the kPCA plots add to the paper.  What dataset is\nrepresented in the figure?  Why did you choose that dataset?  I'm sure\nthere are other datasets for which the separation for rmESN is better\nthan for NVARk.\n\nAn ablation study in which the MTS were augmented with just lags or\nwith just products would help tease out which element is most\nimportant.  Takens says nothing about nonlinear features.\n\nQUESTIONS:\n(1) How sensitive is the approach to hyperparameter choice, both in\nterms of complexity and accuracy?\n\n(2) Are both lags and products important for accuracy?  Why or why not?\n\nLIMITATIONS:\nThere is no such discussion in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a new time series kernel based on Nonlinear Vector AutoRegressive (NVAR) processes, following recent literature on its equivalence to reservoir dynamics. The kernel operates on time delay embeddings and enables the computation of similarity between time series with different lengths. The proposal is paired with SVM models and evaluated in classification tasks for both univariate and multivariate cases. The obtained accuracy and speed results are promising when compared to other kernel methods and a reservoir computing alternative.\n\nSTRENGTHS:\nThe manuscript is well written and organized. The motivation is clear and the tackled problem (building kernels for time series data) is relevant to the community. The discussion of the related methods is comprehensive, although I have missed a nod to Gaussian process models (see 'Questions' section). \n\nThe main proposal seems to merge the contributions by Bianchi et al. (2020) and Bollt (2021), evaluating the NVAR equivalence to RC in the task of time series classification. The simple to understand heuristics that form the NVARk is a relevant strength of the work.\n\nThe main proposal, NVARk, is explained in detail and the provided source code that implements Algorithm 1 is easy to follow. I also praise the provided supplementary material and the qualitative result for KPCA visualization.\n\nWEAKNESSES:\nThe performed experiments are quite extensive, but the results presentation can be improved. For instance, Table 1 could also present the average rank of each method and be paired with boxplots of accuracy results across the 122 univariate datasets. Beside the per dataset results, Table 2 should also include the metrics considered in Table 1 (rank, average score), for the sake of homogeneity.\n\nQUESTIONS:\n- I believe a more thorough discussion on how Taken's theorem holds with random lagged dimension dropping (Eq. (5)) is required. Perhaps some of the content of the supplementary material could be brought to the main text.\n\n- In practice, is there any interplay between the choice of $k$ lagged steps and polynomial order $n$? In which scenarios n > 2 would be useful? \n\n- How the proposed method behave for larger dimensional datasets? It would be interesting to include the dataset dimension in Table 2.\n\n- Lines 315-316: \"NVARk can, in fact, get around both issues with its non-recursive structure and the absence of a training phase.\" -> I suppose the ridge regression step and the paired SVM are not considered here, which should be made clearer.\n\n- Since the tuned NVARk* usually presents even better results, it would be interesting to verify its trade-off by including it in the experiments of Section 4.3 (Execution time and scalability).\n\n- It would be valuable to comment about the use of the introduced NVARk in the context of Gaussian processes models. Maybe a brief discussion on how it would compare with other GP approaches for sequences [1,2,3,4] could be included.\n\nReferences\n\n[1] Frigola-Alcade R. and Rasmussen, C. Integrated pre-processing for Bayesian nonlinear system identification with Gaussian processes. IEEE CDC, 2013.\n\n[2] Mattos, C. et al. Recurrent Gaussian processes. ICLR, 2016.\n\n[3] Al-Shedivat, M. et al. Learning scalable deep kernels with recurrent structure. JMLR, 2017.\n\n[4] Toth, C. and Oberhauser H. Bayesian learning from sequential data using Gaussian processes with signature covariances. ICML, 2020.\n\nLIMITATIONS:\nThe manuscript states sufficiently its limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a feature extraction based on a non-linear vector autoregressive model (called NVAR kernel in the paper). The NVAR method constructs a deterministic feature matrix with the original input time series, lagged versions of the time series (parametrized by the lag and spacing parameters) and the products between lagged and unlagged dimensions, up to some polynomial order. The resulting feature map is then subsampled to obtain a random feature map. A linear readout layer performs next-step prediction on the embedding states and the resulting parameters form the feature map for a radial basis function kernel. The authors form a connection to state space reconstruction theory to justify their approach of subsampling by connecting it to state space reconstruction theory. The method is compared to other kernel choices on univariate (109) and multivariate (16) datasets for classification. The authors also demonstrate the scalability of their approach by measuring execution time and provide heuristics for hyperparameter choices.\n\nSTRENGTHS:\nOriginality\nThe underlying ideas in this paper have been proposed before in the time series space (lags and polynomial features; randomized feature maps), but the combination proposed here is novel for time series classification. The NVAR formalism in time series classification is novel as well (to the best of my knowledge). \n\nQuality\nThe algorithm choices presented in this paper are motivated by referring to previous work and connecting to state space reconstruction theory. While I’m not an expert on this, I appreciate the effort the authors made to justify the randomized subsampling step. The paper evaluates the proposed approach against other kernel choice baselines from different classes (reservoir, model-based, Fourier-based) univariate and multivariate datasets. The authors also analyze the execution time and scalability. I appreciate the efforts from the authors to provide heuristics for hyperparameter choices of their approach and showing the gain of additional cross-validation. I think the paper misses the comparison to a critical baseline, which I will discuss in the Weaknesses section. \n\nClarity\nOverall, the paper was well written but I think the clarity could be improved by introducing some core concepts into the introduction to readers that are not familiar with reservoir computing. For example, the authors state in the introduction that reservoir computing has a “large set of hyperparameters” but do to mention what they mean. I think a bit more detail to introduce the reader would be helpful (although I would consider this largely a stylistic choice as most of this detail comes later in the paper). \n\nSignificance\nTime series classification is an important application of time series research. Especially in the healthcare domain, time series classification methods have to run on devices with limited compute (smartphones or other edge devices). As such, a classification method that does only require minimal compute is significant. However, since this method is specific fairly specific to time series classification, the broader impact is limited.\n\nWEAKNESSES:\nI agree the rationale of the authors to select one representative baseline methods per category (reservoir, model based) etc. However, I think the the MINIROCKET method (Dempster et al. KDD 21: https://dl.acm.org/doi/10.1145/3447548.3467231) shares many properties of the proposed method (per time series computation is the most expensive step, suitable for downstream linear classification, low computational cost). Since this method provides high classification accuracy at low computational cost, I think it should be one of the baselines in the paper (it can also deal with uneven sequence length due to its final pooling operation). I would appreciate if the authors consider to compare against the method. At the very least, I would expect that this method is discussed in the related work (even though it is not from the reservoir computing literature).\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nLimitations are not discussed in the paper (or I missed it). A short discussion of limitations would be appreciated.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn their paper, the authors propose a new method for deriving a kernel from time series data. They combine ideas from reservoir computing with nonlinear vector autoregressive models, based on recent theoretical work exploring their similarities.\nThe primary idea of the paper is to construct a kernel by modeling a time series' dynamics using a nonlinear vector autoregressive model first. The vectorized fit parameters are then used to constitute the kernel representation of the respective time series.\nThe authors investigate the efficacy of this kernel in SVM classification across a range of unimodal and multimodal time series. They demonstrate state-of-the-art performance when compared to three other approaches.\n\nSTRENGTHS:\nOverall, the paper is well-written and easy to follow. The main ideas and contributions are clearly delineated. Further, code is provided with the submission. The method manages to significantly reduce the number of hyperparameters needed to model dynamics compared, for instance, to RC approaches. It also achieves state-of-the-art performance on a range of unimodal and multimodal time series classification tasks, compared to other kernel based methods. It presents an interesting contribution that as far as I can judge is novel, and leverages a recent development in a related field to come up with a new application.\n\nWEAKNESSES:\nComparison methods: If I understood correctly, no hyperparameters were tuned for the comparison methods in the experiment, while you optimized the delay-embedding parameters for your method. While I appreciate that a thorough hyperparameter tuning for some of the comparison methods can be computationally prohibitive and the smaller amount of hyperparameters is one of the strengths of your approach, it would make the comparisons fairer if some tuning took place, or if you explained in more detail why you didn't perform any tuning.\n\nLine 217: \"We propose the region 75 ≲ ¯dr ≲ 100 as a good, though non-optimal, setting\" The cutoff around 75-100 is justified experimentally in the appendix. The phrase \"non-optimal setting\" is repeated in the appendix. What do you mean by \"non-optimal\" in this context? Why would you choose a non-optimal setting? While I understand that this cutoff is motivated by decreased performance due to redundancy, it seems highly dependent on the dataset, and I'm not sure I understand why this reduces performance for the MTS so drastically. Since this is one of the central components/hyperparameters of your algorithm, some more clarification would be useful here.\n\nQUESTIONS:\nRelationship to Taken's Theorem: The choice of methodology is explained with reference to Taken's delay embedding. However, the connection to Taken's theorem in some of the practical cases isn't entirely clear to me. In the original next-generation RC approach, chaotic attractors are considered, where Taken's Theorem applies, while in your setting, many of the time series used in the classification tasks are not derived from something that can be as straightforwardly modeled via an underlying attractor (e.g., Japanese Vowels). It might be interesting to investigate further for which types of time series your approach is particularly suited and outperforms other methods. It's also worth exploring whether this relates to the suitability of a dynamical systems description for the underlying dynamics.\n\nl.56: The resulting model is more accurate, interpretable, and non-recurrent.\nWhat does interpretable refer to in this context? The single k-PCA example does not make this point strong enough in my mind, and it is not apparent to me how the derived parameters are more interpretable than e.g. the parameters of an RC model.\n\nSmall typos:\n226, implies\n236: only the ridge\n328: Computationally, it is exceptionally\n\nLIMITATIONS:\nThe authors explicitly mention that high-dimensional datasets are the main weakness of their approach. I don't foresee any direct negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-24"}