Randomized Signature Layers for Signal Extraction in Time Series Data
Authors: Enea Monzio Compagnoni
Luca Biggio
Antonio Orvieto
Thomas Hofmann
Josef Teichmann
Venue: ICLR 2022 Submitted (2022)
Keywords: signature
random features
time series
SDE
differential equations
rough path

Abstract: Time series analysis is a widespread task in Natural Sciences, Social Sciences, and Engineering. A fundamental problem is finding an expressive yet efficient-to-compute representation of the input time series to use as a starting point to perform arbitrary downstream tasks. 
In this paper, we build upon recent work using the signature of a path as a feature map and investigate a computationally efficient technique to approximate these features based on linear random projections. We present several theoretical results to justify our approach and empirically validate that our random projections can effectively retrieve the underlying signature of a path.
We show the surprising performance of the proposed random features on several tasks, including (1) mapping the controls of Stochastic Differential Equations to the corresponding solutions and (2) using the random signatures as time series representation for classification tasks. Besides providing a new tool to extract signatures and further validating the high level of expressiveness of such features, we believe our results provide interesting conceptual links between several existing research areas, suggesting new intriguing directions for future investigations.

Under review as a conference paper at ICLR 2022
RANDOMIZED SIGNATURE LAYERS FOR SIGNAL EXTRACTION IN TIME SERIES DATA
Anonymous authors Paper under double-blind review
ABSTRACT
Time series analysis is a widespread task in Natural Sciences, Social Sciences, and Engineering. A fundamental problem is finding an expressive yet efficient-to- compute representation of the input time series to use starting point to perform arbitrary downstream tasks. In this paper, we build upon recent works that use the Signature of a path feature map and investigate computationally efficient technique to approximate these features based on linear random projections. We present several theoretical results to justify our approach and empirically validate that our random projections can effectively retrieve the underlying Signature path. We show the surprising performance of the proposed random features several tasks, including (1) mapping the controls of stochastic differential equa- tions the corresponding solutions and (2) using the Randomized Signatures time series representation for classification tasks. When compared to correspond- ing truncated Signature approaches. our Randomizes Signatures are more compu- tationally efficient in high dimensions, and often lead to better accuracy and faster training. Besides providing new tool to extract Signatures and further validating the high level of expressiveness of such features, we believe our results provide interesting conceptual links between several existing research areas, suggesting new intriguing directions for future investigations.
INTRODUCTION
Modeling time series is a common task in finance, physics, and engineering. A frequent challenge is finding a transformation mapping a newly observed time series into a target one (seq2seq modeling) or into a label summarizing its salient properties (classification). In the absence of any principled model describing such mapping, one has to infer from data. The last few years have witnessed the rise of deep neural networks, which have found successful application to problems involving time series in numerous domains (Fawaz et al.) 2019 Gamboa 2017). Nevertheless, their outstanding performance comes the price of over- parametrization, data hungriness, and expensive training cost (Werbos 1990 Brown et al. 2020 Teubert et al. 2021 Neyshabur et al.] 2018 Marcus 2018). Furthermore, even if sufficient data is available, the resulting models learn representations. of the input data that are highly specialized to the training task and difficult to adapt in different contexts. In addition, the remarkable performance of these methods is often the result of substantial engineering effort and is not supported by theoretical results.
Reservoir Computing (RC) (Schrauwen al. 2007) offers an intriguing alternative strategy to cope with the limitations above, yet retaining the universal approximation properties typical of deep neu- ral networks. In RC, the learning is divided into two phases: first, data passed through an untrained reservoir which extracts set of fask-independent features; second, simple and efficient-to- linear map (the readout map) projects such features into the desired output. The critical point is that the design of the reservoir determines the expressiveness of the features, and several alternatives can be found in the literature (see (Gauthier et al. (021) and references therein).
A powerful reservoir is offered by the Signature Transform, often simply referred to as Signature, stemming from rough path theory (Ben Hambly) 2010; Friz and Hairer (020). The Signature of a path consists in enhancing the path with additional curves, which, in the smooth case, corresponds to iterated integrals of the curve with itself. A profound mathematical result (Levin et al.) (0013) guarantees that the solution of (rough) differential equation can be approximated arbitrarily well

---

Under review as a conference paper at ICLR 2022
Input time series
Randomized Signature Layer
Reservoir Features
Linear map
Downstream Tasks
System
Classification
Reconstruction
Figure I: Reservoir computing via random Signature layers, Given input time-series, approach be seen random Signature layer, acting as reservoir, extracting of random features preserving the strong theoretical properties the Signature of the input path. simple linear readout can applied the so extracted features to perfrom arbitrarily downstream tasks.
by a linear map of the Signature of the input signals (a.k.a. controls). In the machine learning jargon, feature map extracting all its geometrical properties
the Signature of a path can be interpreted and thus allowing simple linear map approximate any function of it. On the other hand, it is often the case that the reservoir features are very high dimensional, and hence are particularly expensive to calculate and use in downstream tasks. In addition, the high-dimensionality of the Signature reservoir poses additional challenges for modern gradient-based optimizers as convergence rates suffer from linear dependence in the model dimension (Bottou et al. 2018).
Inspired by the remarkable theoretical properties of the Signature reservoir and motivated to fix its practical pitfalls, (Cuchiero et al.] 2021bla), a recently introduced reservoir of random features. These features prov- ably hold the same geometric properties and approximation power the Signature, yet often more efficient to compute and of lower dimensionality. To extract Randomized Signatures, we nu- merically integrate set of random linear stochastic differential equations driven by the path itself. We assess the expressiveness of the proposed random features on several tasks, including non- parametric (black-box) system identification problems arising from complex controlled nonlinear dynamical systems as well as time series classification.
contribution is to showcase the effectiveness of Randomized Signatures
The paper structured follows: In Section 2, we link the Randomized Signature approach with related research areas in the literature; in Section 3, we present relevant theoretical results motivating our method; in Section 4, we assess the effectiveness Randomized Signatures on various system identification and time series classification tasks, showing promising results when compared to other signature approaches for reservoir computing. Finally, in Section 5, we discuss our conclusions and possible future developments.
RELATED WORKS
Random Features and Reservoir Computing. The idea of extracting features based on random operations is not new and has seen a number of successful applications over the past years. Of particular note, the seminal work Rahimi and Recht (2008) proposes to accelerate kernel machines by learning random features whose inner product matches that of a target shift-invariant kernel. The trade-off between generalization and computational efficiency of learning with random features has then been rigorously studied by Rudi and Rosasco (2017). A conceptually very similar rationale introduced by a parallel series of works exploring the topic of Reservoir Computing (Schrauwen ee al. 2007). Similarly to us, Echo State Networks (Jaeger] 2003) evolve the input state by series of fixed random projection (the reservoir) and generate the output by applying a trainable linear projection over the hidden states. However, we make the additional step of linking the random features to the Signature of the input path and, as shown below, the evolution of the features is dictated by randomly-evolved stochastic differential equation driven by the input path.
Controlled Differential Equations. Our work is also related with a series of recent papers inves- tigating the problems of how to process irregular time series and to condition model on incoming information through the lens of controlled differential equations (CDEs) (Kidger et al.] (2020] Mor- rill et al. 2021). Interestingly, Kidger et al. (2020) show that the action of linear layer the final
2

---

Under review as a conference paper at ICLR 2022
output of trained Neural CDEs the extension of Neural ODEs (Chen al. 2018) to CDEs results in a universal function approximator. Differently from them, our method is less computa- tionally expensive as the only parameters need to train are those of the final linear readout layer. Fundamentally different to our work, the approach of Morrill et (2020) first randomly projects the possibly high dimensional controls into lower dimensional space and then extracts the Truncated Signature from such compressed input. Our method instead extracts a random compression of the signature of the original input controls.
Rough Path Theory. Rough path theory is about describing how possibly highly oscillatory (rough) control path interact with nonlinear systems (Lyons] 2014). The concept of Signature is introduced in this context provide a powerful description of the input path, removing the need to look at the fine structure of the path itself. Recent years have seen a resurgence of these ideas, which have been revisited from machine learning perspective (Bonnier et al.. 2019 Kidger and Lyons] 2021. Our analysis is strongly influenced by the work of Cuchiero et al. (2021a b). who, starting from the observation that the Signature is an infinite dimensional reservoir, establishes that its in- formation content can be efficiently compressed by random projection performed by dynamical system driven by random vector fields. As we show later, this results in a random reservoir which preserves the properties of the Signature while living in finite dimensional space.
3
BACKGROUND
We provide the theoretical tools supporting the Randomized Signature approach for feature extrac- tion in times series. formal discussion can be found (Friz and Hairer 2020) and (Cuchiero) et al. 2021b).
3.1 RANDOMIZED SIGNATURE OF PATII
Let X: [0, - Rd be a continuous piecewise smooth d-dimensional path X = (x!,... ..)).. We will refer to X as the control and to its single components X; controls. We denote by (e,....ee)) the canonical basis of Rd.
Definition (Signature) For any t E [0, T), the Signature of a contimuous piecewise smooth path I (R2) @h where,
X J0, T + Rd on [0, is the countable collection S. :- (1, St, St....
for each k 2 1, the entries Sk are the iterated integrals defined as
E WOEEss-Sns dx ... dxis
St
ei 8 ... Csa.
(LL (I (IN
We define the Truncated Signature of X of order M 2 0 as
S.¹ = (1, S E
=: T. =M())
A practical example of computation of Signatures is presented in Appendix A.3
The definition in the last paragraph suggests that the Signatures of X can be used to approximate any regular enough function of X, for instance the solution to differential equations controlled by X. The following result makes this argument precise in the multidimensional setting.
Theorem (Signature a Reservoir) Let V; Rm + RTh ......dd d be vector fields regular enough s.t. dY, = Σi= V VY()) dxł;, Yo = y E RTD admits a unique solution Y, : [0, + R"" Then, for any smooth test function F RmTh + R and every M > 0 there is time-homogeneooss linear operator L TM (Rd) + IR which depends only on (V1...., Va. F, M, y) s.s.
F (Y)) = L (S + O (tM+1), t [0.
(3)
This theorem suggests the first M entries of the Signature of X are sufficient linearly explain the solution of any differential equation driven by it. In addition to this, it supports the claim that such features are valuable for any downstream usage.
3

---

Under review as a conference paper at ICLR 2022
Pirtt Siguatnee Sexed Siguetion Stpzatuue
Ranskuise: Siganture Randerooee Signatrm
Rasokninn Sigaskart
055
02-
3-
44-
-05
40-
002
0100
Mane
Figure 2: To enhance intuition, we show the Signatures and Randomized Signatures for X (smoothed Brow- nian Motion). Note that X, = S + Xg for each t, and that S and 5 get smaller and smaller magnitude, yet more and more regular (smoother, as a result of iterated integrals). The Randomized Signatures resemble a mix (i.e. random linear combination of) the true Signatures: Z' resembles a shifted version of -S', while Z resembles combination of S and S Indeed, Thm. 2 guarantees that Randomized Signatures approximate the full Signature with no need for computation of iterative integrals. We validate this experimentally in Sec.
3.2 RANDOMIZED SIGNATURE OF A PATH AND ITS THEORETICAL GUARANTEES
Calculating S.M requires the calculation of ا iterated integrals (Appendix A.3 - which in d-1
total quickly becomes computationally expensive. Several computational techniques have been de- veloped to circumvent this problem, see, e.g. (Kidger and Lyons| 2021). The next result provides a practical description of how it is possible to reduce the computational burden without losing too much explanatory power. As such, this results provides the theoretical foundation for our approach.
Theorem (Randomized Signature (Informal)) For any k E big enough and appropriately chosen random matrices A1,-..; Aa in R&xk, random shifts bi.-..,b ,b in Rkx1, random starting point z in IR&x1, and any fixed activation function o, the solution of
dz, (A.Z+
=== LE [0,
calledd the Randomized Signature of X - has comparable approximation power as the Signature itself and maintains its geometric properties.
The formal statement of such a result is the combination of Theorem g Theorem 9 and Definition
10 (see Appendix). A complete formal discussion as well as the original statement of this theorem can be found in the paper (Cuchiero al. 20) In nutshell, the Randomized Signature Z follows a dynamics which provides an efficient and powerful compression of the Signature through low-dimensional random projection. The expressiveness of such projection is guaranteed by argument (Cuchiero et al. 2021a).
standard
Computational complexity and dimensionality of Randomized Signatures. The computational complexity of calculating Z, is O d), while its dimensionality is O(k). However, we should then ask the crucial question "how should we compare k with M?". We show experimentally in Sec- tion 4 that, in order to match the approximation guarantees of the Truncated Signature of order M, the number required Randomized Signatures & is not too big - in particular it is not exponen- tial in M. This confirms that working with Randomized Signatures is often less computationally demanding and results in lover-dimensional features.
We conclude the section remarking the fascinating implications of choosing random A,s and b,s.
Theorem (Density of Randomized Signatures (Informal) For any sequences of time points O S to <... IN < T' and points 2 Zty E RR* we can find a smooth control X such that the solution Z of Equation at time L; is such that Ze, = 2t, for i == O...., N. Additionally, if the control X is a d-dimensional Brownian motion, then the solution of Equation at any point in time > 0 admits a smooth density with respect to Lebesgue measure.
4 THE EXPRESSIVE POWER OF RANDOMIZED SIGNATURES
To compute the Randomized Signature, we rely on an Mallr-Maruyama approximation; details are To start, experimentally validate some of the results presented above. In
given in Algorithm |
4

---

Under review as a conference paper at ICLR 2022
Algorithm 1 Generate Randomized Signature
Require: X E Rd sampled at 0 lo < < IN = T, Randomized Signature dimension k
activation function aT. Initialize: Zo E IR A E pelak by E R² to have iid standard normal entries
for in {1..N} do
Zt, = -: + Ei= 0 (A,2tm-1 bbi) (xi, X Xi.-.)
end for
particular, we show that the Truncated Signature of control lies in the linear span of its Randomized Signatures, thus validating Theorem 2 As a second proof of concept, we reconstruct a control as linear combination of its Randomized Signatures, showcasing the expressiveness of such random features. Finally, we combine l'heorem and Theorem 2 and we use Z to learn the dynamics of complex nonlinear systems of differential equations and to perform time series classification. Our result show that the Randomized Signature is a powerful random reservoir which guarantees both in theory and practice the same geometrical properties and approximation power as the Signature itself, while often being of lower dimensionality and less computationally expensive both extraction and at deployment time.
Note on the choice of a. While the choice of a does not affect the theoretical results (Cuchiero) et al. 20211aba b). we found that careful choice is needed for optimal results. Inspired by seminal works on the stability of deep linear networks (Glorot and Bengio 2010) and by the connection to neural ODEs (Chen al. 2018), we selected be linear, with slope dxvi' This choice, complemented with additional tnn-tuniig, leads to the best performance for downstream tasks.
4.1 LEARNING THE SIGNATURES
In this experiment, we show that we can approximate the Truncated Signature of a control us- ing linear combination of its Randomized Signatures. We an equally-spaced partition D (fo. ... AN of (0, 1] and calculate the truncated Signature of order M of a d-dimensional path X on D, i.. S.A. for = 0..... .... .N. For a fixed value of k, we use Algorithm to generate a realiza- tion of the Randomized Signature Z of X on D. For sake of clarity, the truncated Signatures on D are reshaped into an (N ((((M+1) 1) (dd 1) - 1)) -dimensional matrix called Strue while the Randomized Signatures D is represented an (N x dimeenniial matrix called Z ZRand
Finally, we train a Ridge Regression to find a parameter matrix W R (1x((((M=)-))/(d mapping the Randomized Signatures ZRRnn into the truncated Signatures Strue
More precisely, we consider a 10-dimensional control X = (t, W.) where W is 9-dimensional Brownian motion with independent components, and fix M = 6. For each k E {1,... 200), we generate 2 Z.Rand perform a simple linear regression and calculate the 1 approximation error. 3 indicate that the approximation error decreases as & increases. This confirms that, as ensured by Theorem the random features constructed prescribed by Equation 4 holdd the same information content as the Signature and thus maintain its approximation power and geometric properties. Finally, we highlight that, if we were to be satisfied with approximation error of order 10-4, we could achieve it with roughly with = 190 Randomized Signatures. To conclude, this means that, instead of calculating ((((M+1) - 1) /(d- - 1) = 1111110 integrals per time step, we could just perform 1.2d = 36100 calculations per time step - which is 3 times cheaper. More remarkably, this high feature dimensionality (1111110 for Signature) makes any model for downstream tasks exponentially more complex than the that based on Randomized Signature (only 190 features). Modeling choices and details are reported in Appendix B.2
Results in Figure
4.2 RECONSTRUCTION AND RANDOMIZED SIGNATURE AUTOENCODERS
The Randomized Signature X is rich enough to reconstruct it almost perfectly. In practice, we consider an "autoencoder" architecture, whose encoder is random feature generator and the decoder is a trained linear projection mapping the Randomized Signatures into the original path. For this experiment, we fix an equally-spaced partition D (lo, ,(N) and consider single
5

---

Under review as a conference paper at ICLR 2022
log Entrr
Sumple
2.2
1.5
Z
Estraciad Traek
0.4
306
0.4
Figure 3: Randomized Signature approx- Figure 4: Performance of the Randomized Signature autoencoder.
imates the Signature as & increases.
realization XTrain of an Omsteci-UUllenbeck process. Ultimatel,, we train Ridge Regressor to map the Randomized Signature Z of the one and only train sample XTrain into XTrein itself. Table in Appendix shows the average reconstruction error on Nrest = 10000 test samples as we repeat the experiment for different choices of k. Such results are accordance with Figure 4 which shows an example of the true and generated time series of a test sample. This confirms that the Randomized Signature of path holds an equivalent information content as the path itself while being more usable for any downstream tasks. Modeling choices and details are reported in Appendix B.2
4.3 LEARNING SDE SOLUTIONS WITH A LINEAR MAP ON THE R RANDOMIZED SIGNATURE
In this series of experiments, we provide evidence of the potential of Randomized Signatures in several downstream tasks. First of all, shown in Appendix B prove that our approach can accurately learn the dynamics of several rough differential equations: we highlight that the algo- rithm has only access to the input controls and the output response of the system but it is completely agnostic with respect to the law of the dynamics itself. To start with, we focus on the nonlinear dynamics of 1-dimensional stochastic Double Well. Secondly, we study the multidimensional case of the 4-dimensional Ornstei-UUllebeec. Then, we investigate the performance on the extremely rough dynamics of Fractional Geometric Brownian Motion even for low values of the Hurst param- eter H. Additionally, we obtain promising results in learning the dynamics of an Enzyme-Suber Reactions (Ingalls) 2013). We also demonstrate that this approach has sustainable performance even when the input control is irregularly sampled as well as Neutral Controlled Differential Equations. Due to space constraints, these results are presented in the Appendix. Here, we show the perfor- mance in learning highly nonlinear dynamical systems with real world application.
Tumor Growth Kinetics. In this experiment, we consider predicting the output of a complex controlled nonlinear system using Randomized Signatures. The Simeoni Tumor model (Simeoni et al. 2004) is widely used to predict the evolution of tumor weight Y, under the concentration of a treatment drug Xt. The system is specified by the following equations
dut { Aou? [1 + Ao//Aww))
w -1/4 "
1: Xey di.
duf = [k2 k] di,
(5)
duž = Kn (u] w²) dt, duf = [k1 (u² - -2)) dt, Y, = ut + uz + up + +u².
We fix an equally spaced partition D of [0, 10) and perform Ridge Regression to map 10000 instances of the Randomized Signature of the controls into the respective integrated solution Y, We sample X, from the law of Scaled Squared Brownian motion 0.25 W? where W, is 1-dimensional Brownian Motion (BM) and the left side of Figure shows the comparison of the true and the generated time series on test sample along with the respective path of the control. To test the robustness of our approach, we use controls that do not follow the law used in training. More precisely, the middle of Fig. shows what happens if we use [[W?>1)
input control instead of 0.25 W? while its right does the same as we use W,. These two experiments prove that the algorithm learnt much more than merely predicting the behaviour of the system under ordinary circumstances. The first shows that the algorithm has "understood" that the absence of drug leads to the growth of the tumor while it dies back once the medicine is injected. On the other hand, the second suggests that the algorithm cannot behave properly when the input becomes negative not allowed by the problem definition (negative drug concentration). Further details are reported in Appendix B.2
Electrochemical Battery Model. In this experiment, we learn the dynamics of the electrochemi- cal battery model proposed in (Daigle and Kulkamn 2013) which returns the voltage Y as the current

---

Under review as a conference paper at ICLR 2022
Out of Samale
Our of Sample
DuCO! Semole
Extracted
Extracted True
Extrededd True
Ny
30
20
2
10
Figure 5: Tumor Growth Kinetics stimulated with Scaled Squared BM (left) Step Function (center) BM (right). Model is only trained on scaled squared BMs, so the input in the central plot is out of distribution.
2.10-
Tru
Extrected
2.75
2.08
2.55
2.00
2.06
0.75
NNARX1000
2.00
2.00
0.00
0.5
300
3000
302
400
503
100
300
300
402
500
100
200
300
400
500
Figure 6: Electrochemical Battery Model: Example of fit for a Train Sample (left) Preeditionsss on Test Sample for different & (left) Comparison with NNARX (right)
X drawn from the battery. We use the open source NASA Prognostic Model Package (Teubert) eet al. (02T) to simulate voltage trajectories given input current control paths. On fixed equally spaced partition ID of [0, 500). we model the input current with step functions taking values on random sub-interval of [0, 500]. To test the robustness of our approach, we train Ridge Regres- sion to map NTrain = 1000 instances of d-dimensional Randomized Signature of the controls into the respective solutions to which we add white noise. The left of Figure 6 shows the comparison between the ground truth and our prediction on an in-sample trajectory. In the same figure, we com- pare the generated time series for several choices of & on test sample along with the path of the input control. In Figure 6 we show the performance of our method in comparison with a nonlinear neural ARX (NNARX)
which is often used in the modern control theory/system identification literature as a benchmarch (see c.g. (Masti and Bemporad] 2018)), given its strong theoretical guar- antees (Schoukens and Ljung 2019). Table in Appendix reports the performances in terms of MSE on 1000 test samples for our models compared to two versions of NÑARX: NNARX 22 which has around 1000 trainable parameters just like our model with h: 1000, and NNARX 1000 which matches our best model in terms of MSE but has around 106 trainable parameters. Details are reported in Appendix B.2
Comparison with Neural Controlled Differential Equations. In Appendix B, we benchmark the performance of our model with that of NCDEs (Kidger et al. 2202). a recently proposed competitive deep learning technique for controlled dynamical system modelling. We test the two models on the task of learning the dynamics of an Ornstein- -uastinn-hhlenbcck Process, and showcase the superior performance of our method both in terms of computational complexity and accuracy.
4.4 COMPARISON WITH TRUNCATED SIGNATURES
In this experiment, we learn the dynamics of the Fractional Ornstein-Uhlenbeck process using our Randomized Signature model and benchmark our results against Truncated Signature model of order M. The computational cost of extracting the Truncated Signature of order M = 3 for d controls is (dd)) while for Randomized Signature of extracting features equal across models, we select k d. As result, the number of features of the Signature-based model is (dd) while that our model is Odd. the spirit of Reservoir Computing, we choose both models to be linear. Hence, the number of trainable parameters is equal to the number of features. This implies that, the number of trainable parameters of our model is orders of magnitude smaller than that of our benchmark (left of Figure 7 this severely affects the training time (see e.g. (Bottou et al. 2018). Figure 7 shows that our training time is almost independent d. Instead, the training complexity of the model is exponential in d and
dk:2d). To keep the computational cost
7

---

Under review as a conference paper at ICLR 2022
the Out-of-Samppi performance degenerates as the underlying optimization problem explodes in dimension. This result clearly highlights the data-hung hungrinesss of Signature- based models when the number of dimensions is high. To conclude, the Truncated Signature of a path is 200MB while its Randomized Signature is just 32KB. As a consequence, training the former on NTrain 10000 would take an unpractical amount of time and storage memory while we were able to train ours in 4.8896 seconds using just 320MB and attaining an Out-of-Sample Error on NTest 10000 equal 0.075329. Modeling choices and details are reported in Appendix B.2
emphasize that if d 80, the storage cost of
10"
Sysaem Higrssinn tkg stguabec (Nows
10
112
COOOO 100 aa
22-
Spretare Signsture
224
Spsttaceee =2t)
<Bh face Spetar -00
30P
20 40
20
40
60
20
40 50
50
Figure 7: Randomized Signature vs. Truncated Signature model. (Left) Number of trainable parameters for Randomized Signature is significantly smaller regardless of the number of controls. (Center) Truncated Signature is much slower than Randomized Signature in high dimensions. (Right) As opposed to Randomized Signature, the performance of Truncated Signature degrades as the number of controls increases.
4.5 TIME SERIES CLASSIFICATION
In time series classification (TSC) the goal is to map an input time series to the corresponding label. Given the complex high-dimenn nigh-dimensional nature of temporal signals, the vast majority of classification algorithms aims at effectively finding expressive representations of the original time series and then maps them into the final prediction. This last operation is often realized via a relatively simple map- ping, e.g. a linear map various types of deep convolutional neural networks which extract a hierarchical representation of the input time series by learning the parameters of their filters via backpropgation. These features are finally mapped into the logits associated with each class via a trainable linear projection. Last but not least, it is key to remark that these networks are typically over pverparramerrized and their training can be computationally demanding as well as challenging from an engineering perspective.
shallow neural network. Deep learning algorithms used for TSC include
In this section, we investigate how we can actually replace deep neural networks used for feature extraction with relatively cheap random layers which iteratively generate reservoir. Among other notable works, Rocket (Dempster et al. 2020) arguably proposes the most successful solution.
Rocket. This algorithm prescribes the generation of k >> unidimensional filters characterized by randomly drawn lengths, weights, and jumps. Given filter of length lyster. weights w E R'smeerr and jump j. it can be easily shown that the feature Z' resulting from its action on an input path X, from position in X, evolves according the following iterative update:
z = 2 + X ((x()) X Wk, k E {0, ,fftterttr------
(6)
where Zi = and 6 is a random real number. We define Z to be equal to -ip Depending on the values of and lfttter and on the length lx of the path X, this update produces vector Z = = MM of M == lx - (1 - 1) random features, used to extract only two nonlinear features: mar and is ΣM [[2:00)- Ultimately, this means that if we use k kernels, we extract 2k features per time series which are then used as input to a Ridge classifier to obtain the final prediction. Rocket results in state-of-the-art performances in TSC both in terms of accuracy and computational efficiency, outperforming several deep learning baselines which often involve order of magnitude more parameters to optimize.
An alternative interpretation. Looking at Eq. 6 it is natural to wonder if it can be replaced with an alternative and possibly more principled update scheme driven by the input path and evolved by random projections. With the intent of aligning Rocket to the framework presented in this paper, we replace Eq. [ with the update equation in Algorithm I thus exploiting the expressive power of Randomized Signatures. As detailed in the Appendix B.2 the current implementation of this
8

---

Under review as a conference paper at ICLR 2022
Accuracy
Bonel
recket
modenn
25221
rocket oJurs
twles szaze
32602
resne!
(1111 615ha
lzn
mlp
encodar
Figure 8: Mean rank of the modified version of Rocket described in the text and number of deep learning baselines (results taken from (Fawaz al.] 2019)) 128 datasets from the UCR archive. Our method referred as "rocket ours". The other methods are, from left to right: Time I.--Net (Le Guennee et al. 2016), Multi Channel Deep Convolutional Neural Network (Zheng et al. 2014), Time Warping Invariant Echo State
Network (Tanisaro and Heidemann 2016). Time-CNN (Zhao al. 2017). Multi Layer Perceptron (Wang et al.
2017). Encoder (Serrà al. 2018) Fully Convolutional Neural Network Wang 2017
Residual Network
Waag et al. 2017).
algorithm is equivalent to assuming that the random vector field A is approximately diagonal and autonomous. In other words, the Randomized Signatures are never mixed during their evolution and, differently from the update in Eq. 6 the filter weights do not depend on time. These represent two strong limitations which pave the way to several enhancements in future work. Despite these drastic shortcomings, the resulting algorithm outperforms several deep learning baselines 128 TSC datasets from the UCR archive. as shown in Fig. g The original algorithm using Eq. 6 provides better results than ours but does not enjoy the same theoretical guarantees based on the theory of Randomized Signatures presented in Section 3.
5 CONCLUSIONS
Reservoir Computing and the Concept of Signature. The concept of reservoir computing is based simple and intriguing idea: generating a large number of features and then training linear readout to perform an arbitrary downstream task. This simple approach promises to reduce the number of model parameters yet retaining high level of expressive power. Inspired by results from rough path theory showing that the Signature Transform enjoys the status of reservoir, we propose a random reservoir, i.e. the Randomized Signature, approximate the Signature Transform and thus inheriting its geometrical properties and expressive power. Following rigorous theoretical arguments, we do so by evolving latent differential equation controlled by a path and evolved according to random vector fields. The empirical investigations presented in this paper provide evidence supporting that this strategy is successful and can be deployed on several tasks including non-parametric system identification and TSC.
Future research and open questions. From a theoretical standpoint, Eq. 4 is written in terms of autonomous vector fields A, for i 1..... d, whose entries are drawn according to Gaussian distributions with zero mean and unit variance, and an activation function a. It is natural to wonder if choosing time- dependentt matrices A, leads increased expressive power and what extent the quality of the resulting features depend on particular choice of a. Furthermore, since our exper- iments suggest that the dimensionality of the random features play a significant role in terms of expressive power, it is not trivial to identify rigorous criterion to determine it. A precise character- ization of this quantity and a clever way to select its value could potentially lead to further savings in terms of computational cost.
From a practical point of view, our method proved to be able to accurately predict the response of dif- ferent dynamical systems driven by heterogeneous and possibly irregular control paths, even when such controls were sampled from different laws with respect to the one used in training. In light of these results, it would be interesting to explore whether the method of Randomized Signatures could be extended to model the response of systems governed by controlled partial differential equations. In the context of TSC, we established link with existing works using time series representations based on random features and our approach based on the update in Eq. 4 We implemented a first baseline algorithm combining our Randomized Signatures with the algorithm proposed in (Demp- ster et al., 2020), resulting in superior performance compared to strong deep learning baselines but sub-optimal results compared to the original algorithm. We believe such a gap can be filled by ad- dressing the limitation of the current version of the algorithm, e.g. the assumptions it makes on the vector fields, as discussed at the end of Section 5. As a final consideration, we believe an inter- esting research direction being investigate whether be incorporated into hierarchical models, such as deep neural networks, to form hybrid methods including both random and learned components operating in symbiosis.
random feature extraction pipeline could

---

Under review as a conference paper at ICLR 2022
REFERENCES
T. L. Ben Hambly. Uniqueness for the signature of path of bounded variation and the reduced path group. Annals of Mathematics, 171:109-167, 2010.
F. Biagini, Y. Hu, B. Øksendal, and T. Zhang. Stochastic Calculus for Fractional Brownian Motion and Applications. Springer, 2008.
P. Bonnier, P. Kidger, P. Arribas, C. and T. Lyons. Deep signature transforms. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pages 3 3105- 3115, 2019.
L. Bottou, F. E. Curtis, and J. Nocedal. Optimization methods for large-scale machine learning. Siam Review, 60(2):223-311 211,111 2018.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam,
G. Sastry, A. Askell, Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few- shot learners In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran As- sociates, Inc., 2020.
R. T. Chen, Rubanova, J. Bettencourt, and D. Duvenaud. Neural ordinary differential equations. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 6572-6583, 2018.
C. Cuchiero, Gonon, L. Grigoryeva, J.-P. Ortega, and Teichmann. Expressive power of ran- domized signature. In The Symbiosis of Deep Learning and Differential Equations, 2021a.
C. Cuchiero, L. Gonon, L. Grigoryeva, J.-P. Ortega, and J. Teichmann. Discrete-time signatures and randomness in reservoir computing. IEEE Transactions on Neural Networks and Learning Systems, 2021b.
M. Daigle and C. S. Kulkarni. Electrochemistry based battery modeling for prognostics. In Annual Conference of the PIIM Society, volume 5, 2013.
H. A. Dau, Bagnall, K. Kamgar, C.-C. M. Yeh, Y. Zhu, S. Gharghabi, C. Ratanamahatana, and E. Keogh. The uer time series archive. IEEE/CAA Journal of Automatica Sinica, 2019.
A. Dempster, F. Petitjean, and G. I. Webb. Rocket: exceptionally fast and accurate time series classification using random convolutional kernels. Data Mining and Knowledge Discovery, 34 1554 Jul 2020. ISSN 1573-756X.
doi: 10.
H. I. Fawaz, G. Forestier, J. Weber, L. Idoumghar, and P.-A. Muller. Deep learning for time series classification: review. Data mining and knowledge discovery, 33(4):917-963, 2019.
K. Friz and M. Hairer. A course on rough paths. Springer, 2020.
J. C. B. Gamboa. Deep learning for time-series analysis. arXiv preprint arXiv: 2017.
D. J. Gauthier, E. Bollt, A. Griffith, and W. A. S. Barbosa. Next generation reservoir computing. Nature Communications, 12(1), Sep 2021. ISSN 2041-1723. doi: 10. 10.10388441477-21--58811-
X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256. JMLR Workshop and Conference Proceedings, 2010.
B. P. Ingalls. Mathematical modeling in systems biology: an introduction. MIT press, 2013.
H. Jaeger. Adaptive nonlinear system identification with echo state networks. NIPS, 06 2003.
10

---

Under review as a conference paper at ICLR 2022
P. Kidger and T. Lyons. Signatory: differentiable computations of the signature and logsignature transforms, on both CPU and GPU. In International Conference on Learning Representations, 2021.
Kidger, J. Morrill, J. Foster, and T. Lyons. Neural controlled differential equations for irregular time series, 2020.
A. Le Guennec, Malinowski, and R. Tavenard. Data augmentation for time series classification using convolutional neural networks. In ECML/PKDD workshop on advanced analytics and learning on temporal data, 2016.
D. Levin, T. Lyons, and H. Ni. Learning from the past, predicting the statistics for the future, learning an evolving system. arXiv preprint arXiv:1309.0266, 2013.
Lyons. Rough paths, signatures and the modelling of functions on streams. arXiv preprint 1eXiv:1405.455 2014.
G. Marcus. Deep learning: A critical appraisal. arXiv preprint arXiv:1801.00633 2018
D. Masti and A. Bemporad. Learning nonlinear state-space models using deep autoencoders. In 2018 IEEE Conference on Decision and Control (CDC), pages 3862-3861 IEEE, 2018.
J. Morrill, A. Fermanian, P. Kidger, and T. Lyons. A generalised signature method for multivariate time series feature extraction. arXiv preprint arXiv:2006.00877 2020.
Morrill, P. Kidger, L. Yang, and T. Lyons. Neural controlled differential equations for online prediction tasks. arXiv preprint 1r6.v:2106./1000 2021.
B. Neyshabur, 7. Li, S. Bhojanapalli, Y. LeCun, and N. Srebro. The role of para-aaraattrizzatoon in generalization of neural networks. In International Conference Learning Representations, 2018.
A. Rahimi and B. Recht. Random features for large-scale kernel machines. In J. Platt, D. Koller, Y. Singer, and Roweis, editors, Advances in Neural Information Processing Systems, vol- ume 20. Curran Associates, Inc., 2008.
A. Rudi and L. Rosasco. Generalization properties of learning with random features. In NIPS, pages 3215-3225, 2017.
Schoukens and I. Ljung. Nonlinear system identification: A user-oriented road map. IEEE Control Systems Magazine, 39(6)28-99, 2019.
B. Schrauwen, D. Verstracten, and J. Van Campenhout. An overview of reservoir computing: theory, applications and implementations. In Proceedings of the 15th european symposium on artificial neural networks. p. 47/-482 2007, pages 471-482, 2007.
Serrà, Pascual, and A. Karatzoglou. Towards a universal neural network encoder for time series. In CCIA, pages 120-129. , 2018.
M. Simeoni, P. Magni, C. Cammia, De Nicolao, Croci, E. Pesenti, M. Germani, Poggesi, and M. Rocchetti. Predictive pharmacokinetic-poe phwpokiieettt-paaaannodnnnnnnni modeling of tumor growth kinetics in xenograft models after administration of anticancer agents. Cancer research, 64(3):1094-1100) 2004.
Tanisaro and G. Heidemann. Time series classification using time warping invariant echo state networks. In 2016 15th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 831-836, 2016. doi: 10.1 2.111////CII...011.00011
T. L. Terry J. Lyons, Michael Caruana. Differential Equations Driven by Rough Paths. Springer, 2004.
C. Teubert, M. Corbetta, C. Kulkamn,, and M. Daigle. Prognostics models python package, Aug. 2021.
11

---

Under review as a conference paper at ICLR 2022
Z. Wang. W. Yan, and T. Oates. Time series classification from scratch with deep neural networks: A strong baseline. In 2017 International joint conference on neural networks (IJCNN), pages 1578-1585. IEEE, 2017.
Werbos. Backpropagation through time: what does and how to do it. Proceedings of the IEEE, 18(10):1500-1560 1990. doi: 10. 109//5977383.
B. Zhao, H. Lu, S. Chen, J. Liu, and D. Wu. Convolutional neural networks for time series classification. Journal of Systems Engineering and Electronics, 28(1):162-169, 2017. doi: 10.21699/SEE2221.00111.
Y. Zheng, Q. Liu, E. Chen, Y. Ge, and J. L. Zhao. Time series classification using multi-channels deep convolutional neural networks. In WA/M, 2014.
12

---

Under review as a conference paper at ICLR 2022
Appendix
A THEORY DETAILS
A.1 BASIC DEFINITIONS
First of all, we define the concept of admissible tensor norms, which we assume to have in this work.
Definition 2 (Admissible Tensor Norms) Let E == Rd and @ be tensor product such that the tensor powers of E, (ESk, 1), are equipped with a family (ll |l k 2 1) of norms satis- fying:
1. For j. k E I and all h E E and I Esk
|| 8 11 I ||
2. For any permutation of (1....,k).
||41 ... @ Ea rar (1(11) .... lock)ll
3. For any bounded linear functionals f on E and on E there exists a unique bounded linear functional, denoted as f @ g, E such that for all h V and I E
2 ( & l) f(b)g(1).
family of tensor norms satisfying these conditions is called a family of admissible tensor norms.
Then we define the space in which the Signature of path lies in, that the following Tensor Algebra.
Definition 3 (Tensor Algebra T (R4)) We define the tensor algebra on Rd as
T TRR)=
II (RR4) k=
Ok
as well as its truncated version of order M 2 aa
M
TM (R*) = II
@k (R
k=
where (R1)3t 8k is the space of tensors of shape (d. (d....,d) d) given by Rd @ ... Q Rd for l times.
Definition (Concatenation Operation) We define the Concatenation Operation * such that for any given couple of continuous piecewise smooth paths X [0, Idd and Y [s, Rd. their image through & is the continuous piecewise smooth path X Y [0,t] + IRd defined by
X X.+YY.Y.
if E [0,s] if |s,t
X X+ =
Definition (Inverse Operation) We define the Inverse Operation + such that for any continuous piecewise smooth path X Rd, its image through is the continuous piecewise smooth path X. = X each E [].
Definition (Topological Space) topological space an ordered pair (X, T), where X is a set and is collection of subsets of X, satisfying the following axioms:
1. The empty set and X itself belong T;
13

---

Under review as a conference paper at ICLR 2022
2. Any arbitrary (finite infinite) union of members of still belongs T;
3. The intersection of any finite number of members of still belongs
The elements of are called open sets and the collection is called topology X.
Definition 7 (Arewise Connected Topological Space) A topological space (X,T) is said be cwise connected if any two distinct points z, y E X can be joined by an arc, that is contimuous map [0, 1] X such that a0) and (1) y.
We say that X is uniquely arewise connected if for any two distinct points I, E X, there exists unique path in X that joins them.
Definition 8 (R-tree) An R-tree is uniquely arcwise connected metric space, in which the between two points is isometric to an interval.
Definition (iree-Like) contimuous piecewise smooth path x : =) + Rd is tree-like if there
exists an R-Iree T, continuous map Φ [0, and a map to V such that @(0) (TT) and x = v/o Φ.
A.2 SUPPORTING RESULTS
Lemma (Johnson-Lindenstraus Lemma, (Cuchiero et al.) 2021a)) Given an M M-dimensional
Hilbert space (H, ()) and Q a N -ppoint subset of H, for any < < 1, for each k. E IN
satisfying the so called Bohnson-LLndenstrann constraint
l4 N
k 2
322 2c
there exists a linear map flQ.M.K) И - Rk that embeds Q into Rk in an almost isometric manner. More specifically, we have that
(10MM) (a1) f(Q.M.)) (az) s (1+ [|a] -a2l
(1- [|a] a2ll
for each a,, a2 E Q.
Remark When there is no need to specify the dependence of f(Q.M.N) from Q, M, or k. we will omit them up to referring to is simply as J. Finally, we define 1 IR - H to be the adjoint map of I with respect to fixed inner product (,,)) in R*.
To apply such Lemma in our context, we select M 0, equip TM (R*) with an inner product such that
(ci, @ & Cise Cj @ Cju) == 8: 8imju:
where {ci, .... ....@ar Cias } ii... E ad) is the canonical basis of T (Rd). Therefore, have that MI (Rd),...) is an Hilbert space.
Theorem 5 (Existence and Uniqueness of the Signature, Lemma 2.10 (Terry J. Lyons, 2004) The following controlled differential equation
dS: -s S S. =1
has unique solution which, at each t [0, T) the Signature S. of X on [0, t).
Notation 1 When there is no ambiguity about X, we will refer to its Signature as S, while we use SX is more paths are involved.
Theorem 6 (Theorem 2.29, (Terry J. Lyons 2004)) Given continuous piecewise smooth path X [0, T) Rd, its Signature the banal Signature = (1,0,0,..) if and only if X is tree-like. In particular 8X=X 1
14

---

Under review as a conference paper at ICLR 2022
Now, we provide results to show the relevance of these features. First of all, the following theorem ensures that the Signature of path encodes its essence and characterizes it completely.
According to Theorem 6 the concatenation X * X of X with its inverse has the same Signature as the constant path, but cannot be reparametrised to be constant. Similarly, if X, Y, Z are non-
constant paths, then X..Y.Y=2.Z.. = 1, but X Y Y Z * Z X is not path of the form ? * by for any path 7. The formal definition of tree-like path is given in Definition 10
Theorem (Characterizing Nature the Signature, Corollary 1.4 (Ben Hambly, 2010
Given a couple of continuous piecewise smooth paths X and X, then S S if and only if
X * X is tree-like.
This result is actually much stronger as it implies that the solution of any differential equation con- trolled by X fully determined by the vector fields and S. In particular, Theorem shows that the solution of any differential equation controlled by X is essentially linear in S.
Adapting Theorem III.8. (Cuchiero et al., 2021b) as done in (Cuchiero
2021a) one obtains:
Theorem (Randomized Signature) For any fixed integer M Z O, any fixed partition (to. ... (N ,tt' of t such that 0 S. to < ... < tN < T, let us consider the Truncated Signa- ture of order M of X at such times, that is Q == {S ..xx such that its elements all lie 1, k N satisfying the associated Johnson-
}
in (TM (Ba). (..))) Let us now select O < Lindenstrauss constraint, let f be the implied Johnson-Linde Lindenstrauss map and f adjoint map. Then, the solution of the controlled differential equation in R
dZ, = Σ [ (Z) Zo = f(1) 1=1
on D. that is (Zto.... Zty) }, are called the Randomized Signature of X D. It holds that each Z, is the projection of S. from (TM (R*), ((.-)) onto R$ via f. To conclude, since f (Zr,) is close to SM in the norm of T (R*). we conclude that preserves the geometric properties and the approximation power of S.
Adapting Theorem III.7 in (Cuchiero et al. 2021b) can be be shown that (asymptotically) the JL. projected vector fields stem from random matrices:
Theorem (Z is random dynamical system) For l - oo, for each i E {1,..., d} the lin- vector fields f(0 (f())))))) R* + R* square matrices with asymptotically normally distributed, independent entries.
Leveraging this results, we define the following:
Definition 10 (Localized Randomized Signature) For any random matrices Ai...., ., Aa in Rkxk and shifts bi...., ba in R such that maximal interrbbgyyyy holds on random starting point z E Rk, any fixed activation function o, and d-dimensional control X, the solution of
d7, (A:Z b.) Zo 2, te 10,
called the Localized Randomized Signature of X and we often refer to it as Randomized Signature.
Theorem (Signature is a Reservoir (Restatement of Theorem 1)) Let V, 1mTh + Rm i V (Y;) dx; 2. Yo = y E Rm admits
1,... d be vector fields regular enough such that dY, = =d unique solution Y, : [0, T) R Then, for any smooth test function F : RR R and for every
M 0 there is time-homogeneous linear operator L TM (Rd) R which depends only
(V .V,,.., Va, T,M, u) such that F (Y;) L (SM) + O (1M+1), and [).
X.:= Tx=
15

---

Under review as a conference paper at ICLR 2022
Proof: We provide just a sketch. For every smooth test function F : Rm - R. the formal Taylor
expansion along controls together with the fundamental theorem of calculus gives us that
(X) = F())) + Σ I. (Y. $2 O.
d
(8)
1=0
For y E R" this equation can be inserted into itself leading to a generalized version of Taylor expansion for controlled ordinary differential equations
Σ
E
Vi...Vy.P
Josus
dxtt ... ...x + (F,t)))
(e)....))
for M 2 O, with the remainder satisfying
fo Ststis..stunss
(F,tt =
E
Vi... ViM- F(r,) dx.; ... dd
(1 (E (f.....)))
A.3 PRACTICAL EXAMPLE THE COMPUTATION OF SIGNATURES (ONE DIMENSION).
To enhance intuition, exemplify the computation of Signatures in the simplest setting possible. Let X [0, T] R, then S fo dXx, which is exactly X, Xo. To get S.. instead have to compute the following iterated integral: S S (So dx,,) dX Signatures of higher order S/ are computed in similar way, by iteratively integrating the path j times. As a practical example, let X, = t. Then it is easy to see that S f. Now let (t) be an analytic function of time for which we have Y(t) jino y z
JT. Taylor's theorem combined with the previous computation of the Signatures implies that Y can be approximated as linear map of the Signatures of t. Finally, note that S in this case gets smaller and smaller in magnitude increases. This suggests that the truncated Signature can be safely used to approximate
Computational Complexity and dimensionality of Signatures. The computational complex- ity for computing S. is (ddM). Indeed, consider d = 2: St is a 2 x 2 matrix with ele- ments S (Jo dx.)) dx!. Jó (So dx.)) dx? f (Jo" dx?) dX, and (Jr dx?) dx?- For M 3. the object to compute is instead 2 x 2 2 tensor, containing all integrals of the type Se (f." (fe" dx;) dxx2) / for all i, i2, i3 E (1,2). Hence, the complexity as well as the features dimensionality. scales exponentially in M.
16

---

Under review as a conference paper at ICLR 2022
B EXPERIMENTS
In this section, we prove that our approach can accurately learn the dynamics of several rough dif- ferential equations. First of all, focus on dynamics of 1- 1-dimensional stochastic Double Well which is characterized by highly nonlinear dynamics. Secondly, we study the multidimensional case of the 4-dimensional Omsttein Omnstin-Uhlenbbed Uhlenbeck. Then, we investigate the performance on the rough
dynamics of Fractional Geometric Brownian Motion even for choices of extremely low values of the hurst parameter H. We also obtain promising results in learning the dynamics of an Enzyme- Substrate Reactions. Next, we show how in high dimensions randomized signatures are computa- tionally efficient, and also present one experiment in comparison with neural controlled differential equations (Kidger et al. 2020). To conclude, we show how the randomized signature approach can be used in cases where the time grid is irregularly sampled.
B.1 PRELIMINARY RESULTS: ROUGH DIFFERENTIAL EQUATIONS
1-Dimensional Stochastic Double Well. Let us recall that the dynamics of the 1-Dimensional Stochastic Double Well process is given by
dy, = OY, (1- - Y2? di +oo Yo =0 E ER LE [0,
where W; is 1-dimensional Brownian motion, and (μ, 0, a) EXXR x R+ Let us fix 1 and (μ= 2,0 2,0=10 1. = 1), and the partition D of [0, 1] to have N = 101 equally spaced time steps. We train Ridge Regression with regularization parameter À 0.001 map instances of Randomized Signature of the controls Z into the respective solution Y. We repeat the experiment on different values of the number Nrrain of train samples and dimension of the Randomized Signature. On the left of Figure we plot an example of the trajectory of while, on its right, we plot the comparison out-of-sample case. The following table shows the
of the and the generated time series performance in terms of L relative error on 10000 test samples:
NTrain
Manin
NTain 100
N 1000
Nrain 10000
k 111
0.319443
0.100310
0.007727
0.005854
0.005932
1 222
0.447563
0.357439
0.030950
0.005193
0.004558
k 444
0.504822
0.46000
0.092313
0,005690
0.0043893
Table 1: Double Well: Relative L Error
Rendom Sgrature: 3M111 sartwe)
Samp
10.0
9
7.5
5.0
1.0
0.8
0.5
0.0
2.6
0.5
5.8
2.0
0.0
0.2
0.4
0.8
1.0
Figure 9: Double Well: Randomized Signatures (left) - Test Sample (right).
4- D-Dimensional Ornstein-Uhlenbeck process. Let us recall that the dynamics of the 4- process given by
Dimensional
dY. (μ+ OY,) + EaW, 6, Yo yo E Rİ. i [0,1]
where W, is a 4-dimensional Brownian motion, and (μ, θ Σ) E R² x R4X4 x R4X4, Let fix
Yo = 1, fr = 1, Σ 14, O,,, = i/j. the partition D [0, 1] to have = 101 equally spaced time steps, and h 708. Finally, train Ridge Regression with X 0.001 on Niain train sample and Figure [0 shows the comparison of oft generated and true trajectories while Table 2 reports the performance in terms of 7,2 relative enror 10000 test samples.
17

---

Under review as a conference paper at ICLR 2022
1.50
1.0
0.0
3
0.25
Cstracted
Extracted
0.00
True Y2
True YV
0.2
0.8
Time
Time
3.00
1.0
True Y?
2.75
Extracted
2.50
6.
2.50
1.25
1,00
0.0
0.2
0.4
3.0
0.8
1.0
0.0
0.2
0.4
0.6
3.0
Figure 10: 4- d-Dimensional Ornstein Uhlenbechhhlbckk process: Test Sample.
k
708
NTrin
NTain 10
NTrin 100
NTrin 1000
NTrain 10000
Y
0.025960
0.010615
0.005145
0.001467
0.000929
Y2
0.047323
0.023858
0.009432
0.001838
0.001247
y3
0.033357
0.024392
0.009694
0.002008
0.001228
y
0.027637
0.020390
0.007798
0.001775
0.001 140
Table 2: 4-Dimensional Ohsteen-Uhllenecck process: Relative I Error.
1-Dimensional Geometric Fractional Brownian Motion. Let us recall that the dynamics of the I-Dimensional Geometric Fractional Brownian Motion process given by
dY; Ye (udt odd! Yo yo E RT.
where μ o R (0), and B("); is a one-dimensional fractional Brownian motion of hurst parameter H E (0.1). In this experiment, we fix yo = 1 and (μ = = 1). we let H vary in and the partition D [0, made of N equally spaced times. We highlight that, as per Theorem D.3.2 in (Biagini et al. 2008). when H < 1/ /4, the fractional Brownian motion so rough that it is no longer a Rough Path and its Signature no longer defined. Finally, we train a Ridge Regression with À 0.001 NTrai 10000 train samples of Z into the logarithm of the respective solution log(Y; which implies that we need to then apply the exp function to our predictions. The right of Figure 1T shows the comparison of the generated and true trajectories an sample trajectory while the following table shows the Relative 2 Error as we vary the number of time steps N and hurst parameter H::
(0.1, 0.4, 0.5),
H = 0.1
H = 0.2
H = 0.3
H = 0.4
H = 0.5
N 11
2.44 10-3
7.07
10-4
5.73 -10-
5.79 10-6
2.72 10-8
N 101
3.04 10-4
6.96 10-5
1.58
10-5
1.76
10-6
3.93 10
N 1001
3.78 10-5
1.11 10-5 4.03
10-6
1.46 10-6 1.32 1
Table 3: Relative 2 Error Fractional Geometric Brownian Motion.
Enzyme-Substrate Reactions. First, consider set of highly nonlinear CODEs modeling the interaction between a catalyst (enzyme, with abundance C,) and a substrate (with abundance S.). Substrate (e.g. lactose) is injected via the control variable X,, reacts with the enzyme (e.g. lactase)
18

---

Under review as a conference paper at ICLR 2022
Out OT Sample
Of Sample
9.
3.5
Extracted Thue
2.6
Y:
1.4
2.5
1.2
1.5
1.0
3.8
0.5
10
Figure 11: Double Well: Irregularly Sampled Test Sample (left) Fractional Geometric Brownian Motion Test Sample (right).
and the concentration Y, of the product chemical (c.g. glucose) is observed (Ingalls 2013
ds, (C+ - k1 (1 - Ct)) dt + X,dt
dCc - (C,- K1S: ( (1 C.)) dt - K2Ct
dY, h2Crdt.
Following (Ingalls] 2013) we choose (As,, 12) (30, 10). set (S., Po) = (0, 00,0,0) and consider the evolution on 0 t 1. We fix the time grid to have N 101 equally spaced time steps and the control X2 to follow the law of W? where W, is a 1-dimensional Brownian Motion (to ensure positivity). We train Ridge Regression with regularization parameter
10-3 to map 105 instances of 222-dimensional Randomized Signature 2 of the controls into the respective solution Y, On the left of Figure [2] we plot the comparison of the true and the generated time series test sample. As we can see, the model has learnt to correctly map a trajectory of X, to the respective system response Y.. More surprisingly, the right of such a figure shows that our model is able to predict the correct output even if we stimulate the system with substrate injection that follows a completely different law such as 0.5 (W?>0.5)- This suggests that the system was correctly identified of distribution.
Out Of Sample
1.75
Doncuec Tear
0.5
1.50
0.4
1.25
3 1.00
0.3
0.75
0.2
0.3
0.0
0.2
0.4
0.6
0.8
1.0
0.2
3.0
Time
Figure 12: Enzyme-Substrate Reactions stimulated with Squared Brownian Motion (left) Step
Function (right).
Comparison with Neural Controlled Differential Equations In this experiment, we benchmark the performance of our Randomized Signature model with that of Neural Controlled Differential Equations (NCDEs) (Kidger et al., 2020), a recently proposed competitive deep learning technique for controlled dynamical system modelling. We test the two models on the task of learning the Crmstein-Uhlenbede Process. As usual, to train our model, we fix
dynamics of the 1-dimensional an equally spaced partition D of [0, 1), extract the Randomized Signature of the controls for Nrrain samples, and train Ridge Regression to map them into the respective integrated solutions. The NCDE models instead parametrize the potentials feed-forward neural networks with hidden layer of Anodes. Figure 13 shows the comparison of our model against several version the NCDEs. Additionally, Table and T the number of trainable parameters, average 2 Relative Errors, and training times for all the models for rain 100 and Nyrain = 1000 respectively. We observe that our model is the best one from all aspects and that not even considerable amount of training data beneficial for NCDEs.
a latent controlled differential equation with
19

---

Under review as a conference paper at ICLR 2022
4.0
NCDE1B 100)
NCDE:16, 512)
Tue
3.0
Rand
2.3
1.0
20
-50
50
100
Time
Figure 13: Sat-of-Sample comparison of NCCDE Roodes) models, Randomized Signature model and Ground Truth.
Rand. Sig Model NCDE(4,10)
NCDE(8.100)
NCDE(16,512)
Trainable Param
107
107
1733
16961
Training Time
0.047390
1833.00
1837.858
2485.80
Error
1.23 10
0.081516
0.080829
0.0800839
Table 4: Randomized Signature vs. NCDEs - = 80, NTrain 100
Rand. Sig Model
NCDE(4,10)
NCDE(8,100)
NCDE(16,512)
Trainable Param
107
107
1733
16961
Training Time
0.238423
2853.00
3972.79
6510.95
Error
2.45 10
0,076195
0.072763
0.073845
Table 5: Randomized Signature vs. NCDEs d = 80, Nyrain
1000
B.2 ADDITIONAL. DETAILS
Learning the Signatures. In this experiment, we fix the partition {to..... .t....., [0, 1] to have 201 equally-spaced time steps and the regularization parameter of the Ridge Regression is 0.001.
Reconstruction and Randomized Signature Autoencoders. In this experiment, fix the par- tition D = (oNN) of [o, 1] to have 101 equally-spaced time steps, fix k = 111, and consider single realization XTrain of Ornstein-Uhlenbeck process of asymptotic mean μ 2, mean reversion 0 = 1. and volatility a 1. We train Ridge Regression with regularization parame- ter X= 0.001 map the Randomized Signature Z of the one and only train sample XTrein into XTrain itself.
k: == 111
h:
222
k = 444
Nrest 10000
8.35
10 9
2.26 10
2.27
10
Table 6: Relative L Error Autoencoder.
Comparison with Vanilla Signature Approach Let recall that the dynamics of the d- Dimensional Fractional OmntteinUUllenecck process given by
dY. = (m- oY,) + EdB",
Yo = Yo LE [0.1]
where B. is a d-dimensional Fractional Brownian motion of hurst parameter H (0. 1), and (n, θ Σ) E Rd x (ddxd x Rdxd Let us fix yo 1, l = 1, = 1,, O,, = i/j, the partition of [0, to have N 101 equally spaced time steps, and H 0.3.
20

---

Under review as a conference paper at ICLR 2022
1-Dimensional Stochastic Double Well -rregularly Sampled Time Grid. dynamics of the I-Dimensional Stochastic Double Well process is given by
Let us recall that the
dY, = 0Y, (μ (u--X,) dt + odW, Yo= = yo E R tE [0,
(9)
where We is a 1-dimensional Brownian motion, and (1, o) E R x R+ x RR, For each train and test sample, the partition of [0. 1] is made of N randomly drawn times. More precisely, D {0, ... 1, } such that & = 1/(1 exp(-sx), and ($1. ... , ,NN-1} realizations a uniform distribution [4 1] sorted increasingly. As a result, the probability that two samples share the same D is null. We train a Ridge Regressor with NTain 10000 train samples and an out-of-sample generated and true trajectory. Finally, Table 7 shows the Relative L Error 10000 test samples
are N - independent
Figure [ shows the comparison
vary the number of time steps N and and while comparing it to the respective experiment case the time grid is regularly spaced. As can see, even though the performance are worse than the regularly sampled setup, this technique proves to be anyway reliable on irregularly sampled regimes. In this experiment, we fix yo = 1 and 0.001.
(μ 2,0 1 =) 1) while the regularization parameter of the Ridge Regression
(N, k) (11,
(N = (101, 222) 0.016885
k) (1001,
Irregular
0.082735
0.010902
Regular
0.026759
0.004465
0.003004
Table 7: Irregularly Sampled Double Well: Relative L Error.
Tumor Growth Kinetics. We set (k1, Ao \1, v (ub: "6: ug. u Yo) (2,0, 0, 2). We fix the partition D of [0, 10] to have 101 equally spaced time steps and we again train a Ridge Regression with regularization parameter = 0.001 map 10000 instances of the 222-dimensional Randomized Signature Z of the controls into the respective solution Y,
(10,0.5,00 0.7,20), and
Electrochemical Battery Model. In this experiment, we fix the partition D of [0, 500] to have 1001 equally spaced time steps, the regularization parameter of the Ridge Regression is X 10-3 while E (50, 166, 1000). Regarding the latest experiment, we simply learn to map the Random Signature of the controls into the respective Y to which we added white noise with variance 0.01. Regarding NNARX, for the regressor we follow the results presented in (Masti and Bemporad] 2018), where na = Thp 12, and rıg lead the best results. Regarding the neural network, we use a feedforward neural network with input dimension nt, + nı, + 25 and 2 hidden layers each with either 22 (NNARX22) or 1000 (NNARXION
hidden units. For the loss function, we used the mean square eтror optimized using Adam with learning rate 0.01 for 100 epochs. Table &rpports the performances in terms of MSE on 1000 test samples for our models compared to NNARX.
k 50
k = 166
k
1000
NNARX 22
NNARX 1000
Nrest 1000
3.15 10
2.66
10
2.59
10
3.88 10-4
2.62 10-4
Table 8: Electrochemical Battery Model: MSE Error Comparison.
Time Series Classification
DATASET DETAILS We run our TSC experiments on the UCR archive, one of the most popular benchmarks for TSC methods (Dau al. 2019 Fawaz et al.) 2019). consists of 128 heterogeneous datasets of different sizes comprising possibly irregularly sampled one-dimensional time series of various lengths. Rocket one of the best performing algorithms on this benchmark.
ROCKET When implementing Rocket, we generate h kernels each characterized by certain ran- dom length lftter. random weights w E Ritterr and random jump j. Since the lengths and the jumps are randomly sampled, the length M of the vector = =(',, ..... ...., ... M) of the random fea-
tures varies accordingly. Given that the number of possible combinations of the values of l fatter and
21

---

Under review as a conference paper at ICLR 2022
j is limited, group the kernels based such combinations. As result, among each of these groups, the length M is fixed and, additionally, each kemel is applied to the very same sub-time- series (i+j
( I)xxi) of X starting from position i. If now focus these subgroups and let us assume it contains kernels of length fytter and jump j. As a result of the application of each kernel on X, we obtain the following matrix
of
Z
Z
MM.17
20,2
2
zu,k
Z1.k
ZM.kk
where Z the result of the action of the r-th kernel on the sub-time-series (X,, Xetj. (et(leatee==)xx of X starting from position i. Finally, out of each row, we ex- tract the features described above.
Turning approach, generate & kernels each characterized by certain length length fitters weights (wo, w1) R², and jump j. As before, let us again group them based on all the possi- ble combinations lyater and and focus on only one of such subgroups. As a consequence, we have & kernels each characterized by weights (i].w)). E (1,.... k} with fixed length and jump. The application of each of these k kernels following the update rules proposed in Algorithm is equivalent to extracting 1 1-dimensional Randomized Feature of each of the sub-time-sed series
[X,, Xitj: Xi+t(liuer 1)xj). As result, we obtain the following matrix
Z
Z M.1T
z
z1.2
2
20,6
21,
ZM
where Z is the 1-dimensional Randomized Signature of (Xo, Xj. X( ylaee x)x via the r-th kernel. Let us we define the random matrix guiding the evolution of the Randomized Signature as the ie & diagonal matrix having the weights wh on the diagonal while the random bias is column vector having the wI as entries. Then, the i-th column the k-dimensional Randomized Signature of the sub-time-series (Xt, X (xi+((puee--))ii Finally, out of each row, extract the features described above.
Comparison of Computational Complexities and Dimensions The following table reports key figures regarding the computational complexity and dimensionality of both Signature and Random- ized Signature.
Comp. complexity
Dimension ((dM)
Guarantees
Truncated Sign. (M- components):
(dd
v (Thm
Randomized Sign. (d- dimensional):
O(2dd)
O(k)
(Thm
Table 9: Properties of (Randomized) Signatures of d-dimensional path.
22


Reviews:
[{"review_id": "tfvlOw8rjg4", "reviewer": "Reviewer_iWAG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The manuscript builds on previous work that has showed that the signature is a reservoir by demonstrating the performance of the randomized signature. This adds to the literature on reducing overhead in signature methods, which has grown a lot in the last few years but is still a very active area of research. As it stands however, the theoretical contribution of the manuscript is minimal as far as I can tell, and it is mainly a showcase of what can be done with the randomized signature.\n\nWhile the breadth of experiments is impressive, there is a lack of comparison with other signature based baselines such as neural CDE or RDE methods or low-rank projections. Ideally there should also be some discussion about what the benefits are of using the randomized signature over the other methods and when it is more or less suitable.\n\nMinor points:\n\n - Text is, in my opinion, overly verbose and would probably improve if cut down a bit.\n\n - Many published papers are cited as arXiv preprints.\n\n - Section 3.2 paragraph 1: \"problem, in particular kernelization techniques, see, e.g., Kidger and Lyons (2020)\" This paper does not propose kernelization techniques.\n\nOverall, the paper is decent and might be very useful in certain situations, but it lacks discussion, and the empirical contributions are rather weak considering the lack of technical innovation.  \n", "summary_of_the_paper": "The manuscript proposes that by combining the signature transform with ideas from reservoir computing one may reduce computational cost and spatial overhead. This is demonstrated on some toy problems as well as some real world datasets\n", "summary_of_the_review": "This is a good showcase of the power of reservoir computing in signature methods, but there is room for improvement.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "g8R-99ampP8", "reviewer": "Reviewer_KwmV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nS1: Following Theorem 1 theoretically justifies that Signature of $X$ is suited to solving differential equations driven by $X$.\n\nS2: Following Theorem 2, Randomized Signature approximates Signature while being much cheaper to compute. The approximation power of Randomized Signature to the original Signature is empirically verified in Section 4.1 (in particular Figure 3).\n\nS3: In Section 4.3, Randomized Signature can find solutions of differential equations that are close to ground-truth solutions.\n\nS4: For time series classification, Randomized Signature when combined with Rocket (Dempster et al., 2020) gives better performance than existing related deep learning methods. It however does not perform as well as Rocket.\n\nWeaknesses:\nW1: The paper provides informal versions of Theorems 1 and 2 only. I cannot find the formal versions as well as their proofs in the appendix. I would recommend the authors to provide more details. This will help to make points such as \"comparable approximation power as the Signature\" in Theorem 2 more rigorous.\n\nW2: How to choose activation $\\sigma$? There seems also no explanation about this in Section 4.\n\nW3: From time complexity analysis in Section 3.2, the runtime speed up seems significant. However, the experiment in Section 4.1 reveals using Randomized Signature yields less than an order of magnitude speed up (3 times in particular). This can be because $k$ is quite large compared to $M$. The authors should provide more detailed runtime comparisons between the two types of signature in the experiments in Section 4.3.\n\nW4: Most experiments in Section 4.3 are on data with $d \\leq 5$. How about data with higher number of dimensions such as more than 10? It would be interesting to see scalability to $d$ in both runtime and quality of Randomized Signature. Could the authors provide a summary of the 128 UCI data sets for time series classification? This will help to better understand impact of $d$ on the performance.\n\nW5: Randomized Signature + Rocket performs worse than Rocket, due to limitations of the update scheme in Algorithm 1. The authors already provided some justifications in the last paragraph of Section 4.4.", "summary_of_the_paper": "This paper proposes a Signature-based method for learning feature representation of time series, denoted as $d$-dimensional path $X$. By definition, Signature is a sequence of $M$ tensors where each tensor is a sum of integral terms defined on dimensions of  $X$. Theoretically Signature is guaranteed to linearly explain the solution of any differential equation driven by $X$ which implies that it encodes well $X$. Computing Signature however has time complexity $O(d^M)$ rendering it impractical for learning purposes. To alleviate the issue, the paper introduces Randomized Signature, which can be computed in $O(k^2 d)$ only where $k$ is the number of Randomized Signatures. Each of such Signature has $O(kd)$ dimensions. As stated by Theorem 2 in the paper, Randomized Signature approximates well the original Signature. Experiments on differential equations show Randomized Signature finds solutions close to ground-truth solutions. Experiments on time series classification also show its potential.", "summary_of_the_review": "Using Signature theory for time series analysis is a promising direction with some theoretical guarantees. The paper however is not self-contained and some details are lacking. Please help to address the above comments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "65tlS3s-UB", "reviewer": "Reviewer_5Jna", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "S1：This paper gives significant insights into the randomized signatures by showcasing its performance on non-parametric system identification and time series classification problems.\n\nS2: The background and theory related to randomized signatures are well described.\n\nS3: The experimental results provide evidence of the effectiveness of the presented strategy.\n\nW1: The original contribution of this paper is limited. The methods of reservoir computing and random signature are mature components. The paper just uses these strategies in a trivial manner.\n\nW2: The claimed advantages of the proposed randomized signature layer over the existing solution, such as inferring mapping from data, are not evident. It is expected that the results are provided to show the success of the randomized signature layer on resolving the problems such as the pointed over-parametrization, data hungriness, expensive training cost.\n\nW3: Although the signature is well theoretically guaranteed, it is not clarified how these guarantees can contribute to the problem of time series classification in comparison with the original Rocket where the randomized convolution kernels are used.\n\nW4: A minor issue. As the main contribution of this paper is to showcase the effectiveness of the randomized signature layer, it’s expected that its performance on more representative tasks are evaluated and reported.   \n", "summary_of_the_paper": "This paper showcases the effectiveness of a recently introduced reservoir of random features, i.e., randomized signatures.", "summary_of_the_review": "In summary, I believe this paper is under the acceptance threshold because of its limited contributions as well as some other issues, as I have pointed from W1 to W4. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "2JmhrtTugPs", "reviewer": "Reviewer_g484", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall this paper is very well written and tackles an interesting problem - dealing with the blowup in the dimensionality of the signature.\n\nA lot of recent papers in the ML literature that utilise the signature pay little notice of the severe feature blowup associated with increasing signature feature dimension. The result of this is a number of methods that do not scale well to tasks with large numbers of feature channels. This paper outlines an interesting method of dealing with this by way of random projections applied to the path \n\n## Strengths\n- Well written with a good introductory section to signatures and their usefulness. \n- Randomised signature layers are an interesting way to reduce signature dimension - a current bottleneck for a lot of signature methodologies. \n\n## Weaknesses\n- **Method is not entirely novel** - Lyons et al. \"A feature set for streams and an application to high-frequency financial tick data\" introduce this idea of randomised signatures and it is considered in Morrill et al. \"A Generalised Signature Method for Multivariate Time Series Feature Extraction\". Neither of these papers are mentioned/referenced by the authors. \n- **Performance on time series classification tasks seems poor** -  Whilst the \"rocket_ours\" classifier comes second of all classifiers considered, \"rocket\" was already the top performer. I'm not sure how I should be considering this other than that the randomised signature features make the rocket classifier worse -- why would I use randomised signature features then? \n- **Informal theorem is too informal** - The randomised signature theorem with the claim \"has comparable approximation power as the Signature\" needs more explanation. Whilst I am a fan of informal theorem statements in such papers in general, I need some indication of what such a statement actually means. This can be easily remedied by a high level explanation of what is going on, or a mathematical statement as to what this comparable approximation power actually means. It also needs to reference a full, formal theorem somewhere. ", "summary_of_the_paper": "The authors utilise random projections before computation of the signature resulting in \"Randomised signatures\" from which similar information can be extracted (compared with the standardly applied signature transform) with reduced computational burden and lower overall feature dimension. \n\nThe authors demonstrate the utility of this randomised methodology on a variety of tasks including learning rough differential equations and standard time series classification benchmarks. ", "summary_of_the_review": "This paper would represent the best reference to the idea of \"randomised signature\" features if accepted. However, it is currently lacking in that it does not adequately reference prior work, and it has theorems that are slightly too vague. If these things can be rectified, I would consider raising my score. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper empirically evaluates the performance (in time and accuracy) of randomized signatures for time series, an idea that was developed theoretically in a series of recent paper. While reviewers acknowledge that implementing and testing this idea is relevant, they also consider that the lack of methodological and theoretical novelty, combined with the fact that the experimental results do not convincingly show that randomized signatures outperform existing methods on a variety of tasks, puts the paper below the acceptance bar.
Decision: Reject