Randomized Signature Layers for Signal Extraction in Time Series Data

1 INTRODUCTION . Modeling time series is a common task in finance , physics , and engineering . A frequent challenge is finding a transformation mapping a newly observed time series into a target one ( seq2seq modeling ) or into a label summarizing its salient properties ( classification ) . In the absence of any principled model describing such a mapping , one has to infer it from data . The last few years have witnessed the rise of deep neural networks , which have found successful application to problems involving time series in numerous domains ( Fawaz et al. , 2019 ; Gamboa , 2017 ) . Nevertheless , their outstanding performance comes at the price of over-parametrization , data hungriness , and expensive training cost ( Werbos , 1990 ; Brown et al. , 2020 ; Teubert et al. , 2021 ; Neyshabur et al. , 2018 ; Marcus , 2018 ) . Furthermore , even if sufficient data is available , the resulting models learn representations of the input data that are highly specialized to the training task and difficult to adapt in different contexts . In addition , the remarkable performance of these methods is often the result of a substantial engineering effort and is not supported by theoretical results . Reservoir Computing ( RC ) ( Schrauwen et al. , 2007 ) offers an intriguing alternative strategy to cope with the limitations above , yet retaining the universal approximation properties typical of deep neural networks . In RC , the learning is divided into two phases : first , data is passed through an untrained reservoir which extracts a set of task-independent features ; second , a simple and efficient-to-train linear map ( the readout map ) projects such features into the desired output . The critical point is that the design of the reservoir determines the expressiveness of the features , and several alternatives can be found in the literature ( see ( Gauthier et al. , 2021 ) and references therein ) . A powerful reservoir is offered by the Signature Transform , often simply referred to as Signature , stemming from rough path theory ( Ben Hambly , 2010 ; Friz and Hairer , 2020 ) . The Signature of a path consists in enhancing the path with additional curves , which , in the smooth case , corresponds to iterated integrals of the curve with itself . A profound mathematical result ( Levin et al. , 2013 ) guarantees that the solution of a ( rough ) differential equation can be approximated arbitrarily well by a linear map of the Signature of the input signals ( a.k.a . controls ) . In the machine learning jargon , the Signature of a path can be interpreted as a feature map extracting all its geometrical properties and thus allowing a simple linear map to approximate any function of it . On the other hand , it is often the case that the reservoir features are very high dimensional , and hence are particularly expensive to calculate and use in downstream tasks . In addition , the high-dimensionality of the Signature reservoir poses additional challenges for modern gradient-based optimizers as convergence rates suffer from a linear dependence in the model dimension ( Bottou et al. , 2018 ) . Inspired by the remarkable theoretical properties of the Signature reservoir and motivated to fix its practical pitfalls , our contribution is to showcase the effectiveness of Randomized Signatures ( Cuchiero et al. , 2021b ; a ) , a recently introduced reservoir of random features . These features provably hold the same geometric properties and approximation power as the Signature , yet are often more efficient to compute and of lower dimensionality . To extract Randomized Signatures , we numerically integrate a set of random linear stochastic differential equations driven by the path itself . We assess the expressiveness of the proposed random features on several tasks , including nonparametric ( black-box ) system identification problems arising from complex controlled nonlinear dynamical systems as well as time series classification . The paper is structured as follows : In Section 2 , we link the Randomized Signature approach with related research areas in the literature ; in Section 3 , we present relevant theoretical results motivating our method ; in Section 4 , we assess the effectiveness of Randomized Signatures on various system identification and time series classification tasks , showing promising results when compared to other signature approaches for reservoir computing . Finally , in Section 5 , we discuss our conclusions and possible future developments . 2 RELATED WORKS . Random Features and Reservoir Computing . The idea of extracting features based on random operations is not new and has seen a number of successful applications over the past years . Of particular note , the seminal work of Rahimi and Recht ( 2008 ) proposes to accelerate kernel machines by learning random features whose inner product matches that of a target shift-invariant kernel . The trade-off between generalization and computational efficiency of learning with random features has then been rigorously studied by Rudi and Rosasco ( 2017 ) . A conceptually very similar rationale is introduced by a parallel series of works exploring the topic of Reservoir Computing ( Schrauwen et al. , 2007 ) . Similarly to us , Echo State Networks ( Jaeger , 2003 ) evolve the input state by a series of fixed random projection ( the reservoir ) and generate the output by applying a trainable linear projection over the hidden states . However , we make the additional step of linking the random features to the Signature of the input path and , as shown below , the evolution of the features is dictated by a randomly-evolved stochastic differential equation driven by the input path . Controlled Differential Equations . Our work is also related with a series of recent papers investigating the problems of how to process irregular time series and to condition a model on incoming information through the lens of controlled differential equations ( CDEs ) ( Kidger et al. , 2020 ; Morrill et al. , 2021 ) . Interestingly , Kidger et al . ( 2020 ) show that the action of a linear layer on the final output of a trained Neural CDEs – the extension of Neural ODEs ( Chen et al. , 2018 ) to CDEs – results in a universal function approximator . Differently from them , our method is less computationally expensive as the only parameters we need to train are those of the final linear readout layer . Fundamentally different to our work , the approach of Morrill et al . ( 2020 ) first randomly projects the possibly high dimensional controls into a lower dimensional space and then extracts the Truncated Signature from such compressed input . Our method instead extracts a random compression of the signature of the original input controls . Rough Path Theory . Rough path theory is about describing how possibly highly oscillatory ( rough ) control path interact with nonlinear systems ( Lyons , 2014 ) . The concept of Signature is introduced in this context to provide a powerful description of the input path , removing the need to look at the fine structure of the path itself . Recent years have seen a resurgence of these ideas , which have been revisited from a machine learning perspective ( Bonnier et al. , 2019 ; Kidger and Lyons , 2021 ) . Our analysis is strongly influenced by the work of Cuchiero et al . ( 2021a ; b ) , who , starting from the observation that the Signature is an infinite dimensional reservoir , establishes that its information content can be efficiently compressed by a random projection performed by a dynamical system driven by random vector fields . As we show later , this results in a random reservoir which preserves the properties of the Signature while living in a finite dimensional space . 3 BACKGROUND . We provide the theoretical tools supporting the Randomized Signature approach for feature extraction in times series . A formal discussion can be found in ( Friz and Hairer , 2020 ) and ( Cuchiero et al. , 2021b ) . 3.1 RANDOMIZED SIGNATURE OF A PATH . Let X : [ 0 , T ] → Rd be a continuous piecewise smooth d-dimensional path X = ( X1 , · · · , Xd ) . We will refer to X as the control and to its single components Xi as controls . We denote by { e1 , . . . , ed } the canonical basis of Rd . Definition 1 ( Signature ) For any t ∈ [ 0 , T ] , the Signature of a continuous piecewise smooth path X : [ 0 , T ] → Rd on [ 0 , t ] is the countable collection St : = ( 1 , S1t , S 2 t , . . . ) ∈ ∏∞ k=0 ( Rd ) ⊗k where , for each k ≥ 1 , the entries Skt are the iterated integrals defined as Skt : = ∑ ( i1 , ... , ik ) ∈ { 1 , ... , d } k ( ∫ 0≤s1≤···≤sk≤t dXi1s1 . . . dX ik sk ) ei1 ⊗ · · · ⊗ eik . ( 1 ) We define the Truncated Signature of X of order M ≥ 0 as SMt : = ( 1 , S1t , . . . , S M t ) ∈ M∏ k=0 ( Rd ) ⊗k = : T M ( Rd ) . ( 2 ) A practical example of computation of Signatures is presented in Appendix A.3 . The definition in the last paragraph suggests that the Signatures of X can be used to approximate any regular enough function of X , for instance the solution to differential equations controlled by X . The following result makes this argument precise in the multidimensional setting . Theorem 1 ( Signature is a Reservoir ) Let Vi : Rm → Rm , i = 1 , . . . , d be vector fields regular enough s.t . dYt = ∑d i=1 V i ( Yt ) dX i t , Y0 = y ∈ Rm , admits a unique solution Yt : [ 0 , T ] → Rm . Then , for any smooth test function F : Rm → R and for every M ≥ 0 there is a time-homogeneous linear operator L : T M ( Rd ) → R which depends only on ( V1 , . . . , Vd , F , M , y ) s.t . F ( Yt ) = L ( SMt ) +O ( tM+1 ) , t ∈ [ 0 , T ] . ( 3 ) This theorem suggests the first M entries of the Signature of X are sufficient to linearly explain the solution of any differential equation driven by it . In addition to this , it supports the claim that such features are valuable for any downstream usage . 3.2 RANDOMIZED SIGNATURE OF A PATH AND ITS THEORETICAL GUARANTEES . Calculating SMt requires the calculation of dM+1−1 d−1 iterated integrals ( Appendix A.3 ) – which in total quickly becomes computationally expensive . Several computational techniques have been developed to circumvent this problem , see , e.g . ( Kidger and Lyons , 2021 ) . The next result provides a practical description of how it is possible to reduce the computational burden without losing too much explanatory power . As such , this results provides the theoretical foundation for our approach . Theorem 2 ( Randomized Signature ( Informal ) ) For any k ∈ N big enough and appropriately chosen random matrices A1 , . . . , Ad in Rk×k , random shifts b1 , . . . , bd in Rk×1 , random starting point z in Rk×1 , and any fixed activation function σ , the solution of dZt = d∑ i=1 σ ( AiZt + bi ) dX i t , Z0 = z ∈ Rk , t ∈ [ 0 , T ] . ( 4 ) – called the Randomized Signature of X – has comparable approximation power as the Signature itself and maintains its geometric properties . The formal statement of such a result is the combination of Theorem 8 , Theorem 9 , and Definition 10 ( see Appendix ) . A complete formal discussion as well as the original statement of this theorem can be found in the paper ( Cuchiero et al. , 2021a ) . In a nutshell , the Randomized Signature Z follows a dynamics which provides an efficient and powerful compression of the Signature through a low-dimensional random projection . The expressiveness of such projection is guaranteed by a standard Johnson-Lindenstrauss argument ( Cuchiero et al. , 2021a ) . Computational complexity and dimensionality of Randomized Signatures . The computational complexity of calculating Zt is O ( k2d ) , while its dimensionality is O ( k ) . However , we should then ask the crucial question “ how should we compare k with M ? ” . We show experimentally in Section 4 that , in order to match the approximation guarantees of the Truncated Signature of order M , the number of required Randomized Signatures k is not too big — in particular it is not exponential in M . This confirms that working with Randomized Signatures is often less computationally demanding and results in lower-dimensional features . We conclude the section remarking the fascinating implications of choosing a random Ais and bis . Theorem 3 ( Density of Randomized Signatures ( Informal ) ) For any sequences of time points 0 ≤ t0 < . . . < tN ≤ T and points zt0 , . . . , ztN ∈ Rk we can find a smooth control X such that the solution Z of Equation 4 at time ti is such that Zti = zti , for i = 0 , . . . , N . Additionally , if the control X is a d-dimensional Brownian motion , then the solution of Equation 4 at any point in time t > 0 admits a smooth density with respect to Lebesgue measure .

Summary: This paper proposes a Signature-based method for learning feature representation of time series, denoted as $d$-dimensional path $X$. By definition, Signature is a sequence of $M$ tensors where each tensor is a sum of integral terms defined on dimensions of  $X$. Theoretically Signature is guaranteed to linearly explain the solution of any differential equation driven by $X$ which implies that it encodes well $X$. Computing Signature however has time complexity $O(d^M)$ rendering it impractical for learning purposes. To alleviate the issue, the paper introduces Randomized Signature, which can be computed in $O(k^2 d)$ only where $k$ is the number of Randomized Signatures. Each of such Signature has $O(kd)$ dimensions. As stated by Theorem 2 in the paper, Randomized Signature approximates well the original Signature. Experiments on differential equations show Randomized Signature finds solutions close to ground-truth solutions. Experiments on time series classification also show its potential.