Signatory: differentiable computations of the signature and logsignature transforms, on both CPU and GPU

1 INTRODUCTION . The signature transform , sometimes referred to as the path signature or simply signature , is a central object in rough path theory ( Lyons , 1998 ; 2014 ) . It is a transformation on differentiable paths1 , and may be thought of as loosely analogous to the Fourier transform . However whilst the Fourier transform extracts information about frequency , treats each channel separately , and is linear , the signature transform exacts information about order and area , explicitly considers combinations of channels , and is in a precise sense ‘ universally nonlinear ’ ( Bonnier et al. , 2019 , Proposition A.6 ) . The logsignature transform ( Liao et al. , 2019 ) is a related transform , that we will also consider . In both cases , by treating sequences of data as continuous paths , then the ( log ) signature transform may be applied for use in problems with sequential structure , such as time series . Indeed there is a significant body of work using the ( log ) signature transform in machine learning , with examples ranging from handwriting identification to sepsis prediction , see for example Morrill et al . ( 2019 ) ; Fermanian ( 2019 ) ; Király & Oberhauser ( 2019 ) ; Toth & Oberhauser ( 2020 ) ; Morrill et al . ( 2020b ) . Earlier work often used the signature and logsignature transforms as a feature transformation . See Levin et al . ( 2013 ) ; Chevyrev & Kormilitzin ( 2016 ) ; Yang et al . ( 2016a ; b ) ; Kormilitzin et al . ( 2016 ) ; Li et al . ( 2017 ) ; Perez Arribas et al . ( 2018 ) for a range of examples . In this context , when training a model on top , it is sufficent to simply preprocess the entire dataset with the signature or logsignature transform , and then save the result . However , recent work has focused on embedding the signature and logsignature transforms within neural networks . Recent work includes Bonnier et al . ( 2019 ) ; Liao et al . ( 2019 ) ; Moor et al . ( 2020 ) ; Morrill et al . ( 2020a ) ; Kidger et al . ( 2020 ) among others . In this context , the signature and logsignature transforms are evaluated many times throughout a training procedure , and as such efficient and differentiable implementations are crucial . Previous libraries ( Lyons , 2017 ; Reizenstein & Graham , 2018 ) have been CPU-only and single-threaded , and quickly become the major source of slowdown when training and evaluating these networks . 1And may be extended to paths of bounded variation , or merely finite p-variation ( Lyons et al. , 2004 ) . 1.1 CONTRIBUTIONS . We introduce Signatory , a CPU- and GPU-capable library for calculating and performing functionality related to the signature and logsignature transforms . To our knowledge it is the first GPU-capable library for these operations . The focus is on machine learning applications . Signatory is significantly faster than previous libraries ( whether run on the CPU or the GPU ) , due to a combination of parallelism and novel algorithmic improvements . In particular the latter includes both uniform and asymptotic rate improvements over previous algorithms . Additionally , Signatory provides functionality not available in previous libraries , such as precomputation strategies for efficient querying of the ( log ) signature transform over arbitrary overlapping intervals . The library integrates with the open source PyTorch ecosystem and runs on Linux or Windows . Documentation , examples , benchmarks and tests form a part of the project . Much of the code is written in C++ primitives and the CPU implementation utilises OpenMP . The backward operations are handwritten for both speed and memory efficiency , and do not rely on the autodifferentiation provided by PyTorch . The source code is located at https : //github.com/patrick-kidger/signatory , documentation and examples are available at https : //signatory.readthedocs.io , and the project may be installed directly via pip . This paper is not a guide to using Signatory—for that we refer to the documentation . This is meant as a technical exposition of its innovations . 1.2 APPLICATIONS . Signatory has already seen a rapid uptake amongst the signature community . Recent work using Signatory include Morrill et al . ( 2020b ) ; Perez Arribas et al . ( 2020 ) who involve signatures in neural differential equations , or Moor et al . ( 2020 ) ; Min & Ichiba ( 2020 ) who study deep signature models ( Bonnier et al. , 2019 ) . Meanwhile Ni et al . ( 2020 ) apply Signatory to hybridise signatures with GANs , and Morrill et al . ( 2020a ) create a generalised framework for the “ signature method ” . As a final example , Signatory is now itself a dependency for other libraries ( Kidger , 2020 ) . 2 BACKGROUND . We begin with some exposition on theory of the signature and logsignature transforms . We begin with definitions and offer intuition afterwards . Also see Reizenstein & Graham ( 2018 ) for an introduction focusing on computational concerns , and Lyons et al . ( 2004 ) and Hodgkinson et al . ( 2020 ) for pedagogical introductions to the motivating theory of rough paths . 2.1 THE SIGNATURE TRANSFORM . Definition 1 . Let Rd1⊗Rd2⊗· · ·⊗Rdn denote the space of all real tensors with shape d1×d2×· · ·× dn . There is a corresponding binary operation ⊗ , called the tensor product , which maps a tensor of shape ( d1 , . . . , dn ) and a tensor of shape ( e1 , . . . , em ) to a tensor of shape ( d1 , . . . , dn , e1 , . . . , em ) via ( Ai1 , ... , in , Bj1 , ... , jm ) 7→ Ai1 , ... , inBj1 , ... , jm . For example when applied to two vectors , it reduces to the outer product . Let ( Rd ) ⊗k = Rd ⊗ · · · ⊗Rd , and v⊗k = v⊗ · · · ⊗ v for v ∈ Rd , in each case with k− 1 many ⊗ . Definition 2 . Let N ∈ N. The signature transform to depth N is defined as SigN : { f ∈ C ( [ 0 , 1 ] ; Rd ) ∣∣ f differentiable } → N∏ k=1 ( Rd ) ⊗k , SigN ( f ) = ∫ · · · ∫ 0 < t1 < ··· < tk < 1 df dt ( t1 ) ⊗ · · · ⊗ df dt ( tk ) dt1 · · · dtk 1≤k≤N . ( 1 ) Most texts define the signature transform using the notation of stochastic calculus . Here , we sacrifice some generality ( that is not needed in this context ) in favour of more widely-used notation.2 The signature transform may naturally be extended to sequences of data . Definition 3 . The space of sequences of data over a set V is S ( V ) = { x = ( x1 , . . . , xL ) |L ∈ N , xi ∈ V for all i } . An interval of ( x1 , . . . , xL ) ∈ S ( V ) is ( xi , . . . , xj ) ∈ S ( V ) for some 1 ≤ i < j ≤ L. Definition 4 . Let x = ( x1 , . . . , xL ) ∈ S ( Rd ) with L ≥ 2 . Let f : [ 0 , 1 ] → Rd be the unique continuous piecewise affine function such that f ( i−1L−1 ) = xi for all i , and is affine on the pieces in between . Let N ∈ N. Then define SigN ( x ) = SigN ( f ) . In this way we interpret SigN as a map SigN : S ( Rd ) → N∏ k=1 ( Rd ) ⊗k . Note that the choice of i−1L−1 is unimportant ; any L points in [ 0 , 1 ] would suffice , and in fact the definition is invariant to this choice ( Bonnier et al. , 2019 , Definition A.10 ) . 2.2 THE GROUPLIKE STRUCTURE . With A0 = B0 = 1 ∈ R on the right hand side , define by3 : ( N∏ k=1 ( Rd ) ⊗k ) × ( N∏ k=1 ( Rd ) ⊗k ) → N∏ k=1 ( Rd ) ⊗k , ( A1 , . . . AN ) ( B1 , . . . , BN ) 7→ k∑ j=0 Aj ⊗Bk−j 1≤k≤N . Chen ’ s identity ( Lyons et al. , 2004 , Theorem 2.9 ) states that the image of the signature transform forms a noncommutative group with respect to . That is , given a sequence of data ( x1 , . . . , xL ) ∈ S ( Rd ) and some j ∈ { 2 , . . . , L− 1 } , then SigN ( ( x1 , . . . , xL ) ) = Sig N ( ( x1 , . . . , xj ) ) Sig N ( ( xj , . . . , xL ) ) . ( 2 ) Furthermore the signature of a sequence of length two may be computed explicitly from the definition . Letting exp : Rd → N∏ k=1 ( Rd ) ⊗k , exp : v → ( v , v⊗2 2 ! , v⊗3 3 ! , . . . , v⊗N N ! ) , then SigN ( ( x1 , x2 ) ) = exp ( x2 − x1 ) . With Chen ’ s identity , this implies that the signature transform may be computed by evaluating SigN ( ( x1 , . . . , xL ) ) = exp ( x2 − x1 ) exp ( x3 − x2 ) · · · exp ( xL − xL−1 ) . ( 3 ) 2.3 THE LOGSIGNATURE , INVERTED SIGNATURE , AND INVERTED LOGSIGNATURE . The group inverse we denote −1 . Additionally a notion of logarithm may be defined ( Liao et al. , 2019 ) , where log : image ( SigN ) → N∏ k=1 ( Rd ) ⊗k . ( 4 ) 2Additionally , many texts also include a k = 0 term , which is defined to equal one . We omit this as it does not carry any information , and is therefore irrelevant to the task of machine learning . 3Most texts use ⊗ rather than to denote this operation , as it may be regarded as an generalisation of the tensor product . That will not be important to us , however , so we use differing notation to aid interpretation . This then defines the notions of inverted signature transform , logsignature transform and inverted logsignature transform as InvertSigN ( x ) = SigN ( x ) −1 , LogSigN ( x ) = log ( SigN ( x ) ) , InvertLogSigN ( x ) = log ( SigN ( x ) −1 ) respectively . We emphasise that the inverted signature or logsignature transforms are not the inverse maps of the signature or the logsignature transforms . The logsignature transform extracts the same information as the signature transform , but represents the information in a much more compact way , as image ( log ) is a proper subspace4 of∏N k=1 ( Rd ) ⊗k . Its dimension is w ( d , N ) = ∑N k=1 1 k ∑ i|k µ ( k i ) di , which is known as Witt ’ s for- mula ( Lothaire , 1997 ) . µ is the Möbius function .

Summary: The paper presents the first GPU-capable library implementing the _"signature"_ and _"log-signature"_ functions as well as their gradients. It introduces these transformations to a machine learning audience, as well as their recent uses in ML, then proposes algorithmic improvements that reduce the necessary computation. The resulting library is benchmarked against existing implementations, and the code, benchmarks, and proofs are included in supplementary materials.