Path Development Network with Finite-dimensional Lie Group
Authors: Hang Lou
Siran Li
Hao Ni
Venue: Accepted by TMLR ()

Abstract: Signature, lying at the heart of rough path theory, is a central tool for analysing controlled differential equations driven by irregular paths. Recently it has also found extensive applications in machine learning and data science as a mathematically principled, universal feature that boosts the performance of deep learning-based models in sequential data tasks. It, nevertheless, suffers from the curse of dimensionality when paths are high-dimensional. 

We propose a novel, trainable path development layer, which exploits representations of sequential data through finite-dimensional Lie groups, thus resulting in dimension reduction. Its backpropagation algorithm is designed via optimization on manifolds. Our proposed layer, analogous to recurrent neural networks (RNN), possesses an explicit, simple recurrent unit that alleviates the gradient issues. 


Our layer demonstrates its strength in irregular time series modelling. Empirical results on a range of datasets show that the development layer consistently and significantly outperforms signature features on accuracy and dimensionality. The compact hybrid model (stacking one-layer LSTM with the development layer) achieves state-of-the-art against various RNN and continuous time series models. Our layer also enhances the performance of modelling dynamics constrained to Lie groups. Code is available at https://github.com/PDevNet/DevNet.git.

Published in Transactions on Machine Learning Research (08/2024)
Path Development Network with Finite-dimensional Lie Group
Hang Lou Deportment of Mathematics University College London
louhang39@gmail.omm com
Siran Li
s44025 (..0255mmy..d
Department of Mathemalics Shanghai Jiao Tong University
Hao Ni
h.ni@ucl.ac. uk
Department of Mathematics University College London
Reviewed on OpenReview: https: /ttps:///penrevvtee net/ forum? YoWBLu747L
Abstract
Signature, lying at the heart of rough path theory, central tool for analysing controlled differential equations driven by irregular paths. Recently has also found extensive ap- plications in machine learning and data science as mathematically principled, universal feature that. boosts the performance of deep learning-based models in sequential data tasks. It, nevertheless, suffers from the curse of dimensionality when paths high-dimensional.
We propose novel, trainable path development layer, which exploits representations of sequential data through dimite-dimensional Lie groups, thus resulting in dimension reduction. Its backpropagation algorithm is designed via optimization on manifolds. Our proposed layer, analogous to recurrent neural networks (RNN), possesses an explicit, simple recurrent unit that alleviates the gradient issues.
Our layer demonstrates its strength in irregular time series modelling. Empirical results on a range of datasets show that the development layer consistently and significantly outperforms signature features on accuracy and dimensionality. The compact hybrid model (stacking one- layer LSTM with the development layer) achieves state-of-the-ar against various RNN and continuous time series models. Our layer also enhances the performance of modelling dynamics constrained to Lie groups. Code is available at /https://githun www.a/b..coo//eeeeee///eeelleeee git.
1 Introduction
Signature-baeed methods are an emerging tool for the modelling of time series data. The signature a path, originated from rough path theory stochastic analysis (ef. Lyons et al. (2007): Couttnn & Qian (2002) and the many references cited therein), has shown promisc as an efficient feature representation of time series data, facilitating prediction performance when combined with suitable machine learning models, e.g., deep learning and tree- based models (Xie et al. (2017); Arribas et al. (2018); Morrill et al. (2019)).
The signature of path X [0, + Rd is the central concept in the theory of rough paths, which aims at providing rigorous mathematical lools for defining and analysing solutions to the controlled differential equations (CDE) driven by ascillatory paths rougher than semimartingales. Such solutions are of very low regularity general, hence have remained impenetrable via classical analytical tools for CDE. The celobratod theory of regularity structure (Hairer) (2014)) underpins the theoretical contribution of the rough path theory in pure mathematics. Roughly speaking, the signature of path serves a prineipled feature that offers

---

Published in Transactions on Machine Learning Research (08/2024)
top-down description of the path. Just the monomial basis of R", the signature viewed their noncommutative analogue - constitutes a basis for the path space. More specifically, the signature of X is defined as an infinite sequence (1, (o.T)>"
x
(k) (o.T)...)) where, providing that the integrals are well defined,
X 0.71 I
JostrettusT dXe, @ .... dXu.
We refer the reader to Section 2.1 and Appendix [ for the precise definition signature of paths of bounded variation (BV-paths). See e.g. Lyons (2014) and the many references cited therein for the general case of paths of finite p-variation; p
In applications to time series analysis, the signature is a mathematically principled feature representation, in contrast to certain deep learning-basse models that are computationally expensive and difficult to interpret. The signature is universal and enjoys desirable analytic-geometric properties (cf. Lyons (2014); Levin et al. (2013)) ; hence can be used as a deterministic, pluggable layer in neural networks (Kidger et al. (2019) Nevertheless, the signature encompasses three major challenges practice:
It suffers from the curse of dimensionality dimension of the truncated signature up to the Ath term, iee. d' dh=1_1 grows geometrically in the path dimension d.
d-1
It is not data-adaptive, and hence appears ineffective in cortain learning tasks.
It incurs potential information loss due to finite truncation of the signature feature.
The main objective of this paper is to address the above issues. We propose a novel trainable feature representation of time series, termed as the path development layer, which is mathematically principled, data- adaptive, and suitable for high-dimensional time series. The theoretical foundation of the path development layer proposed in our paper is rooted in the concept of the development of a path (a.ka. Cartan development; see, e.g., Driver 1995)). It has recently been explored in theoretical studies of rough paths, especially for signature inversion (Lyons & Xu (2017)) and uniqueness of signature (Boedihardjo Geng (020)) Hambly & Lyons [210 Chevyrev Lyons (2016)).
One way introduce the development of path X Rd is as follows. Consider matrix Lie group G. It shall serve the range for the development. Then let M be linear transform from Rd to g, the Lie algebra of G. The path development of X can be viewed as generating function of the signature:
M -
M (x ()))),
k20
where M (c) & 02 8 ... 2 vn) = M(vi) (12) ..... M(vn) with U, Rd for i E {1,... K)) The product is
the matrix multiplication.
An important remark is in order. As we are working with specific matrix Lie groups G' and the corresponding Lie algebras g, it is natural to view both G and as subsets of gi(m; R), the space of m x m real matrices, i.e. the Lie algebra of the general linear group GL(m;R), which consists of invertible m x m matrices. This viewpoint significantly simplifies our theories practice.
The preservation of favourable geometric and analytic properties makes the path development promising feature representation of sequential data. Specifically, non-commutativity of the multiplication in & reflects the irreversibility of the order of events. It has been established in Chevyrev & Lyons (2016) that, with suitably chosen, the path development constitutes universal and characteristic features. Moreover, in rantrast to the infinite dimensionality the hath signature, the development takes values in finiteddimensional Lie groups with dimensions independent of the path dimension d.
(x x
Motivated by the discussions above, we construct the path development layer, which the central objective of this paper. This layer transforms any sequential data (2o, ,NN) [Nx(N+)) to the path development.
2

---

Published in Transactions on Machine Learning Research (08/2024)
1dm 2
2,3
Zeo
Ze,
Zzz
Zis
Output: (2ex) matrix Lie Group
e trainable Lie algebra weights
dY, (dX,)
AX
AX
AX
Input: Data/Path (Xe).
Figure 1: A high-level summary of the proposed development layer. Output and trainable weights take values in the matrix Lie group G and Lie algebra g, respectively. (Left) It can be interpreted as solution to the linear controlled differential equation driven by driving path (Definition 2.2 which is continuous lift of sequential data. (Right) It can be viewed as an analogy of the RNNs, but with a simpler form (Eq. 11
2 = (2o, ... nN) under a trainable linear map Mo : Rd > g. where each 2; lies in the Lie group G. For cach n E (0, .. NJ we set
1 := 2, exp (Mg zn)), 20 Idm-
(1)
where the model parameter, Idm is the identity matrix, and exp the matrix exponential. possesses a recurrence structure analogous to that of RNNs.
To optimise the model parameters of the development layer, exploit the recurrence structure in Eq. (1 and the Lie group-valued output to design an efficient gradient- based optimisation method. We combine backpropagation through time RNNs and "trivialisation", an optimisation method on manifolds [Lezcano Casado (2019)). ) In particular, when g is the Lie algebra of the orthogonal group, we can establish boundedness of the gradient. This alleviates the gradient vanishing/explldingg problems of backpropagation through time, thus leading more stable training process.
To the best of our knowledge, this paper is the first of the kind to
construct trainable layer based on the path development; and
design, taking into account adequate Lic group structures, the backpropagation of the development layer.
Key ad advantages of our proposed path development layer include the following:
1. it provides mathematically principled features that are characteristic and universal (Theorem 2.1 and Theorem B.1
is dala-adaptive, trainable layer pluggable into general neural network architectures;
is applicable to high-dimensional time series;
4. it helps stabilise the training process; and
can model the dynamics on oon-Euclidean spaces by exploiting appropriate Lie group structure.
Numerical results reported in this paper validate the efficacy of the development layer in comparison to the signature layer and several other continuous models. Morcover, the hybrid model obtained by stacking together LSTM with the development layer consistently achieves outstanding performance with more stable training processes and less need for hyper-parameter tuning. We also provide toy examples for simulated
3

---

Published in Transactions on Machine Learning Research (08/2024)
Brownian motions on S and N-body motions. They serve as evidence for the idea that equivariance structures inherent. to learning tasks can be effectively incorporated into the development model by properly choosing Lie groups to result in performance boost. This may offer a novel, promising class of models based on development modules for time series on manifolds or trajectorics of moving data clouds.
1.1 Related works
Recurrent Neural Networks (RNNSs RNNs and their variants are popular models for sequential data, which achieve superior empirical performance on various time series tasks (Hochreiter Schmidhuber (1997);
Cho et al. (2014); Bai et al. (2018)). ). They show excellent capacity for capturing lemporal dynamic behaviour, thanks to the recurrence structure of hidden states. However, they also prone to difficulty in capturing long-term temporal dependency and issues of banishing/explodiinn gradients (Bengio et al. (1994). Restricting weight matrices of RNNs to the unitary group U(n) may circumvent such issues, as shown in Arjovsky et al.
(2022). The development layer propased in this work possesses recurrence structures similar to those of the RNNs (soc Eq. ([)), but in an explicit and much simpler form. Furthermore, the outputs of our layer are in the suitably chosen Lie groups, which has positive effects on modelling long-ten long-term temporal dependency and stabilising the training processes.
(016): Lezcano-Casado (2019); Kiani et al.
Geometric Deep Learning (GDL). Recent advances in GDL have gained enormous attention by extending neural networks to handle complex, non-Euclidean (e.g., manifold-valued) data. See Monti et al. (017
Bronstein et al. 2017]; Cao et al. 2020p. A notable example is Riemannian ResNet in Katsman et al. 2024
which extends the construction of Residual Neural Network (ResNet) to general Riemannian manifolds.
Our proposed development resembles Riemannian ResNet - both employ the exponential map and exhibit recurrence structures. However, the recurrence of Riemannian ResNet lies in two consecutive layers, whereas the recurrence occurs between two consecutive times in single development layer. Also, in contrast Ricmannian ResNet, the development layer is designed specifically for times series input.
State space models (SSMs). State space models, originated from the approximation of linear dynamical systems, serve as sequence layer that can be stacked for various time series tasks (Rangapuram et al. (2018)). More recently, the structured state space sequence ("S4") layers and its simplified version "S5" have been introduced by imposing new paramelerisation of linear coefficient matrices in SSM and utilising low-rank approximation. The S4 and S5 lead to significant improvement on computational efficiency and stato-of-the-art performance on long-range sequence modeling tasks. See Gu et al. (2021a): [Smihh et al. (2022). Analogous to the SSM models, the development layer proposed in our work also originated from linear differential equations. Nonethe the output of our development layer takes values in matrix groups at each time, while the output of SSMs are vector-valued. The geometric structure of the matrix Lie groups prove to be crucial, from both theoretical and algorithmic perspectives, for the development layer proposed in this paper.
Continuous time series modelling. Continuous time series models have attracted increasing attention due to their strengths on treating irregular time scries. Popular differential cquation-inspired models include Neural ODEs (Haber & Ruthotto (2017); Chen et al. (2018)), Neural SDEs (Liu et al. (2019)) ), and Neural CDEs (controlled differential equations: Kidger et al. (2020); Morrill et al. (2021)). The model based on path signature and development we propose here, similar in spirit to the above, takes a continuous perspective it embeds discrete time scries to the path space and solves for lincar CDEs driven by the path. We shall show its advantages in coping with time series which are irregularly sampled, of variable length, and/or invariant under time reparameterisation. Notably, recent work has also extended Neural ODEs to handle other data types, such as graph data Gravina et al. Eliasof et al. (2024).
1.2 Organisation of the paper
The remaining parts of the paper are organised as follows: In $ we collect some background materials on path signature, path development, and optimisation on manifolds. In S we propose our path development layer. Algorithms for forward/bacde backward pass are described in detail with rigorous mathematical justification. Next, in $4 numerical experiments sequential data. imaging and dynamics manifolds reported. Brief
4

---

Published in Transactions on Machine Learning Research (08/2024)
concluding remarks given in proofs and experimental details.
$
The four appendices at the end of the paper present various mathematical
2
Preliminaries
2.1
Path Signature
We present here self-contained, brief summary of the path signature, which can be used as principled, efficient. feature of time series data. See Lyons et al. (2007) Levin et al. (2013): Chevyrev & Kormilitzin 2016): Kidger et al. (2019) and the references cited therein.
Denote by Vi (10, 7), R2)) the space of continuous paths on I&d of finite length. Write T ((R²) = Dz20 (Rd) 994 for the tensor algebra equipped with tensor product and component-wis addition. Signature takes values in T (RR)), and its zeroth component is always 1.
Definition 2.1 (Path Signature). Let J C [0, T) be compact interval and X E Vi (10, 7J, R2)) The signature of X over is defined as
S(X), = (1, (1,
where X
dXa, 2 ....8 dx for each 2 as Räemann -tinnnt--tieetees integrals.
a1 NISK EJ
The truncated signature of X of order is (X), == X3, ,,)). Its dimension is Σ d' d++i_ which grows exponentially in k. The signature can be regarded as a non-commutative version of the exponential map defined on the space of paths. Indeed, consider exp IR + R, exp(t) = e' = E सí it is the unique C*. solution to the linear differential equation d cxp(t) cxp Analogously, the signature map t S(X)o. solves the following linear differential equation:
d
dS(()o, S(X),, @ dX,,
= = 1= (1,0,0.... 111,0.-1:
(2)
The range of signature of all paths in Vi (10, TJ, is denoted S 511 ([.,T..R.)
The signature of path is faithful and universal feature representation and enjoys favourable algebraic and analytic properties, e.g. multiplicative property, characteristic property, and time invariance, etc. These propertics distinguish the signature as a useful fcature set for time serics (ef. [yons et al. (2007); Levin et al. (2013) Chevyrev & Kormilitzin (2016); Kidger el al. (2019); see also Appendix|A) A).
2.2 Path Development on matrix Lie groups
Let G be a finite-dimensional Lie group with Lie algebra Assume throughout this paper that is a matrix Lie algebra, namely that a Lie subalgebra of gl(m; F), such as the following:
gl(m; F) = {m x m matricos over F} = Fmxm
so (m, R) (m,, R) = {A pmxm m T = 0}
A Jm + Jm 0} 62m, C) : {A 2mmx 2m
su C) = {A E C
tr(A) 0, 4" + A = 0}
1m In addition, for vector spaces V1, V2, denote Here and hereafter, IR or C, and Jm : (-9 -Im 0
by L(V1, 12) the space of linear transforms V1 V2. For gl(m;F) (hence its subsets) we always take the Hilbert-Schmid norm:
|||||| == VIT(A**)
Σ A
f,==1
5

---

Published in Transactions on Machine Learning Research (08/2024)
Remark 2.1. The Lie group Sp(2m, C) corresponding to the algebra of symplectic matrices sp(2m, C) is not compact. In this work we shall somelimes work with
Sp(2m, R) == Sp(2m, C) n (2m,, C)
known the compact symplectic group. It is compact real form of Sp(2m, C), namely that Sp(2m, IR) is compact real Lie group whose Lie algebra t satisfies e := f &R C sp(2m, C).
Definition 2.2 (Path Development). Fir integer m 2 1. Let M Rd g C gl(m;; F) be linear map and let X E Vi (10, [1, be path. The path development (a.ka. the Cartan development) of X G under M is the solution to the equation
dZ, Ze MdX,)) for all t [0, with Zo = e,
(3)
where G is the group identity and is the matrix maltiplication..
Write Das (X) for the endpoint Zr of the path development of X under M.
Example 2.1. For lincar path X Vi (1,, R"),, its development matriz Lie group G under M E L(R",g) g) is
(M),) = exp (M( - -Xo)).
This is because t + cxp(M(XX Xo)) is the unique solution to (3)
Lemma 2.1 (Multiplicative property of path development).
Let X E V1 (I0, s), Rd) and Y V1 (1s, t), Denole by X Y their coneatenation: (X ()))) X(v) for u [0, 8] and Y(u) Y(s) + X(s) for [s,
Then DM (X Y) (Y) for all M L ())
By Lemma 2.1 and Example 2.1 the development of piecewise linear paths can be analytically computed. In the example below, the development takes value in the (isometry group of the) hyperboloid. It is a useful tool for studying uniqueness of signature and expected signature (Hambly & Lyons 2010); Boedihardjo et al.
2021
Li Ni 2022)
1002 x y/
Definition 2.3 (Hyperbolic Development). Let M R so(1,2) be the map M (x. v)
0
where 20(1, is the Lie algebra of the group of orientation- isometries of the hyperbolic plane H in the hyperboloid model:
H : (x1)² 2 ((xs)² -1, Iz 0}.
Set p (0, 0, 1)" HP? The hyperbolic development of X is the path
t DMs(X)oo
The path development shares several properties with signature (summarised below), which are relevant and useful to applications in machine learning. Proofs for Lemma 2.2 and Lemma 2.3 will be provided Appendix B The comparison between signature and development will be given the end of this subsection.
For M E L (Rd, gl(m;;))) we have the canonical extension:
M E L (T(RR)))
M (ci, C22 @ ci = M(e,) M(ee.) ((e.)))......((.))
(4)
Here (ex) is the standard Cartesian basis for R2d and is the matrix multiplication.
Lemma 2.2 (Link with signature). Let E Vi (I0, 71, Rd) be a path and M LRR,, g) be a linear transform.
Then Da (x) M(S()).
The development of X can be thought of as M + Exso M (S(X))) the "generating function" of S(X).
Here Te is the projection onto the 4th level of tensor algebra element.

---

Published in Transactions on Machine Learning Research (08/2024)
piecenie path Octy.8
Hyperbalic denelopmant with 1000000
25
04
0.3
0.2
01
-025 000 025 030
00
Figure Left panel: 2-dimensional piecewise linear path X; Right panel: the hyperbolic development (M).).
Lemma 2.3 (Invariance under timnereparrmmetiiatton)) Let E V1 (10, Rd) and be non-deccreaingg C³ diddiffeomorphiim from [o, 7) onto S). Define Xi =xxx for 1 E [0, T-- Then for all M E L (IR g) and
8, E [), we have Dai (X (x DM (X).a): where [(e.)] denotes X restricted to the interual [s. 4]-
Remark 2.2. Lemma 2.3 shows that path development, similar to path signature, is invariant under time re-
parumeeerisation. Thus, the development feature can remove the redundancy caused by the speed of traversing the path, hence bringing about massive dimension reduction and robustness effects online handwritten character recognition and haman action recognition, among other tasks. If, however, the speed information is relevant to the prediction laski one may simply add the time dimension to the input dala.
When choosing adequate Lie algebras, the space of path developments constitutes a rich enough model space to approximate contimuous functionals on the signature space.
Theorem 2.1 (Characteristic property of path development, Theorem 4.8 in Chevyrev & Lyons 2016 Let æ (zo, ...) E T ((RR)) such that Is * 0 for some k 2 O. For any m 2 max k/3), there erists
M [Ra, sp(m, C)) such that Mxx) # O. In particular,
(Vi Dep ( (Vi ((O, ((,T)) = U mmLm
M Me (R" sp C
separates points over S (Vi (J,
Remark 2.3. Theorem 2.1 shows that if two signatures differ at 1tth level, one can find M : T ((R4) - G
that separates the two signatures, whose resulting development has dimension S (max((, })) This number is typically much smaller than the dimension of the truncated signature up to degree k, namely d-1
1+1_1
Chevyrev Lyons (2016) also proved that the unitary representation of path development is universal approximate continuous functionals on the signature space. One may refer to Theorem B.] in the Appendix.
2.3 Optimisation on Lie Group
Now we briefly discuss a gradient-basod optimisation method on Riemannian manifolds introduced Lezcano- Casado (2019 Pezcano-Casado Martuez--ubbl 2019). We shall focus on matrix Lie groups; see
Sizcano-Casade 2019 for general manifolds.
Definition 2.4 (Trivialisation). Let G be matria Lie group. A surjection o Rdd G is said to be
trivialisation.
Trivialisations allow us to reduce an optimisation problem constrained on a manifold to an unconstrained problem on a vector space, namely that
min f(x)
min fó(()). aER
#EG

---

Published in Transactions on Machine Learning Research (08/2024)
The right-hand side can be solved numerically by gradient descent once V(f o Φ) is computed.
For a compact connected matrix group G with Lie algebra the matrix exponential exp serves as the "canonical" trivialisation. For the noncompact case where exp is not surjective, it can still be used as a "local trivialisation.. See Theorem 4.7 in LezcanooCasadee (2019).
The formulae in Theorem 2.2 below enable fast numerical implementation of the gradient V(f exp) over matrix groups.
Theorem 2.2. Let f gl(m; C) = Cmxm + R be scalar function, let exp be the matric exponential, and let A be any matrix in gi(m;; C). We have the following
formula for gradient
LezaanoCasaad (2019)):
V( exx) = (dexp)AT (VS(exp(A)))
formula for the differential of exponential
Rossmann 2002));
(dexp)a(X)) = Σ ad(A))
(k+ I1) (cxp(A)x),
where ad is the adjoint action ad(X)Y XY YX.
A variant of the first formula in Theorem 2.2 particularly useful for our numerical experiments in $ Here and hereafter, denote the nearest point projection from gl(m,, C) to given Lie subalgebra g
Proj: gl(m,, C) +
(5)
For instance, Proj(M) (M - (M-M)) for g= so(m, R). We then have
Lemma 2.4 (Gradient on matrix Lie algebra). Let f gl(m, C) - R be a smooth scalar field and Proj gi(m, C) + g be the projection as before. Then
V (A)j()) = Projfdar exp (v(J (xxe)))((
where A gl(m; C) and Proj(A). Here we use exp to denote both the exponential maps g + G and gl(m, C) - GL(m,,C)
Proof. For each gi(m, C) it holds that
V o expo Proj(VMM[J oex))),
where M = Proj(A) E C gl(m, C). Morcover,
VM exp) = dur exp (V(feexp(M))
due to Proposition 6.1 in Lezcano-Casado 2019
3 Path Development Layer
The main objective of this paper to propose a novel neural network layer, termed as the path development layer. It is generic, trainable module for time series data.
8

---

Published in Transactions on Machine Learning Research (08/2024)
3.1 Network architecture
In practice, one oflen observes discrete time series. Let x (zo. ,NN E Rdx(N+1) be d-dimensional time series. Once the Lie algebra g and M E L(R g) are specified, one can define the development of æ under M via the continuous lincar interpolation of x. We parameterise the linear map M E L(R g) of the development Da by its linear coefficients 0 E g". That is, given (01, .00.:..0))) E g", define
Mo Mo (..)).... vl1))
00 E
The path development layer will be constructed below in a recursive fashion. Definition 3.1 (Path development layer). Fir a malric group with Lie algebra g. The path development layer defined as a map Dø [N+( + "N+ G, resp.] I = (zo. IN) + 2 (20. , [or nn, resp.] such that for each E NN.),,,
in+1 = 2n exp (Mo - -x)),, 20 = Idm
(6)
Here exp is the matriz exponential, GN+1 is the (N + 1)-fold Cartesian product of G, and A e gd constitutes trainable model weights.
The path development layer is designed to take the form of Eq. 6 mainly for two reasons: (1), the multiplicative property of development (Lemma 2.1]; (2) the explicit solution for linear paths (Example 2.1 Notice too that the output of development at step admits an analytic formula:
= exp(Mo(21 zo)) exp(Mo(x2 - x1)) exx((M - xn-1)) EG.
Remark 3.1 (Connection with RNN). The recurrence structure of the development in Eq. 6 resembles that of the recurtent neural network (RNN); the development oulpul in plays the role as the hidden neuron of RNNs ha. In contrast, our path development layer does not require fully connected neural network to construct the hidden neurons.
EapRNN [Lezeano- Casado 2019)), a geometrie variant of RNN model, also possesses the buill-in Lie group structure and restricts the model parometers to the orthogonal Lie group in the recurrence relation hn+1 = (0h,, TTn41). The hidden neurons of EapRNNs, as opposed to the path development layer, may fail to live in the Lie group.
Remark 3.2 (Comparison with path signature layer). The signature layer of the path maps time series x
[20: ,NN) to s = (so.... ,8)) according to the sollowing equation, which is of a similar form to
Eq.
6
Sp Sn-1 @ exp(xn+1 - xn) for all n E (E,.
50 1
In comparison with the delerménistic signature layer, the path development has trainable weighls. When choosing the matrix Lie algebra g C. F), the resulting development feature lies in GL(m, Note that dimy GL(m, F)) = m², independently of the path dimension d. This is in stark contrast with the signature, whose range (al bruncated order n) has dimension depending geometrically on d.
The development layer allows for both sequential output (z)) NN++ and static output nn G. Its forward evaluation is summarised in Algorithm I below.
Algorithm Forward Pass of Path Development Layer
1: Input: E gd C mxxmx (model parameters), x (Nu,... ,NN) E (N+(1++1) (input time scries), m E N
(order of the matrix Lie algebra), (d, N) are the feature and time dimensions of æ, respectively.
2: 2p <- Id
3: for n E (1.....N N} do
2n t- Zn-1 exp(MM(xx xn-1))
end for Output: (o...,,2N) N+1 m=xmx(N+1) (sequential output) 2N G C R mx (static
output).
9

---

Published in Transactions on Machine Learning Research (08/2024)
3.2 Model parameter optimisation
We shall utilise method introduced in Lezcano-Casado (2019); Lescano-Casado & Martmez-Rubio (2019) to optimise model parameters of the development layer, which effectively leverages the Lie group-valued outputs and facilitates efficient gradient computation by exploring the recurrence structure via backpropagation through time.
More specifically, consider scalar field v GN+1 + R and an input x:= == (Ir) (Re))++ The goal is to seek optimal parameters 0" minimising (D)((1)); in formula,
N+1
0 argmingg ((D((s)),
where Dg is the development layer in Eq. 6
Due to the Lie group structure the output, the gradient descent Euclidean spaces not directly applicable here. We adapt the method of trivialisation Cezzano-Casadee (2019); Lezcano-Casaan & Martmez- (2019)) to do gradient computations. Taking into account the recurrence structure of the development, we express the gradients of the development layer in form similar to the Recurrent Neural network, and thus implement the corresponding backpropagation through time algorithm for parameter optimisation.
To describe the optimisation method on Lie groups in full detail, first let us fix some notations. Recall that 2 =: (z, ) E GN+1 denotes the output of the development layer Do, whose input is x = (xn) E Rdx((N+1), The variables 2n have the recursive structure on = Zn-1 exp(MM( (Ax,)); )); see Eq. We introduce the Step-i update function:
G: : G x g" - G, (=, exp( (Mop())((m))))
The output of development layer (=) can be expressed by the update function:
2 (=
f) for each i E (1......)
20 Id
When there is no ambiguity, we simply write 6,(2-1) for 6,(2--1.)).
Denote by pr, the projection of GN+ onto the 7th coordinate. This notation selected distinguish it from Ty: the level truncation of the signature (Eq. (8), the projection Proj from square matrices to g (Eq. (5). For scalar field vb GN++1 R, the correct definition for its "2th partial derivative" is (pr;) dib, the pushforward of dø via pr,
(pr ) deb Tr,,(>) - TeO for each = E GN+1
For each n E {0, 1.......,
N --1 1} we also introduce the function Gh : G + R as follows:
(2n(2) v6 ((20) ...: Zn. Ga41(Zn), G,+2 Gn+1(Zn), GN (nt1(
(7)
Recall from 6 that Zns the output of development layer, is defined recursively with respect to n. As a consequence, the exterior differential of V. also has a recursive structure.
Theorem 3.1 (Recursive structure of the differential at 2n). Lel x (an, ... ,N)) E (IR N+1 and (zo...., ,2N GN+ be the input and output of the development layer as before, where G is a matriz Lie group. Let 1): N+1 + be a smooth scalar field and th be as in Eq. For each n E {1.-... ..).., .N} we can express the 1-form dv., on G as follows:
(zn.E) -)) + draei Sotzi (5 2nt1) for each g. d,, En - 2n) (pr,) 4 del (zn
Here Ar == Int1 - Fn and is the matric multiplication.
In the above dea Tz,C0 G + T~ T
A generic element of the domain T G takes the form 2n . dC., (E) for & E e == TaaG, where L... G + + G is the left multiplication. One may view dib, as a 1-form G or, equivalently,
element. of g"
10

---

Furthermore, the Published in Transactions on Machine Learning Research (08/2024)
By virtue of Theorem 3.1 and the duality between gradient and differential, are able to determine the gradient of Vn, which is the main ingredient, of the Riemannian gradient descent algorithm of the development layer (i.e., Algorithm
2
below). Before further developments, let us first comment on the Riemannian gradient
on matrix Lie groups.
Remark 3.3. Given scalar field f (M.9) + E, recall that its gradient Vf is the vector field on M determined by
VVV..V) = df(v)
for any vector field V.
Throughout this paper, when M = gl(, for F or C, we always take the Riemannian metric given by the Hilbert Schmidt norm of matrices.
Proposition 3.1 (Gradient with respect to model parameter 0). Let E (R N+1 N+1 2 GNG++1 and 0 E g" be the input, output, and model parameter of the development layer Do (Rd) + GN+1, respectively. The gradient of + Do(x) for ony scalor field vb GN+1 R determined by the expression
V Do(s)) = Σ Vo v
where Gn is the Step-n update function and Un is the updated version of as in Eq. right-hand side can be computed via
Vo Vo(To Vn o Gn(=n-1,0)) = Proj (daaiisee)) exp {[v[[[V
Vee, vin (Mo(Aun))
=n-1))-
With the gradient computation at hand (in particular, Theorem 3.1 and Proposition 3.1 we are now ready to describe the backpropagation of development layer in Algorithm
Algorithm Backward Pass of Path Development Layer
1: Input: (ro, .N) E (NN+1)xd (input time series), (zo.... ,N) E GN+1 (output series
by the forward pass), 0 = (0,,-.-,00 E gd C m (model parameters), n E R (learning rate), Φ: GNN1 + R (loss function).
2: Initialize <- 0
represents da, Hn > w represents Vo (bb Do(x))
3: Initialize w + 0
4: for n {N, ,1] do
5: Compute m < exp (Mo(Ar,)
6:
Compute a + (prn) deb y=z +a aa-a
>> by Theorem 3.1
Compute w w dm exp(a) Arn
by Proposition 3.1 and Theorem
2.2
end for
0+00 0 1
10: for E ... d) do
0, < Proj (0,) Eg.
b by Equation
11: end for
12: Output: return 0 (updated model parameter).
One crucial remark is in order here. In view of Theorem 3.1 and Proposition 3.1 the development layer proposed in this paper possesses a recurrence structure analogous to that the RNNs. This is the key structural feature of Algorithm 2 However,
well known that the RNNs are, in general, prone problems of vanishing and/or exploding gradients (see Bengio al. (1994)). We emphasise that when G is the orthogonal unitary group, the gradient issues are naturally alleviated for the development layer.
Remark 3.4. The differential of the update function is an isometry when G S0(m) or U(m). That is, when the matria group G is equipped with the Hilbert Schmiddt. norm, dG; TG TG has operator norm
11

---

Published in Transactions on Machine Learning Research (08/2024)
for any i E 10, 1} and GN+1, (Here d6, is understood as d 0.) for fized 0; E g"; suppress the dependence on 0; for simplicity.) This is because.
dz, (L.(:(((2))44) 2; 1) exp (Mo., Ami+1) for any rj E
and, on the right-hand side, both the left medtiptication by 2s and the right multippicaaiion by exp (Mor Ari+1) e G are isometries for G O(m) or U (m). More concretely,
|| ((C,,, ||l|l for any g.
The implementation of the development layer can be flexibly adapted to general matrix Lie algebras via Proj in Eq. (5) The corresponding Lie groups used in our implementation include the special orthogonal, unitary, real symplectic, and special Euclidean groups, as well as the group of crientation-preserviig isometries of the hyperbolic space.
We adopt https://github www/Leecano/eee/eeeeccc expRNN for the PyTorrc implementation of matrix exponential and its differential in Al-Mohy & Higham 2009; 2010) to efficiently evaluate and train the development. layer. A araling-and-square trick for matrix exponential computation (Higham (2005) ) is used to facilitate stabilisation of model optimisation. To sum up, the computation of Algorithuss and 2 requires O(N) time and O(N) storage complexity (N = the length of time series). The complexity of Algorithm ] may be optimised by pre-computing the Lie group-valued path increments in parallel and then computing the path development output iteratively.
4 Numerical experiments
We begin this section with validating the model performance the proposed development layer (DEV) and the hybrid model, constructed by stacking L.TM with the development layer (DV) for general sequential data tasks in Section 4.1 We then proceed with two examples of simulated data to articulate its capability of modelling trajectories non-Euelidean spaces in Section 4.2 which may fail to be accomplished by RNN or its existing geometric variants (e.g., ExpRNN).
In the subsequent experiments, we consider the development with special orthogonal, real symplectic, and special Euclidean groups, denoted as DEV,SO), DEV(Sp), and DEV(SE) respectively. The standard loss functions are chosen: cross entropy for the time series classification tasks in Section 4.1 and mean squared error (MSE) for the regression tasks in Scction Full implementation details of the experiments can be found in Appendix [ Additionally, have included the relevant codes in the supplementary material to ensure reproducibility.
4.1 Time series and sequential image modelling
The path development network with suitably chosen matrix Lie group has a broad range of capabilities (e.g., capturing long-term dependencies, handling irregular time scries, improving training stability and convergence rale) for a wide range of applications (e.g. character trajectories, audio, images). This is demonsttated using the following datasets. (1) Character Trajectories [Bagnall et al. (2018)) (2) Speech Commands dataset.
(Warden (2018)). (3) Sequential MNIST et al. (2015)), permuted sequential MNIST (Le et al. (0015)). and sequential CIFAR-10 (Chang et al. (2017)).
We benchmark the proposed development models with various models, including RNN-based and/or continuous time series models (e.g., LSTM, NCDE, GRU-ODE) as baselines and state-of-the-art (SOTA) models for each task.
4.1.1 Speech Command dataset
We first demonstrate the performance of the path development network on the Speech Commands datasct. [Warden| (2018)) as an example of high-dimensional long time series. We follow Kidger et al. (2020) precompute frequeccyy cepstrum coefficients of the audio recording and consider balanced classification
12

---

Published in Transactions on Machine Learning Research (08/2024)
task of predicting a spoken word. This processed dataset consists of 34975 samples of secondly audio recordings, represented in regularly spaced time series of length 169 and path dimension 20.
Table 1: Test accuracies (%) of the linear models using multiple orthogonal path developments on the Speoch Commands dataset w.r.t. the different matrix sizes m and number of path development layers N.
Matrix size
Number of developments
70.0
77.3
81.5
83.4
10
81.4
83.8
84.3
85.2
20
84.4
86.0
86.5
85.6
Path development as a universal representation with dimension reduction. We apply linear modols on both path signature and path development to Speech Command and Character trajectorics datasots. Both features achieve high performance on their own, thus validating universality (see Theorem B.1p. As shown in Table ! increasing the number of developments or the matrix size leads to an improvement in classification accuracy in general. The only exception is that for matrix size 20, the accuracy is reduced by 0.9% if we increase the number of developments from 4 to 8, which may result from overfitting. In Figure the curve of test accuracy against feature dimension of the single development (Red) is consistently above that of the signature (Blue), demonstrating that development is a more compact feature than signature. The development layer with the matrix group of order 30 achieves an accuracy about 86% of that of the signature, but with dimension reduced by factor of 9 (900 v.s. 8420).
Model Development 8 Sgnature
DEPTH
80%
Test Accuracy
30%
20%
27
180
seoss
200000
Feature Dimension
Figure 3: Test accuracy v.s. the feature dimension curves of the linear models using the development and signature representation on Speech Commands dataset. See full results in Table
State-of-the-art performance. We first compare the proposed development model with other continuous time series model baselines, including GRU-ODE (De Brouwer et al. (2019)), ODE-RNN (Rubanova al. (2019)). ), and NCDE (Kidger et al. (2020)). Our model significantly outperforms GRU-ODE and ODE-RNN this task, whilst achieve comparable test accuracy with NCDE. The average accuracy of NCDE is 1.9% higher than our development model. However, the standard deviation of the NCDE model is 2.5%, higher than that of the development model (0.1%), thus making the accuracy difference of these two models insignificant. Note that NCDEs fully parameterize the vector field of the controlled differential equations (CDEs) by a neural network, while our proposed development satisfics linear CDEs with trainable weights constrained the matrix Lie algebra. The difference in complexities of the relevant CDEs may contribute to the higher average test accuracy and higher variation of the prediction by NCDE.
13

---

Published in Transactions on Machine Learning Research (08/2024)
Compared with the strong baseline LSTM, development layer alone has lower accuracy by 7.8%. However, our hybrid model (ISTM+DEV) further improves the LSTM performance by 0.9%. The LSTM-DDV model achieves 96.8% test accuracy, which is slightly lower than that of the state-of-the-ar FLEXTTN (Romero cet al. (0211a)). But, involving only a quarter of parameters the latter, our model is significantly more compact.
Table 2: Test accuracies (%) (mean std, computed across 5 runs) on Speech Commands dalaset
Model
Test accuracy(%)
# Params(K)
ODE RNN Rubanova et al. 2019
65.9 #35.6
89
NCDE al. 2020
89.8 12.5
89
LSTM
95.7 10.2
88
EXPRNN Lezcano-Casia Casado & Martmez- (2019
82.1
270
LipschitzRNN Erichson et al.
2020
88.38
270
CKConv Romero et al.
2021b
95.3
100
FlexTCN Romero et al.
2021a
97.7
373
S4 Gu et al.
2021a
95.3
260
LSSI et (2021b)
93.58
330
Signature DEV(SO) LSTM+DEV(SO)
85.7.10.1
84
87.9+0.1
87
96.8+0.1
86
4.1.2 Character Trajectories data
Next we investigate the performance of the development network on irregular time series in terms of accuracy, robustness and training stability. We consider the Character Trajectories from the UEA time series classification archive (Bagnall et al. (201))). This dataset consists of Latin alphabet characters written in single stroke with their 2-dimensional positions and pen tip force. There are 2858 time scries with a constant length of 182 and 20 unique characters to classify. Following the approach in Kidger al. (2020), randomly drop 30%, 50%, 70% of the data in the same way for every model and every repeat, which results in irregular sampled time series. In addition, we validate the robustness of model to timescale change following
Gu et al. 2020).
State-of-the-art accuracy. Similar to Speech Commands dataset, the development layer itself outperforms the signature layer, although still underperforms NCDE and L.TTM in terms of test accuracy. However, the proposed hybrid model (LSTM+DEV) with special orthogonal group achieves the statc-of-tho-art accuracy, with around 0.6-0.7% and 0-1.1% performance gain in comparison to the best NCDE and CKCNN models, respectively.
Robustness. The path development layer exhibits robustness and improves the robustness of LSTM in the following two settings: (1) data dropping; (2) timescale distribution shift.
Robustness to dropping data. Table demonstrates that, as observed for other continuous time series models, the accuracy of DEV and LSTM+DEV remains similar across different drop rates, thus exhibiting robustness against data dropping and irregular sampling of time series. LSTM achieves competitive accuracy of ~94% when the drop rate is relatively low, with drastic performance decrease (6%) for the high drop rate (70%). However, by adding the development layer, especially the one with the special orthogonal group, one significantly improves the test accuracy of LSTM by 11.4% to high drop rate (70%).
Robustness to timescale distribution shift. We follow the approach in Gu et al. (2020), where the train and test character trajectories are sampled at different timescales. Table 3 shows that, while all the baselines suffer from severe deterioration test accuracy in this sctting, the path development layer retains high performance, similar to HIPPO Gu et al. (2020)) and improves the accuracy of LSTM by ~60%. This is because the path development invariant under time-reparamettrrsationn (Lemma 2.3
14

---

Published in Transactions on Machine Learning Research (08/2024)
Table Test (c on the Character Trajectories dataset for different data dropping rate and time scale distribution shifts. means taken from our reproduced baseline models.
MODEL
TEST ACCURACY(%6)
TASK
DROP RATE
SAMPLING RATE
30%
50%
70%
1-+
1/2 1
1/2
GRU-ODE DE BROUWER ET AL. 2019
92.6 =1.6
$7.6 #3.9
89.9 =3.7
96.2
23.1
NCDE KIDGER
2020
98.8
98.6 =0.4
98.8
44.7
LSTM
94.0 + 3.9
94.6 2.0
87.8
6.3
91.3
31.9
28.2
EXPRNN LEZCANO-CASE & MARTINEZ RUBIO 2019
95.8 + 0.7
96.6 0.9
95. 95.640.7
97.2*
17.3+
11.0s
CKCNNN
ROMERO ET AL.
2021B
99.3
98.8
98.1
HIPPO GT RT AT. 2020
88.8
90.1
SIGNATURE
92.1 =0.2
92.3 + 0.5
90.2+:0.4
92.9
92.3
91.5
DEV (SO)
92.7+ 01.6
92.6+0.9
91.9 +0.9
93.2
91.6
92.3
LSV (SO)
99.3# 0.3
99.3 0.1
99.24 0.3
99.5
94.6
91.3
LSV (Sp)
97.74 0.2
97.7 + 0.2
97.44 0.4
97.9
94.2
89.3
Figure 4: The comparison plot of LSTM and LSTM+DEV on Character Trajectories with 30% drop rate. (Left) The evolution of validation loss against training time. (Middle) The evolution of validation accuracy against training time. The mean curve with + std indicated by the shaded area is computed over runs. (Right) The boxplot of the validation accuracy for varying learning rate.
1.0
1.0
0.9
0.8
2.0
0.8
validaton 1052
Lest eccurety
2.5
0.6
0.7
M
1.0
0.5
0.4
0.3
training seconds
Improvement of training LSTMs by Development Layer (LSTM+DEV). Apart from model robustness and compactness, the additional development layer improves the LSTM in terms of (1), training stability and convergence; and (2), less need for hyper-parameder tuning. This has led to better model performances with faster training.
Training stability and convergence. The left and middle subplots of Figure g show the oscillatory evolution of the loss and accuracy the LSTM model during training, which can be significantly stabilised by adding DEV(SO) and DEV(Sp). In particular, the performance of DEV(SO)) is better than that of (EV(Sp) in terms of both training stability and prediction accuracy. It suggests that the development layer may facilitate the saturation of hidden states of LSTMs and resolving the gradient vanishing/ exploding issues, by virtue of the boundedness of gradients resulting from 20 1 of DEV(SO) (Remark 3.4). It can thus serve as effective means for alleviating long temporal dependency issues of LSTM. Morcover, our hybrid model has a much faster training convergence rale than LSTM.
vanishing/cxploddngg
2n|
Sensitivity of learning rate. Figure 4 (Right) shows that the LSTM performance highly sensitive to the learning rate. So, extensive, time-consuming hyper-parameter tuning is a must to ensure high performance of LSTM. However, the LSTM+DEV model training is much more robust to the learning rate, leading less hyper-parameter tuning and consistently superior performance over LSTM.
15

---

Published in Transactions on Machine Learning Research (08/2024)
4.1.3 Sequential image classification
We consider three time scries datasets of large scale for the sequential image classification: sequential MNIST et al. (2015)); permuted sequential MNIST (Le et al. (2015)), and sequential CIFAR-10 (Chang et al. (0017)), consisting of sequences of pixel-by-pixe with lengths 784 and 1024, respectively. Each has 50000 training samples and 10000 test samples.
Accuracy comparison and model compactness. On the sequential MNIST and CIFAR-10 datasets, stacking a path development layer of a 10 x 10 matrix to the LSTM significantly increases the performance of the baseline LSTM by 7-11%. Among RNN-based models, the hybrid model achieves similar results as the state-of-the-art Lipschitz RNN model, but with only 1/5 model parameters.
In comparison with the state-of-the-art models FlexTCN (Romero et al. (2021a)), S4 (Gu el al. (2021a) ), and LSSL (Gu et al. 2021b)), the proposed LSTM-DEV model slightly underperforms on the sequential MINST and permuted MNIST data by 0.7% and 2.6% respectively. However, note it only uses 72k model parameters compared with larger number of model parameters required for sota models, i.c., 375k (FlexTCN), 6000k (S4) and 200k (T.STT.). However, there is stiill a significant gap between LSTM+DEV and the stoa models on the Cifar10 dataset, which merits further investigation in future.
Table 4: Test accuracies (%) of the sequential image classification, indicates the results from reproduced baseline models
MODra
TEST ACCURACY (%)
TIME: (8/RPOCH)
MieMor((Ms)
PARAMS(K)
DATASET
=MNIST
PMNIST
sCIFAR-10
*MNIST/=CIFAR-ID
LST BALET AL
2018
87.2
85.7
57.6s
25/29-
1638/1798
70
L-STAM TINH ET
2018
98.
95.2
72.2
500
DTRIV PACRO- ASADO
4119
98.9
96.5
30.0+
120/131+
1887/2183+
130
EXPRNNLI LIZUANO- CASA
ARTENR RUmnO
2019
98.4
96.2
35.9+
119/125+
1887/2183+
130
LIPSCHFTRNN ERicHSON RT AL.
2020
99.1
96.3
64.2
260
CKCокV ROMERO BT Al..
2021
99.3
98.1
02.3
98
FLEXTCN ROMERO ET
20211
99.6
98.6
80.8
375
S4 27 AL. 1411A
99.6
98.7
91.1
6000
LSSICG
6
99.5
98.8
84.7
200 22000
LSTM DEV(SO)
98.9tt 0.2
96.2 0.3
64.31 0.9
62/63
1957/2207
72
Training time & Memory. The training time of the development layer is comparable to that of LSTM due to the similar recurrent structure. The DEV layer training is times faster than the continuous-ting time series baseline NCDE Kidger et al. (2020)). ). The training time and memory consumption of LSTM+DEV(SO) sequential image data are reported in Table 4 The memory of LEVS+)EV(SO) only increases roughly 25% compared to the plain LSTM, with about doubled training time. In comparison to ExpRNN, a geometric RNN model alleviate gradient issues of RNNs, LSTM+DEV has only half of the model parameters but outperforms consistently, especially in sequential CIFAR-10 data.
4.2 Modelling dynamics on non-Euclidean spaces
Brownian motion on $ and N-body motions are simulated to demonstrate the efficacy of the development network, taking into account the group structure. Our DSTM+DEV model, with Lie groups appropriately chosen, can constrain the prediction to prescribed non- non-Euclidean space and significantly boost the performance over baseline LSTM models.
4.2.1 Brownian motion on S
We simulated 20000 samples of the discretised Brownian motion B on the unit 2-sphere S with time length L = 500 and equal time spacing At = 2e by the random walk approach [Novikov et al. (2020)). ). The driving path for simulating B is a random walk X of length L RR.. The task of predicting the Brownian motion from the given driving random walk is formulated as a requence-to-sqquen supervised learning problem with input X and output B. We benchmark our LSTM+DEV model with SO (eereeeerennnnnonnn
16

---

Published in Transactions on Machine Learning Research (08/2024)
LSTM_ DEV(SO)
ExpRNN
1.0
1.0
1.0
1.0
0.5
0.5
0.5
1.0
-10
1.0
1.0
1.0
0.5
0.5
0.5
0.5
-1.0-0.5
0.0
-1.0
0.0
-0.5
-1.0
0.0
-1.0-05
-1.0
0.0
0.0
0.0
y
0.0
-0.
y
0.5
-1.0
0.5
-1.0
0.5
-1.0
0.5
-10
Figure True and predicted sample trajectories of the Brownian motion on $2 on the test set. The test. MSE of LSTM+DEV(SO(3),, LSTM and ExpRNN are 0.0184, 0.109 and 0.11 respectively.
against the baselines LSTM and ExpRNN. Essentially, the DEV(SO(3)) layer is used as a final pooling layer to map the output trajectories to the constrained Lie group. See Appendix D.5 for details on data simulation and model architecture.
As illustrated in Figure 5 the proposed LSTM+ reduces the test MSE of both baselines by around 83% Its predicted trajectories always stay the sphere thanks to the built-in Lie group structure of the development output. It is highlighted that ExpRNN enforces model weights to the orthogonal group, but it can not constrain the output to live in the desired Lie group (see the red sample leaves from the sphere).
Table 5: Test MSE (mean + std, computed across runs) on the N-body simulation datasel
Test MSE (1e-3)
Prediction steps
100
300
500
Statie
7.79
61.1
LSTM
3.54+ 0.22
22.1 1.5
63.2 +1.4
LEV(SDE
5.50 0.01
47.5+0019
128+
LETM+DEV(SE(2)
0.505:0.01
6.94 0.16
25.6:t 1.5
4.2.2 N-body simulations
Following Kipf et al. (2018),
the simulated dataset of 5 charged particles and consider regression task of predicting the future k-step location of the particles. The input sequence consists of the location and velocity of particles in the past 500 steps. As coordinate transforms from the current location 2, to the future location &++ lie in SE(2) (the special Euclidean group), we propose lo use the LSTM+DEV with represttttttnntatiionn to model such transforms. Details of data simulation and model architecture can be found in Appendix DD66 To benchmark our proposed method, we consider two other baselines: (1) the current. location (static) and (2) the LSTM model. To justify the choice of the Lie algebra here, we compare our method with DEV(SO2))))
17

---

Published in Transactions on Machine Learning Research (08/2024)
LSTM
+STM+DEVISEE2)
832
X;
predicted Xi.s
predicted X1+
-0.25
-0.5
0.50
0.30
-1.0
-1.0
Figure 6: Prediction comparison between LSTM (left) and LSTM+DEV (SE(2)) (right) in the 5-body simulattons.
As shown in Table 5 the proposed LSTM-DEV(SEE2)) consistently and significantly outperforms both in terms of MSF for varying & (100,300, 500). Figure provides visual
baselines and LSTM demonstration to show the superior performance of the estimator of future location by our LSTM+DEV(SE(2)) in comparison to the strongest bascline LSTM, thus manifesting the efficacy of incorporating development layers with inherited group structures in learning tasks.
5
Discussions and future work
5.1
Vector field
The development layer has linear driving vector field (in terms of controlled differential equations, CDE), which enables us to derive explicit, analytic solution and use it for fast computation of the development layer. However, this may impose limitation on the development layer an efficient standalone model for modelling complex time series. In this regard, one may consider extending the development layer by parameterising the vector filld via a neural network taking values in the Lie algebma and computing it by solving a nonlinear CDE in analogy with NCDE (Kidger et al. (020
5.2
Lie group representation
Although our exposition focuses on matrix groups, our methodology and implementation of the development layer can be naturally extended general Lie groups. In view recent works on incorporating Lie group representations to neural networks (Fuchs al. (2020): Thomas al. (2018)), we expect that techniques developed therein may be utilised to increase the expressiveness of the development layer and further enhance its performance.
5.3 Hybrid model architecture
This work focuses the proposed hybrid LSTM+Dev model, showing superior performance various time scries learning tasks. However, there is scope to design hybrid models combining the path development layer with other lypes of neural works for modelling complex spatio-lemporal data, such as video or bkesed action sequences. The selection of suitable hybrid model architecture could potentially enhance performance by leveraging appropriate neural networks (c.g., Convolutional Neural Networks and Graph Neural Networks) capture spatial dependencies, while utilizing the path development layer extract temporal dynamics information.
18

---

Published in Transactions on Machine Learning Research (08/2024)
6 Conclusion
In this paper, we propose for sequential data tasks a novel, trainable path development layer with finite- dimensional matrix Lic groups built in. Empirical experiments demonstrate that the path development layer, by virtue of the accuracy and training stability of LSTM models. Based on the path development layer, a compact hybrid LSTM+DEV has been designed, which exhibits competitive performance and superior robustness. It may open up doors to efficient, accurate modelling and prediction of wide range of time series data with equivariance structures arising from the particular geometries of the underlying domains, such Hamiltonian dynamical systems, rigid body motion, and molecular dynamics, on various non-Euclidean spaces with nontrivial symmelry groups.
structures and recurrent nature, proves advanlageous
improving
Acknowledgments
HN supported by the EPSRC under the program grant EP/S026317/1 and the Alan Turing Institute under the EPSRC grant EP EP/N510129/L The research of SL supported by NSFC Project # #12201399, National Natural Science Foundation of China, and Shanghai Frontier Rescarch Institute for Modern Analysis. SL and the Royal Society CB5/NSFCC/233077 LH is supported by University College London
and HN are both supported by the SJTU-UCL joint seed fund WH610160507/ International Exchanges 2023 grant and the China Scholarship Council under the UCL-CSC scholarship (No. 201908060002). The authors extend their gratitude to Terry Lyons for insightful discussions. HN thanks Kevin Schlegel and Jiajie Tao for proofreading the paper. Moreover, HN grateful to Jing Liu for her help with Figure
References
Awad H Al-Mohy and Nicholas J Higham. Computing the Fréchet derivative of the matrix exponential, with application to condition number estimation. SIAM Journal Matriz Analysis and Applications, 30(4): 1639 1657, 2009.
Awad H Al-Mohy and Nicholas J Higham. A new scaling and squaring algorithm for the matrix exponential. SIAM Journal on Matriz Analysis and Applications, 31(3):970 989, 2010.
Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In International Conference Machine Learning, pp. 1120 1128, 2016.
Imanol Perez Arribas, Guy M Goodwin, John R. Geddes, Terry Lyons, and Kate EA Saunders. A signature- based machine learning model for distinguishing bipolar disorder and borderline personality disorder. Translational psychiatry, 8(1):1 7, 2018.
Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The UEA multivariate time series classification archive, 2018. arXio preprint arXiv:1811.00077 2018.
Shaojic Bai, J Zico Kolter, and Vladlen Koltun. An empirical ovaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint orXiv.1803.01977 2018.
Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependoncics with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157- 166, 1994.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL sattps://wwit wandb. www.d.commmo//////// Software available from wandb.com.
Horatio Boedihardjo and Xi Geng. SL (Lx(())-eveloommen and Signature Asymptotics for Planar Paths with Bounded Variation. arXiv preprint arXiv:2009. 13082, 2020.
Horatio Boedihardjo, Joscha Dichl, Mare Mezzarobba, and Hao Ni. The expected signature of brownian motion stopped on the boundary of circle has finite radius of convergence. Bulletin of the London Mathematical Sociely, 53(1):285 299, 2021.
19

---

Published in Transactions on Machine Learning Research (08/2024)
Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: going beyond euclidean data. IEEF Signal Processing Magazine, 34(4):18-44) 2017.
Wenming Cao, Zhiyue Yan, Zhiquan He, and Zhihai He. A comprehensive survey on geometric deep learning. IEFE Access, 1455929-55444 2020.
Shiyu Chang, Yang Zhang, Wei Han, Mo Yu, Xiaoxiao Guo, Wei Tan, Xiaodong Cui, Michael Withrock, Mark A Hasegawa-Johnson, and Thomas S Huang. Dilated recurrent neural networks. Advances in neural information processing systems, 30, 2017.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Proceedings of the 32nd International Conference on Newral Information Processing Systems, pp.
2018.
Ilya Chevyrev and Andrey Kormilitzin. A primer on the signature method in machine learning, aiv preprinl 1xXi::603.00788.. 2016.
Nly Chevyrev and Terry Lyons. Characteristic functions of measures on geometric rough paths. The Annals of Probability, 44(6):4049 4082, 2016.
Kyunghyun Cho, Bart Van Merriënboer, Damitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv: 1409.129, 2014.
Laure Coutin and Zhongmin Qian. Stochastic analysis, rough path analysis and fractional brownian motions. Probability theory and related fields, 122(1):108 140, 2002.
Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series. Advances in Neural Information Processing Systems 32, 2019.
Bruce K Driver. A primer on Ricmannian geometry and stochastic analysis on path spaces. University of California, San Diego, 1995.
Moshe Eliasof, Eldad Haber, and Eran Treister. Feature transportation improves graph neural networks In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 11874-11882, 2024.
N Benjamin Erichson, Omri Azencot, Alejandro Queiruga, Liam Hodgkinson, and Michael W Mahoney. Lipschitz recurrent neural networks. arXiv preprint vrXiv:2006.12007 2020.
Fabian B Fuchs, Daniel E Worrall, Volker Fischer, and Max Welling. trans-trrasformmrr:: 3d roto-translation equivariant attention networks. arXiv preprint XrXiv22006.10500 2020.
Alessio Gravina, Davide Bacciu, and Claudio Gallicchio. Anti-symmetric dgn: a stable architecture for deep graph networks. In The Eleventh International Conference on Learning Representations.
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with
optimal polynomial projections. Advances in Neural Information Processing Systems, 33:1474 1487, 2020.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces.
arXiu preprint arXi::211.000:: 2021a.
Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-time models with linear state space layers. Advances in Newral Information Processing Systems, 34, 2021b.
Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks. Inverse problems, 34(1):
014004, 2017.
Martin Hairer. A theory of regularity structures. Invent. Math., 198:269 504, 2014.
Ben Hambly and Terry Lyons. Uniqueness for the signature of a path of bounded variation and the reduced path group. Annals of Mathematies, pp. 109-167, 2010.
20

---

Published in Transactions on Machine Learning Research (08/2024)
Nicholas J Higham. The scaling and squaring method for the matrix exponential revisited. SIAM Journal Matric Analysis and Applications, 26(4):1179-11 1199,, 2005.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural compulation, 9(8):1735 1780, 1997.
Isay Katsman, Eric Chen, Sidhanth Holalkere, Anna Asch, Aaron Lou, Ser Nam Lim, and Christopher M
De Sa. Riemannian residual neural networks. Advances in Neural Information Processing Systems, 36, 2024.
Bobak Kiani, Randall Balestriero, Yann LeCun, and Seth Lloyd. projunn: efficient method for training deep networks with unitary matrices. Advances in Neural Information Processing Systems, 35:14448 14463, 2022.
Patrick Kidger, Patric Bonnier, Imanol Perez Arribas, Cristopher Salvi, and Terry Lyons. Deep signature transforms. Advances in Neural Information Processing Systems, 32, 2019.
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. foirth Conference on Neural Informalion Processing Systems, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprinl 141. 2014.
Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In International Conference on Machine Learning, pp. 2688 2697. PMLR, 2018.
Quoc V Le, Navdeep Jaitly, and Geoffrey E Hinton. A simple way to initialize recurrent networks of rectified linear units. arXiv preprint arXiv: 00944, 2015.
Daniel Levin, Terry Lyons, and Hao Ni. Learning from the past, predicting the statistics for the future, learning an evolving system. arXic preprint orXiw:1309. 0260, 2013.
Mario Sercano-Cassado Trivializations for gradient- based optimization on manifolds. Advances in Neural Information Processing Systems, 32:9157 9168, 2019.
Mario Lezzano-Casade and David Martmez--ubbe Rubio. Cheap orthogonal constraints neural networks: A simple parametrization of the orthogonal and unitary group. In International Conference on Machine Learning, pp. 3794-3803. PMLR, 2019.
Siran Li and Hao Ni. Expected signature of stopped brownian motion on d-dimensional c².0 -dommnns has finite radius of convergence everywhere: 2 S d 8. Journal of Funclional Analysis, 282(12):109447, 2022.
Xuanqing Liu, Si Si, Qin Cao, Sanjiv Kumar, and Cho-Jui Hsieh. Neural SDE: Stabilizing neural ODE networks with stochastic noise. arXio preprint arXiv: 02355, 2019.
Terry Lyons. Rough paths, signatures and the modelling of functions on streams. the Proceedings of the International Congress of Mathematicians 2014, Korea, 2014.
Terry Lyons and Weijun Xu. Hyperbolic development and inversion of signature. Journal of Functional Analysis, 272(7):2933) 2955, 2017.
Terry Lyons, Michael Caruana, and Thierry Lévy. Differential equalions driven by rough palhs. Springer, 2007.
Federico Monti, Davide Boscaïni, Jonathan Masci, Emanuele Rodola, Jan Svoboda, and Michael M Bronstein. Geometric deep learning on graphs and manifolds using mixture model cnns. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5115-5124, 2017.
James Morrill, Andrey Kormilitzin, Alejo Gevado-Holgade Sumanth Swaminathan, Sam Howison, and Terry Lyons. The signature- based model for early detection of sepsis from electronic health records the intensive care unit. In 2019 Computing in Cardiology (CinC), pp. Page-1. 1. IEFF, 2019.
21

---

Published in Transactions on Machine Learning Research (08/2024)
James Morrill, Cristopher Salvi, Patrick Kidger, and James Foster. Neural rough differential equations for long time series. In International Conference on Machine Learning, pp. 7829-7833 PML.R, 2021.
Alexei Novikov, Dmitri Kuzmin, and Omid Ahmadi. Random walk methods for monte carlo simulations of brownian diffusion sphere. Applied Mathematics and Compulation, 364:124670, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeriing Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-PPeformance Deep Learning Library. In Advances in Newral Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forccasting. Aduances in neural information processing systems, 31, 2018.
David W Romero, Robert-Jan Bruintjes, Jakub M Tomezak, Erik J Bekkers, Mark Hoogendoorn, and Jan C van Gemert. Flexconv: Continuous kernel convolutions with differentiable kernel sizes. arXiv preprint 100:21100.0000 10.00009, 2021a.
David W Romero, Anna Kuzina, Erik J Bekkers, Jakub M Tomczak, and Mark Hoogendoorn. Ckconv: Continuous kernel convolution for sequential data. arXiv preprint arXio:2102.0205 2021b.
Wulf Rossmann. Lie Groups: An Introduction Through Lineor Groups. Oxford graduate texts in mathematics;
5. Oxford University Press, 2002. ISBN 0198596839.
Yulia Rubanova, Ricky TQ Chen, and David Duvenaud. Latent ODEs for irregularly-sammpeed time series. Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 5320-5330, 2019.
Jimmy TH Smith, Andrew Warrington, and Scott Linderman. Simplified state space layers for sequence modeling. In The Fleventh International Conference on Learning Representations, 2022.
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and trauslation-eequivrriant neural networks for 3d point clouds. arXiv preprint arXiv.1802.08211 2018.
Trien Trinh, Andrew Dai, Thang Luong, and Quoe Lc. Learning longer-term dependencies in mns with auxiliary losses. In International Conference on Machine Learning, pp. 4965 4974. PMLR, 2018.
Pete Warden. Speech commands: dataset for limitad-vocabulary speech recognition. arXiv preprint XXXi::804400200 2018.
Zecheng Xie, Zenghui Sun, Lianwen Jin, Hao Ni, and Terry Lyons. Learning spatial-semantic context with fully convolutional recurrent network for online handwritten chinese text recognition. IEEE transactions on pattern analysis and machine intelligence, 1098):1903,, 2017.
22

---

Published in Transactions on Machine Learning Research (08/2024)
A
The signature of a path
In this section, we briefly introduce the signature of a path, which can be used as a principled and efficient feature of time series data Lyons et al. (2007): 5 Levin et al. (2013); Chevyrev & Kormilitzin (2016); Kidger
et al. 2019
Denote by V1 ()) the space of continuous paths [0, T Rd with finite length. Write T ((R*) for lensor algebra space:
T ((RR)) = (R") 000
k20
equipped with tensor product and component-wise addition. The signature takes values in T' ((R4)), and its zeroth component is always 1.
Definition A.1 (Path Signature). Let J C [0, T be compact interval and X E V1 [J.,T). The signature of X over J, denoted by S(X)s, is defined as follows:
S(X), (1,X), x3."..)
where for each k 2 1, X5
dX., 8 ... @ dX
The truncated signature of X of order & is
(1, (1,,, x
(8)
Its dimension equals Σ d' 1612 which grows exponentially in k. The range of signature of all paths in Vi (l0, TJ, IR.) is denoted as ((), TJ, Example A.1 (Linear paths). For a linear path X [0. 7] + RRd, its signature is given by
S(X), exp (XT Xo),
where exp is the tensor exponential. Equivalently, the h level of the signature equals
X (Xr Xo
Example A.2 (Piecewise linear paths). Let = (Io, aND E R be discrete d-dimensional time series. Embed it in the piecewise linear path X via linear interpolation. By the multiplicative properly, the signature of X is given by
S(X) = exp(x1 - xo) @ exp(22 x1) @ exp(xx IN-1).
For a general continuous path of bounded variation, its signature can be obtained by taking limits of signatures of their piecewise linear approximations as the time mesh tends to 0. For the more general case of paths of bounded p-variation (p 2 1), see Lyons et al. (2007).
The signature can be deemed as a non-commutative version of the exponential map defined on the space of paths. Indeed, consider exp : IR - R, exp(t) = e' = Σt= ; it is the unique -oosutntion to the linear differential equation d dexp(t)= = exp( Analogously, the signature map t ++ 5(Xo,) solves the following linear differential equation:
dSS()),, S(X),, @ dx,, S(X)o,o = (1,0,0....)
The signature provides a top-down description of the path. It serves as an attractive tool for machine learning in light of the properties manifested in the three results below.
23

---

Published in Transactions on Machine Learning Research (08/2024)
Lemma A.1 (Invariance under time reparamcterization; Lyons et al. (2007) ). Let X E V ((O.T) T1 Rd) and x: )0 ( T2] be non decreasing surjection and define X = Xx, d, for the representation of X under X. Then for every t E [0,
S(X)A,,) (
(10)
Thcorem A.1 (Uniqueness of signature; Hambly & Lyons (2110). Let X E (1,, Then S(X) delermines up to the tree-like equivalence.
Theorem A.2 (Signature Universal Approximation: Lyons et al. 2007)). Suppose f S1 + R is continuous function, where S1 is a compact subset of (V"(J, and C R an interval. The topology on vi (J, (d) is chosen such that the signature map is continuuus. Then for any 0 there is a linear functional
I E T ((Re))* such that.
[f(a) L(a)l se for every E St.
B
The development of a path
B.1 Connection with signature
Recall that V1 (10, Tj, Rd) denote the space of continmous paths [0, T] - Rd of bounded 1-variation and is (matrix) Lie algebra.
Lemma B.1 (Lemmn 2.2 Let X Vi (R) and M L (R4 g)) The development of the path X under M is given by
DM = MS())).
Proof. By definition of the development, DMs (X) is the endpoint of (Ze)eeo,.)) which satisfies the linear controlled differential equation
dz (dX,,,,),
Zo = ld.
By Picard iteration, obtains
E/
Z-= = ld DM(X) = ZT
M(dX.,) ... M(d...).
n21 Jocc ET <T
Thus, by lincarity of M, the right-hand side of the above equation coincides with (S(x)).
B.2 Dimension comparison with signature
The dimension of the signature of d-dimensional path of depth & is d++1-1 if if 2 and k d When specifying the Lie algebra g of the development as matrix algebra, the dimension of development of order m is m² (independent of d). The dimension of trainable weights of the development layer with static output is dm?, while the signature layer does not have any trainable parameter.
B.3 Invariance under time-reparameterisstion
Lemma B.2 (Lemma 2.3 Let X E V1 (), and let be a non-decreasing C¹ -'ddffooomorphsn from [0, S] onto [0, Define X = Xx, for t E [0, Then for all M E L (Rd,g)
(M DM (X).).
Proof. It follows directly from Lemma 2.2 and the invariance under time reparameterisation of the signature. See Chevyrev & Kormilitzin (016))
24

---

Published in Transactions on Machine Learning Research (08/2024)
B.4 Universality
Let W be vector space. In the sequel, we denote by g((W) the space of endomorphisms of W equipped with the Lie bracket (01,0)) == 01 0 01.
Definition B.1 (Lie algebra representation). Let be a Lie algebra. A Lie algebra representation of on a vector space W is a Lie algebra homomorphism + g((). That is, a linear map satisfying p([ = ((X))))((X))) for all X2 g.
To fix the idea, endow with (Rdy@k for each & N the projective norm; see Chevyrev & Lyons (0166). Let & be the space of tensor elements with infinite radius of convergence:
no
| < for all A A20 A
3 x: (zo, T((R))
Σ
Theorem B.1 (Universality of path development). Let G be the space of group- elements in & and K c. g be compact subsel. For ony conlinuous Sunction K. - C and > O, there exist and Li...., LN E L (gl(m, C); C) such that
E C))
sup a
f(x)
Li o M((x) <C. Σ
Here recall from Eq. 44 that M, E (l, C)) is the algebra morphism extended from M, e L (IR4 g((m, C)) by naturality. As per Definition 2.1 in Chevyrev & Lyons (0016), equip T(R")) with the coarsest topology such that all algebra morphisms M e L (G,A) which arise from M E L (R A), with A ranging through all normed algebras, are continuous.
A Also, note that when the algebraic homeomorphisms M, have their domains restricted to G, they indeed take values in the group GL(m, C) of invertible matrices.
Proof. Let us consider
A
finite-dimensional matrix representations of T((R))
}
arising from extensions all M E L Rd.
gl,m, C)
m=1
By Chevyrev Lyons 2016
Theorem 4.8, A separates points over T((R)). Then define
} C (R4,C).
Ao M: M and AEL glmm, C);C
(11)
1=
We claim that B is a complex unital -laleerr..
We first explain how to conclude the proof assuming the claim. Indeed, since separates points over T(RR)) know that B separates points over T((R)) too. By Corollary 2.4 in Chevyrev Lyons (2016). we know that G is Hausdorff, so K. C G is compact Hausdorff space. It thus follows from the Stone Weierstrass theorem - i.e., for a compact, Hausdorff topological space X, if 5 (X, C) separates points, then the complex unital "-algebra generated by S is dense in (2' C) that for any f: K. + C and e > 0, there exists S E B (viewed as a function over the domain XC) such that || 11f-Spp (T1C)(()))) < </2. In addition, as both have countable bascs, B is separable. Thus one may find M.. My E L(G, gl(m, C)) and Li...., Ln E L (gl(m, C); C) satisfying - Di Li M, (x) < </22.
L (RR: + 0 gl(m, C)) and L ( gl(m,
This concludes the proof.
Now let prove the claim.
25

---

Published in Transactions on Machine Learning Research (08/2024)
A contains the unit, which is the trivial representation. The unit of L (4m=1 gl(m, C)) is the constant map 1. So B contains the unit too, which is also the constant map 1.
The product on A is given by the tensor product, f.e., the Kronecker product for matrices: pi ()))(x) : pr (x) @ p2(x)) for Pt, PI,22 E A. The involution in A corresponds to the dual representation, namely that p² p²(x)
- [p[x]]] (the adjoint matrix) for p E A. By the definition of M M, we have M1 @ M2) (z) (M1 @ 1 + @ (2)( where is the constant map which maps to the identity of the group.
Thus, for A1, 22 E L O m gl(m, C): and M1, M2 E
gl(m, C)), we may define the R + Dm=
natural product and involution in as follows:
d1 MM
\1 M
o = (A1 12) M2 @ Mz)
It is casy to check that they are well defined and satisfy the desired properties as algebraic optiratioss on unital *-algebra over C. In particular, their continuity follows from that of M + M, which is part of the definition of the topology on T)(R)).
The claim is proved.
Two remarks are in order concerning the above theorem. Remark B.1.
1. It may appear odd that the product in A has the structure of tensor product of group representations instead of Lie algebra representations. This is intentionally designed to make A (with domain restricted to G) a Hopf algbera representation of the space of group-like elements G. Indeed, one characlerisation of G that.
G = fx T((R")) \ {0}: =x)=
where is the Hopf algebra coproduct on the tensor algebra, extended from (v) = & on Rdd In this provision (pr & P2)(x) = (p1 & p2) (x)). On the other hand, the sign in the dual representation corresponds to the antipode of the Hopf algebra.
Our version of A is taken only for convenience of exposition; it is larger than necessary. In fact, we may adopt verbatim Chevyree & Lyons (016),, Definition 1.1 for the definition of A, which only M laking values in unilary operutors on fimensoaiiaa complex Hilbert spaces are taken into account. In this way, we may replace gl(m, C) in Theorem B.1 by the unitary Lie algebra u(m). The proof is essentially the same, the Hopf algebra representations of g into +m, u(m) remains complex unital "-algebra separating points over G.
C Backpropagation of the development layer
C.1 Backpropagation of the development
Consider a scalar field sb N+1 + RR and an input x = (I E N+1 We aim to find optimal parameters 0* minimising (Do(x))) formula,
0" arrgmnnegs ((D((1)),
where Da is the development layer defined in Eq. ( Without loss of generality, focus on the case where the output is the full sequence of path development. Euclidean gradient descent is not directly applicable here due to the Lie group structure of the output.
The following identity allows us to compute the gradient of Lie Sigebra-valued parameters in our development layer by adapting the method of trivialisation in "Lezcano-Casale (2019): Lezeano- Cazcano-Casado Casado & Martieez Rubio (2019). Consider smooth map & N + M between Riemannian manifolds (M, 91) and (N.gz). Denote its differential by dΦ TN TM, and let the adjoint. of d& with respect to Φ1; g2 be dd" TM TN.
26

---

Published in Transactions on Machine Learning Research (08/2024)
Corollary C.1. Let c : + MM be smooth map between Riemannian manifolds and let f M R. Then V(fo d (VS).
C.2 Proof of Theorem 1
Proof. We first consider more general statement: let M and N be differentiable manifolds, let f : N be a smooth function, and let sp M + R be scalar field factors through the graph of f. That is, there is a smooth function & M x N + Il such that v(x) = (xx,) (())) for each x E M. Then we have the following identity on T M:
dē (pri) du (prr) dı odf.
(12)
Here we view (pri) de E M the canonical projections from M X onto .M and N, respectively.
@ (0} and (PrI)) # deb ods E (0) (0) TM TM; the symbols pr, and prn denote
The demonstration for Eq. [2] is straightforward. For functions fi E + S and f2 E + S2 where E, S1, S2 are arbitrary sets, we denote f1 3 f2 E + (Si x S2) as the function (fi @ f2)(2) == (fi(z), In this notation one has
$ = th (TdM 1 f)
Using the chain rule we then deduce that
dv = dyb
range (Idmes) od (IdM @ f) = =dyb graph (IdTM @ df),
with IdTM: TM- + TM TM @ {0)] and df TM + [TN = (0) @ TNJ. This is tantamount to the right-hand side of Eq. 12 by the definition of pr, and pru-
Let now prove the lemma from the identity 6,41 @ (Gx42
Indeed, by taking pr, in place of pr and f o... G2 Gn1)), where M = G and N = NN-n we deduce that
12
dvhn (pr,) de (pr,+1) d dG + (praaz) # dub .d (G.+2 Gn+1)
+ (Pr dw) d(G .... Gn+2 6n41)
= -pr,) du (pr d dG + (pr dub d6n=2 d6.+1
+ (prw) d d (G GN-10 . Gn42) d6a+1.
Here we have used the fact that for C. G G. (h) = gh, its differential dh, is also the left-multiplication by which is given by matrix multiplication when is matrix algebra. The second equality follows from the chain rule.
On the other hand, notice that the terms in the right-most expression above possess a recursive structure too: by taking pr place of pr, and 6,+2 (6 o Gn42) @ (GN o... o GG422 in Eq. 12 we have
(pr,+1) de + (pr } dG (prr)) d d(G GN- Gn+2) d
The above computations establish the following equality on G:
d = (pr,), # dy + d dG
Thus, evaluating this identity at 2n G, we infer from the definition of the update function Gn+1(zn) 2n41 that
de vn (pr,) dub in + den.i Vnt1 de, GD+1-
We now conclude by the recurrence relation 2nt1 2n exp(eMoo (Avn+1)) in Eq. 6
27

---

Published in Transactions on Machine Learning Research (08/2024)
A remark notations is in order: In the last line of the proof above, (formally) replaced ds, G,+1 with exp(Ma...) (Arn+1)). The reason is that
Gn+i(Zn)= 2n+1 2n exp( (A (Axn+1)).
where Ga4t - G. Viewing G as subset of vector space gl(m, (), may regard 6,+1 linear mapping in its argument. Thus, dz,, 6at1 Ts,, G + T2 G equals the right multiplication by the matrix exp (Mo..1 (Arx+1)).
C.3 Proof of Proposition 3.1
Before proceding the proof of Proposition 3.1 given by the Hilbert Schmidt norm of matrices in M = gl(m,,) F) for F = R or C. Equivalently, identify gl(m, F) with the Euclidean space pmxm via the diffeomorphism v gl(m, F) + FmX"
remark that the Riemannian metric taken here is the
V (14)) = [A], A2. A/n AR, Ag.
A
AT AT .....
The pullback metric is preciscly the Hilbert-Schmide metric (B, C) tr( tr(BC*) for any B,C Tagi(m, F) gl(m, FF) with arbitrary A E IF)
The choice of Hilbert Schmidt metric on matrix Lie groups is natural in our context. Indeed, for inclusions S0(m) C gi(m, R) and (mm C gl(m, C), pullback metrics of the Hilbert-Schmidt metric under the natural inclusions are exactly the bi-invariant Riemannian metries on (0 and (mm, respectively.
In provision of this remark, one can compute the gradient of &n in Eq. 7 by expressing Veva in terms of the differential exponential map exp g + G, which in turn computed via Theorem 2.2 and Lemma 2.4 Recall the projection Proj: gl(m, C) > from Eq.
As usual, denote by f. G G for each gi G the left regular action; ie., Las (922) = 9192.
Proof. The expression
Ve (
Σ Vo Un (m((nn1,))
follows directly from the definition of 2n in Eq.
To further compute the right-hand side, notice that Vo 55 (2( 8)) a vector field on the Euclidean manifold gd (see Remark 3.3 Recall also that
Zn G = -zn-n exp (Mo (Mo(Ar,)))
where Mo(Ax,) takes values in g Proj(al(m, C)). Moreover, set A gd g to be the mapping A(0) = Mo (Ax,) and put f = Un Lin-I. We compute that
Vo (wh G, (=n-111,0) Proj A V(F oxx) (xx)())))
Proj A# {dmiott ( (A(0)) }
The second line follows from Theorem 2.2 and the first line holds by the general rule - let M and A be
Riemannian manifolds and p M - N, N R be smooth functions. Then V(ho (( = *#((H) (Vh) where
* is the pullback operator.
To procoed, let us compute V.. Since for any g G the left regular action Cg satisfics d., = Lg whenever G is a mabria group, we have that
thn 2n-1;
28

---

Published in Transactions on Machine Learning Research (08/2024)
where is the matrix multiplication. We put the subscript 2n here only to emphasise that the gradient of sin is taken with respect to this variable.
It remains compute (A)))(0), where we write V == doo [[(exp (A(0)] for abbreviation. Recall that V and V are vector fields on and g", respectively. Elementary computations in differential geometry then yield that
(0) - V (A - for each E ge.
It is crucial here that A is a linear mapping from gd to gi thus, the pushforward A# coincides with A(0).
We may now complete the proof.
Experimental details
D.1 General remarks
Optimisers. All experiments used the ADAM optimiser as in Kingma & Ba 2014 batch sizes vary along experiments and models. The lcarning rate in certain experiments may be applied with a constant exponential decay rale, and the training process is terminated when the metric failed to improve for some large number of epochs. See individual sections for details. We saved model checkpoints after every epoch whenever the validation set performance (the performance metric varies from experiment to experiment) gets improved and loaded the best performing model for evaluation on test set.
Learning rates and
Architectures. In our numerical experiments, we consider the development modols (DEVs) and the hybrid model (LSTM+DEVs) by stacking the LSTM with the development layer together. To benchmark our development- based models, we use the signature model and the LSTM model
baselines.
For the Speech Commands and Character Trajectories experiments, all the above mentioned models (DEVs, LSTM+DEVs, Signature, and LSTM) all include linear output layer to predict the estimated probability of each class. For a fair comparison, we keep the total number of parameters across different models comparable and then compare the model performance in terms of accuracy and training stability.
Similarly, we also keep the model complexity of the baseline LSTM and our proposed hybrid model (LSTM+DEVSE(2) comparable in the N-body simulation task. See the specific architecture of each individual model for those three datasets in Appendix D.2 Appendix D.3] and Appendix D.6 respectively.
Hyperparameter tunning. Once our model architecture is fixed for fair comparison, hyperparameter tunning focuses mainly on learning rate and batch size. Specifically, the learning rate is first set to 0.003 and reduces until the good performance is achieved. Then we search the batch size with a grid of [32, 64, 128]. The decay factor of learning rate was applied to those models with severe overfitting observed on the validation set. Throughout the experiments, we notice that the hyperparameters have minimal effects on the development. network model, while the RNN-based models are very sensitive hyperparameters.
Loss. We used cross-entropy loss applied to the softmax function of the output of the model for the multi-classes classification problem. We used mean squared error loss for the regression problem in the N-Body simulation.
Computing infrastructure. All experiments were run on five Quadro RTX 8000 GPUs. We ran all models with PyTorch 1.9.1 Paszke et al. (2019) and performed hyperparameter tunning with Wandb Biewald 2020 Codes. The codes for reproducing all experiments are included in supplementary material.
D.2 Speech Commands
We follow the data generating procedure in Kidger et al. (2020) and Look a 115% soain/validaation/eee split. The batch size was 128 for every model. Dev(SO), LSTM+Dev(SO) and signature models used a constant lcarning rate 0.001 throughout. LSTM was trained with learning rate 0.001 and exponential docay rate 0.997. Training terminates if the validation accuracy stops improving for 50 epochs. Set maximum training epochs to 150.
29

---

Published in Transactions on Machine Learning Research (08/2024)
Signature. We apply the signature transform up to depth on the imput, which converts the imput time series to vector of size 8420 before passing to a output linear layer. The model has 84210 parameters in total.
LSTM. The input of the model is passed to a single layer of LSTM (135 hidden units). The output of the LSTM (last time step) is passed to a output linear layer. has 86140 parameters total.
DEV.SO). The input of the model is passed the special orthogona development lyear (54 by 54 matrix hidden units). Then the output of the development (final time step) is passed to final linear output layer. It has 87490 parameters in total.
LSTM-DEV(SO), The input of the model is passed to single LSTM layer (56 hidden units). Then we pass the output of the LSTM (full sequence) to the special orthogonal development layer (32 by 32 matrix hidden units). Then the output of the development (final time step) is passed to final linear output layer. It has 85066 paramcters in total.
D.3 Character Trajectories
We follow the approach in Kidger et al. (2020), in which we combined the train/test split of the original dataset and then took a 70% 15% 15% train/ validation stestt split.
The batch size used 32 for every model. Dev(SO), DeVM+))v(SO) and signature model used constant learning rate of 0.001 throughout the training. LSTM was trained with learning rate of 0.003, with exponential decay rate of 0.997. If the validation accuracy stops improving for 50 epochs, we terminate the training process. The maximum training epochs is set to be 150.
Signature. We apply the signature transform up to depth on the imput, which converts the input time series to a vector of size 340, then passed a linear output layer. The model has 6820 parameters total.
LSTM. The input of the model is passed to single layer of LSTM (40 hidden units). The output of the LSTM (last time step) is passed to a linear output layer. The model has 8180 parameters in total.
ExpRNN. The input of the model is passed to a single layer of ExpRNN (78 hidden units). The output of the ExpRNN (last time step) is passed to a linear output layer. The model has 8054 parameters in total.
(EV.SO). The model's input passod to the special orthogonal development layer (20 by 20 matrix hidden units). Then the output of the development (final time step) is passed to a final linear output layer. The model has 7760 parameters in total.
LSTM-DEV(SO), The model's input is passed to single LSTM layer(14 hidden units). Then we pass the output of the LSTM (full sequence) to the special orthogonal development layer (14 by 14 matrix hidden units). Then the output of the development (final time step) is passed to a final linear output layer. The model has 8084 parameters in total.
D.4 Sequential MNIST and CIFAR10
The MNIST dataset is a large collection of handwritten digits, with a training set of 60,000 examples and a test set of 10,000 examples. Each 28 x 28 pixel valued image is flattened to a sequence of length 784. The permuted task (P-MNIST) applies the same random permutation on the sequence of size 784. The CIFARIO dataset consists of 60000 (50000 training + 10000 test samples) 32 by 32 colour images with 10 classes. Like the sequential MNIST, the CIFAR10 task flattens the images to pixel by pixel sequences of length 1024.
The batch size used was 128 for both sequential MNIST and CIFAR10. LSTM+DEV(SS) was trained with an inital learning rate of 0.002 and an exponential decay rate of 0.997. We terminate the training process if the validation accuracy stops improving for 50 epochs. The maximum training epochs is set to be 200.
LETM+DEV (SO). common model architecture is used for both datasets. The input is passed to single LSTM layer (120 hidden units); the output of the LSTM (full sequence) is passed to the special orthogonal development layer (10 by 10 matrix hidden units). Then, the output of the development, (final time step) is
30

---

Published in Transactions on Machine Learning Research (08/2024)
passed to final linear output layer. The model has 72050 and 73010 parameters for sequential MNIST and CIFAR10, respectively.
D.5 Brownian motion on the unit 2-sphere g
We simulated 20000 samples of discretised Brownian motion on S with time length L = 500 and equal time spacing Al 2 by the random walk approach (Novikov et al. (2020)).
The discretised Brownian motion B (B,., ) with in nAt Sa is simulated by driving random walk X (Xt, 1 R², Let Bi (0. 1) - We first simulate a 2D random walk X by i.d.. increments AX, == X uniformly distributed (-5..5, 0.5). Then rescale X by the factor V12 to approximate 33-valued Brownian motion over the time interval At. For generic point b (b(1), b,2), blo) the following basis for T,S² C R3, the tangent plane of S at b:
S?
we use
-bulcb -6(1)(2) ([rocy + (b(3)) b(208(3)
6²
1
cf
of
Votu) + (b(3))2
V(baa)22 (b(3))2
611
At each step {1, L 1}, perform standard random walk on the tangent plane orthogonal to the orientation vector of Br, and project Br, + V12AAXX, to B,, as described in Novikov et al. (2020) In this manner we simulate, by Monte-Carde independent samples of the pair of driving random walk and corresponding Brownian motion on S
The simulated input/output trajectories are split into vrain/validation//ees with ratio 10%/11%/10% All models were trained with learning rate 0.003 and exponential docay rate 0.998. The batch size was 64 for every model. Training was terminated if the validation MSE stopped lowering for 100 epochs. The maximum number of epochs was set to be 300.
For all models, we passed the input through two dense layers (32 hidden units), followed by single LSTM/orthogonaIRRNN layer (64 hidden units).
LSTM. For the LSTM model, we pass the LSTM output (full sequence) through one dense layer (64 hidden units) and a single lincar output layer.
ExpRNN. For the ExpRNN model, we pass the ExpRNN output (full sequence) through one dense layer (64 hidden units) and single linear output layer.
DEVM+ we pass the LSTM output (full sequence) to a SO(3) development layers (each has by 3 matrix hidden units). The output (full sequence) of the development. layer is a sequence Lie group element in SO(3). Then we take the last column of the SO(3) matrix as the final sequential output.
D.6 N-body simulations
We followed Kipf et. al. (2018) to simulate 2-dimensional trajectories of the five charged, interacting particles. The particles carry positive and negative charges, sampled with uniform probabilities, interacting according to the relative location and charges. We simulated 1000 training trajectories, 500 validation trajectories and 1000 test trajectories, each having 5000 time steps. Instead of inferring the dynamics of the complete trajectories as in Kipf et al. (2018), we consider it a regression problem with sequential input: the input data is a sequence of locations and velocities in the past 500 time steps (downsampled to the length of 50). The output the particle's positions in the future ls time steps for E {100, 300, 500).
(6) be the 2-D location and velocity of the 7th particle at time t. The input More specifically, let x and iy
5
and (#(4k) i=!' respectively. In and output of the regression problem are xx = (((=5) (0) s0t-500 i=1
2-D the coordinate transform is described by the special Fuclidean group SF((2, which consists of rotations
31

---

Published in Transactions on Machine Learning Research (08/2024)
and translations:
x
R
3/
T
by
y
(13)
0
0
1
1
where T E SE( and R is a 2-D rotation matrix.
The proposed LSTM+DEV(SE(2)) network architecture models the map 2/ (5)
Titk hence predicting the
(9) The model weights change over future location by multiplying the output of the development layer with x2 development layers for different particles.
The other three baselines include (1) the current location Iy (static estimator, assuming no further movements of the particle); (2) the LSTM; and (3) the S2(2) development layer.
All models are trained with a learning rate of 0.001 with a 0.997 exponential decay rate. The batch size used was 128 for every model. The training was terminated if the validation MSE stopped lowering for 50 epochs. The maximum number of epochs was set to be 200.
For all models, we passed the input through two dense layers (16 hidden units), followed by a single LSTM layer (32 hidden units).
LSTM. For the LSTM model, we pass the LSTM output (last time step) through one dense layer (32 hidden units) and a single linear output layer.
LSTM+DEV(SE(2)) We pass the LSTM output (full sequence) to the five independent SE(2) development layers (each has 3 by 3 matrix hidden units). The output of each independent, development layer (final time step) is a Lie group element in SE(2). We apply the learned SE(2) element to the last observed location of each particle to estimate the future locations of the five particles, according to Equation (13.
LSS)))) We pass the LSTM output (full sequence) to the five independent SO(2) development layers (cach has by 2 matrix hidden units). The output of each independent development layer (final time step) is Lie group element in SO(2). We multiply the learned SO(2) element with the last observed location of each particle to estimate the future locations of the five particles.
32

---

Published in Transactions on Machine Learning Research (08/2024)
E
Supplementary numerical results
Here we report the supplementary numerical results on speech Commands datasct (Table 6
Table 6: Test accuracies of the linear model on development and signature baselines on Speech Commands dataset.
SIGNATURE
DEPTH (n)
TEST ACCURACY
# FEATURE
12.3%
20
2
75.4%
420
3
55.7%
8420
88.9%
168420
DEVELOPMENT (SO)
ORDER (m)
TEST ACCURACY
# FEATURE
70.5%
25
81.3%
100
20
84.6%
400
30
86.2%
900
50
87.5%
2500
100
89.0 %
10000
DEVELOPMENT (SP)
ORDER (m)
TEST ACCURACY
# FEATURE
19%1%
36
10
83.7%
100
20
85.5%
400
30
87.1%
900
88.5%
2500
100
86.3%
10000
33


Reviews:
[{"review_id": "fekxnufE6f", "reviewer": "Reviewer_Kb6t", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- The authors' method is principled, and the paper presents a very careful and\n  complete derivation of the method from basic principles from rough path\n  theory and controlled differential equations theory. The mathematical writing\n  is clear and precise, and examples and remarks given along the way make this\n  material to a technical but out-of-area reader. Complete proofs of claims are\n  provided in appendices. These derivations actually seem to represent a useful\n  introduction to some of these concepts (e.g. the signature) versus other more\n  technical papers in the literature.\n- The authors demonstrate their method on a range of experiments, from standard\n  time series benchmarks (e.g., audio data with cepstrum features) to dynamics.\n  The method effectively improves performance and scalability with regards to\n\n## Weaknesses\n\n- The numerical performance of the authors' method is often significantly below\n  that of pure black-box deep learning approaches (e.g., LSTM or SSMs), whereas\n  a 'hybrid' approach involving their layer (playing the role of a kind of\n  'feature extractor') can often improve over this baseline. The authors mention\n  that their method is more robust and more easily trainable, as well, but\n  I have some doubts about this type of claim: it seems more likely to me (e.g.,\n  some evidence suggested by experiences with past networks with a similar\n  flavor in the image/CNN context, such as the scattering networks) that any\n  benefits along these two axes are due to inherent constraints on the capacity\n  of networks derived from the authors' layer, and therefore at odds with\n  performance (which is not the tradeoff one desires).\n\n- The authors' layer is presented as a complete novelty, but it shares some\n  functional similarities with other types of layers that have been inspired by\n  Riemannian optimization: for example, the Riemannian ResNet (https://arxiv.org/abs/2310.10013) proposes to use\n  the exponential map similar to a skip connection. One major difference here\n  appears to be the focus on time series data and the principled foundation in\n  terms of the signature, but the authors should compare to these prior works in\n  order to situate their network architecture contribution within the deep\n  learning literature.\n\n## Detailed Comments\n\n- Theorem 2.1: $x$ is defined with subscripts but the hypothesis involves $x$\n  with a superscript (is it supposed to be the same?). The authors should\n  comment here on how this dimension reduction functions in practice, given that\n  the theorem concerns a noncompact Lie group and an earlier remark in the paper\n  stated that noncompactness could be a problem in practice; can a result like\n  Theorem 2.1 hold for a compact Lie subalgebra as well? It would be most useful\n  to have a more \"quantitative\" version of this result as well (e.g., is it\n  possible to assert 'how well' I can approximate a path using $k$ terms of\n  the signature sequence and $m \\times m$ operators, for any $k, m$?)\n- Claim before Remark 3.1: the use of the product shorthand notation here\n  suggests that these matrix exponentials are commuting; but isn't it the case\n  that they only commute when $M_{\\theta}$ has a far more restrictive structure?\n- Table 1 and Figure 3 give a good empirical sense of the role of the choice of\n  operator dimension and number of developments in determining performance of\n  the authors' method. However, given the very significant mathematical flavor of\n  the paper, it feels like there is a missed opportunity to demonstrate some toy\n  examples in which this tradeoff can be mathematically-precisely understood. Are there\n  natural simple curves (e.g. like in the toy Figure 2 example, which does not\n  really play a role in the current text) where the kind of 'low dimensional\n  structure' inherent in the signature is present and moreover captured by the\n  authors' proposed layer at certain choices of $m$? This would also go a long\n  way to demonstrating what the limitations of the authors' approach might be.\n\n## Minor Comments\n\n- The beginning of the intro feels slightly inaccessible to a non-specialist in\n  rough path theory. Two points that feel improvable: (i) in the sentence\n  \"roughly speaking\" that describes the signature, the description seems both\n  overly formal and insufficiently precise (the formalism presented is a bit\n  unwieldy with notation that can't be understood based on the description\n  given; it also suppresses/does not define the \"iterated integral\"\n  notation appearing here), maybe prefer to add a pointer to appendix A here\n  and add clarification of iterated integral notation to appendix A; (ii)\n  although it will be spun out in the remainder of the paper, it would be\n  helpful to strengthen the transition from the mathematical idea of the\n  signature to the idea of the signature as a feature map by explaining more\n  clearly how one computes with/approximates the signature (the current 'roughly\n  speaking' formal description does not appear to admit an easy efficient\n  approximation).\n- The paper's primary focus and contributions are around proposing a new\n  computational method for computing with time series data which is a scalable\n  alternative to the signature -- hence practical -- but the paper is written in\n  an overly-mathematical style that at times introduces too much distracting,\n  irrelevant detail with regards to this goal. For example, it seems like\n  section 3.2 could be replaced by a reference to the standard Absil, Mahoney,\n  Sepulchre and maybe Remark 3.3 kept (because this is simply a standard\n  implementation detail, which the matrix exponential implementation should\n  handle regardless). On the other hand, this style of writing is helpful in\n  sections that are closer to the conceptual novelty of the authors' approach\n  (e.g. section 2).\n\n## Typos/etc.\n\n- Page 2 second graf: \"will shall\" (sic)\n- display on page 2: what is the 0,T subscript's meaning (is this a typo and it\n  should be $[0, T]$)?\n- Equation (4) and Lemma 2.2 - there might be a 'types' issue here? ($\\tilde{M}$\n  is defined on infinite sequences of tensors of increasing \"order\"\n  $T(\\mathbb{R}^d)$, but (4) only shows how $\\tilde{M}$ acts on a single element\n  of this infinite sequence, and for 2.2 to be true, I think the expression in\n  (4) needs to be summed over \"order\" (as in the display on page 2?).\n- Lemma 2.3 - is there a typo in the conclusion? ($X$ is defined as a path on\n  $[0, T]$; what is the meaning of the double subscript $X_{s, t}$ (etc.)?)\n- Lemma 2.4 - please correct me if I'm mistaken, but this seems wrong by\n  pattern-matching against Theorem 2.2, first result (e.g., the chain rule in\n  euclidean space gives that the gradient of x \\mapsto f(A(x)) for a linear\n  map A is A^* \\nabla f(A(x)) (^* denotes adjoint); so why is there no Proj\n  operation \"on the inside\" of the composition here?). I am also not sure why\n  the differential notation is different here than in Theorem 2.2 (why d_{M^T}\n  instead of df_{M^T}?).\n- page 13: broken label (latex issue) for Figure 3 (the referrer is incorrectly\n  written \"Section 4.1.1\")", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a novel neural network architecture (via a novel layer) for\ntime series data, called the path development layer, which is derived from the\nnotion of the development of a path in rough path theory. This formalism\nessentially corresponds to a generating function for the signature of a path on\n$\\mathbb{R}^d$ -- a useful feature representation for suitably-encoded time\nseries data, but which is not very scalable -- with monomials in\na sub-Lie-algebra of $\\mathfrak{gl}(n)$. The idea is then to consider\na discretization of an arbitrary path into linear segments, in which case the\ndynamics associated to the path's development have a simple closed-form\nexpression in terms of the matrix exponential, and interpret the resulting\ndiscrete iteration as a neural network (composed of the so-called path\ndevelopment layers). As a result, the dimensionality of this network depends\nonly on $n$, not $d$; this can lead to significant savings over calculating the\nnaive signature, without sacrificing too much 'capacity' of the underlying\nrepresentation.\n\nThe authors' contributions are to clearly derive this formalism from basic\nprinciples in rough path theory, then describe a computational implementation\nfor it using ideas from Riemannian optimization (this is used for learning the\n'parameters' of the path development layer, which correspond to matrix-valued\nmaps from tangent vectors along the discretized path to elements of the Lie\nalgebra). They demonstrate the resulting layer numerically on several standard\nbenchmarks from this area, showing that it leads to improved performance and\nscalability versus previous deep approaches for approximating the signature,\nas well as competitive performance with black-box approaches when used in\ncombination with a LSTM.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Discussion of similar Riemannian-inspired NN architectures mentioned above\n- If possible, a more formal example of the tradeoffs between $m$/number of\n  developments in a specific toy case that gives more precise insight than Table\n  1/Figure 3 would be very helpful", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "d2o0BIPwGV", "reviewer": "Reviewer_cmMa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The method proposed in this paper looks novel to me.\n- The theory seems correct and provides understanding about the method\n- The experimental section is strong\n\nWeaknesses:\n- Missing references and qualitative comparisons with literature in the field of neuroODEs. For example see [1] which is a prior work to Chen et al. (neuroODEs) and [2,3] that discuss weight sharing for learning hidden states similar to the paper here. A discussion should be added to the revised paper.\n\n- The paragraph \"The preservation of favourable geometric and analytic properties makes the path development a promising\nfeature representation of sequential data. Specifically, non-commutativity of the multiplication in ....\nreflects the irreversibility of the order of events. It has been established in Chevyrev & Lyons (2016) that,\nwith g suitably chosen, the path development constitutes universal and characteristic features. Moreover, in\ncontrast to the infinite dimensionality of the path signature, the development takes values in finite-dimensional\nLie groups with dimensions independent of the path dimension d.\"  is vague and should be expanded.\n\n- The method uses matrix exponentials, which is an expensive thing to compute. How do you cope with it? \n\n- In the key advantages: What does characteristic mean? do you mean universal in approximation sense? \n\n- The authors do not discuss the complexity and runtimes of the methods. It should be added to the paper including a comparison with other methods. \n\nQuestions:\n- From your notations, in the end of page 2, it looks like there is only time dimension. Does your work support spatial data in addition to temporal data? If so, how? If not, can you elaborate on why?\n\nMinor:\n- Typo in the sentence \"Consider a matrix Lie group G. It\nwill shall serve as the range for the development.\"\n\n- Typo in \"it can model the dynamics on on-Euclidean spaces by exploiting appropriate Lie group structure.\" ?\n\n- Table 1 should be resized, its font looks weird.\n\n\nReferences:\n\n[1] Stable Architectures for Deep Neural Networks  \n\n[2] Anti-Symmetric DGN: a stable architecture for Deep Graph Networks  \n\n[3] Feature Transportation Improves Graph Neural Networks", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new method for path development using finite dimensional Lie group algebra.\n\nThe idea of the paper is to learn hidden states that conform to a dynamical system inspired behavior, while using an RNN like mechanism, while providing theoretical understanding and guarantees regarding the behavior of the network, as well as allowing to operate on non-Euclidean spaces. \n\nThe authors verify their method by numerous experiments, showing the strong performance of the method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see in my review.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8X6iiqYGrU", "reviewer": "Reviewer_BQ81", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n-----------\n* The paper is technically correct.\n* The paper's subject of time series data is very important and has a wide audience range.\n\nWeaknesses\n--------------\n* While technically correct, the actual contribution is quite simple. In particular, it is just a lie group parameterized RNN. \n* Related to the above, I also think the presentation is far too involved/obfuscated. I would like for there to be a simpler presentation of the background, and the method could be simplified as well.\n* Related to the above, but it would likely be much more accessible if the paper was presented with rough path theory as the motivation (with the actual contribution being framed closer to the RNN perspective) than the rough path theory motivation. Notably, the rough path stuff feels a bit superfluous.\n* The experimental results are not very impressive.\n* There is a slight breach of anonymity, as the conclusion includes named information.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel lie-group based approach to learning the path signature of data. This is applied to time-series data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would like a comparison with https://arxiv.org/abs/2203.05483, as the methods seem broadly related (ie RNN + Lie Group optimization).\n\nI would also emphasize simplifying the presentation and filling in harder to access information.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a new layer for time series neural networks termed the path development layer, which leverages tools from rough path theory to provide more efficient estimates in the case of high-dimensional paths, along with suitable optimization procedures. The reviewers found the theory and experiments novel, interesting, and thorough, though the presentation is a little technical, and the empirical results slightly underwhelming.
Decision: Accept (poster)