Rough Transformers: Lightweight and Continuous Time Series Modelling through Signature Patching
Authors: Fernando Moreno-Pino
Alvaro Arroyo
Harrison Waldon
Xiaowen Dong
Alvaro Cartea
Venue: NeurIPS 2024 poster (2024)
Keywords: efficient time series modelling
efficient sequence modelling
continuous-time temporal modelling
patching
self-attention
transformers
computational efficiency
spatial processing
Primary area: deep_learning_architectures
TL;DR: We propose a way to make the self-attention mechanism operate on continuous-time representations of temporal signals and incurs significantly lower computational costs.

Abstract: Time-series data in real-world settings typically exhibit long-range dependencies and are observed at non-uniform intervals. In these settings, traditional sequence-based recurrent models struggle. To overcome this, researchers often replace recurrent models with Neural ODE-based architectures to account for irregularly sampled data and use  Transformer-based architectures to account for long-range dependencies. Despite the success of these two approaches, both incur very high computational costs for input sequences of even moderate length. To address this challenge, we introduce the Rough Transformer, a variation of the Transformer model that operates on continuous-time representations of input sequences and incurs significantly lower computational costs. In particular, we propose multi-view signature attention, which uses path signatures to augment vanilla attention and to capture both local and global (multi-scale) dependencies in the input data, while remaining robust to changes in the sequence length and sampling frequency and yielding improved spatial processing. We find that, on a variety of time-series-related tasks, Rough Transformers consistently outperform their vanilla attention counterparts while obtaining the representational benefits of Neural ODE-based models, all at a fraction of the computational time and memory resources.

Rough Transformers: Lightweight and Continuous Time Series Modelling through Signature Patching
Fernando Moreno-Pino !. Álvaro Arroyo 1:2,
Harrison Waldon
Xiaowen Dong
Álvaro Cartea
Oxford-Man Institute, University of Oxford
2 Machine Learning Research Group, University of Oxford Mathematical Institute, University of Oxford
Abstract
Time-series data in real-world settings typically exhibit long-range dependencies and are observed at non-uniform intervals. In these settings, traditional sequence- based recurrent models struggle. To overcome this, researchers often replace recurrent architectures with Neural ODE-based models to account for irregularly sampled data and use Transformer based architectures to account for long-ram long-range dependencies. Despite the success of these two approaches, both incur very high computational costs for input sequences of even moderate length. To address this challenge, we introduce the Rough Transformer, a variation of the Transformer model that operates on continuous-time representations of input sequences and incurs significantly lower computational costs. In particular, we propose multi-view signature attention, which uses path signatures to augment vanilla attention and to capture both local and global (multi-scale) dependencies in the input data, while remaining robust to changes in the sequence length and sampling frequency and yielding improved spatial processing. We find that, on variety of time-series- related tasks, Rough Transformers consistently outperform their vanilla attention counterparts while obtaining the representational benefits of Neural ODE-based models, all at a fraction the computational time and memory resources, |
Introduction
Real-world sequential data in areas such healthcare 665 finance [36] and biology 28 often are irregularly sampled, of variable length, and exhibit long-range dependencies. Furthermore, these data, which may be drawn from financial limit order books [8] or EEG readings (85), often sampled at high frequency, yielding long sequences of data. Hence, many popular machine learning models struggle model real-world sequential data, due to input dimension inflexibility, memory constraints, and computational bottlenecks. Rather than treating these data as discrete sequences, effective theoretical models often assume data generated from some underlying continuous-Ne process 53 66). Hence, there is an increased interest developing machine learning methods that continuous- representations analyze sequential data.
One recent approach modelling continuous-time data involves the development of continuous-time analogues of standard deep learning models, such as Neural ODEs 12 and Neural CDEs (45). which extend ResNets [37] and RNNs (30], respectively, to continuous-tine settings. Instead of processing discrete data directly, these models operate latent continuous-tin time representation
"Equal contribution.
Email: [fernando. mor.mo-piiooooommmmmmmmmmmoooom. alroo www.arooooo@@oooo...o.o Code available at: https://gittub.co htt.a//oo.oootaattoooorrr///eeee
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

of input sequences. This approach is successful in continuous-tine modelling tasks where standard deep recurrent models fail. In particular, extensions of vanilla Neural ODEs the time-series setting
770 45 succeed in various domains such as adaptive uncertainty quantification
[59] counterfactual
inference (79), or generative modelling 77.
In many practical settings, such as financial market volatility [20 54] or heart rate fluctuations [35], continuous-time data also exhibit long-range dependencies. That is, data from the distant past may impact the system's 's current behavior. Deep recurrent models struggle in this setting due to vanishing gradients, whereas continuous-
analogues of these models have been shown address this difficulty [46]. Several recent works [52] 58] also successfully extract long-range dependencies
from sequential data with Transformers [6), which learn temporal dependencies of a tokenized
representation of input sequences. Extracting such temporal dependencies requires positional encoding of input data, because the attention mechanism permutation invariant, which projects data into some latent space. The parallelizable nature of the Transformer allows for rapid training and evaluation on sequences of moderate length and it contributes to its success in fields such as natural language processing (NLP).
While the above approaches succeed in certain settings, several limitations hinder their wider applications. On the one hand, Neural ODEs and their analogues
)o.o M(X);
45 70 bear substantial computational costs when modelling long sequences of high dimension; see [57] On the other hand, Transformers operate on discrete-time representations of input sequences, whose rel- ative ordering is represented by the positional encoding. This representation may inhibit their expressivity in continuous- time data modelling tasks (11. Moreover, Transformer-based models suffer from number of difficulties, including (i) in- put sequences must be sampled at the same times, (ii) the sequence length must be fixed, and (iii) the computational cost scales quadratically in the length of the input sequence. These difficulties severely limit the application of Transformers to continuous- -ime data modelling.
(X)ea-1.4
5(X) en 1sth
((X)o..
Local View VW
Global View hww
Contributions 1) We introduce Rough Transformers, variant of the Transformer architecture amenable the processing of continuous-time signals, which can be easily integrated into existing code-bases. The Rough Transformer built upon the path signature from Rough Path Theory (5]). We define novel, multi-scale transformation which projects discrete input data to continuous timm path and compresses the input data with minimal information loss. Moreover, this transformation is an efficient feature representation of continuous time paths, because linear functionals of path signatures approximate con- tinuous functions of paths arbitrarily well (see Theorem A.2 in Appendix[A] [A].
Mmm
t
Figure 1: A representation of the multi-view signature. The continuous-ting
path is irregularly sampled at points marked with a red x. The local and global signatures of a linear interpolation of these points are computed and concatenated to form the multi-view signature. The multi-view signature transform con- sists of L multi-view signatures.
2) We introduce the multi-view attention mechanism to extract both local and global dependencies of very long time-series efficiently. This mechanism operates directly on continuous- time representations of data without the need for expensive numerical solvers or constraints the smoothness of the data stream. Moreover, the multi-view attention mechanism is provably robust irregularly sampled data.
3) We carry out extensive experimentation on long and irregularly sampled time-series data. In particular, we show that Rough Transformers (i) improve the learning dynamics of the Transformer, making it more sample-efficient and allowing it to achieve better out-of-sample results, (ii) reduce the training cost by factor of up to 25x when compared with vanilla Transformers and more when compared with Neural ODE based architectures, (iii) maintain similar performance when data are irregularly sampled, where traditional recurrent-basee models suffer a substantial decrease in

---

performance 770. and (iv) yield improved spatial processing, accounting for relationships between different temporal channels without having to pre-define specific inter-channel relation structure.
Background and Methodology
Problem Formulation. In many real-world scenarios, sequential data are time-series sampled from some underlying continuous-time process, so datasets consist of long, irregularly sampled sequences of varied lengths. In these settings, the problem of sequence modelling is described as follows. Let C(R Rd) - {g : R+ + R4 continuous), and consider X E C( ;R) which we call a continuous-time path. A time-series of length L with sampling times Tx (4:) C R+
defined X ((t), X1), ...: (tL, Xt)), where X; (ts) E Rd. Now, define continuous function paths ;(RR;:RRd) - RR. Next define dataset D { ((+)) We seek to approximate the function f from the set 'D for some downstream task. Importantly, we do not assume that 7x= TY for all X, Y D, so that D may be irregularly sampled.
Sequence Modelling with Transformers. Transformers are used extensively baseline architec- ture approximate functions of discrete-time sequential data and are successfully applied to settings when input sequences are fixed in length, relatively short, and sampled at regular intervals. First, the Transformer projects input time series X E 2xd to high nigh-dimensional space X + T(X) E RLxd' for d' >> d using some linear positional encoding T R rxd R f.xa". Next, latent repre- sentation of the encoded sequence is learned by a multi-headed self-attention mechanism which splits T(X) into H distinct query, key, and value sequences: Qs = T
h. KA = T A Vn T( respectively, with h ... H and weight matrices W W W E Rd''d' The multi-head self-attention calculation for each head is given by
QuK
O₂ softmax
Vn.
Vdl.
and the latent representation is projected to the output space RR using multi-layer perceptron (MLP).
The input length L of the MLP and the Transformer is fixed by assumption. To evaluate the Transformer time-series X with |Tx| * n. must perform extrapolation, etc.) which may degrade the performance of the model. Furthermore, the memory and time complexity of the Transformer is of order OLL d), which presents substantial difficulty in modelling long sequences.
transformation (interpolation,
Rough Path Signatures. Broadly, the difficulties faced by the Transformer in modelling time- series stem from time-series being sampled from underlying continuous-tin objects, while the attention mechanism underpinning the Transformer is designed to model discrete sequences. To address these difficulties, Rough Transformers augment standard Transformers by lifting the input time-series the space of continuous time functions and performing the self attention calculation in this
space. To achieve this, we use the path signature from Rough Path Theory.
For a continuous-time path X E C2 (R+; IR and times s, E R+, the path signature of X from to denoted S x) s,,, is defined as follows. First, let
Ia (1.,,... ..., ip) ig (l,..., d) j and E N)
(2)
denote the set of all d-multi-indices and Iy ( E Ia ||| n). Next, set S(X)g, == 1 and for any I E Int define
S(X)1, = ff J. Cenusenncuee " (wn)... ... (x,) dun dup
x"
(3)
where X² =d8 /dt. Abusing notation, define level n of the signature as
("())s.t -1
(4)
3

---

and define the signature as the infinite sequence
S(X);, = (S( st.), (())),,..,,,,),,.......
(5)
Finally, define the truncation of the signature S()) En = (S(x)) )2.t..... "(, S(X),,), where S ) can be interpreted element of the extended tensor algebra of Rd:
T((R))) = {(ao.... ap...) ...: : an E Rdd3a)
(6)
Analogously, we say that 5F E s(. A central property of the signature is that is invariant with respect to time-rep aim-rrppramettrrizaton [51]. That is, let [0, 7] - [0, be surjective, continuous, and non-decreasing. Then we have
S(X),r = S(X 7),,T,
which will be crucial to demonstrate the Rough Transformer's robustness to irregularly sampled data.
In contrast wavelets Fourier transforms, which parameterize paths on functional basis, the signature provides a basis for functions of continuous paths. Hence, the path signature is well-suited sequence modelling tasks in which seeks learn function of the underlying functional. For a more rigorous presentation of signatures and a description of additional properties, see Appendix A and Lyons al. [5]].
Rough Transformers
Now, we construct the Rough Transformer, Transformer- architecture that operates on continuous- catiiuoous-tmme sequential data by means of the path signature.
Let 'D be dataset of inregularly sampled time-series. To project discretized time-series X D to continuous-time object, let X denote the piecewise- interpolation of X Next, for t T, define the multi-view signature
M = ((X)
(8)
In what follows, we refer to the components (S(8)o. 5(8) (0) as global and local, respec-
efficient representation of long-term information (see Theorem A.2 Appendix A). and the local component as type of convolutional filter that is invariant to the sampling rate of the signal. Now, define the multi-view sig- nature transform M (X)= =
tively; Figure I1 Intuitively,
interpret the global component
...,, M (X))). L)) and denote by M(X) Sn the truncated signature for a truncation level n. Next, define the multi-view attention mechanism, which uses the multi-view signature transform to extend the standard attention mechanism to the space of continuous functions
[51] First, fix truncation level N, and let e N be such that M (X) ) Rdd For h = 1,.., let Q.K.K R.V E dxd for some d' E N, and let
a M() W K, = M(X)
V% = M(X)SWW..
(9)
Then, the attention calculation is given by
Qn Va
O₂ softmax
V
(10)
Notice that the attention calculation is similar to I however, we stress that the multi-view attention built on contimuus--th time objects, the signatures, while the standard attention mechanism acts discrete objects. The multi-view signature provides compressed representation of the time series, minimizing the computational costs associated to quadratic scaling without excessive loss of representational capacity, see Appendix
2 Any continuous-tin time interpolation of X can be used, e.g. splines. However, the signature computation of Seccewise-linedd paths is particularly fast; see Appendix

---

S750 (LAz Smale)
S/1 (Leg Scata)
3000
3000
2000
0.00
00.0
2000
1000
2-2
Inpert Length Stale)
oifirer + Dhaa 303m
Figure 2: Seconds per epoch for growing input length and for different model types on the sinusoidal dataset. Left: Log Scale. Middle: Regular Scale. Right: Log-log scale. When a line stops, it indicates an OOM error.
3.1 Advantages of Rough Transformers
Computational Efficiency. As demonstrated in Section 4 multi-view attention mechanism can substantially reduce the computational cost of vanilla Transformers. In particular, the attention calculation decreases from O d) in the vanilla case to O d), where L << with Rough Transformers. This enables both faster wall-clock training time and the ability to process long input sequences which would otherwise yield out-of-memory errors for the vanilla Transformer, see Figure Moreover, the multi-view attention mechanism does not require backpropagation through the signature calculation, which can be computed offtine. This is significantly more computationally efficient compared with the complexity of computing signatures batch-wise in every training step. Finally, the signature of piecewise-linee paths can be computed explicitly, see Appendix A and there are a number of Python packages devoted optimized signature calculation 43
Variable Length and Irregular Sampling. The multi-view signature transform underpinning Rough Transformers is evaluated by constructing continuous-time interpolation of input data and computing a series of iterated integrals of this interpolation. The bounds of these integrals are a fixed set of time points, meaning that the sequence length of the multi-view attention mechanism is fixed and independent of the sequence length of input samples. Furthermore, the following proposition shows that the output of the Rough Transformer for two (possibly irregular) samplings of the same path is similar.
Proposition 3.1. Let T. be a Rough Transformer. Suppose X [0 + Rd is contimuous-time process, and let y [0, 27 + [0, denote a tine-reparamettrizaaionn Suppose X and X are samplings of X and X 7, respectively. Then T(X Rs T(X)).
Proof. By 5(X) ) = S( o y),,, for all 8, E [0, [) Hence, one has M(X') x M(X²). Finally, T(X) RJ T(X2) because the attention mechanism and final MLP are both continuous. O
Hence, the Rough Transformer is robust irregular sampling. In many tasks, the sampling times convey important information about the time-series. In these settings, one may augment the input time-series with its sampling times, that is, write X ((lo, Xo), (4,)
Spatial Processing. While an interpolation of input data could be sampled to make vanilla Trans- formers independent of the length of the input sequence, important locality information could be lost, see Appendix F.2 Instead, Rough Transformers summarize spatial interactions between channels by means of the multi-view signature transform. One may notice that in (5). the dimension of the signature grows exponentially in the level of the signature n. particular, when X; E Rdd (S()) Jo, d(d
(, so the multi-view attention calculation is of order O(E2d"). In many practical time-series modelling problems, however, the value of not very large. The signature terms also decay factorially in the signature level (see Proposition| A.3 in Appendix A) so in practice, one may take the value of n to be small without sacrificing performance. The majority of computational savings result from the reduction of the sequence length to L, and in practice, take L <<<> L.
d-i
When the dimension d is large, there are three possible remedies to maintain computational efficiency. First, instead of computing the signature in M (X); (S(X)o,4 8(Xe)1,ta) may compute
5

---

the log-stie signanuaa,e, which is a compressed version of the signature [67]. When the dimension is large enough such that the sig-signature computationally infeasible, one may instead compute the univariate signatures of features coupled with the time channel. That is, consider X E C[ T); R²), with (t) (X1 (t).......
Denote the time-added function (t) == (t, (t)). Then we define the univariate nulti-view signature
- (Xxa)a)
(11)
The attention mechanism in this case is constructed as before. Fixing the maximum signature depth to be some value n²', one sees that the number of features in the univariate multi-view signature is approximately 2n d. In practice we find that n² < 5 provides sufficient performance, so the order of the attention calculation is O L d) for C 2" Finally, one may use randomized signatures to reduce dimension by using a Johnson- LinissonLnndenttraus----o type projection a low-dimensional latent space and computing the signature in this space, as in [21 19
4 Experiments
In this section, we present empirical results for the effectiveness of the Rough Transformer, hereafter denoted RFormer, on variety of time-series-related tasks. Experimental and hyperparameter details regarding the implementation
the method are in Aapendices/(and]] We consider long multivariate time-series as main experimental setting because we expect signatures to perform best in this scenario. Additional experimentation on long-range reasoning tasks on image-based based datasets is left for future work, as these would likely require additional inductive biases.
To benchmark RFormer, we consider both discrete-time and continuous- models. particular, we include as main baselines traditional RNN models (GRU (15)), ODE-based methods designed for sequential data (Neural CDE (45)), as well as ODE-based methods explicitly designed for long time-series (Neural-RDE (57) Furthermore, we compare against vanilla Transformer [86]
which the RFormer backbone. Finally, we present comparisons with a recent continuous-time Transformer model, Cont Former [13). to highlight the computational efficiency gap between RFormer and similar continuous-time models. We note that the first two tasks focus on evaluating the performance improvement of RFormer over the Transf ormer baseline. For other mangragge tasks, we include comparisons to recent state-space models [3 62 81]. In the irregular sampling regime, we benchmark against state-of-ther of-thh-art models tailored to that setting [61 78) See Appendix for additional discussion on related models and more details about our experimental choices.
4.1 Time Series Processing
Frequency Classification. Our first exper- iment is based on a set of synthetically gen- erated time series from continuous paths of the form
10-
1.0
0.8
0.5
Traufimer
Accureey
0.6
NRD8
{(t)= = g(t sin(w + +v + mtt),
(12)
0.4
GRU
where g(t) is non-linear trend component, y and 7) are two noise terms, and w is the frequency. Here, the task of the model is to classify the time-series according to its fre- queney We consider 1000 samples in 100 classes with w evenly distributed from 10 to 500. Each time-series is regularly sampled with 2000 times-steps on the interval [0, 1).
0.2
0.0
230
300
730
1000
250
300
700
1000
Epoch
Epoch
Figure 3: Test accuracy per epoch for the frequency classification task across three random seeds. Left: Sinusoidal dataset. Right: Long Sinusoidal dataset.
This synthetic experiment is similar to others in recent work time-series modelling
49
89
55
We include an additional experiment in which we alter the signal in (12) so its frequency is Wp for t < to and Wi afterward, where the task is to classify the sinusoid based on the first frequency. We call this dataset the "long sinusoidal" dataset. This extension of the original experiment aims to test the ability of the model to perform long-range reasoning effectively. Note that for this task, we also add ODE-RNI [70] to the previously mentioned baselines.
'e only benchmark Neural- -DE models in settings where time series are of relatively short length, due to the computational demands of this model for longer sequences.

---

Figure 3 shows that the inclusion of both local and global in- formation with the multi-view signature enhances the sample model,
Table Test RMSE (mean + std) computed across five seeds on the Heart Rate (HR) dataset.
efficiency of the RFormer over the vanilla Transf even though the attention mechanism now operating on a much shorter sequence. When compared with other models, we see that GRU and ODE-RNN fail to capture the information the signal, and are not able to obtain any meaningful performance improvement throughout the training period. This highlights the shortcomings of most RNN-baseds models when processing sequences of mod- erate length, which are very common in real-world applications. Both Neural-CDS and Neural -RDral-RDE capture some useful depen- dencies in the time series but fall short compared with both vanilla Transformer and RFormer.
HR
Model
RMSE 1
ODE-RNN
13.06 0.00
Neural-CDF:
9.82 + 0.34
Neural-RDFF RDE
2.97 + 0.45
GRU
13.06 1 0.00
ODE-RNNI
13.06 0.00
Neural-RDE'
4.04 + 0.11
HR dataset. Next, we consider the Heart Rate dataset from the TSR archive 83 originally sourced from Beth Israel Deaconess Medical Center (BIDMC). This dataset consists of time-series sampled from patient ECG readings, and each model is tasked to perform a regression by forecasting the patient's 's heart rate (HR) at the sample's conclusion. The data, sampled at 125Hz, consists of three- cireecchanne time-series (including time), each spanning 4000 time steps. We used the L2 loss metric to assess the performance, Table I shows the results, where denotes the results from Morrill et al. [57] and f our reproduction. The sequences in the HR dataset are sufficiently short to remain within memory when running the Transf ormer model. The baseline Transformer model improves however, is less competitive when compared with Neural-RDE, suggesting that the Transformer is not particularly well-suited for this type of task. However, the RF RFormer model improves over the baseline Transformer by 67%. Across all tasks, we see significant improvements in efficiency as consequence of the signature computation. We elaborate on this in more detail in the following subsection.
Transformer
8.24 + 2.24
ContiFormer
OOM
RFormer
2.66 0.21
GRU, and ODE-RNN,
Long Time Series Classification. We now evaluate the performance of RFormer five long time series classification tasks from the UEA time series classification archive [3]. A summary these datasets is provided in Table 13 in Appendix [ As previously done in (57). the original train and test datasets are merged and then randomly divided into new train, validation, and test sets, following a 70/15/15 split. The resulting performance metrics are summarized in Table 2
In this setting, we see that RFormer generally matches or slightly outperforms the continuous-time and SSM baselines. Due to the scaling problems of ContiFormer with respect to sequence length, we were unable to run this baseline within GPU memory constraints in most cases, and thus no results reported (see Appendix G.2 for efficiency comparisons between models). In contrast, RFormer can cheaply train on the same device (see Section 4.2 for details) due to its ability to take advantage of the parallel nature of GPU processing and compress the original time series. This is especially models (Neural-CDS Neural-RDE LogCDE), which sometimes orders of magnitude slower than model and consistently report lower similar results. Additional experimental details can be found in Appendix G as well as some experiments on hyperparameter sensitivity.
noticeable when compared to continuous-
Table 2: Classification performance on various long context temporal datasets from UCR TS archive.
Dataset
LRU
S5
S6
Mamba
NCDE
NRDE
LogNCDE
Transformer
RFormer
SCP1
82.6. 1 3.4
89.9
1 4.6
82.8
1 2.7
80.7
: 1.4
79.8
1. 5.6
80.9
1 2.5
83.1
1 2.8
84.3 6.3
81.2 1: 2.8
SCP2
51.2
50.5
+ 26
49.9
+ 9.5
48.2
= 3,9
53.0
+ 28
53.7
+ 6.9
53.7
+ 4.1
49.1 = 2.5
$2.3 : 3.7
MI
5.0
47.7
5.5
51.3
1 4.7
47.7
4.5
49.5
1
2.8
47.0
5.7
53.7
5.3
50.5 3.0
55.8 1. 6.6
EW
2.8
81.1
3.7
85.0
+: 16.1
70.9
+
15.8
75.0
+: 3.9
83.9
+ 73
85.6
00M
90.3 + 0.1
ETC
21.5
2.1
24.1
1:
4.3
26.4
1 6.4
27.9
1: 4.5
29.9
1
6.5
25.3
1
1.8
34.4
1.
40.5
6.3
34.7
1 4.1
HB
78.4
18.4667 6.7
77.7
5.5
76.5
= 8.3
76.2
= 388
73.9
+ 2.6
72.9
+ 488
75.2 4.6
70.5 + 0.1
72.5 = 0.1
Av.
61.7
61.8
62.0
$8.6
60.2
60.6
64.3
59.0
64.5
'We note that baseline results for this task taken from
87
7

---

4.2 Training Efficiency
Here, we focus on the computational gains of the model when compared with vanilla Transformers and methods that require numerical ODE solvers.
Attention- based architectures are highly parallelizable on modern GPUs, as opposed to traditional RNN models which require sequential updating. However, vanilla attention experiences bottleneck in memory and time complexity as the sequence length L grows. As covered above in Section 3 variations of the signature transform allow the model to operate a reduced sequence length L without increasing the dimensionality in a way that would become problematic for the model. This allows us to bypass the quadratic complexity of the model without resorting to sparsity techniques commonly used in the literature [26 49].
Tables 3 show that RFormer is competitive when mod- elling datasets with extremely long sequences without an explosion in the memory requirements. RFormer ex- ploits the parallelism of the attention mechanism to sig- nificantly accelerate training time, as the length of the input sequence is decreased substantially. In particular, we observe speedups of 1.4x to 26.17 1..11x x with respect to standard attention, and higher when compared with all methods requiring numerical solutions to ODEs. The com- putational efficiency gains of RFormer are attained due to the signature transform reducing the length of the time- series with minimal information loss. The effectiveness of this transformation can be seen from the ablation study car- ried out in Appendix [F] This contrasts with NRDEs (57), which augment NCDEs with local signatures of input data, and find that smaller windows often perform better. Fur- thermore, NRDEs do not experience the same computational gains as RFormer because they must perform many costly ODE integration steps.
Table 3: Seconds per epoch for all mod- els considered.
Sec. Epoch
Model
Sine
EW
HR
GRU
0.12
0.25
1.07
ODE-RNN
5.39
48.59
50.71
Neural-CDE
9.83
Neural-RDD
0.85
5.23
9.52
Transformer
0.77
OOM
11.71
RFormer
0.55
0.11
0.45
Speedup
1.4x
26.11x
In Figure tional efficiency of RFormer compared to vanilla Trans- formers [6). continuous-timg Transformers [3], and other continuous-time RNNs [45] 57 when processing sequences from L= 100 samples up L. 10K. As seen, RFormer is significantly more efficient than its continuous-time and vanilla counterparts, even when performing the signature computation online, which involves computing the signa- tures for each batch during training, resulting in significant redundant computation. When signatures are precomputed just once before training, the computational time of each epoch remains constant across input all sequence lengths including I, 10K (see the exact signature computation times for different datasets in Table We also stress the fact that RFormer also scales gracefully for extremely long sequences (up to L = 250K) with both online and offline computation of the signatures, shown in Appendix G Finally, we highlight that ContiFormer has sample complexity of O OLL"d S), where represents the normalized number of function evaluations of the numerical ODE solver, which makes Former orders of magnitude more computationally intensive when compared to RF Formerr and prevents the model from running on sequences longer than 500 points without running out of memory (see device details in Appendix C).
showcase the improvements in computa-
Table 4: Dataset processing times for training, validation, and testing phases.
Dataset
Train
Val
Test
Eigenworms
1.11 s. 0.19
0.19
HR
4.23 s. 0.84 s. 0.85
Sine (1k)
0.39 s. 0.39 s. 0.39 s.
Sine (5k)
0,51 s. 0.51 s.
0.51 s.
Sine (20k)
1.64 s. 1.64
1.64 s.
Sine (100k)
5.74 s.
5.74
5.74 s.
4.3 Irregular Time Series Classification
So far, we mainly focused on the efficiency and inductive bias afforded to the model through the use of signatures. However, key element of RFormer is that it can naturally deal with irregularly sampled sequences without expensive numerical ODE solvers. This property follows from the fact that signatures are invariant to time reparameterization see Proposition 3.. this subsection, we empirically test this property by training the model on the same datasets but randomly dropping percentage of the data points. This test intends to find if the model able to learn continuous-time representations of the original input seriss. The results can be found in Table We find that
8

---

RFormer consistently results in the best performance, with a small performance drop when compared the full dataset. Importantly, this property achieved in conjunction with the efficiency gains afforded to the model and without the use of expensive numerical ODE solvers Finally, we perform an additional set of experiments on the 15 univariate classification datasets from the UEA time series classification archive and compare our model with recent state-of-the-a art models for irregular time series [61] 78). Across the board, we find that our model is both faster and more accurate than the continuous-time benchmark despite having discrete-time Transformer backbone, as shown in Figure which introduces Continuous Recurrent Units (CRU) 78) as an extra baseline. For more details and more exhaustive experimentation random data drops, see Appendix G
Table 5: Performance of all models under a random 50% drop datapoints per epoch.
50% Drop Performance
Model
EW (%) f
HR |
Sine (%)
Sine Long (%)
GRU
35.90
13.06
0.96
1.16
ODE-RNN
37.61
13.06
1.06
1.23
Neural-RDE
60.68
4.67
0.94
0.87
Transformer
OOM
12.73
7.37
20.23
RFormer
87.69
2.96
59.57
93.17
5
Reasons for improved model performance
In this final section, we provide explanations for the superior inductive bias of the RFormer model compared to its vanilla Transformer counterpart, despite its lower computational cost.
5.1 Spatial Processing
First, highlight that a key reason the model achieves significant compression benefits in the tasks considered is its ability to jointly account for temporal and spatial interactions through the self-attention mechanism and signature terms, respectively. In particular, we believe that for certain datasets, the relationships between different channels of the time series may hold more importance than the temporal information itself, which can often be redundant. This is exemplified in the Ei gigenworms dataset, which experiences 20% performance drop when employing univariate signatures, but able to achieve state-of-the-art performance with a 600 compression rate in temporal dimension when signatures are applied across all channels, as shown in Figure To this end, we draw parallels between the use of signatures and the field temporal graph processing, all channels can be seen as a fully connected graph, capturing
where the use of the signature information from channels, and the univariate signature would correspond to a graph with only self-connections between the nodes, as depicted in Figure 5 In our view, this hints towards the idea of using sparse graph learning techniques 17 23 to reduce the explosion in signature terms while retaining the ability to perform effective spatial processing.
To empirically test these claims, we design a synthetic experiment using 2-channel time se- ries. Each channel contains a signal of the form sin(w,t + 14), 2, where Wi and i4 ran- domly sampled from the interval [0, 2r). For half of the dataset, the last 1% of temporal sam- ples in the second channel are set to match the frequency of the first channel. The task is to classify whether the samples in this final inter- val are of the same frequency. As shown in Figure 5 RFormer demonstrates greater sam- ple efficiency and achieves higher test accuracy compared to its vanilla Transformer counterpart, highlighting the effectiveness of signatures in spatial processing.
CRU
Acomar (%)
Nounl LSDE
80-
Newal LNSDE
Neural GSDE
Nowal CDE
30%
50%
70%
Doop Poresttgg
Figure 4: Average performance of all models on the 15 univariate datasets from the UEA Time Se- ries archive under different degrees of data drop.
*ee use our own reproduction to test the performance of all models in irregularly sampled datasets. Random dropping requires window sizes larger than 2 because signatures cannot be computed over single point. The best step size chosen in accordance with performance on the validation dataset, see Appendix D

---

210
380
100
0.5
0.0
BFanno TrenSeeser
0.0
210
100
200
(30
1993
2ome
Spreck
Figure 5: Left: Graph connectivity structures for multivariate, univariate and sparse signature. Middle: Example samples for synthetic task. Right: Performance spatial synthetic experiment.
5.2 Sequence Coarsening as an Inductive Bias for Transformers
In addition to the benefits of higher-order signature terms, empirically observe that using level-one signature terms resulted in performance improvements when compared to processing sequences without any transformation. We believe that the reduction in input signal length, achieved without significant information loss through the signature transform is another important factor in the improved inductive bias of RFormer. This finding aligns with the concurrent work of 44, which highlights some of the drawbacks of decoder-only Transformers for long sequences in terms of both oversquashing and representational collapse.
100
24
Mor 1201
5003 13000
Figure 6: Left: Dirichlet energy as a function of window size for the Eigenworms dataset. Right: Original and hidden representation after signature layer for two examples in the EW dataset.
To measure the degree of coarsening in the sequence, we find that interpreting the temporal sequence as a path graph and using ideas from the oversmoothing literature (75) serves as a good way to measure the similarity of the representations being fed to the Transformer. In particular, we compute Σm |X, X,-1lla of the temporal sequence resulting from taking increasing window sizes the global signature. An example this shown in Figure 6 for the Eigenworms dataset, where compared different numbers of windows (from 2 to 18k). Interestingly, we found that the "elbow" the Dirichlet energy corresponded to 30 windows in this dataset, which we found empirically to be of the most performant settings. This hints at the idea of the Dirichlet energy being used for signature hyperparameter tuning as well.
the Dirichlet Energy [74], defined in this
E(X)
Conclusion
We introduced the Rough Transformer, variant of the original Transformer that allows the processing of discrete-time series as continuous-time signals through the use of multi-view signature attention. Empirical comparisons showed that Rough Transformers outperform vanilla Transformers and continuous-time models variety of time-series tasks and are robust to the sampling rate of the signal. Finally, we showed that RFormer provides significant speedups in training time compared to regular attention and ODE-based methods, without the need for major architectural modifications or sparsity constraints.
Impact Statement
This work is unlikely to result in any harmful societal repercussions. Its primary potential lies in its ability enhance and advance existing data modelling and machine learning methods.
Acknowledgements
We thank Christopher Salvi, Antonio Orvieto, Yannick Limmer, and Benjamin Walker discussions at different stages of the project. AA acknowledges support from the Rafael Del Pino Foundation and a G-Research travel grant. XD acknowledges support from the Oxford-Mr Institute of Quantitative Finance.
10

---

References
[1] M. Arjovsky, A. Shah, and Y. Bengio. Unitary evolution recurrent neural networks. In International conference machine learning, pages 1120-1128. PMLR, 2016.
[2] I. P. Arribas. Derivatives pricing using signature payoffs. arXiv preprint 1809:::8000000000 2018.
[3] A. Bagnall, H. A. Dau, J. Lines, M. Flynn, J. Large, A. Bostrom, P. Southam, and E. Keogh. The uea multivariate time series classification archive, 2018, arXiv preprint arXiv: 1800005 2018.
[4] F. Barbero, A. Banino, S. Kapturowski, D. Kumaran, J. G. Araújo, A. Vitvitskyi, R. Pascanu, and P. Veličković. Transformers need glasses! information over-squashing in language tasks. arXiv preprini arXiv:2406.0422 04266777 2024.
[5] I. Beltagy, M. E. Peters, and A. Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.055.50 2020.
[6] M. Biloš, J. Sommer, S. S. Rangapuram, T. Januschowski, and S. Günnemann. Neural flows: Efficient alternative to neural odes. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processsing Systems, volume 34, pages 21325 21337. Curran Associates, Inc., 2021. URL URL URL/attps://prroceddnng neurips.com cc/ paper_
Paper pdf
[7] S. Calvo-Ordonez, J. Huang, L. Zhang, G. Yang, C.-B. Schonlieb, and A. I. Aviles-Rivero. Beyond u: Making diffusion models faster & lighter. arXiv preprint arXiv:2310.20092. 2023.
[8] Á. Cartea, S. Jaimungal, and J. Penalva. Algorithmic and high-frequency trading. Cambridge University Press, 2015.
[9] Á. Cartea, G. Martin, and L. Sánchez- Betancourt. Detecting toxic flow. arXiv preprint arXiv:23/2.05827, 2023.
[10] B. Chang. M. Chen, E. Haber, and E. H. Chi. Antisymmetricrmm: A dynamical system view on recurrent neural networks. In International Conference on Learning Representations, 2018.
[11] P. Chang, G. DurannMartin, A. Y. Shestopaloff, M. Jones, and K. Murphy. Low-rank extended kalman filtering for online learning of neural networks from streaming data. arXiv preprint arXiv:2305.199 19535, 2023.
[12] R. T. Chen, Y. Rubanova. J. Bettenncour,, and D. K. Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018
[13] Y. Chen, K. Ren, Y. Wang, Y. Fang, W. Sun, and D. Li. Contiformer: Continuous time trans- former for irregular time series modeling. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[14] R. Child, S. Gray, A. Radford, and I. Sutskever. Generating long sequences with sparse transformers. arXiv preprini 1904::9904..0555 2019.
[15] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using mn encoder-decoder for statistical machine translation. arXiv preprint 1406. 1078, 2014.
[16] K. M. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins, J. Q Davis, A. Mohiuddin, L. Kaiser, et al. Rethinking attention with performers. In International Conference on Learning Representations, 2020.
[17] A. Cini, I. Marisca, D. Zambon, and C. Alippi. Taming local effects in graph-based spatiotem- poral forecasting. Advances Neural Information Processing Systems, 36, 2024.
[18] N. M. Cirone, A. Orvieto, B. Walker, C. Salvi, and T. Lyons. Theoretical foundations of deep selective state-space models. arXiv preprint arXiv:2402. 19047, 2024.

---

[19] E. M. Compagnomi, A. Scampicchio, L. Biggio, A. Orvieto, T. Hofmann, and J. Teichmann. On the effectiveness of randomized signatures as reservoir for learning rough dynamics. In 2023 International Joint Conference Neural Networks (IJCNN), pages 1-8. IEEE, 2023.
[20] Corsi. simple approximate oong-memory model of realized volatility. Journal of Financial Econometrics, 7(2):174-196 2009.
[21] C. Cuchiero, L. Gonon, L. Grigoryeva, J.-P. Ortega, and J. Teichmann. Discrete-tiie tieeete-timeeeee signatures and randomness in reservoir computing. IEEE Transactions on Neural Networks and Learning Systems, 33(1 1):6311-6330, 2021.
[22] T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré. Flashattention: Fast and memory-efficien exact attention with io-awareness. Advances in Neural Information Processing Systems, 35: 16344-16359, 2022.
[23] H. S. de Ocáriz Borde, A. Arroyo, and I. Posner. Projections of model spaces for latent graph inference. ICLR 2023 Workshop on Physics for Machine Learning, 2023.
[24] E. Dupont, A. Doucet, and Y. W. Teh. Augmented neural odes. Advances in neural information processing systems, 32, 2019.
[25] N. B. Erichson, O. Azencot, A. Queiruga, L. Hodgkinson, and M. W. Mahoney. Lipschitz recurrent neural networks. In International Conference on Learning Representations, 2020.
[26] A. Feng, I. Li, Y. Jiang, and R. Ying. Diffuser: efficient transformers with multi-hop attention diffusion for long sequences. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pages 12772-12780, 2023.
[27] A. Fermanian. Embedding and learning with signatures. Computational Statistics & Data Analysis, 157:107148, 2021.
[28] C. Fleming, D. Sheldon, W. Fagan, P. Leimgruber, T. Mueller, D. Nandintsetseg, M. Noonan, K. Olson, E. Setyawan, A. Sianipar, et al. Correcting for missing and irregular data home- range estimation. Ecological Applications, 2003):1003-1010, 2018.
[29] E. Fons, A. Sztrajman, Y. El-Laham, A. Iosifidis, and S. Vyetrenko. Hypertime: Implicit neural representation for time series. arXiv preprint arXiv:2208.05888 2022.
[30] K.-i. Funahashi and Nakamura. Approximation of dynamical systems by continuous time recurrent neural networks. Neural networks, 6(6):801-806, 1993.
[31] A. Gu and T. Dao. Mamba: Linear-time sequence modeling with selective state spaces, arXiv preprint arXiv:2312.00752. 2023.
[32] A. Gu, K. Goel, and C. Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2021.
[33] B. Hambly and T. Lyons. Uniqueness for the signature of path of bounded variation and the reduced path group. Annals of Mathematics, pages 109-167, 2010.
[34] R. Hasani, M. Lechner, A. Amini, D. Rus, and R. Grosu. Liquid time-constant networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7657-7666. 2021.
[35] J. M. Hausdorff and C.-K. Peng. Multiscaled randomness: A possible source of 1/f noise in biology. Physical review E, 54(2):1154, 1996.
[36] N. Hautsch. Modelling irregularly spaced financial data: theory and practice of dynamic duration models. Springer Science & Business Media, 2004.
[37] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
12

---

[38] M. Henaff, A. Szlam, and Y. LeCun. Recurrent orthogonal networks and long- memory tasks. In International Conference on Machine Learning, pages
PMLR, 2016.
[39] M. Höglund, E. Ferrucci, C. Heránddzz, A. M. Gonzalez, C. Salvi, L. Beabbeez-,etacooutt,, and Y. Zhang. A neural rde approach for continuous-time non-markovian stochastic control problems. In /CML. Workshop on New Frontiers in Learning, Control, and Dynamical Systems, 2023.
[40] S. I. Holt, Z. Qian, and M. van der Schaar. Neural laplace: Learning diverse classes of differential equations in the laplace domain. In International Conference on Machine Learning, pages 8811-8 PMLR, 2022.
[41] A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret. Transformers are mns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pages 5156-5165. PMLR, 2020.
[42] T. A. Keller, L. Muller, T. Sejnowski, and M. Welling. Traveling waves encode the recent past and enhance sequence learning. In The Twelfth International Conference on Learning Representations, 2023.
[43] Kidger and T. Lyons. Signatory: differentiable computations of the signature and logsignature transforms, both cpu and gpu. arXiv preprint arXiv:2001.00 00776, 2020.
[44] Kidger, P. Bonnier, I. Perez Arribas, C. Salvi, and T. Lyons. Deep signature transforms. In H. Wallach, H. Larochelle, A. Beygelzimer, F. I'Alcéé-Buc, E. Fox, and R. Garnett, ed- itors, Advances in Neural Information Processing Systems, volume 32. Curran Associates. Inc.. 2019. URL httphttss//pprcoodiingg wwwp:/iinnnnn..un....... cc/ wtt.:pcppppppp/p.. http:/ppppaeelsseeeeeeeeaaleeeep wwt.ffffee//pppppe httpa///lllss/pppp////////// 2Bod644477877770222256446444325 Paper. .dff
[45] P. Kidger, J. Morrill, J. Foster, and T. Lyons. Neural controlled differential equations for irregular time series. Advances Neural Information Processing Systems, 33:6696-6707, 2020.
[46] M. Lechner and R. Hasani. Learning long-term dependencies in irreuulaly-samppled time series. arXiv preprint arXiv:2006.04418, 2020.
[47] M. Lemercier, C. Salvi, T. Cass, V. Bonilla, Damoulas, and T. J. Lyons. Siggpde: Scaling sparse gaussian processes on sequential data. In International Conference on Machine Learning, pages 6233-6244 PMLR, 2021.
[48] M. Lezcano-Casado and D. Martinez-Rin Rubio. Cheap orthogonal constraints in neural networks: A simple parametrization of the orthogonal and unitary group. In International Conference on Machine Learning, pages 3794-3880000 PMLR, 2019.
[49] S. Li, X. Jin, Y. Xuan, X. Zhou, W. Chen, Y.-.. Wang, and X. Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in neural information processing systems, 32, 2019.
[50] Z. Li, N. B. Kovachki, K. Azizzadenesheli, K. Bhattacharya, A. Stuart, A. Anandkumar, et al. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2020.
[51] J. Lyons, M. Caruana, and T. Lévy. Differential equations driven by rough paths. Springer, 2007.
[52] V. Melnychuk, D. Frauen, and S. Feuerriegel. Causal transformer for estimating counterfactual outcomes. In International Conference on Machine Learning, pages 15293-15329, PMLR, 2022.
[53] M. Corariu-Patrichl and M. S. Pakkanen. State-dependent hawkes processes and their applica- tion limit order book modelling. Quantitative Finance, 22(3):563-582 2022.
[54] Moreno- and S. Zohren. Deepvol: Volatility forecasting from high-frequency data with dilated causal convolutions. arXiv preprint arXiv:2210.04797,, 2022.
13

---

[55] F. Moreno-Pino, P. M. Olmos, and A. Artés-Rodriguez. Deep autoregressive models with spectral attention. Pattern Recognition, 133:109014, 2023.
[56] F. Moreno-Pino, Á. Arroyo, H. Waldon, X. Dong, and Á. Cartea. Rough transformers for continuous and efficient time-series modelling. arXiv preprint arXiv:2403. 10288, 2024.
[57] J. Morrill, C. Salvi, P. Kidger, and Foster. Neural rough differential equations for long time series. In International Conference on Machine Learning, pages
PMLR, 2021.
[58] T. Nguyen and A. Grover. Transformer neural processes: Uncertainty-aware meta learning via sequence modeling, In International Conference on Machine Learning, pages 16569-16594. PMLR, 2022.
[59] A. Norcliffe, C. Bodnar, B. Day, Moss, and P. Liò. Neural ode processes. In International Conference Learning Representations, 2020.
[60] A. Norcliffe, C. Bodnar, B. Day, N. Simidjievski, and P. Liò. On second order behaviour in augmented neural odes. Advances in neural information processing systems, 33:591 1-5921, 2020.
[61] Y. Oh, D. Lim, and Kim. Stable neural stochastic differential equations analyzing irregular time series data. In The Twelfth International Conference on Learning Representations.
[62] A. Orvieto, S. L. Smith, A. Gu, A. Fernando, C. Gulcehre, R. Pascanu, and S. De. Resurrecting recurrent neural networks for long sequences. International Conference on Machine Learning, pages 16670-26699) PMLR, 2023.
[63] Y. Park, J. Choi, C. Yoon, M. Kang, et al. Learning pde solution operator for continuous modeling of time-series. arXiv preprint urXiv:2302.00854 2023.
[64] I. Perez Arribas, G. M. Goodwin, J. R. Geddes, T. Lyons, and K. E. Saunders. A signature- machine learning model for distinguishing bipolar disorder and borderline personality disorder. Translational psychiatry, 8(1):274, 2018.
[65] S. Perveen, M. Shahbaz, T. Saba, K. Keshavjee, A. Rehman, and A. Guergachi. Handling irregularly sampled longitudinal data and prognostic modeling of diabetes using machine learning technique. IEEE Access, 8:21875 21885, 2020.
[66] R. Ratcliff, P. L. Smith, S. D. Brown, and G. McKoon. Diffusion decision model: Current issues and history. Trends in cognitive sciences, 20(4):260-281, 2016.
[67] J. Reizenstein. Calculation of iterated-integral signatures and log signatures. arXiv preprint XrIi 2017.
[68] J. Reizenstein and B. Graham. The iisignature library: efficient calculation of iterated- -irtteddintegaal signatures and log signatures. arXiv preprint 102.02280,.,9252, 2018.
[69] D. W. Romero, A. Kuzina, E. Bekkers, M. Tomezak, and M. Hoogendoorn. Ckconv: Continuous kernel convolution for sequential data. In International Conference on Learning Representations.
[70] Y. Rubanova, R. T. Chen, and K. Duvenaud. Latent ordinary differential equations for irreuularl-samppled time series. Advances in neural information processing systems, 32, 2019.
[71] T. K. Rusch and S. Mishra. Coupled oscillatory recurrent neural network (cornn): An accu- rate and (gradient) stable architecture for learning long time dependencies. In International Conference Learning Representations, 2020.
[72] T. K. Rusch and S. Mishra. Unicornn: A recurrent model for learning very long time de- pendencies. In International Conference on Machine Learning, pages 9168-9178, PMLR, 2021.
[73] K. Rusch, S. Mishra, N. B. Frichson, and M. W. Mahoney. Long expressive memory for sequence modeling. In International Conference Learning Representations, 2021.
14

---

[74] T. K. Rusch, B. Chamberlain, J. Rowbottom, S. Mishra, and M. Bronstein. Graph-coupled oscillator networks. In International Conference on Machine Learning, pages 18888-18909. PMLR, 2022.
[75] T. K. Rusch, M. M. Bronstein, and S. Mishra. A survey on oversmoothing in graph neural networks. arXiv preprint arXiv:2303. 10993, 2023.
[76] H. Sáez de Ocáriz Borde, A. Arroyo, I. Morales, I. Posner, and X. Dong, Neural latent geometry hhasdorfff-nffrmmed bayesian optimization.
search: product manifold inference via Advances in Neural Information Processing Systems, 36, 2024.
[77] C. Salvi, M. Lemercier, C. Liu, B. Horvath, T. Damoulas, and T. Lyons. Higher order kernel mean embeddings to capture filtrations of stochastic processes. Advances in Neural Information Processing Systems, 34: 1663--66667, 2021.
[78] M. Schirmer, M. Eltayeb, S. Lessmann, and M. Rudolph. Modeling irregular time series with continuous recurrent units. In International conference on machine learning, pages 19388- 19405. PMLR, 2022.
[79] N. Seedat, F. Imrie, A. Bellot, Z. Qian, and M. van der Schaar. Continuous- time modeling of counterfactual outcomes using neural controlled differential equations. In International Conference Machine Learning, pages 19497-19521. PMLR, 2022.
[80] V. Sitzmann, J. Martel, A. Bergman, D. Lindell, and G. Wetzstein. Implicit neural representations with periodic activation functions. Advances in neural information processing systems, 33: 2020.
[81] J. T. Smith, A. Warrington, and S. Linderman. Simplified state space layers for sequence modeling. In The Eleventh International Conference on Learning Representations.
[82] C. Tallec and Y. Ollivier. Can recurrent neural networks warp time? arXiv preprint arXiv:1804.111 11 2018.
[83] C. W. Tan, C. Bergmeir, F. Petitjean, and G. I. Webb. Monash university, uea, uer time series extrinsic regression archive. arXiv preprint arXiv:2006. 10996, 2020.
[84] A. Tong, T. Nguyen-Tang for deep hedging, In Proceedings of the Fourth ACM International Conference on Al in Finance, pages 124-132, 2023.
Lee, T. M. Tran, and J. Choi. Sigformer: Signature transformers
[85] A. Vahid, M. Mückschel, S. Stober, A.-K. Stock, and C. Beste. Applying deep learning single- trial eeg data provides evidence for complementary theories on action control. Communications biology, 3(1):112, 2020.
[86] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[87] B. Walker, A. D. McLeod, T. Qin, Y. Cheng, H. Li, and T. Lyons. Log neural controlled differential equations: The lie brackets make a difference. arXiv preprint arXiv:2402. 12, 2024.
[88] S. Wang, B. Z. Li, M. Khabsa, H. Fang. and H. Ma. Linformer: Self-attention with linear complexity. arXiv preprint wrXiv:20006044666 2020.
[89] Yoon, D. Jarrett, and M. Van der Schaar. Time-series generative adversarial networks. Advances in neural information processing systems, 32, 2019.
[90] M. Zaheer, G. Guruganesh, K. A. Dubey, J. Ainslie, C. Alberti, S. Ontanon, P. Pham, A. Ravula, Q. Wang, L. Yang, et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 13:1323--1229 2020.
[91] A. Zeng, M. Chen, L. Zhang, and Q Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 11 11121-11128 2023.
15

---

A Properties of Path Signatures
First, recall that the path is uniquely determined by its signature, which motivates its feature map.
Proposition A.1. Given path 8 [0,
Rd, then the map P [0, - R Rl+d where P(() (1, (()) uniquely determined by it's signature ((P)o,T-
The proof can be found Hambly and Lyons
For Rough Transformers, several features of path signatures are important. First, linear functionals on path signatures possess universal approximation properties for continuous functionals.
Theorem A.2. Fix T > O, and let K C Cj (I; R")) Let f K - IR be continuous with respect to the sup-norm topology on C3 (0, T); )d). Then for any > O. there exists linear functional such that
(f(X) - (C,5 Jo, (e, Sc,
(13)
for any X K, where X denotes the time-added uugmentation of 8.
For proof of A.2 see Arribas [2]. Even though Theorem A.2 guarantees that linear functionals are sufficient for universal approximation, linear models are always sufficient practice. This motivates the development of nonlinear models built upon the path signature which efficiently extract path behavior.
The second feature is that the terms of the path signature decay factorially, as described by the following proposition.
Proposition A.3. Given & ((0, T): Rd), for any s, E [0, T), we have that for any I T3
|S(8),rr1= = O (1/nl).
(14)
151. Hence, the number of terms in the signature grows exponentially in the level of the signature, but the tail the signature is well-hehaved, so only a few levels in truncated signature necessary to adequately approximate continuous functionals.
For proof of Proposition A.3
A.1 Signatures of Piecewise Linear Paths.
In the Rough Transformer, we use linear interpolation input time-series to get a continuous-time representation of the data. As mentioned in Section the signature computation in this case is particularly simple.
Suppose Ng [tx, tx41 Rdd linear interpolation between two points Xx, X Rdd. That is,
(a Xx
t t 1++1 la
(Xx41 - Xx)
(15)
Then the signature of Xx is given explicitly by
S ((Kk)entee = (1, - X Xk: ] (X
1
002 31
(X - 00 /. (Xx+1 - Xx) o0n
X2)
(16)
where @ denotes the tensor product. Let 8x 8 R denote the concatenation of Xx and 8 That is, X * Xx+1 [tx, 1x+2] + Rd is given by
(x(()) EE [L,,, 41+11 (x+1(t)
Ne (t+1
(17)
te (tz, tx+2].
The signature of the concatenation Ex 8e41 is given by Chen's relation. whose proof is in (51). To state this result, we first note that S(X) )r, can be interpreted as an element of the extended tensor algebra Rd;
T(RR)) ((ao .... an....) an E ld ddd
(18)
16

---

Proposition A.4 (Chen's 's Relation). The following identity holds:
S(Xx X841)cenu1 S ((Kn)tuntes & 14x
(19)
where for elements A, B E T((R)) with = (Au, A1. A2, and = (Bo, B1, B2, ...... the tensor product @ is defined
A@ B =
Σ J=0
A3 & Bk-j
(20)
R20
Let X = represented as the concatenation of finite number of linear paths:
be a time-series. Then the linear interpolation X :0,2] + R2d can be
x ==
80 81-11
(21)
Hence, the signature is
S(X) = S(Vo)o. .... 0...8((6-iiamam
(22)
B Related Work, Experimental Choices, and Impact Statement
Continuous-time models. Since their introduction 12]. Neural ODEs were extended in various
ways to facilitate modelling continuous time-series data 770 60 34 40 79). While Neural ODEs and their extensions are successful in certain tasks they are burdened with a high computational cost, which makes them scale very poorly to long sequences in the time-series setting, Various authors propose methods and augmentations to vanilla Neural ODEs to decrease their computational overhead [24 6). Other approaches augmenting deep learning methods to modelling continuous data include implicit neural representations [80] 29 continuous kernel convolutions [69], or Fourier neural operators [50 63).
Transformers. First proposed in 86), the Transformer has been exceptionally successful discrete sequence modelling tasks such natural language processing (NLP). Key to the success of the Transformer in NLP is the attention mechanism, which extracts long-range dependencies. There number of extensions to improve efficiency and decrease the computation cost of the attention mechanism
16.
Signatures in machine learning. The path signature originates from theoretical stochastic analysis [51 and has since become a popular tool in machine learning. Path signatures are regarded as effective feature transformations for sequential data [64 27 44. Additionally, signatures help mitigate the computational cost of Neural CDEs in long time-series
[57] and non Markovvakn stochastic control problems (39] Other more recent works in this direction include [18. 87). Approaches such as randomized signatures [2 19 and the signature kernel (77) have been developed to mitigate the curse of dimensionality inherent in path signature computations, Rough Transformers provide a first step towards incorporating path signatures for continuous-tima sequence modelling using Transformers.
We also note that contemporary work 84 employs a Transformer architecture with signature features for the task of deep hedging. However, work differs in several key aspects. First, introduce the multi-view attention mechanism, which uses signatures to extract both global and local information, which we found be necessary in our experimentation, as Transformers extracting local information (see Figure 7). whereas their work just uses a global signature. Moreover, their work computes the signature at every time step, strictly dilating input data. This is particularly problematic for long, multi-variatt sequences, for reasons discussed above, and can actually negatively impact performance. Our work. however, compresses data using the multi-view signature transform, and we find that this compressed representation can actually improve performance. Finally, their work relies on the assumption that data regularly sampled, as the signature computed every time step, in contrast to our work which is robust to irregular sampling.
known struggle in
For a preliminary version of this paper, also direct the reader to [56].
17

---

Long-Range Sequence modelling. A highly relevant line of research related to enhancing recurrent
neural networks' capability to capture long-term dependencies involves the development of various
models. These include Unitary RNNs Π. Orthogonal RNNs [38] expRNNs
(48] ChronoL.TT
[2]. antisymmetric RNNs [Q], Lipschitz RNNs 25]. coRNNs 771, unicoRNNs [72], LEMs 73
waveRNN [4). Linear Recurrent Units 62]. and Structured State Space Models [22 3T. While we utilize many benchmarks and synthetic tasks from these works to test our model, it is important to note that our work is not intended to compete with the state-of-the theh-att in these tasks. Therefore, not directly compare our model with the models mentioned above. Instead, this paper seeks to show that the baseline Transformer architecture more computationally efficient, (ii) being invariant to the sampling rate of the signal, and (ii) having good inductive bias for temporal and spatial processing. Furthermore, brings alternative benefits, such as the ability to perform spatial processing effectively, which is a setting in which long-range sequence models typically struggle.
do
benefit from the use of signatures by (i) becoming
highlight that RFormer
Efficient Attention Variants. There are several efficient self-attention variants that have emerged over the years, including Sparse Transformer [4]] Longformer [5], Linear Transformers [41] BigBird [90]. Performer [16], or Diffuser (26). In our setting, we highlight that central part of this paper is to showcase how signatures significantly reduce the computational requirements of vanilla attention and empirically demonstrate that this also results in improved learning dynamics and invariance to the sampling frequency the signal. Given the large efficiency gains that we observed with this approach when employed on vanilla attention, we did not consider that further experimentation on other forms of "approximate" attention was needed. Since most variants of attention seek to make the operation more efficient through several approximations (e.g., lincarization sparsification techniques), believe that a first attempt at showcasing the power of multi-view signatures on vanilla attention is already significant. However, other variants of attention (such as the ones outlined before) could be added on top of the signature representations to obtain even better efficiency gains.
Limitations and Future Work. While found RFormer to very performant in our experiments, much of this performance gain relies on heavy hyperparameter tuning, especially when it comes the choice of window sizes and signature level. However, this could be handled using Neural Architecture Search (NAS) techniques, such as those employed in [76]. Furthermore, despite the computational gains we achieve for low-dimensional sequences, additional work would be required to scale this method to larger dimensions. We should also note that the experiments and results presented in this paper are constrained by the relatively small scale of the models studied.
C Experimental Details
All experiments are conducted on an NVIDIA GeForce RTX 3090 GPU with 24,564 MiB of memory, utilizing CUDA version 12.3. Hyperparameters used to produce the results Table are reported in T The timings presented in all tables are obtained by executing each model independently for each dataset and averaging the resulting times across 100 epochs.
Table 6: Hyperparameters used for Table where G and I. refer to the Global and Local signature components, respectively.
SCP1
SCP2
MI
EW
ETC
HB
Batch Size
20
10
50
10
20
Embedded Dim
10
5
20
20
20
5
Multi- View Terms
[G]
[G]
[L]
[L]]
[G]
[G, L]
Learning Rate
4.08e-3
1.38e-3
4.08e-3
6.73e-3
1.00e-3
7.72e-3
Num. Heads
3
3
I
3
Num. Layers
3
3
2
1
3
Num. Sig Windows
100
50
200
10
400
30
Sig Level
2
3
2
2
1
2
Univariate
true
true
true
false
false
true
Num. Epoch
110
10
26
39
200
16
18

---

Table 7: Hyperparameters validation on remaining datasets.
Dataset
Learning Rate Number of Windows
Sig. Depth
Sig. Type
Univariate/Muutivrriatte Sig.
Sinusoidal
1 x 10-3
75
6
Multi-
HR
1 10-3
75
4
Local
Multivariate
To prevent excessive growth in signature terms, we use the univariate signature LOB datasets. As an alternative, one could employ randomized signatures
[9]
or low-rank approximations 9 [1]
D Baselines Validation
This section collects the validation of Step and Depth for the Neural-RDE model. Optimal values are selected for evaluation on test-set. Early-stopping is used with the same criteria [57]
Table 8: Validation accuracy the sinusoidal dataset.
Acc. Val
Step
Depth
Memory Usage (Mb)
Elapsed Time (s)
17.26
778.9
6912.7
12.21
770.3
1194.43
16.35
382.2
2702.48
19.27
386.16
574.97
20.99
193
1321.36
24.02
3
194.17
332.17
17.15
97.13
136.43
21.59
16
3
98.17
156.93
17.46
65.96
105.94
20.59
24
3
66.68
98.97
Table 9: Validation accuracy on the long sinusoidal dataset.
Acc. Val
Step
Depth
Memory Usage (Mb)
Elapsed Time (s)
11.10
4017.22
2961.98
9.59
3
4008.33
2779.52
10.39
2001. 76
1677.78
10.19
3
2006.80
1615.64
14.03
1004.07
665.55
15.34
3
1005.72
723,41
1.61
503.66
125.85
1.92
16
3
505.28
120.63
1.51
339.80
58.87
2.12
24
3
341.90
69.35
19

---

Table 10: Validation aceuracy on the EW dataset.
Acc.
Val
Step
Depth
Memory Usage (Mb)
Elapsed Time (s)
84.62
5799,40
21289 ,99
87.18
6484.93
25925.80
79.49
2891.61
11449.
82.05
3
3240.99
9055. 12
82.05
1446.94
4143.26
76.92
3
1624.73
3616.43
82.05
16
724.35
1909.69
76.92
16
3
817.04
1924.27
79.49
483.92
1098.21
74.36
24
3
543.78
987.02
Table 11: Validation loss the HR dataset.
Acc.
Val
Step
Depth
Memory Usage (Mb)
Elapsed Time (s)
2.44
2
5044.44
56492.33
3.03
5059.28
39855.19
3.67
2515.40
10765.58
16.04
2531.44
7157.20
5.35
2
1259.30
3723.94
2.70
1268.60 60
18682.82
3.58
16
632.08
3518.96
3.64
16
636.64
7922.96
3.86
24
422.74
3710.95
3.55
24
3
426.83
6567.02
Table 12: Validation loss on the LOB dataset (IK), included an additional experiment in Appendix
??.
Val Loss
Step
Depth
Memory Usage (Mb)
Elapsed Time (s)
0.58
2
1253.55
180.79
1.74
1447.57
308.52
1.58
4
2
623.87
71.18
32.90
754.05
87.81
2.94
8
2
317.40
61.27
4.84
406,88
62.71
2.24
16
2
164.70
18.67
6.26
16
234.92
24.20
3.82
24
112.80
12.69
15.35
24
176.68
14.92
E Long Temporal Datasets Details
Table 13 summarises the long temporal modeling datasets from the UEA time series classification archive B used in Section
20

---

Table 13: Summary of datasets used in the long time-series classification task.
Dataset
#Sequences
Length
#Classes
#Dimensions
SeliRegulationSCPI (SCPI)
561
896
2
6
SelfRegulationSCP2 (SCP2)
380
1152
2
7
Motorlmagery (MI)
378
3000
2
64
Eigen Worms (EW)
259
17984
5
6
EthanolConcennrattion (ETC)
524
1751
4
3
Ablation Studies
Global and Local Signature Components
In this section, we ablate the use the multi-view signature transform over both global and local transformations of the input signal. The results for the sinusoidal datasets are shown in Figure 7 In cases, the of both local and global components improves the performance of RFormer. This choice, however, can be seen as hyperparameter and will be dataset-dependent.
1.0
0.8
0.8
Accuracy
0.6
Accuracy
0.6
0.4
0.4
Multi-View
0.2
Global
0.2
Local
0.0
0.0
500
1000
500
1000
Epoch
Epoch
Figure 7: Ablation of local and local components of the multi-view signature for the sinusoidal datasets. Left: Sinusoidal dataset. Right: Long Sinusoidal dataset.
F.2
Signature Level and Naive Downsampling
One of the main points of the paper is that the shorter representation of the time-series endowed by the signatures helps to significantly reduce the computational cost of the self-attention operation with minimal information loss (and with improved performance in many of the experiments). By equation (16), one sees that the first level of the signature of linear function is the difference between its endpoints. Hence, using multi-view attention with signature level operates the increments of piecewise-linear interpolated data, which corresponds to naive downsampling. To test that higher levels of the signature provide improvements in performance, compare the result of using the signature on the datasets tested in Table 14 below.
Table 14: Comparative performance of different methods datasets.
Dataset
Linear-Interpolation + Vanilla Rough Transformer with sig level (n) Improvement
Eigen Worms
64.10%
90.24% (2)
40.775 77%
HR
10.56
2.66 (4)
74.81 %
There significant performance gain in considering higher levels the signature because one can capture the higher-order interactions between the different time-series.
21

---

G Additional Experiments and Comparisons
G.1 Random Drop Experiments
Furthermore, conduct new of experiments in which we dropped 30% and 70% of the dataset for RFormer. Note that even with a 70% drop rate in the Eigen Worms dataset, the vanilla Transformer fails run due memory limitations. Therefore, to provide results for the Transformer model on the Eigen Worms dataset, we conduct experiments with an 85% drop rate. This comparison highlights the performance gap between the vanilla Transformer and our proposed model under these conditions, with the RFormer model yielding superior results. All results are computed across five seeds and are summarized in the tables and figure below.
Table 15: Performance of models under various data drop scenarios for EW dataset.
Model
Full
30% Drop
50% Drop
70% Drop
85% Drop
Transformer
OOM
OOM 87.86% 3.28
OOM
OOM
72.45% + 3.36
RFormer
90.24% + 2.15
87.69% + 4.97 83.35% + 2.86
82.74% 2.13
Table 16: Performance consistency of RFormer under data drop scenarios for HR dataset.
Model
Full
30% Drop
50% Drop
70% Drop
RFormer 2.66 1 0.21 2.72 1 0.19 2.82 1 0.05 2.98 + 0.08
Table 17: Epoch-wise performance under different data drop scenarios for the sinusoidal dataset.
Epoch 100
Epoch 250
Epoch 500
Epoch 1000
30% Drop
48.6%
82.5% 56.8%
91.4% 64.9%
99.3%
70% Drop
35.7%
67.8%
Table 18: Epoch-wise performance under different data drop scenarios for the long sinusoidal dataset.
Epoch 100
Epoch 250
Epoch 500
Epoch 1000
30%
Drop
39.1%
72.6%
96.2% 78.5%
98.2%
70% Drop
27.5%
66.7%
85.3%
0.6
0.8
Transformer
Accuray
0.4
RF Former NRDE
Acturay
0.6
NCDE
0.4
ODE-RNY
0.2
0.2
0.0
0.0
500
1000
500
1000
Epoch
Fiooh
Figure 8: Test accuracy epoch for the frequency classification task across three random seeds for sinusoidal datasets with 50% random drop per epoch. Left: Sinusoidal dataset. Right: Long Sinusoidal dataset.
22

---

Finally, Table 19 compares CRU and RFormer in an irregularly sampled synthetic data setting, featuring shorter sinusoids and fewer classes than the experiments in Section 4.1 Additionally, Table 20 presents the hyperparameter validation for CRU (see Table 21 for training time analysis). These experiments demonstrate that recurrent models perform well with short sequences. Note that despite RFormer's superior performance, our model is significantly faster than other continuous-tima models, as shown in Appendix G.2 particularly in Table 21
Table 19:
Comparison of RFormer and CRU (two best and simplest performing instances [Num, Buun.basisBanndwidthh 20/3)) at different random drop percentages.
Random Drop
RFormer
CRU (LSDD=10)
CRU (ISID=20)
0%
100.00%
100%
100.00%
30%
98.60%
65.90%
99.60%
100
50%
97.80%
34.40%
94.70%
70%
96.10%
43.00%
78.60%
85%
85.50%
32.30%
57.30%
0%
100.00%
100.00%
100%
30%
99.90%
42.95%
94.90%
250
50%
99.40%
43.65%
77.30%
70%
98.30%
45.40%
94.40%
85%
86.20%
38.80%
83.60%
0%
100.00%
100.00%
OOM
30%
99.90%
47.15%
OOM
500
50%
99.70%
48.80%
OOM
70%
99.30%
55.15%
OOM
85%
87.70%
46.50%
OOM
Table 20: CRU's hyperparameters (L 100) (latent state dimension (LSD), number of basis matrices
(Num.basis basis), and their bandwidth).
LSD
Num. basis
Bandwidth
Ace (30 Epochs)
15
78%
15
10
10
20
3
100%
20
10
15
81.30%
15
91.70% 10
20
20
100% 3
20
99.90% 10
15
3
99.90%
15
97.50%
40
20
3
100%
20
10
100%
G.2 Additional Efficiency Experiments and Discussion
We conduct additional experiments compare the runtime of Rough Transformers with other models. In this experiment, we use the synthetic sinusoidal dataset considered in our paper and compute the runtime per epoch for varying sequence lengths. We demonstrate results for two variants RFormer: "online", which corresponds to computing the signatures of each batch during training (resulting in significant redundant computation), and "offline", which corresponds to computing the signatures in one go at the beginning of training. We include recent RNN-based model as basis for comparison with high- performiffoo RNN baselines. In addition the models discussed in Section 4 we introduce Continuous Recurrent Units (CRU) [78 new baseline. See Table 21 for summary of the results.
23

---

Table 21: Seconds per epoch for growing imput length and for different model types on the sinusoidal dataset.
S/E for Varying Context Length
Model
L=100
L=250
L=500
L=1000
L-2500
L=5000
L=7.5k
L=10k
NRDE
5.87
11.67
20.27
44.01
103.11 11
201.21
312.31
467.47
NCDE
42.59
121.82
225.14
458.09
1126.77
2813.42
4199.50
5345.39
GRU
1.56
1.55
1.65
1.63
1.78
2.37
3.65
4.79
CRU
59.22
199.15
789.28
OOM
OOM
OOM
OOM
OOM
ContiFormer
61.36
248.31
1165.02
OOM
OOM
OOM
OOM
OOM
Transformer
0.75
0.79
0.82
0.95
1.36
5.31
9.32
16.32
RFormer (Online)
0.75
0.88
0.94
1.03
1.28
1.55
1.83
2.35
RFormer (Offline)
0.67
0.64
0.63
0.65
0.60
0.59
0.62
0.60
We remark that previous running times obtained with batch size of 10. Further, the ContiFormer model could be run for L = 1000 decreasing the batch size to 2 (which significantly affects the parallelization process), avoiding OOM issues and resulting in 4025 seconds/epoch, which is several orders of magnitude larger than RFormer, As an additional experiment, we tested the epoch time (S/E) of RFormer for extremely oversampled sinusoidal time series. We show our results in the table below.
Table 22: Seconds per epoch for very large input length.
S/E for Varying Context Length
Model
L=25k
L=50k
L=100k
L=250k
RFormer (Online)
5.39
9.06
19.95
45.20
RFormer (Offline)
0.60
0.61
0.60
0.63
Thus, the time needed to compute the signature is inconsequential when compared with the time required to train standard models on the full or even downsampled datasets, since this step has to be carried out only once. To put this into context with an example, we note that takes 4s to compute the signature representations for the HR dataset (which is about half the time it takes for the Vanilla Transformer to go through one epoch) and results in 26x increase in computational speed for RFormer when compared to the vanilla Transformer.
Table 23: Processing times for different sizes the sinusoidal dataset.
Size
100
250
500
1k
2.5k
5k
7.5k
10k
25k
50k
75k
100k
Time 0.15 s 0.21 0.24 s 0.39 0.42
0.51
0.70
1.09
1.64
2.94
4.49 s
5.74
To showcase that this is the case for not only sequences of moderate length but also extremely long sequences, we also carry out the following experiment where we compute the signature representation for the sine dataset, with progressively increasing number datapoints. seen in Table 23 this does not cause explosion in computational time.
24

---

3/ (Lac Scale)
S/8 Scak))
3300
36000
100000
150005 200000 Togur Eength
352000
Jagus Leaall (Log Scale
Figure Seconds per epoch for growing input length and for different model types on the sinusoidal dataset for extremely long lengths (up to 250k) Left: Log Scale. Middle: Regular Scale. Right: Log-log scale. When a line stops, it indicates an OOM error.
G.3 Additional ContiFormer Comparisons
Also, to provide some context of the performance of ContiFormes compared with method (and not only results on complexity and training times), we run the model on the sinusoidal classification task for signals of length L 100 and L 250. Due to the slow running time of the ContiFormes model, we did not consider sequence lengths of L > 250. We evaluate the using head. However, given the subpar results we obtain, we also test with four heads, using the hyperparameters originally used in the paper for their irregularly sampled time series classification experiments. By contrast, all variations RF ororrer tested in this paper for this experiment employ only one head, but reported significantly better results.
model
Table 24: Model performance for 100.
Model
Epoch 100
Epoch 250
Epoch 500
ContiFormer (1 Head)
2.3%
2.8%
3.1% %
ContiFormer (4 Heads)
8.5%
17.3%
20.0%
Transformer (1 Head)
13.7%
40.1%
82.8%
RFormer (I Head)
38.7%
81.1%
92.3%
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, they are an accurate reflection of the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, the limitations ae discussed in Appendix B.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
26

---

The paper contains only one theoretical result, and a complete and correct proof is provided.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All experimental details are in Appendix C.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
27

---

reproducible, they Answer: [Yes]
Justification: All datasets used are publicly available and the associated code can found in the following anonymized repo: httpa://anonymouss 4open science/r/rforme: su submission- 2546
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.co cc/ public/guidess Saeil////auidee/dddeebbbsssssss/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 7/nips. //nips.cc/pubb httppss.ccpublli/guuueeee Sttp://///oooeSeeessiiiiiiisssie
ssioiiooiiiiioiociicc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: All details specified in Appendices C and D.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: All experiments are run with several seeds, and the standard deviation is reported alongside the average results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
28

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All experiments were conducted on an NVIDIA GeForce RTX 3090 GPU with 24,564 MiB of memory, as outlined in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We conform with the NeurIPS code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The societal impact statement is included in Appendix B.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
29

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper does not involve the release of data models that have high risk for misuse. However, as mentioned in our impact statement included in Appendix B, we acknowledge potential misuses of our advancements in time series analysis and advocate for ethical application and regulatory oversight.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The owners of the datasets and packages used are acknowledged in the code and in the manuscript.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
30

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not include any experiments involving crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not include any research involving human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
31


Reviews:
[{"review_id": "Uvt4U0hkV3", "reviewer": "Reviewer_wNkv", "summary": "The paper proposes Rough Transformer (RFormer), an extention of the Transformer architecture towards operating on continuous-time representations of time-series data. RFormer employs a novel technique called multi-view signature attention, which performs attention on path signatures pre-computed from input data offline, thereby capturing both local and global dependencies across observations. Experiments on various real-world time-series datasets shows that RFormer enjoys superior predictive performance as well as computational efficiency compared to previous methods, while being robust to changes in sequence length and irregular sampling.", "questions": "- [Q1] **Large overlaps in representation space.** Basec on the multi-view signature formulation in Figure 1 and Equation 8, it seems the earlier observations would be covered by a large number of input tokens. As shown in the right plots in Figure 3, this would result in large \"representational overlap\" similarly to the oversmoothing phenomenon in graph representation learning [B]. Considering this, could restricting the global view to a few previous points rather than all previous points be a viable option? or does the theoretical and empirical robustness of RFormer to variable lengths and irregular sampling require that all previous points be covered in the global view?\n\n- [Q2] **Discussion on input length of MLP and Transformer.** L108 states that the input length $L$ of the MLP and Transformer is fixed by assumption, but is this true? Sequences with different lengths can be processed in a single batch via padding.\n\n[B] Rusch et al., A Survey on Oversmoothing in Graph Neural Networks. arXiv 2023.", "limitations": "The authors have adequately addressed the limitations in Appendix B.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- [S1] **Good novelty.** To the best of my knowledge, incorporating rough path theory to time-series representation learning is a novel approach, and would be of great interest to the machine learning community. \n\n- [S2] **Great empirical performance.** Experiments on a wide variety of real-world datasets show large improvements in both accuracy and efficiency, demonstrating strong utility of proposed multi-view signatures in time-series modeling.", "weaknesses": "- [W1] **Questionable motivation of synthetic frequency classification experiments.** The first experimental section tests RFormer on two synthetic datasets with which the task is to classify input time-series based on their ground-truth frequencies. While L233-234 mentions the second setup in particular is designed towards testing the long-range reasoning capabilities of RFormer, but it is unclear whether this indeed the case. For the second synthetic dataset, in particular, how can identifying the frequency be a proxy for long-range reasoning when the frequency $\\omega_0$ is used for $t < t_0$ only? The results on Figure 2 showing that methods that are \"tailor-made for long-range time series modeling\" (L254) such as Neural-CDE and Neural-RDE underperforming significantly also indicates that the designed task is not really representative of long-range reasoning.\n\\\n\\\nMore interesting questions to ask could be: What makes RFormer sample-efficient vs. vanilla Transformer particularly on the Sinusoidal dataset and not so much on the Long Sinusoidal dataset? What makes RFormer more robust to changes in sampling frequency compared to Neural-CDE and Neural-RDE? Table 1 of [A] shows Neural-CDE is also quite robust to dropped data, but is this characteristic not emergent for Sinusoidal and Long Sinusoidal datasets?\n\n- [W2] **Missing analysis on interpolation methods.** By default, RFormer uses piecewise-linear paths for computing path signatures, but as mentioned in L140, it seems any continuous-time interpolation can be used. As such, it would be interesting to discuss (1) whether any other interpolation techniques can be deployed efficiently similarly to piecewise-linear paths and (2) if they lead to any boosts in predictive accuracy, but these discussions are missing in the current draft (i.e., is the currently used piecewise-linear interpolation \"pareto-optimal\" under performance-efficiency trade-offs?).\n\n[A] Kidger et al., Neural Controlled Differential Equations for Irregular Time Series. NeurIPS 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback, as it has been very relevant in updating our manuscript (especially the comment on oversmoothing). We hope that the reviewer will be satisfied with the additional experiments carried out and will be inclined to raise the score.\n\n**[W1.1] Motivation of synthetic frequency classification.** \n\nThe idea of this experiment was to have the relevant information for the classification experiment at the beginning of the time series, as it would have to be propagated forward in order to correctly perform the classification. This is the type of task where we expect traditional recurrent models to fail due to vanishing gradients if the sequence is relatively long (2000 points in this case). This is likely the reason for the poor performance of Neural CDEs, which are a continuous-time analog of RNNs. However, we highlight that this was useful to test if RFormer was able to retain the long-range capabilities of the Transformer despite operating on a very compressed representation. This was further evaluated in a number of very long datasets (most notably EigenWorms), where RFormer was not only the fastest but also the most performant.\n\nIf the reviewer would suggest an alternative that might be more suitable in their view, we would be more than happy to run additional experiments for this task.\n    \n**[W1.2] Clarification on RFormer’s sample efficiency on Sinusoidal vs. Long Sinusoidal datasets.**\n\nUpon inspection, we found a bug in our code for this particular experiment, where we were not adding the time information to the dataset before passing it to RFormer (this was only the case in this experiment, due to an incorrect local version of the code). After re-running the experiment, we see similar sample efficiency gains from RFormer in this task for the same hyperparameters. Furthermore, we found that some additional tuning of the signature level in this new setting led to even greater sample efficiency gains (Thank you for this!). The results can be found in Figure 2 in the associated PDF. We have also included an experiment investigating the effect of signature hyperparameters in Figure 2 of the PDF as well.\n\n**[W1.3] RFormer's robustness to sampling frequency changes vs. Neural-CDE/RDE.** \n\nThe theoretical reason behind why RFormer can process irregularly sampled sequences is shown in Proposition 3.1 (L175). In terms of the performance of Neural CDE and Neural RDE, we note that they experience a similar drop for this task as in the other tasks considered. To better understand how our model compares to NRDEs and NCDEs in settings in which these baselines perform well, we ran an additional random dropping on 15 experiments, which are shown in Table 4 of the PDF. We find that RFormer is not only better performing, but also several orders of magnitude faster than the rest of the SoTA benchmarks considered.\n\n**[W2] Missing analysis on interpolation methods.** \n\nWe agree that investigating the impact of different interpolation methods in the multi-view signature computation is interesting. Our choice to use linear interpolation of data was due to the efficient signature computation of paths of this form. This efficiency is due to 1) linear interpolation being a *local* operation and 2) as noted in the Appendix (L644), the signature of piecewise-linear functions can be computed explicitly in terms of the data points. Continuous interpolations such as splines are non-local, meaning the computational burden is high for long time series. Deriving a convenient, computationally efficient method for computing the signature is also non-trivial.  As such, we could consider this approach to be \"pareto-optimal\", as the reviewer suggests.\n\nSince we focused our experiments on long time series, we decided against investigating this in our paper. However, based on your review, we will add a small section detailing this choice.\n\n**[Q1] Large overlaps in representation space.** \n\nWe thank the reviewer for this excellent question. We have investigated this matter further, and this has led to some very interesting new insights that we feel should be brought to the attention of all the reviewers. For this reason, we have included our response to this question in our General Response.\n\nAdditionally, we would also note that the multi-view signature transform is only an instance of a broader idea of presenting the model with information at different scales, which we found to improve performance (see Ablation in Appendix E.1). There are many other ways of providing this multi-scale information (such as restricting the global view to a few previous points rather than all previous points, as suggested by the reviewer), but this would be dataset and task-dependent. In the limit, we hope that these parameters can be tuned with the downstream task loss, but we leave this for future work.\n\nWe have included a discussion of our hypothesis of why RFormer achieves substantial efficiency gains in our manuscript, as well as a related citation [1]. We thank the reviewer again for the very helpful comment on oversmoothing. \n\n**[Q2] Discussion on input length of MLP and Transformer.** \n\nThank you for bringing this point to our attention. We agree with the reviewer that padding can be used as an ad-hoc solution to guarantee the same sequence length in the batch, even though it has been found that other models (recurrent and convolutional) sometimes struggle with this solution, see [2].\n\nWe wanted to highlight this as a weakness of the Transformer model when comparing it to RFormer, which will always give the same representation to the model due to the flexibility of the signature. However, we will tone down our wording in accordance with the reviewer's suggestion.\n\n[1] Rusch, T. Konstantin, et al. \"Graph-coupled oscillator networks.\" International Conference on Machine Learning. PMLR, 2022.\n\n[2] Dwarampudi, Mahidhar et al. \"Effects of padding on LSTMs and CNNs.\" arXiv preprint arXiv:1903.07288 (2019).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "meNJ8oUcbu", "reviewer": "Reviewer_tNoo", "summary": "The paper proposes Rough Transformers, an attention-based model for long continuous-time signals. The model utilizes ideas from rough path theory to extract path signatures from the continuous-time signal (obtained by interpolation of the original signal). Two types of signatures are extracted: global and local. The global signature extracts long-term information from the signal whereas the local signature extracts local information. Self-attention is then used on this \"multi-view\" signature. Experiments on classification and regression tasks show improved performance over existing model architectures.", "questions": "- Can the authors clarify by what they mean by \"input sequences must be sampled at the same times, (ii) the sequence length must be fixed\"? These do not seem to be limitations of transformers, especially (ii). \n\nSee weaknesses for other questions.", "limitations": "There is a brief discussion on limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper views the problem of modeling long continuous-time signals through the lens of rough path theory. While this in itself is not novel, the combination of rough path signatures with attention is a novel combination.\n- Experiments on classification and regression tasks show that RFormer improves over existing models in terms of accuracy and significantly more compute efficient. (I have some concerns and questions about the empirical analysis, please see Weaknesses)", "weaknesses": "- The technical contribution is limited. In such scenarios, the empirical analysis needs to be sufficiently strong.\n- The empirical analysis has been conducted on a few toyish datasets. While the results are definitely promising, more experimental support is needed to validate the model.\n    - Although the model is motivated from a continuous-time and irregularly-sampled data perspective, the actual investigation of these settings is limited. As per my understanding, all experiments under 4.1 have been conducted on regularly sampled time series (please correct me, if I am wrong). I find it surprising that simple RNN-based methods do not perform well in these settings. If this is indeed the case, some simple CNN-based method should be studied. Recent models based on state space layers (e.g., S4, Mamba) can also be explored as baselines.\n   - OOM for important baselines is not really helpful to draw any conclusions. To highlight efficiency of RFormer, please conducted experiments where you increased the context length or other parameters to show where baselines run OOM and how do they perform before that.\n   - More experiments are needed, particularly for the forecasting task to understand how well the model understands the dynamics. \n   - I took a brief look at the code and it looks like hyperparameter tuning was conducted for RFormer. Was such tuning also performed for the baselines? If no (which is a valid response), how did you selected the baseline parameters? How sensitive is the model to different hyperparameters? \n- The discussion on related work needs to be moved to the main text and improved. Please contrast RFormer with the related works, particularly the ones that are closely related such as NRDE and ContiFormer. Discussion on some closely related works [1, 2, 3] is missing. Ideally there should also be a comparison with at least of these methods (e.g., CRU).\n\n[1] Schirmer, Mona, et al. \"Modeling irregular time series with continuous recurrent units.\" International conference on machine learning. PMLR, 2022.   \n[2] Ansari, Abdul Fatir, et al. \"Neural continuous-discrete state space models for irregularly-sampled time series.\" International Conference on Machine Learning. PMLR, 2023.   \n[3] Oh, YongKyung, Dongyoung Lim, and Sungil Kim. \"Stable Neural Stochastic Differential Equations in Analyzing Irregular Time Series Data.\" arXiv preprint arXiv:2402.14989 (2024).    \n\nI am happy to update my score, if my concerns are adequately addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for the engaging review and valuable feedback. We believe we have incorporated the reviewer's suggestions into our manuscript and hope our changes warrant an increase in the score.\n\n**[W1] Need for stronger empirical analysis.**\n\nWe have added 19 new datasets, 8 extra baselines, and new synthetic experiments (Tables 4-5/1-3 in the PDF). We hope these additions address the reviewer's concerns and provide a stronger empirical analysis.\n\n**[W2.1] Investigation of continuous-time and irregularly-sampled data limited.**\n\nMost tasks are indeed regularly sampled, except for the Limit Order Book data. However, we performed random drop experiments in Section 4.3 and Appendix F, expanding Section 4.1 to irregularly sampled settings.\n\nWe conducted additional experiments to enhance our model's empirical trustworthiness. For regularly sampled data, our model was tested against SOTA models (S5, Mamba, LRU) across 5 datasets, showing very competitive performance (Table 5 PDF). \n\nFor irregularly sampled settings, we compared our model against 4 additional baselines, including [1,3] as suggested by the reviewer, across 15 datasets from the UCR time-series archive. Table 4 (PDF) shows our model's performance and average rank (due to space constraints), consistently outperforming the baselines. \n\nFurther, Table 2 (PDF) compares CRU and RFormer in an irregularly sampled synthetic data setting (with shorter sinusoids and a smaller number of classes than the experiments in Section 4.1), supported by CRU's hyperparameters validation in Table 1 and training time analysis in Table 3. These experiments demonstrate that recurrent models perform well with short sequences. Note that despite its superior performance, our model is significantly faster than other continuous-time models, as shown in Appendix F.2 and Table 3 (PDF).\n\n**[W2.2] OOM results and efficiency plots.**\n\nWe thank the reviewer for raising this point. Regarding OOM errors for ContiFormer, Appendix E of the ContiFormer paper demonstrates its memory requirements scale exponentially with sequence length, making it impossible to run on long time-series datasets considered in our paper. We felt this is worth noting because we think ContiFormer is most similar in spirit to RFormer in that it augments the attention mechanism in a continuous-time fashion. We hope you find that with the additional baselines, this point is made clearer. \n\nWe want to highlight that efficiency experiments can be already found in the paper in Figure 4 (Section 4.2. Training efficiency), showing seconds per epoch of all models. When a line stops,  it indicates an OOM error. Appendix F.2 contains more information on computational times, showing RFormer can be used on context lengths of 250k time steps. We apologize if this was not clear and will modify the text accordingly.\n\n**[W2.3] More experiments, particularly for forecasting.**\n  \nWe have significantly increased the number of experiments during this rebuttal, adding 19 new datasets and 8 new baselines (see responses to [W1,2.1]) and new synthetic experiments.\n\nRegarding forecasting, we should remark that forecasting pipelines using Transformers typically involve iteratively masking the input representation to predict the next time step. In our setting, since we compress the input representation, these pipelines cannot be directly applied and would require extensive modifications, which is not feasible during the rebuttal timeframe. However, we include a step-ahead forecasting of the Apple Limit Order Book volatility (see General Response). Further, we point the reviewer to the HR task, which contains classical temporal dynamics (e.g. cyclicality).\n\nWe would like to highlight that the main points of our paper on efficiency, inductive bias, and continuous processing should hold with our experiments. Furthermore, we would like to point out that other models in the area (such as [1,3]) have also been applied in the same tasks and have not been tested for forecasting, which requires extensive tuning.\n\n**[W2.4] Hyperparameter tuning for baselines.**\n    \nIn short, yes, we did. We thoroughly validated the step and depth parameters used to compute the log-signatures for the NRDE model (Appendix D). We did this as these are the only hyperparameters for which we performed tuning in our model.\n\nFor the ODE-RNN and NCDE/NRDE models, we validated the architectural choices. Due to occasional sub-optimal performance in our replications, we included the original results for shared datasets (see Tables 1,3). For ContiFormer, we used hyperparameters from the official repository, testing with both 1 and 4 heads (our model uses 1 head), as detailed in Appendix F.3. For the new datasets, we used optimal hyperparameters from manuscripts/repositories, except for CRU, which we validated ourselves.\n\nRegarding the sensitivity of the model to hyperparameters, we direct the reviewer to Appendix E and have included new sensitivity experiments in Figure 2 of the PDF.\n\n**[W3] Improving related work section.**\n\nFor the final version, we will move the discussion on related work to the main text as suggested and we will contrast our work with other models, highlighting (i) the recurrent vs. attention-based structure, (ii) computational efficiency, and (iii) the treatment of the signature with local vs. global windows in the case of NRDEs.\n\n**[Q1] Clarifications.**\n\nWhat we mean here is that the input data should be regularly sampled (to retain decent performance), and each input sequence must be sampled the same number of times. In many practical scenarios (e.g., medical data), time-series may vary wildly in their length. For Transformers, which have a fixed context window, this poses a non-trivial challenge of how to standardize the input data such that it may be encoded by the model. As suggested by reviewer wNkv, this can be improved through padding. We will add some discussion along these lines and tone down some of our wording.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5VSQTpjN0w", "reviewer": "Reviewer_hdr3", "summary": "This paper introduces the Rough Transformer, a variant of the original Transformer that allows the processing of discrete-time series as continuous-time signals through the use of multi-view signature attention. Empirical comparisons shows that Rough Transformers outperform vanilla Transformers and continuous-time models on a variety of time-series tasks and are robust to the sampling rate of the signal.", "questions": "- I was wondering how effective is the proposed model in autoregressive generative tasks and time series forecasting (see https://arxiv.org/abs/2311.04147)? These are natural tasks in the domain of sequence modeling \n- While the proposed method can improve the modified transformer model in sequence modeling, why would the method be attractive for practitioners when they could just use more sophisticated models like SSMs for sequence modeling?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Overall the paper is well written and is easy to follow\n- The idea of using signature transform within an attention mechanism is interesting", "weaknesses": "- Empirical evaluations are limited, casting doubts on the true potential of the proposed architecture\n- The tasks considered are rather simple, and it is not clear whether the proposed architecture will give favorable tradeoffs between accuracy and efficiency in the more challenging tasks (see below) \n- Missing evaluations on time series forecasting tasks (only classification and regression tasks are considered)\n- Missing comparisons with recent RNN models (such as https://arxiv.org/abs/2110.04744, https://arxiv.org/abs/2212.00228), Transformer models (e.g., those studied in https://arxiv.org/abs/2011.04006), State Space models (https://arxiv.org/abs/2111.00396 and the more recent variants such as Mamba: https://arxiv.org/abs/2312.00752) and other sequence models (https://arxiv.org/abs/2305.01638, https://arxiv.org/abs/2209.10655) \n- Missing ablation studies on the components of the proposed architecture, particularly the role of the global and local components in the multi-view signature, truncation level n, etc.\n- Missing related work; e.g., the papers mentioned above, https://link.springer.com/article/10.1007/s40304-017-0103-z and https://arxiv.org/abs/1710.10121 for continuous-time DL models, https://arxiv.org/abs/2006.12070, https://arxiv.org/abs/2102.04877 for continuous-time RNN models, https://dl.acm.org/doi/abs/10.5555/3546258.3546305 for using path signatures to understand continuous-time RNNs", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback. We are slightly surprised by some of their comments, which we believe are already addressed in the manuscript. We hope that this response will help to clarify any misunderstandings. In line with the reviewer's comments, we have carried out an extensive set of new experiments which we believe address all of the reviewer's concerns. We hope the reviewer will take this into account and consider raising the score accordingly.\n\n**[W1, W4] Empirical evaluations are limited / Missing comparisons with recent models.**\n\nIn line with the reviewer's feedback, we have added comparisons to LRU, S5, and MAMBA on 5 long temporal modeling new datasets, where we find RFormer to perform very competitively. These results are included in Table 5 of the attached PDF to this rebuttal.\n\nThe reason we focus on these tasks is because our method is tailored to time-series processing, and other image or text-based benchmark datasets (such as LRA/sMNIST, etc), which typically test model memory, are not as suitable in this setting. We will add a comment along these lines in the paper as well.\n\nIn terms of performance on random dropping, we have added experiments on 15 new datasets, comparing against SoTA models for continuous time series processing. We report the performance and average rank of our model in Table 4 (PDF), where our model consistently outperforms the baselines. We find that our model performs very competitively, as well as being orders of magnitude faster than the baselines.\n\nIn terms of the small empirical evaluation, we would like to stress that we have now tested our model on a wide variety of datasets. We  have also run additional experiments whose results can be found in Tables 1, 2, and 3 of the PDF comparing RFormer with CRU [1]. Additionally, Table 4 (PDF) reports the performance of 8 extra baselines on 15 new datasets with different random dropping rates. We consider this extensive evaluation very much in line with the experiments carried out by other sequence modeling architectures  (e.g. NCDE, NRDE, coRNN, noisy RNN). In terms of the tasks considered, we also find that they are consistent with the experiments in these papers, which focused on classification and regression tasks. Nonetheless, we have extended our results into the forecasting scenario through a step-ahead forecasting task, see response to [W2,W3] and the General Response.\n\n**[W5] Missing ablation studies on the components of the proposed architecture, particularly the role of the global and local components in the multi-view signature, truncation level \\(n\\), etc.**\n\nWe would like to clarify that these results are already in the paper. In particular, they can be found in Appendix E: Ablation Studies. We have also carried out a set of experiments on the sensitivity of the model to hyperparameters, which can be found in Figure 2 of the associated PDF document.\n\n**[W6] Missing related work.**\n\nWe would like to clarify that we have a dedicated related work section in Appendix B, where we have already included citations to many of the works that the reviewer mentions. In particular, we include a discussion of Unitary RNNs, Orthogonal RNNs, expRNNs, chronoLSTM, antisymmetric RNNs, Lipschitz RNNs, coRNNs, unicoRNNs, LEMs, waveRNN, Linear Recurrent Units, and Structured State Space Models (S4 and Mamba) on lines L678 - L690. Furthermore, we discuss efficient attention variants such as Sparse Transformer, Longformer, Linear Transformers, BigBird, Performer, and Diffuser in lines L691 - L702. We will add the remaining citations the reviewer suggests.\n\n**[W2, W3] The tasks considered are rather simple / Missing evaluations on time series forecasting tasks.**\n\nWe would like to echo our previous statement on how similar models (NCDE, NRDE, coRNN, noisy RNN) have tested exclusively in classification and regression tasks and how these models have become widely accepted by the community.\n\nIn terms of forecasting, we would like to highlight that time series forecasting pipelines using Transformers typically train by masking the input representation iteratively to predict the next time step. In our setting, since we are compressing the input representation, these pipelines cannot be straightforwardly implemented for our model. We intend to work towards a model that uses these principles just for time series forecasting, but note this would require a completely new training pipeline as well as heavy tuning. That said, we reiterate that the focus of this work is to showcase the benefits of using Rough Path Signatures within the attention mechanism for efficient and continuous time-series processing, as demonstrated through the widely accepted experimental settings of time-series classification and regression.\n\nWe have, however, included a step-ahead forecasting task on the Apple Limit Order Book volatility to extend our results into te forecasting scenario. The results can be found in the General Response.\n\n**[Q2] While the proposed method can improve the modified transformer model in sequence modeling, why would the method be attractive for practitioners when they could use more sophisticated models like SSMs for sequence modeling?**\n\nWe thank the reviewer for raising this point.\n\nWe believe that showing the significant benefits of RFormer when compared to the traditional Transformer is beneficial to the community. There seems to be a wider adoption of hybrid models (SSM + Transformer) by the community given their superior performance [2]. In tasks that require temporal processing, the use of RFormer could potentially make these models have improved inductive biases, allow for continuous-time processing, and increase efficiency and inference speed.\n\n[1] Schirmer, Mona, et al. \"Modeling irregular time series with continuous recurrent units.\" International conference on machine learning. 2022.\n\n[2] Lieber, Opher, et al. \"Jamba: A hybrid transformer-mamba language model.\" arXiv preprint (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the reviewers for their valuable feedback. We have carefully reviewed their comments and incorporated their suggestions into a new version of the manuscript. Additionally, we are encouraged by the positive feedback provided by the reviewers on the novelty, efficiency, and performance of the proposed method, as evidenced by comments such as:

- Good novelty. [...] a novel approach [...] of great interest to the machine learning community.
- Great empirical performance. Experiments on a wide variety of real-world datasets show large improvements in both accuracy and efficiency, demonstrating strong utility of proposed multi-view signatures in time-series modeling.
- [...] the combination of rough path signatures with attention is a novel combination.
- Experiments on classification and regression tasks show that RFormer improves over existing models in terms of accuracy and is significantly more compute efficient.
- Overall the paper is well written and is easy to follow.
- The idea of using signature transform within an attention mechanism is interesting.

We believe RFormer constitutes a prime example of the general effort in combining effective modelling techniques from different research fields for addressing practical machine learning tasks, which has the potential to inspire further similar approaches. 

In this general response, we aim to address some of the common questions among reviewers, as well as new insights gained during the rebuttal process as a result of excellent observations from the reviewers. 

- **New Datasets and Baselines**

A common sentiment noted among reviewers was the need for additional datasets and baselines. To address this, we have implemented RFormer in 19 new datasets from the UCR time-series archive, consisting of 15 shorter time-series datasets and 4 long time-series datasets. We benchmarked against a number of new methods including SoTA methods for irregular temporal processing such as Neural {SDE, LSDE, LNSDE, GSDE} [1] and CRU [2], as well as SoTA state-space models (S5, LRU, and Mamba). For the irregular sampling experiments, we randomly dropped datapoints at rates of 30%, 50%, and 70%. 

The results of these experiments can be found in Tables 4 and 5 in the accompanying PDF. RFormer consistently outperforms the Neural ODE-based methods on the 15 short time-series datasets and the state-space models on the long time-series datasets. We also include an additional comparison between CRU and RFormer in an irregularly sampled synthetic data setting (supported by CRU's hyperparameters validation in Table 1 and an analysis of training times in Table 3, both in the PDF). We present the results in Table 2 of the PDF. Note that shorter sinusoids with a smaller number of classes than the experiments in Section 4.1 are used here, which demonstrates that recurrent models perform well with short sequences in this synthetic example. Also, as well as demonstrating superior performance, our model is significantly faster than other continuous-time models, as shown in Appendix F.2 and Table 3 of the PDF.

- **Large Overlaps in Representation Space** 

We thank Reviewer wNkv for raising the excellent connection between the representational overlap of the global signature transform and the oversmoothing phenomenon present in graph representation learning. We investigated this question deeply and attained new results which we feel give new insights into the performance of our model and should be brought to the attention of all reviewers. 

In our experiments, we found that Transformers work better with more coarse representations of input data, and we believe that this is the reason behind some of the performance gains that we observe in RFormer. By coarsening the representation offered to the Transformer backbone, the model is able to learn better and faster.

However, as Reviewer wNkv points out, this only occurs if the intervals at which windows are taken are sufficiently large. Otherwise, the signatures may exhibit some form of "representational collapse". We note that in our experiments on very long time-series, we took the signature over windows that were very spaced apart, which prevented global representations from being too similar and seemed to yield better performance. 

To offer a more quantitative evaluation of oversmoothing in this setting, we measured the Dirichlet energy by interpreting the temporal signal as a directed path graph. We compared different numbers of windows (from 2 to ~18k) of the global signature in the EigenWorms dataset. The results are shown in Figure 1 of the attached PDF. Interestingly, we found that the "elbow" of the Dirichlet energy corresponded to 30 windows in this dataset, which we found empirically to be the most performant setting. This hints at the idea of the Dirichlet energy being used for signature hyperparameter tuning as well. 

We thank Reviewer wNkv again for this very interesting suggestion. We will add a dedicated section discussing these conclusions in the paper, as we believe it could shed light on the internal mechanisms of Transformers as well as motivate our method more strongly. We have also added some citations to works associated to oversmoothing.

- **Time-Series Forecasting**

To assess the model's ability to understand the dynamics of the time series, we have carried out an additional experiment of forecasting the next-step intraday volatility of an Apple Limit Order Book. The results are shown below:

|     | RFormer | Transformer | NRDE  |
|----------|---------|-------------|-------|
| **RMSE**     | 32.33   | 33.45       | 37.22 |



[1] Oh, Y., et al. Stable Neural Stochastic Differential Equations in Analyzing Irregular Time Series Data.Twelfth International Conference on Learning Representations.

[2] Schirmer, Mona, et al. "Modeling irregular time series with continuous recurrent units." International conference on machine learning. PMLR, 2022.

Meta-review: This paper proposes a Transformer-based architecture for continuous-time signals, using multi-view attention on path signatures (from rough path theory) to capture both local and global dependencies. The authors effectively addressed the reviewers' concerns during the author-reviewer discussion phase. After the discussion, the majority of reviewers support the acceptance of this paper. AC agrees with the reviewers' positive opinions: this paper presents a novel approach by incorporating rough path theory into time-series representation learning, provides theoretical support for robustness, and empirically demonstrates the effectiveness of the proposed architecture. Although concerns from one negative reviewer remain, AC thinks the author rebuttal well addressed the concerns. For example, the authors added new comparisons with recent baselines as the reviewer suggested.

Therefore, AC recommends acceptance and highly encourages the authors to incorporate all experimental results (e.g., new datasets, new baseline models) and analyses from the author-reviewer discussion into the final manuscript, as they would further strengthen the contributions of this paper.
Decision: Accept (poster)