Title: Rough Transformers: Lightweight and Continuous Time Series Modelling through Signature Patching
Venue: NeurIPS 2024
Accepted: yes
Keywords: efficient time series modelling

efficient sequence modelling

continuous-time temporal modelling

patching

self-attention

transformers

computational efficiency

spatial processing
Abstract: Time-series data in real-world settings typically exhibit long-range dependencies and are observed at non-uniform intervals. In these settings, traditional sequence-based recurrent models struggle. To overcome this, researchers often replace recurrent models with Neural ODE-based architectures to account for irregularly sampled data and use  Transformer-based architectures to account for long-range dependencies. Despite the success of these two approaches, both incur very high computational costs for input sequences of even moderate length. To address this challenge, we introduce the Rough Transformer, a variation of the Transformer model that operates on continuous-time representations of input sequences and incurs significantly lower computational costs. In particular, we propose multi-view signature attention, which uses path signatures to augment vanilla attention and to capture both local and global (multi-scale) dependencies in the input data, while remaining robust to changes in the sequence length and sampling frequency and yielding improved spatial processing. We find that, on a variety of time-series-related tasks, Rough Transformers consistently outperform their vanilla attention counterparts while obtaining the representational benefits of Neural ODE-based models, all at a fraction of the computational time and memory resources.

Rough Transformers: Lightweight and Continuous Time Series Modelling through Signature Patching

Abstract

Time-series data in real-world settings typically exhibit long-range dependencies and are observed at non-uniform intervals. In these settings, traditional sequencebased recurrent models struggle. To overcome this, researchers often replace recurrent architectures with Neural ODE-based models to account for irregularly sampled data and use Transformer-based architectures to account for long-range dependencies. Despite the success of these two approaches, both incur very high computational costs for input sequences of even moderate length. To address this challenge, we introduce the Rough Transformer, a variation of the Transformer model that operates on continuous-time representations of input sequences and incurs significantly lower computational costs. In particular, we propose multi-view signature attention , which uses path signatures to augment vanilla attention and to capture both local and global (multi-scale) dependencies in the input data, while remaining robust to changes in the sequence length and sampling frequency and yielding improved spatial processing. We find that, on a variety of time-seriesrelated tasks, Rough Transformers consistently outperform their vanilla attention counterparts while obtaining the representational benefits of Neural ODE-based models, all at a fraction of the computational time and memory resources.

1 Introduction

Real-world sequential data in areas such as healthcare [65], finance [36], and biology [28] often are irregularly sampled, of variable length, and exhibit long-range dependencies. Furthermore, these data, which may be drawn from financial limit order books [8] or EEG readings [85], are often sampled at high frequency, yielding long sequences of data. Hence, many popular machine learning models struggle to model real-world sequential data, due to input dimension inflexibility, memory constraints, and computational bottlenecks. Rather than treating these data as discrete sequences, effective theoretical models often assume data are generated from some underlying continuous-time process [53, 66]. Hence, there is an increased interest in developing machine learning methods that use continuous-time representations to analyze sequential data.

One recent approach to modelling continuous-time data involves the development of continuous-time analogues of standard deep learning models, such as Neural ODEs [12] and Neural CDEs [45], which extend ResNets [37] and RNNs [30], respectively, to continuous-time settings. Instead of processing discrete data directly, these models operate on a latent continuous-time representation

fernando.moreno-pino , alvaro.arroyo }@ eng.ox.ac.uk

Code available at:

https://github.com/AlvaroArroyo/RFormer

of input sequences. This approach is successful in continuous-time modelling tasks where standard deep recurrent models fail. In particular, extensions of vanilla Neural ODEs to the time-series setting [70, 45] succeed in various domains such as adaptive uncertainty quantification [59], counterfactual inference [79], or generative modelling [7].

In many practical settings, such as financial market volatility [20, 54] or heart rate fluctuations [35], continuous-time data also exhibit long-range dependencies. That is, data from the distant past may impact the system's current behavior. Deep recurrent models struggle in this setting due to vanishing gradients, whereas continuous-time analogues of these models have been shown to address this difficulty [46]. Several recent works [52, 58] also successfully extract long-range dependencies from sequential data with Transformers [86], which learn temporal dependencies of a tokenized representation of input sequences. Extracting such temporal dependencies requires a positional encoding of input data, because the attention mechanism is permutation invariant, which projects data into some latent space. The parallelizable nature of the Transformer allows for rapid training and evaluation on sequences of moderate length and it contributes to its success in fields such as natural language processing (NLP).

While the above approaches succeed in certain settings, several limitations hinder their wider applications. On the one hand, Neural ODEs and their analogues [45, 70] bear substantial computational costs when modelling long sequences of high dimension; see [57]. On the other hand, Transformers operate on discrete-time representations of input sequences, whose relative ordering is represented by the positional encoding. This representation may inhibit their expressivity in continuoustime data modelling tasks [91]. Moreover, Transformer-based models suffer from a number of difficulties, including (i) input sequences must be sampled at the same times, (ii) the sequence length must be fixed, and (iii) the computational cost scales quadratically in the length of the input sequence. These difficulties severely limit the application of Transformers to continuous-time data modelling.

Contributions 1) We introduce Rough Transformers , a variant of the Transformer architecture amenable to the processing of continuous-time signals, which can be easily integrated into existing code-bases. The Rough Transformer is built upon the path signature from Rough Path Theory [51]. We define a novel, multi-scale transformation which projects discrete input data to a continuous-time path and compresses the input data with minimal information loss. Moreover, this transformation is an efficient feature representation of continuous-time paths, because linear functionals of path signatures approximate continuous functions of paths arbitrarily well (see Theorem A.2 in Appendix A).

2) We introduce the multi-view attention mechanism to extract both local and global dependencies of very long time-series efficiently. This mechanism operates directly on continuoustime representations of data without the need for expensive numerical solvers or constraints on the smoothness of the data stream. Moreover, the multi-view attention mechanism is provably robust to irregularly sampled data.

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

2

1

0

1

2

t

Global View

0

2

4

6

8

10

12

2

1

0

1

2

Local View

8

9

10

11

12

13

1.5

1.0

0.5

0.0

0.5

S

( ˜

X

)

t

k

-

1

,t

k

S

( ˜

X

)

0

,t

k

M

(

X

)

k

=

[

S

( ˜

X

)

0

,t

k

S

( ˜

X

)

t

k

-

1

,t

k

]

3) We carry out extensive experimentation on long and irregularly sampled time-series data. In particular, we show that Rough Transformers (i) improve the learning dynamics of the Transformer, making it more sample-efficient and allowing it to achieve better out-of-sample results, (ii) reduce the training cost by a factor of up to 25 × when compared with vanilla Transformers and more when compared with Neural ODE based architectures, (iii) maintain similar performance when data are irregularly sampled, where traditional recurrent-based models suffer a substantial decrease in

performance [70], and (iv) yield improved spatial processing, accounting for relationships between different temporal channels without having to pre-define a specific inter-channel relation structure.

2 Background and Methodology

Problem Formulation. In many real-world scenarios, sequential data are time-series sampled from some underlying continuous-time process, so datasets consist of long, irregularly sampled sequences of varied lengths. In these settings, the problem of sequence modelling is described as follows. Let C ( R + ; R d ) = { g : R + → R d | g continuous } , and consider ̂ X ∈ C ( R + ; R d ) which we call a continuous-time path . A time-series of length L with sampling times T X = { t i } L i =1 ⊂ R + is defined as X = (( t 1 , X 1 ) , ..., ( t L , X L )) , where X i = ̂ X ( t i ) ∈ R d . Now, define a continuous function on paths f : C ( R + ; R d ) → R k . Next define a dataset D = { ( X i , f ( ̂ X i )) N i =1 } . We seek to approximate the function f from the set D for some downstream task. Importantly, we do not assume that T X = T Y for all X , Y ∈ D , so that D may be irregularly sampled.

Sequence Modelling with Transformers. Transformers are used extensively as a baseline architecture to approximate functions of discrete-time sequential data and are successfully applied to settings when input sequences are fixed in length, relatively short, and sampled at regular intervals. First, the Transformer projects input time series X ∈ R L × d to a high-dimensional space X ↦→ T ( X ) ∈ R L × d ′ for d ′ >> d using some linear positional encoding T : R L × d → R L × d ′ . Next, a latent representation of the encoded sequence is learned by a multi-headed self-attention mechanism which splits T ( X ) into H distinct query, key, and value sequences: Q h = T ( X ) W Q h , K h = T ( X ) W K h , V h = T ( X ) W V h , respectively, with h = 1 , ..., H and weight matrices W Q h , W K h , W V h ∈ R d ′ × d ′ . The multi-head self-attention calculation for each head is given by

and the latent representation is projected to the output space R k using a multi-layer perceptron (MLP).

̸

The input length L of the MLP and the Transformer is fixed by assumption. To evaluate the Transformer on a time-series X with |T X | = n , one must perform some transformation (interpolation, extrapolation, etc.) which may degrade the performance of the model. Furthermore, the memory and time complexity of the Transformer is of order O ( L 2 d ) , which presents a substantial difficulty in modelling long sequences.

Rough Path Signatures. Broadly, the difficulties faced by the Transformer in modelling timeseries stem from time-series being sampled from underlying continuous-time objects, while the attention mechanism underpinning the Transformer is designed to model discrete sequences. To address these difficulties, Rough Transformers augment standard Transformers by lifting the input time-series to the space of continuous-time functions and performing the self-attention calculation in this infinite-dimensional space. To achieve this, we use the path signature from Rough Path Theory.

For a continuous-time path ̂ X ∈ C 1 b ( R + ; R d ) and times s, t ∈ R + , the path signature of ̂ X from s to t , denoted S ( ̂ X ) s,t , is defined as follows. First, let

denote the set of all d -multi-indices and I n d = { I ∈ I d : | I | = n } . Next, set S ( ̂ X ) 0 s,t := 1 and for any I ∈ I d , define

where ˙ ̂ X j = d ̂ X j /dt . Abusing notation, define level n of the signature as

and define the signature as the infinite sequence

Finally, define the truncation of the signature S ( ̂ X ) ≤ n s,t = ( S ( ̂ X ) 0 s,t , ..., S ( ̂ X ) n s,t ) , where S ( ̂ X ) n s,t can be interpreted as an element of the extended tensor algebra of R d :

Analogously, we say that S ( ̂ X ) ≤ n s,t ∈ T (( R d )) ≤ n . A central property of the signature is that is invariant with respect to time-reparameterization [51]. That is, let γ : [0 , T ] → [0 , T ] be surjective, continuous, and non-decreasing. Then we have

which will be crucial to demonstrate the Rough Transformer's robustness to irregularly sampled data.

In contrast to wavelets or Fourier transforms, which parameterize paths on a functional basis, the signature provides a basis for functions of continuous paths. Hence, the path signature is well-suited to sequence modelling tasks in which one seeks to learn a function of the underlying functional. For a more rigorous presentation of signatures and a description of additional properties, see Appendix A and Lyons et al. [51].

3 Rough Transformers

Now, we construct the Rough Transformer, a Transformer-based architecture that operates on continuous-time sequential data by means of the path signature.

Let D be a dataset of irregularly sampled time-series. To project a discretized time-series X ∈ D to a continuous-time object, let ˜ X denote the piecewise-linear interpolation of X . 2 Next, for t k ∈ T , define the multi-view signature

In what follows, we refer to the components ( S ( ˜ X ) 0 ,t k , S ( ˜ X ) t k -1 ,t k ) as global and local , respectively; see Figure 1. Intuitively, one can interpret the global component as an efficient representation of long-term information (see Theorem A.2 in Appendix A), and the local component as a type of convolutional filter that is invariant to the sampling rate of the signal. Now, define the multi-view signature transform M ( X ) = ( M ( X ) 1 , ..., M ( X ) ¯ L ) , and denote by M ( X ) ≤ n the truncated signature for a truncation level n . Next, define the multi-view attention mechanism , which uses the multi-view signature transform to extend the standard attention mechanism to the space of continuous functions [51]. First, fix a truncation level n ∈ N , and let ¯ d ∈ N be such that M ( X ) ≤ n k ∈ R ¯ d . For h = 1 , ..., H let W ˜ Q, ˜ K, ˜ V h ∈ R ¯ d × ¯ d ′ for some ¯ d ′ ∈ N , and let

Then, the attention calculation is given by

Notice that the attention calculation is similar to (1), however, we stress that the multi-view attention is built on continuous-time objects, the signatures, while the standard attention mechanism acts on discrete objects. The multi-view signature provides a compressed representation of the time series, minimizing the computational costs associated to quadratic scaling without excessive loss of representational capacity, see Appendix F.

0

2000

4000

6000

8000

10000

Input Length

0

1000

2000

3000

4000

5000

S/E

Transformer

ContiFormer

RFormer (Online)

RFormer (Offline)

NCDE

0

2000

4000

6000

8000

10000

Input Length

2

-1

2

3

2

6

2

9

2

12

S/E (Log Scale)

Transformer

ContiFormer

RFormer (Online)

RFormer (Offline)

NCDE

2

7

2

8

2

10

2

12

2

13

2

-1

2

3

2

6

2

9

2

12

S/E (Log Scale)

Transformer

RFormer (Online)

NCDE

ContiFormer

RFormer (Offline)

3.1 Advantages of Rough Transformers

Computational Efficiency. As demonstrated in Section 4, multi-view attention mechanism can substantially reduce the computational cost of vanilla Transformers. In particular, the attention calculation decreases from O ( L 2 d ) in the vanilla case to O ( L 2 d ) , where L << L with Rough Transformers. This enables both faster wall-clock training time and the ability to process long input sequences which would otherwise yield out-of-memory errors for the vanilla Transformer, see Figure 2. Moreover, the multi-view attention mechanism does not require backpropagation through the signature calculation, which can be computed offline . This is significantly more computationally efficient compared with the complexity of computing signatures batch-wise in every training step. Finally, the signature of piecewise-linear paths can be computed explicitly, see Appendix A, and there are a number of Python packages devoted to optimized signature calculation [43, 68].

Variable Length and Irregular Sampling. The multi-view signature transform underpinning Rough Transformers is evaluated by constructing a continuous-time interpolation of input data and computing a series of iterated integrals of this interpolation. The bounds of these integrals are a fixed set of time points, meaning that the sequence length of the multi-view attention mechanism is fixed and independent of the sequence length of input samples. Furthermore, the following proposition shows that the output of the Rough Transformer for two (possibly irregular) samplings of the same path is similar.

Proposition 3.1. Let T be a Rough Transformer. Suppose ̂ X : [0 , T ] → R d is a continuous-time process, and let γ : [0 , T ] → [0 , T ] denote a time-reparameterization. Suppose X and X ′ are samplings of ̂ X and ̂ X ◦ γ , respectively. Then T ( X ) ≈ T ( X ′ ) .

Proof. By (7), S ( ̂ X ) s,t = S ( ̂ X ◦ γ ) s,t for all s, t ∈ [0 , T ] . Hence, one has M ( X 1 ) ≈ M ( X 2 ) . Finally, T ( X 1 ) ≈ T ( X 2 ) because the attention mechanism and final MLP are both continuous.

Hence, the Rough Transformer is robust to irregular sampling. In many tasks, the sampling times convey important information about the time-series. In these settings, one may augment the input time-series with its sampling times, that is, write X = (( t 0 , X 0 ) , ..., ( t L , X L )) .

Spatial Processing. While an interpolation of input data could be sampled to make vanilla Transformers independent of the length of the input sequence, important locality information could be lost, see Appendix F.2. Instead, Rough Transformers summarize spatial interactions between channels by means of the multi-view signature transform. One may notice that in (5), the dimension of the signature grows exponentially in the level of the signature n . In particular, when X i ∈ R d , | S ( ˜ X ) ≤ n 0 ,t | = d ( d n -1) d -1 = O ( d n ) , so the multi-view attention calculation is of order O ( ¯ L 2 d n ) . In many practical time-series modelling problems, however, the value of d is not very large. The signature terms also decay factorially in the signature level n (see Proposition A.3 in Appendix A), so in practice, one may take the value of n to be small without sacrificing performance. The majority of computational savings result from the reduction of the sequence length to ¯ L , and in practice, we take ¯ L << L .

When the dimension d is large, there are three possible remedies to maintain computational efficiency. First, instead of computing the signature in M ( X ) k = ( S ( X ) 0 ,t k , S ( ˜ X t k -1 ,t k ) , one may compute

the log-signature , which is a compressed version of the signature [67]. When the dimension is large enough such that the log-signature is computationally infeasible, one may instead compute the univariate signatures of features coupled with the time channel. That is, consider ̂ X ∈ C ([0 , T ]; R d ) , with ̂ X ( t ) = ( ̂ X 1 ( t ) , ..., ̂ X d ( t )) . Denote the time-added function X i ( t ) := ( t, ̂ X i ( t )) . Then we define the univariate multi-view signature

The attention mechanism in this case is constructed as before. Fixing the maximum signature depth to be some value n ∗ , one sees that the number of features in the univariate multi-view signature is approximately 2 n ∗ d . In practice we find that n ∗ ≤ 5 provides sufficient performance, so the order of the attention calculation is O ( C ¯ L 2 d ) for C ≤ 2 n ∗ . Finally, one may use randomized signatures to reduce dimension by using a Johnson-Lindenstrauss-type projection to a low-dimensional latent space and computing the signature in this space, as in [21, 19].

4 Experiments

In this section, we present empirical results for the effectiveness of the Rough Transformer, hereafter denoted RFormer , on a variety of time-series-related tasks. Experimental and hyperparameter details regarding the implementation of the method are in Appendices C and D. We consider long multivariate time-series as our main experimental setting because we expect signatures to perform best in this scenario. Additional experimentation on long-range reasoning tasks on image-based datasets is left for future work, as these would likely require additional inductive biases.

To benchmark RFormer , we consider both discrete-time and continuous-time models. In particular, we include as main baselines traditional RNN models ( GRU [15]), ODE-based methods designed for sequential data ( Neural-CDE [45]), as well as ODE-based methods explicitly designed for long time-series ( Neural-RDE [57]). 3 Furthermore, we compare against a vanilla Transformer [86] which is the RFormer backbone. Finally, we present comparisons with a recent continuous-time Transformer model, ContiFormer [13], to highlight the computational efficiency gap between RFormer and similar continuous-time models. We note that the first two tasks focus on evaluating the performance improvement of RFormer over the Transformer baseline. For other long-range tasks, we include comparisons to recent state-space models [31, 62, 81]. In the irregular sampling regime, we benchmark against state-of-the-art models tailored to that setting [61, 78]. See Appendix B for additional discussion on related models and more details about our experimental choices.

4.1 Time Series Processing

Frequency Classification. Our first experiment is based on a set of synthetically generated time series from continuous paths of the form

where g ( t ) is a non-linear trend component, ν and η are two noise terms, and ω is the frequency. Here, the task of the model is to classify the time-series according to its frequency ω . We consider 1000 samples in 100 classes with ω evenly distributed from 10 to 500 . Each time-series is regularly sampled with 2000 times-steps on the interval [0 , 1] .

0

250

500

750

1000

Epoch

0.0

0.2

0.4

0.6

0.8

1.0

Transformer

RFormer

NCDE

ODE-RNN

GRU

0

250

500

750

1000

Epoch

0.0

0.2

0.4

0.6

0.8

1.0

This synthetic experiment is similar to others in recent work on time-series modelling [49, 89, 55]. We include an additional experiment in which we alter the signal in (12) so its frequency is ω 0 for t < t 0 and ω 1 afterward, where the task is to classify the sinusoid based on the first frequency. We call this dataset the 'long sinusoidal" dataset. This extension of the original experiment aims to test the ability of the model to perform long-range reasoning effectively. Note that for this task, we also add ODE-RNN [70] to the previously mentioned baselines.

Figure 3 shows that the inclusion of both local and global information with the multi-view signature enhances the sample efficiency of the RFormer over the vanilla Transformer model, even though the attention mechanism is now operating on a much shorter sequence. When compared with other models, we see that GRU and ODE-RNN fail to capture the information in the signal, and are not able to obtain any meaningful performance improvement throughout the training period. This highlights the shortcomings of most RNN-based models when processing sequences of moderate length, which are very common in real-world applications. Both Neural-CDE and Neural-RDE capture some useful dependencies in the time series but fall short compared with both vanilla Transformer and RFormer .

HR dataset. Next, we consider the Heart Rate dataset from the TSR archive [83], originally sourced from Beth Israel Deaconess Medical Center (BIDMC). This dataset consists of time-series sampled from patient ECG readings, and each model is tasked to perform a regression by forecasting the patient's heart rate (HR) at

Table 1: Test RMSE (mean ± std) computed across five seeds on the Heart Rate (HR) dataset.

the sample's conclusion. The data, sampled at 125Hz, consists of three-channel time-series (including time), each spanning 4000 time steps. We used the L2 loss metric to assess the performance. Table 1 shows the results, where ⋄ denotes the results from Morrill et al. [57] and † our reproduction. The sequences in the HR dataset are sufficiently short to remain within memory when running the Transformer model. The baseline Transformer model improves over GRU , and ODE-RNN , however, it is less competitive when compared with Neural-RDE , suggesting that the Transformer is not particularly well-suited for this type of task. However, the RFormer model improves over the baseline Transformer by 67% . Across all tasks, we see significant improvements in efficiency as a consequence of the signature computation. We elaborate on this in more detail in the following subsection.

Long Time Series Classification. We now evaluate the performance of RFormer on five long time series classification tasks from the UEA time series classification archive [3]. A summary of these datasets is provided in Table 13 in Appendix E. As previously done in [57], the original train and test datasets are merged and then randomly divided into new train, validation, and test sets, following a 70/15/15 split. The resulting performance metrics are summarized in Table 2. 4

In this setting, we see that RFormer generally matches or slightly outperforms the continuous-time and SSM baselines. Due to the scaling problems of ContiFormer with respect to sequence length, we were unable to run this baseline within GPU memory constraints in most cases, and thus no results are reported (see Appendix G.2 for efficiency comparisons between models). In contrast, RFormer can cheaply train on the same device (see Section 4.2 for details) due to its ability to take advantage of the parallel nature of GPU processing and compress the original time series. This is especially noticeable when compared to continuous-time models ( Neural-CDE , Neural-RDE , LogCDE ), which are sometimes orders of magnitude slower than our model and consistently report lower or similar results. Additional experimental details can be found in Appendix G, as well as some experiments on hyperparameter sensitivity.

4.2 Training Efficiency

Here, we focus on the computational gains of the model when compared with vanilla Transformers and methods that require numerical ODE solvers.

Attention-based architectures are highly parallelizable on modern GPUs, as opposed to traditional RNN models which require sequential updating. However, vanilla attention experiences a bottleneck in memory and time complexity as the sequence length L grows. As covered above in Section 3, variations of the signature transform allow the model to operate on a reduced sequence length ¯ L without increasing the dimensionality in a way that would become problematic for the model. This allows us to bypass the quadratic complexity of the model without resorting to sparsity techniques commonly used in the literature [26, 49].

Tables 1-3 show that RFormer is competitive when modelling datasets with extremely long sequences without an explosion in the memory requirements. RFormer exploits the parallelism of the attention mechanism to significantly accelerate training time, as the length of the input sequence is decreased substantially. In particular, we observe speedups of 1 . 4 × to 26 . 11 × with respect to standard attention, and higher when compared with all methods requiring numerical solutions to ODEs. The computational efficiency gains of RFormer are attained due to the signature transform reducing the length of the timeseries with minimal information loss. The effectiveness of this transformation can be seen from the ablation study carried out in Appendix F. This contrasts with NRDEs [57], which augment NCDEs with local signatures of input data, and find that smaller windows often perform better. Fur-

thermore, NRDEs do not experience the same computational gains as RFormer because they must perform many costly ODE integration steps.

In Figure 2, we showcase the improvements in computational efficiency of RFormer compared to vanilla Transformers [86], continuous-time Transformers [13], and other continuous-time RNNs [45, 57] when processing sequences from L = 100 samples up to L = 10 K. As seen, RFormer is significantly more efficient than its continuous-time and vanilla counterparts, even when performing the signature computation online, which involves computing the signatures for each batch during training, resulting in significant redundant computation. When signatures are precomputed just once before training, the computational time of each epoch remains constant across input all sequence lengths including L = 10 K (see the exact signature computation times for different datasets in Table 4). We also stress the fact that RFormer also scales gracefully for extremely long sequences (up to L = 250 K) with both online and offline computation of the signatures, as shown in Appendix G. Finally, we highlight that ContiFormer has a sample complexity of O ( L 2 d 2 S ) , where S represents the normalized number of function evaluations of the numerical ODE solver, which makes ContiFormer orders of magnitude more computationally intensive when compared to RFormer and prevents the model from running on sequences longer than 500 points without running out of memory (see device details in Appendix C).

4.3 Irregular Time Series Classification

So far, we mainly focused on the efficiency and inductive bias afforded to the model through the use of signatures. However, a key element of RFormer is that it can naturally deal with irregularly sampled sequences without expensive numerical ODE solvers. This property follows from the fact that signatures are invariant to time reparameterization , see Proposition 3.1. In this subsection, we empirically test this property by training the model on the same datasets but randomly dropping a percentage of the data points. This test intends to find if the model is able to learn continuous-time representations of the original input time-series. The results can be found in Table 5. We find that

RFormer consistently results in the best performance, with a small performance drop when compared to the full dataset. Importantly, this property is achieved in conjunction with the efficiency gains afforded to the model and without the use of expensive numerical ODE solvers. 5 Finally, we perform an additional set of experiments on the 15 univariate classification datasets from the UEA time series classification archive and compare our model with recent state-of-the-art models for irregular time series [61, 78]. Across the board, we find that our model is both faster and more accurate than the continuous-time benchmark despite having a discrete-time Transformer backbone , as shown in Figure 4, which introduces Continuous Recurrent Units (CRU) [78] as an extra baseline. For more details and more exhaustive experimentation on random data drops, see Appendix G.

5 Reasons for improved model performance

In this final section, we provide explanations for the superior inductive bias of the RFormer model compared to its vanilla Transformer counterpart, despite its lower computational cost.

5.1 Spatial Processing

First, we highlight that a key reason the model achieves significant compression benefits in the tasks considered is its ability to jointly account for temporal and spatial interactions through the self-attention mechanism and signature terms, respectively. In particular, we believe that for certain datasets, the relationships between different channels of the time series may hold more importance than the temporal information itself, which can often be redundant. This is exemplified in the Eigenworms dataset, which experiences a 20% performance drop when employing univariate signatures, but is able to achieve state-of-the-art performance with a 600 × compression rate in the temporal dimension when signatures are applied across all channels, as shown in Figure 6. To this end, we draw parallels between the use of signatures and the field of temporal graph processing, where the use of the signature over all channels can be seen as a fully connected graph, capturing information from all channels, and the univariate signature would correspond to a graph with only self-connections between the nodes, as depicted in Figure 5. In our view, this hints towards the idea of using sparse graph learning techniques [17, 23] to reduce the explosion in signature terms while retaining the ability to perform effective spatial processing.

To empirically test these claims, we design a synthetic experiment using a 2-channel time series. Each channel contains a signal of the form sin( ω i t + ν i ) , i = 1 , 2 , where ω i and ν i are randomly sampled from the interval [0 , 2 π ] . For half of the dataset, the last 1% of temporal samples in the second channel are set to match the frequency of the first channel. The task is to classify whether the samples in this final interval are of the same frequency. As shown in Figure 5, RFormer demonstrates greater sample efficiency and achieves higher test accuracy compared to its vanilla Transformer counterpart, highlighting the effectiveness of signatures in spatial processing.

30%

50%

70%

40

50

60

70

Accuracy (%)

RFormer

CRU

Neural SDE

Neural LSDE

Neural GSDE

Neural CDE

Neural RDE

0

200

400

600

800

1000

-0.5

0.0

0.5

0

200

400

600

800

1000

Time

-0.5

0.0

0.5

0

200

400

600

800

1000

Epoch

50

60

70

80

90

100

Accuracy (%)

RFormer

Transformer

5.2 Sequence Coarsening as an Inductive Bias for Transformers

In addition to the benefits of higher-order signature terms, we empirically observe that even using level-one signature terms resulted in performance improvements when compared to processing sequences without any transformation. We believe that the reduction in input signal length, achieved without significant information loss through the signature transform is another important factor in the improved inductive bias of RFormer . This finding aligns with the concurrent work of [4], which highlights some of the drawbacks of decoder-only Transformers for long sequences in terms of both oversquashing and representational collapse .

0

5000

10000

15000

Number of Windows,

N

0

100

200

300

400

Dirichlet Energy,

E

(

X

(

n

))

0

100

200

300

400

500

Number of Windows,

N

100

200

300

400

0

5000

10000

15000

Time

-5

0

5

Original Rep.

0

10

20

30

Time

-500

0

500

Sig. Rep.

To measure the degree of coarsening in the sequence, we find that interpreting the temporal sequence as a path graph and using ideas from the oversmoothing literature [75] serves as a good way to measure the similarity of the representations being fed to the Transformer. In particular, we compute the Dirichlet Energy [74], defined in this case as E ( X ) = 1 N ∑ N i =1 || X i -X i -1 || 2 of the temporal sequence resulting from taking increasing window sizes of the global signature. An example of this is shown in Figure 6 for the Eigenworms dataset, where we compared different numbers of windows (from 2 to 18k). Interestingly, we found that the "elbow" of the Dirichlet energy corresponded to 30 windows in this dataset, which we found empirically to be one of the most performant settings. This hints at the idea of the Dirichlet energy being used for signature hyperparameter tuning as well.

6 Conclusion

We introduced the Rough Transformer, a variant of the original Transformer that allows the processing of discrete-time series as continuous-time signals through the use of multi-view signature attention. Empirical comparisons showed that Rough Transformers outperform vanilla Transformers and continuous-time models on a variety of time-series tasks and are robust to the sampling rate of the signal. Finally, we showed that RFormer provides significant speedups in training time compared to regular attention and ODE-based methods, without the need for major architectural modifications or sparsity constraints.

Impact Statement

This work is unlikely to result in any harmful societal repercussions. Its primary potential lies in its ability to enhance and advance existing data modelling and machine learning methods.

References

M. Arjovsky, A. Shah, and Y. Bengio. Unitary evolution recurrent neural networks. In International conference on machine learning , pages 1120-1128. PMLR, 2016.

I. P. Arribas. Derivatives pricing using signature payoffs. arXiv preprint arXiv:1809.09466 , 2018.

A. Bagnall, H. A. Dau, J. Lines, M. Flynn, J. Large, A. Bostrom, P. Southam, and E. Keogh. The uea multivariate time series classification archive, 2018. arXiv preprint arXiv:1811.00075 , 2018.

F. Barbero, A. Banino, S. Kapturowski, D. Kumaran, J. G. Araújo, A. Vitvitskyi, R. Pascanu, and P. Veliˇ ckovi´ c. Transformers need glasses! information over-squashing in language tasks. arXiv preprint arXiv:2406.04267 , 2024.

I. Beltagy, M. E. Peters, and A. Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 , 2020.

M. Biloš, J. Sommer, S. S. Rangapuram, T. Januschowski, and S. Günnemann. Neural flows: Efficient alternative to neural odes. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 21325-21337. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/ paper_files/paper/2021/file/b21f9f98829dea9a48fd8aaddc1f159d-Paper.pdf .

S. Calvo-Ordonez, J. Huang, L. Zhang, G. Yang, C.-B. Schonlieb, and A. I. Aviles-Rivero. Beyond u: Making diffusion models faster & lighter. arXiv preprint arXiv:2310.20092 , 2023.

Á. Cartea, S. Jaimungal, and J. Penalva. Algorithmic and high-frequency trading . Cambridge University Press, 2015.

Á. Cartea, G. Duran-Martin, and L. Sánchez-Betancourt. Detecting toxic flow. arXiv preprint arXiv:2312.05827 , 2023.

B. Chang, M. Chen, E. Haber, and E. H. Chi. Antisymmetricrnn: A dynamical system view on recurrent neural networks. In International Conference on Learning Representations , 2018.

P. Chang, G. Duràn-Martín, A. Y. Shestopaloff, M. Jones, and K. Murphy. Low-rank extended kalman filtering for online learning of neural networks from streaming data. arXiv preprint arXiv:2305.19535 , 2023.

R. T. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems , 31, 2018.

Y. Chen, K. Ren, Y. Wang, Y. Fang, W. Sun, and D. Li. Contiformer: Continuous-time transformer for irregular time series modeling. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

R. Child, S. Gray, A. Radford, and I. Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509 , 2019.

K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 , 2014.

K. M. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins, J. Q. Davis, A. Mohiuddin, L. Kaiser, et al. Rethinking attention with performers. In International Conference on Learning Representations , 2020.

A. Cini, I. Marisca, D. Zambon, and C. Alippi. Taming local effects in graph-based spatiotemporal forecasting. Advances in Neural Information Processing Systems , 36, 2024.

N. M. Cirone, A. Orvieto, B. Walker, C. Salvi, and T. Lyons. Theoretical foundations of deep selective state-space models. arXiv preprint arXiv:2402.19047 , 2024.

E. M. Compagnoni, A. Scampicchio, L. Biggio, A. Orvieto, T. Hofmann, and J. Teichmann. On the effectiveness of randomized signatures as reservoir for learning rough dynamics. In 2023 International Joint Conference on Neural Networks (IJCNN) , pages 1-8. IEEE, 2023.

F. Corsi. A simple approximate long-memory model of realized volatility. Journal of Financial Econometrics , 7(2):174-196, 2009.

C. Cuchiero, L. Gonon, L. Grigoryeva, J.-P. Ortega, and J. Teichmann. Discrete-time signatures and randomness in reservoir computing. IEEE Transactions on Neural Networks and Learning Systems , 33(11):6321-6330, 2021.

T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems , 35: 16344-16359, 2022.

H. S. de Ocáriz Borde, A. Arroyo, and I. Posner. Projections of model spaces for latent graph inference. In ICLR 2023 Workshop on Physics for Machine Learning , 2023.

E. Dupont, A. Doucet, and Y. W. Teh. Augmented neural odes. Advances in neural information processing systems , 32, 2019.

N. B. Erichson, O. Azencot, A. Queiruga, L. Hodgkinson, and M. W. Mahoney. Lipschitz recurrent neural networks. In International Conference on Learning Representations , 2020.

A. Feng, I. Li, Y. Jiang, and R. Ying. Diffuser: efficient transformers with multi-hop attention diffusion for long sequences. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 12772-12780, 2023.

A. Fermanian. Embedding and learning with signatures. Computational Statistics & Data Analysis , 157:107148, 2021.

C. Fleming, D. Sheldon, W. Fagan, P. Leimgruber, T. Mueller, D. Nandintsetseg, M. Noonan, K. Olson, E. Setyawan, A. Sianipar, et al. Correcting for missing and irregular data in homerange estimation. Ecological Applications , 28(4):1003-1010, 2018.

E. Fons, A. Sztrajman, Y. El-Laham, A. Iosifidis, and S. Vyetrenko. Hypertime: Implicit neural representation for time series. arXiv preprint arXiv:2208.05836 , 2022.

K.-i. Funahashi and Y. Nakamura. Approximation of dynamical systems by continuous time recurrent neural networks. Neural networks , 6(6):801-806, 1993.

A. Gu and T. Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 , 2023.

A. Gu, K. Goel, and C. Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations , 2021.

B. Hambly and T. Lyons. Uniqueness for the signature of a path of bounded variation and the reduced path group. Annals of Mathematics , pages 109-167, 2010.

R. Hasani, M. Lechner, A. Amini, D. Rus, and R. Grosu. Liquid time-constant networks. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 7657-7666, 2021.

J. M. Hausdorff and C.-K. Peng. Multiscaled randomness: A possible source of 1/f noise in biology. Physical review E , 54(2):2154, 1996.

N. Hautsch. Modelling irregularly spaced financial data: theory and practice of dynamic duration models . Springer Science & Business Media, 2004.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770778, 2016.

M. Henaff, A. Szlam, and Y. LeCun. Recurrent orthogonal networks and long-memory tasks. In International Conference on Machine Learning , pages 2034-2042. PMLR, 2016.

M. Höglund, E. Ferrucci, C. Hernández, A. M. Gonzalez, C. Salvi, L. Sánchez-Betancourt, and Y. Zhang. A neural rde approach for continuous-time non-markovian stochastic control problems. In ICML Workshop on New Frontiers in Learning, Control, and Dynamical Systems , 2023.

S. I. Holt, Z. Qian, and M. van der Schaar. Neural laplace: Learning diverse classes of differential equations in the laplace domain. In International Conference on Machine Learning , pages 8811-8832. PMLR, 2022.

A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning , pages 5156-5165. PMLR, 2020.

T. A. Keller, L. Muller, T. Sejnowski, and M. Welling. Traveling waves encode the recent past and enhance sequence learning. In The Twelfth International Conference on Learning Representations , 2023.

P. Kidger and T. Lyons. Signatory: differentiable computations of the signature and logsignature transforms, on both cpu and gpu. arXiv preprint arXiv:2001.00706 , 2020.

P. Kidger, P. Bonnier, I. Perez Arribas, C. Salvi, and T. Lyons. Deep signature transforms. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/file/ d2cdf047a6674cef251d56544a3cf029-Paper.pdf .

P. Kidger, J. Morrill, J. Foster, and T. Lyons. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems , 33:6696-6707, 2020.

M. Lechner and R. Hasani. Learning long-term dependencies in irregularly-sampled time series. arXiv preprint arXiv:2006.04418 , 2020.

M. Lemercier, C. Salvi, T. Cass, E. V. Bonilla, T. Damoulas, and T. J. Lyons. Siggpde: Scaling sparse gaussian processes on sequential data. In International Conference on Machine Learning , pages 6233-6242. PMLR, 2021.

M. Lezcano-Casado and D. Martınez-Rubio. Cheap orthogonal constraints in neural networks: A simple parametrization of the orthogonal and unitary group. In International Conference on Machine Learning , pages 3794-3803. PMLR, 2019.

S. Li, X. Jin, Y . Xuan, X. Zhou, W. Chen, Y.-X. Wang, and X. Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in neural information processing systems , 32, 2019.

Z. Li, N. B. Kovachki, K. Azizzadenesheli, K. Bhattacharya, A. Stuart, A. Anandkumar, et al. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations , 2020.

T. J. Lyons, M. Caruana, and T. Lévy. Differential equations driven by rough paths . Springer, 2007.

V. Melnychuk, D. Frauen, and S. Feuerriegel. Causal transformer for estimating counterfactual outcomes. In International Conference on Machine Learning , pages 15293-15329. PMLR, 2022.

M. Morariu-Patrichi and M. S. Pakkanen. State-dependent hawkes processes and their application to limit order book modelling. Quantitative Finance , 22(3):563-583, 2022.

F. Moreno-Pino and S. Zohren. Deepvol: Volatility forecasting from high-frequency data with dilated causal convolutions. arXiv preprint arXiv:2210.04797 , 2022.

F. Moreno-Pino, P. M. Olmos, and A. Artés-Rodríguez. Deep autoregressive models with spectral attention. Pattern Recognition , 133:109014, 2023.

F. Moreno-Pino, Á. Arroyo, H. Waldon, X. Dong, and Á. Cartea. Rough transformers for continuous and efficient time-series modelling. arXiv preprint arXiv:2403.10288 , 2024.

J. Morrill, C. Salvi, P. Kidger, and J. Foster. Neural rough differential equations for long time series. In International Conference on Machine Learning , pages 7829-7838. PMLR, 2021.

T. Nguyen and A. Grover. Transformer neural processes: Uncertainty-aware meta learning via sequence modeling. In International Conference on Machine Learning , pages 16569-16594. PMLR, 2022.

A. Norcliffe, C. Bodnar, B. Day, J. Moss, and P. Liò. Neural ode processes. In International Conference on Learning Representations , 2020.

A. Norcliffe, C. Bodnar, B. Day, N. Simidjievski, and P. Liò. On second order behaviour in augmented neural odes. Advances in neural information processing systems , 33:5911-5921, 2020.

Y. Oh, D. Lim, and S. Kim. Stable neural stochastic differential equations in analyzing irregular time series data. In The Twelfth International Conference on Learning Representations .

A. Orvieto, S. L. Smith, A. Gu, A. Fernando, C. Gulcehre, R. Pascanu, and S. De. Resurrecting recurrent neural networks for long sequences. In International Conference on Machine Learning , pages 26670-26698. PMLR, 2023.

Y. Park, J. Choi, C. Yoon, M. Kang, et al. Learning pde solution operator for continuous modeling of time-series. arXiv preprint arXiv:2302.00854 , 2023.

I. Perez Arribas, G. M. Goodwin, J. R. Geddes, T. Lyons, and K. E. Saunders. A signature-based machine learning model for distinguishing bipolar disorder and borderline personality disorder. Translational psychiatry , 8(1):274, 2018.

S. Perveen, M. Shahbaz, T. Saba, K. Keshavjee, A. Rehman, and A. Guergachi. Handling irregularly sampled longitudinal data and prognostic modeling of diabetes using machine learning technique. IEEE Access , 8:21875-21885, 2020.

R. Ratcliff, P. L. Smith, S. D. Brown, and G. McKoon. Diffusion decision model: Current issues and history. Trends in cognitive sciences , 20(4):260-281, 2016.

J. Reizenstein. Calculation of iterated-integral signatures and log signatures. arXiv preprint arXiv:1712.02757 , 2017.

J. Reizenstein and B. Graham. The iisignature library: efficient calculation of iterated-integral signatures and log signatures. arXiv preprint arXiv:1802.08252 , 2018.

D. W. Romero, A. Kuzina, E. J. Bekkers, J. M. Tomczak, and M. Hoogendoorn. Ckconv: Continuous kernel convolution for sequential data. In International Conference on Learning Representations .

Y. Rubanova, R. T. Chen, and D. K. Duvenaud. Latent ordinary differential equations for irregularly-sampled time series. Advances in neural information processing systems , 32, 2019.

T. K. Rusch and S. Mishra. Coupled oscillatory recurrent neural network (cornn): An accurate and (gradient) stable architecture for learning long time dependencies. In International Conference on Learning Representations , 2020.

T. K. Rusch and S. Mishra. Unicornn: A recurrent model for learning very long time dependencies. In International Conference on Machine Learning , pages 9168-9178. PMLR, 2021.

T. K. Rusch, S. Mishra, N. B. Erichson, and M. W. Mahoney. Long expressive memory for sequence modeling. In International Conference on Learning Representations , 2021.

T. K. Rusch, B. Chamberlain, J. Rowbottom, S. Mishra, and M. Bronstein. Graph-coupled oscillator networks. In International Conference on Machine Learning , pages 18888-18909. PMLR, 2022.

T. K. Rusch, M. M. Bronstein, and S. Mishra. A survey on oversmoothing in graph neural networks. arXiv preprint arXiv:2303.10993 , 2023.

H. Sáez de Ocáriz Borde, A. Arroyo, I. Morales, I. Posner, and X. Dong. Neural latent geometry search: product manifold inference via gromov-hausdorff-informed bayesian optimization. Advances in Neural Information Processing Systems , 36, 2024.

C. Salvi, M. Lemercier, C. Liu, B. Horvath, T. Damoulas, and T. Lyons. Higher order kernel mean embeddings to capture filtrations of stochastic processes. Advances in Neural Information Processing Systems , 34:16635-16647, 2021.

M. Schirmer, M. Eltayeb, S. Lessmann, and M. Rudolph. Modeling irregular time series with continuous recurrent units. In International conference on machine learning , pages 1938819405. PMLR, 2022.

N. Seedat, F. Imrie, A. Bellot, Z. Qian, and M. van der Schaar. Continuous-time modeling of counterfactual outcomes using neural controlled differential equations. In International Conference on Machine Learning , pages 19497-19521. PMLR, 2022.

V. Sitzmann, J. Martel, A. Bergman, D. Lindell, and G. Wetzstein. Implicit neural representations with periodic activation functions. Advances in neural information processing systems , 33: 7462-7473, 2020.

J. T. Smith, A. Warrington, and S. Linderman. Simplified state space layers for sequence modeling. In The Eleventh International Conference on Learning Representations .

C. Tallec and Y. Ollivier. Can recurrent neural networks warp time? arXiv preprint arXiv:1804.11188 , 2018.

C. W. Tan, C. Bergmeir, F. Petitjean, and G. I. Webb. Monash university, uea, ucr time series extrinsic regression archive. arXiv preprint arXiv:2006.10996 , 2020.

A. Tong, T. Nguyen-Tang, D. Lee, T. M. Tran, and J. Choi. Sigformer: Signature transformers for deep hedging. In Proceedings of the Fourth ACM International Conference on AI in Finance , pages 124-132, 2023.

A. Vahid, M. Mückschel, S. Stober, A.-K. Stock, and C. Beste. Applying deep learning to singletrial eeg data provides evidence for complementary theories on action control. Communications biology , 3(1):112, 2020.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

B. Walker, A. D. McLeod, T. Qin, Y. Cheng, H. Li, and T. Lyons. Log neural controlled differential equations: The lie brackets make a difference. arXiv preprint arXiv:2402.18512 , 2024.

S. Wang, B. Z. Li, M. Khabsa, H. Fang, and H. Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768 , 2020.

J. Yoon, D. Jarrett, and M. Van der Schaar. Time-series generative adversarial networks. Advances in neural information processing systems , 32, 2019.

M. Zaheer, G. Guruganesh, K. A. Dubey, J. Ainslie, C. Alberti, S. Ontanon, P. Pham, A. Ravula, Q. Wang, L. Yang, et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems , 33:17283-17297, 2020.

A. Zeng, M. Chen, L. Zhang, and Q. Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence , volume 37, pages 11121-11128, 2023.

A Properties of Path Signatures

First, we recall that the path is uniquely determined by its signature, which motivates its use as a feature map.

Proposition A.1. Given a path ̂ X : [0 , T ] → R d , then the map P : [0 , T ] → R 1+ d where P ( t ) = ( t, ̂ X ( t )) is uniquely determined by it's signature S ( P ) 0 ,T .

The proof can be found in Hambly and Lyons [33].

For Rough Transformers, several features of path signatures are important. First, linear functionals on path signatures possess universal approximation properties for continuous functionals.

Theorem A.2. Fix T > 0 , and let K ⊂ C 1 b ([0 , T ]; R d ) . Let f : K → R be continuous with respect to the sup-norm topology on C 1 b ([0 , T ]; R d ) . Then for any ϵ > 0 , there exists a linear functional ℓ such that

for any ̂ X ∈ K , where X denotes the time-added augmentation of ̂ X .

For a proof of A.2, see Arribas [2]. Even though Theorem A.2 guarantees that linear functionals are sufficient for universal approximation, linear models are not always sufficient in practice. This motivates the development of nonlinear models built upon the path signature which efficiently extract path behavior.

The second feature is that the terms of the path signature decay factorially, as described by the following proposition.

For a proof of Proposition A.3, see [51]. Hence, the number of terms in the signature grows exponentially in the level of the signature, but the tail of the signature is well-behaved, so only a few levels in a truncated signature are necessary to adequately approximate continuous functionals.

A.1 Signatures of Piecewise Linear Paths.

In the Rough Transformer, we use linear interpolation of input time-series to get a continuous-time representation of the data. As mentioned in Section 3, the signature computation in this case is particularly simple.

Suppose ̂ X k : [ t k , t k +1 ] → R d is a linear interpolation between two points X k , X k +1 ∈ R d . That is,

Then the signature of ̂ X k is given explicitly by

where ⊗ denotes the tensor product. Let ̂ X k ∗ ̂ X k +1 denote the concatenation of ̂ X k and ̂ X k +1 . That is, ̂ X k ∗ ̂ X k +1 : [ t k , t k +2 ] → R d is given by

The signature of the concatenation ̂ X k ∗ ̂ X k +1 is given by Chen's relation , whose proof is in [51]. To state this result, we first note that S ( ̂ X ) n s,t can be interpreted as an element of the extended tensor algebra of R d :

Proposition A.4 (Chen's Relation) . The following identity holds:

where for elements A,B ∈ T (( R d )) with A = ( A 0 , A 1 , A 2 , ... ) and B = ( B 0 , B 1 , B 2 , ... ) the tensor product ⊗ is defined

Let X = ( X 0 , ..., X L ) be a time-series. Then the linear interpolation ˜ X : [0 , T ] → R d can be represented as the concatenation of a finite number of linear paths:

B Related Work, Experimental Choices, and Impact Statement

Continuous-time models. Since their introduction in [12], Neural ODEs were extended in various ways to facilitate modelling continuous time-series data [70, 60, 34, 40, 79]. While Neural ODEs and their extensions are successful in certain tasks they are burdened with a high computational cost, which makes them scale very poorly to long sequences in the time-series setting. Various authors propose methods and augmentations to vanilla Neural ODEs to decrease their computational overhead [24, 6]. Other approaches to augmenting deep learning methods to modelling continuous data include implicit neural representations [80, 29], continuous kernel convolutions [69], or Fourier neural operators [50, 63].

Transformers. First proposed in [86], the Transformer has been exceptionally successful in discrete sequence modelling tasks such as natural language processing (NLP). Key to the success of the Transformer in NLP is the attention mechanism, which extracts long-range dependencies. There are a number of extensions to improve efficiency and decrease the computation cost of the attention mechanism [49, 88, 22, 41, 16].

Signatures in machine learning. The path signature originates from theoretical stochastic analysis [51] and has since become a popular tool in machine learning. Path signatures are regarded as effective feature transformations for sequential data [64, 27, 44]. Additionally, signatures help mitigate the computational cost of Neural CDEs in long time-series [57] and non-Markovian stochastic control problems [39]. Other more recent works in this direction include [18, 87]. Approaches such as randomized signatures [21, 19] and the signature kernel [47, 77] have been developed to mitigate the curse of dimensionality inherent in path signature computations. Rough Transformers provide a first step towards incorporating path signatures for continuous-time sequence modelling using Transformers. 6

We also note that contemporary work [84] employs a Transformer architecture with signature features for the task of deep hedging. However, our work differs in several key aspects. First, we introduce the multi-view attention mechanism, which uses signatures to extract both global and local information, which we found to be necessary in our experimentation, as Transformers are known to struggle in extracting local information (see Figure 7), whereas their work just uses a global signature. Moreover, their work computes the signature at every time step, strictly dilating input data. This is particularly problematic for long, multi-variate sequences, for reasons discussed above, and can actually negatively impact performance. Our work, however, compresses data using the multi-view signature transform, and we find that this compressed representation can actually improve performance. Finally, their work relies on the assumption that data is regularly sampled, as the signature is computed at every time step, in contrast to our work which is robust to irregular sampling.

Hence, the signature is

Long-Range Sequence modelling. Ahighly relevant line of research related to enhancing recurrent neural networks' capability to capture long-term dependencies involves the development of various models. These include Unitary RNNs [1], Orthogonal RNNs [38], expRNNs [48], chronoLSTM [82], antisymmetric RNNs [10], Lipschitz RNNs [25], coRNNs [71], unicoRNNs [72], LEMs [73], waveRNN [42], Linear Recurrent Units [62], and Structured State Space Models [32, 31]. While we utilize many benchmarks and synthetic tasks from these works to test our model, it is important to note that our work is not intended to compete with the state-of-the-art in these tasks. Therefore, we do not directly compare our model with the models mentioned above. Instead, this paper seeks to show that the baseline Transformer architecture can benefit from the use of signatures by (i) becoming more computationally efficient, (ii) being invariant to the sampling rate of the signal, and (iii) having a good inductive bias for temporal and spatial processing. Furthermore, we highlight that RFormer brings alternative benefits, such as the ability to perform spatial processing effectively, which is a setting in which long-range sequence models typically struggle.

Efficient Attention Variants. There are several efficient self-attention variants that have emerged over the years, including Sparse Transformer [14], Longformer [5], Linear Transformers [41], BigBird [90], Performer [16], or Diffuser [26]. In our setting, we highlight that a central part of this paper is to showcase how signatures significantly reduce the computational requirements of vanilla attention and empirically demonstrate that this also results in improved learning dynamics and invariance to the sampling frequency of the signal. Given the large efficiency gains that we observed with this approach when employed on vanilla attention, we did not consider that further experimentation on other forms of 'approximate' attention was needed. Since most variants of attention seek to make the operation more efficient through several approximations (e.g., linearization or sparsification techniques), we believe that a first attempt at showcasing the power of multi-view signatures on vanilla attention is already significant. However, other variants of attention (such as the ones outlined before) could be added on top of the signature representations to obtain even better efficiency gains.

Limitations and Future Work. While we found RFormer to be very performant in our experiments, much of this performance gain relies on heavy hyperparameter tuning, especially when it comes to the choice of window sizes and signature level. However, this could be handled using Neural Architecture Search (NAS) techniques, such as those employed in [76]. Furthermore, despite the computational gains we achieve for low-dimensional sequences, additional work would be required to scale this method to larger dimensions. We should also note that the experiments and results presented in this paper are constrained by the relatively small scale of the models studied.

C Experimental Details

All experiments are conducted on an NVIDIA GeForce RTX 3090 GPU with 24,564 MiB of memory, utilizing CUDA version 12.3. Hyperparameters used to produce the results in Table 2 are reported in Tables 6. The timings presented in all tables are obtained by executing each model independently for each dataset and averaging the resulting times across 100 epochs.

To prevent excessive growth in signature terms, we use the univariate signature in LOB datasets. As an alternative, one could employ randomized signatures [19] or low-rank approximations [9, 11] .

D Baselines Validation

This section collects the validation of Step and Depth for the Neural-RDE model. Optimal values are selected for evaluation on test-set. Early-stopping is used with the same criteria as [57].

E Long Temporal Datasets Details

Table 13 summarises the long temporal modeling datasets from the UEA time series classification archive [3] used in Section 4.

F Ablation Studies

F.1 Global and Local Signature Components

In this section, we ablate the use of the multi-view signature transform over both global and local transformations of the input signal. The results for the sinusoidal datasets are shown in Figure 7. In most cases, the use of both local and global components improves the performance of RFormer . This choice, however, can be seen as a hyperparameter and will be dataset-dependent.

0

500

1000

Epoch

0.0

0.2

0.4

0.6

0.8

1.0

Multi-View

Global

Local

0

500

1000

Epoch

0.0

0.2

0.4

0.6

0.8

1.0

F.2 Signature Level and Naive Downsampling

One of the main points of the paper is that the shorter representation of the time-series endowed by the signatures helps to significantly reduce the computational cost of the self-attention operation with minimal information loss (and with improved performance in many of the experiments). By equation (16), one sees that the first level of the signature of a linear function is the difference between its endpoints. Hence, using multi-view attention with signature level one operates on the increments of piecewise-linear interpolated data, which corresponds to naive downsampling. To test that higher levels of the signature provide improvements in performance, we compare the result of using the signature on the datasets tested in Table 14 below.

There is a significant performance gain in considering higher levels of the signature because one can capture the higher-order interactions between the different time-series.

G Additional Experiments and Comparisons

G.1 Random Drop Experiments

Furthermore, we conduct a new set of experiments in which we dropped 30% and 70% of the dataset for RFormer. Note that even with a 70% drop rate in the EigenWorms dataset, the vanilla Transformer fails to run due to memory limitations. Therefore, to provide results for the Transformer model on the EigenWorms dataset, we conduct experiments with an 85% drop rate. This comparison highlights the performance gap between the vanilla Transformer and our proposed model under these conditions, with the RFormer model yielding superior results. All results are computed across five seeds and are summarized in the tables and figure below.

0

500

1000

Epoch

0.0

0.2

0.4

0.6

Transformer

RFormer

NCDE

ODE-RNN

GRU

0

500

1000

Epoch

0.0

0.2

0.4

0.6

0.8

Finally, Table 19 compares CRU and RFormer in an irregularly sampled synthetic data setting, featuring shorter sinusoids and fewer classes than the experiments in Section 4.1. Additionally, Table 20 presents the hyperparameter validation for CRU (see Table 21 for training time analysis). These experiments demonstrate that recurrent models perform well with short sequences. Note that despite RFormer's superior performance, our model is significantly faster than other continuous-time models, as shown in Appendix G.2, particularly in Table 21.

G.2 Additional Efficiency Experiments and Discussion

We conduct additional experiments to compare the runtime of Rough Transformers with other models. In this experiment, we use the synthetic sinusoidal dataset considered in our paper and compute the runtime per epoch for varying sequence lengths. We demonstrate results for two variants of RFormer: 'online', which corresponds to computing the signatures of each batch during training (resulting in significant redundant computation), and 'offline', which corresponds to computing the signatures in one go at the beginning of training. We include a recent RNN-based model as a basis for comparison with high-performing RNN baselines. In addition to the models discussed in Section 4, we introduce Continuous Recurrent Units (CRU) [78] as a new baseline. See Table 21 for a summary of the results.

Weremark that previous running times are obtained with a batch size of 10. Further, the ContiFormer model could be run for L = 1000 if decreasing the batch size to 2 (which significantly affects the parallelization process), avoiding OOM issues and resulting in 4025 seconds/epoch, which is several orders of magnitude larger than RFormer . As an additional experiment, we tested the epoch time (S/E) of RFormer for extremely oversampled sinusoidal time series. We show our results in the table below.

Thus, the time needed to compute the signature is inconsequential when compared with the time required to train standard models on the full or even downsampled datasets, since this step has to be carried out only once. To put this into context with an example, we note that it takes 4s to compute the signature representations for the HR dataset (which is about half the time it takes for the Vanilla Transformer to go through one epoch) and results in a 26 × increase in computational speed for RFormer when compared to the vanilla Transformer.

To showcase that this is the case for not only sequences of moderate length but also extremely long sequences, we also carry out the following experiment where we compute the signature representation for the sine dataset, with a progressively increasing number of datapoints. As seen in Table 23, this does not cause an explosion in computational time.

0

50000

100000

150000

200000

250000

Input Length

0

1000

2000

3000

4000

5000

S/E

Transformer

ContiFormer

RFormer (Online)

RFormer (Offline)

NCDE

0

50000

100000

150000

200000

250000

Input Length

2

-1

2

3

2

6

2

9

2

12

S/E (Log Scale)

Transformer

ContiFormer

RFormer (Online)

RFormer (Offline)

NCDE

2

7

2

9

2

12

2

15

2

18

2

-1

2

3

2

6

2

9

2

12

S/E (Log Scale)

Transformer

RFormer (Online)

NCDE

ContiFormer

RFormer (Offline)

G.3 Additional ContiFormer Comparisons

Also, to provide some context of the performance of ContiFormer compared with our method (and not only results on complexity and training times), we run the model on the sinusoidal classification task for signals of length L = 100 and L = 250 . Due to the slow running time of the ContiFormer model, we did not consider sequence lengths of L > 250 . We evaluate the ContiFormer model using one head. However, given the subpar results we obtain, we also test it with four heads, using the hyperparameters originally used in the paper for their irregularly sampled time series classification experiments. By contrast, all variations of RFormer tested in this paper for this experiment employ only one head, but reported significantly better results.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Yes, they are an accurate reflection of the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Yes, the limitations ae discussed in Appendix B.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The paper contains only one theoretical result, and a complete and correct proof is provided.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All experimental details are in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All datasets used are publicly available and the associated code can be found in the following anonymized repo: https://anonymous.4open.science/r/rformer_ submission-2546 .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All details are specified in Appendices C and D.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All experiments are run with several seeds, and the standard deviation is reported alongside the average results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All experiments were conducted on an NVIDIA GeForce RTX 3090 GPU with 24,564 MiB of memory, as outlined in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We conform with the NeurIPS code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The societal impact statement is included in Appendix B.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper does not involve the release of data or models that have a high risk for misuse. However, as mentioned in our impact statement included in Appendix B, we acknowledge potential misuses of our advancements in time series analysis and advocate for ethical application and regulatory oversight.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The owners of the datasets and packages used are acknowledged in the code and in the manuscript.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not introduce new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not include any experiments involving crowdsourcing or human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not include any research involving human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Rough Transformer, a variant of the original Transformer that allows the processing of discrete-time series as continuous-time signals through the use of multi-view signature attention. Empirical comparisons shows that Rough Transformers outperform vanilla Transformers and continuous-time models on a variety of time-series tasks and are robust to the sampling rate of the signal.\n\nSTRENGTHS:\n- Overall the paper is well written and is easy to follow\n- The idea of using signature transform within an attention mechanism is interesting\n\nWEAKNESSES:\n- Empirical evaluations are limited, casting doubts on the true potential of the proposed architecture\n- The tasks considered are rather simple, and it is not clear whether the proposed architecture will give favorable tradeoffs between accuracy and efficiency in the more challenging tasks (see below) \n- Missing evaluations on time series forecasting tasks (only classification and regression tasks are considered)\n- Missing comparisons with recent RNN models (such as https://arxiv.org/abs/2110.04744, https://arxiv.org/abs/2212.00228), Transformer models (e.g., those studied in https://arxiv.org/abs/2011.04006), State Space models (https://arxiv.org/abs/2111.00396 and the more recent variants such as Mamba: https://arxiv.org/abs/2312.00752) and other sequence models (https://arxiv.org/abs/2305.01638, https://arxiv.org/abs/2209.10655) \n- Missing ablation studies on the components of the proposed architecture, particularly the role of the global and local components in the multi-view signature, truncation level n, etc.\n- Missing related work; e.g., the papers mentioned above, https://link.springer.com/article/10.1007/s40304-017-0103-z and https://arxiv.org/abs/1710.10121 for continuous-time DL models, https://arxiv.org/abs/2006.12070, https://arxiv.org/abs/2102.04877 for continuous-time RNN models, https://dl.acm.org/doi/abs/10.5555/3546258.3546305 for using path signatures to understand continuous-time RNNs\n\nQUESTIONS:\n- I was wondering how effective is the proposed model in autoregressive generative tasks and time series forecasting (see https://arxiv.org/abs/2311.04147)? These are natural tasks in the domain of sequence modeling \n- While the proposed method can improve the modified transformer model in sequence modeling, why would the method be attractive for practitioners when they could just use more sophisticated models like SSMs for sequence modeling?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Rough Transformers, an attention-based model for long continuous-time signals. The model utilizes ideas from rough path theory to extract path signatures from the continuous-time signal (obtained by interpolation of the original signal). Two types of signatures are extracted: global and local. The global signature extracts long-term information from the signal whereas the local signature extracts local information. Self-attention is then used on this \"multi-view\" signature. Experiments on classification and regression tasks show improved performance over existing model architectures.\n\nSTRENGTHS:\n- The paper views the problem of modeling long continuous-time signals through the lens of rough path theory. While this in itself is not novel, the combination of rough path signatures with attention is a novel combination.\n- Experiments on classification and regression tasks show that RFormer improves over existing models in terms of accuracy and significantly more compute efficient. (I have some concerns and questions about the empirical analysis, please see Weaknesses)\n\nWEAKNESSES:\n- The technical contribution is limited. In such scenarios, the empirical analysis needs to be sufficiently strong.\n- The empirical analysis has been conducted on a few toyish datasets. While the results are definitely promising, more experimental support is needed to validate the model.\n    - Although the model is motivated from a continuous-time and irregularly-sampled data perspective, the actual investigation of these settings is limited. As per my understanding, all experiments under 4.1 have been conducted on regularly sampled time series (please correct me, if I am wrong). I find it surprising that simple RNN-based methods do not perform well in these settings. If this is indeed the case, some simple CNN-based method should be studied. Recent models based on state space layers (e.g., S4, Mamba) can also be explored as baselines.\n   - OOM for important baselines is not really helpful to draw any conclusions. To highlight efficiency of RFormer, please conducted experiments where you increased the context length or other parameters to show where baselines run OOM and how do they perform before that.\n   - More experiments are needed, particularly for the forecasting task to understand how well the model understands the dynamics. \n   - I took a brief look at the code and it looks like hyperparameter tuning was conducted for RFormer. Was such tuning also performed for the baselines? If no (which is a valid response), how did you selected the baseline parameters? How sensitive is the model to different hyperparameters? \n- The discussion on related work needs to be moved to the main text and improved. Please contrast RFormer with the related works, particularly the ones that are closely related such as NRDE and ContiFormer. Discussion on some closely related works [1, 2, 3] is missing. Ideally there should also be a comparison with at least of these methods (e.g., CRU).\n\n[1] Schirmer, Mona, et al. \"Modeling irregular time series with continuous recurrent units.\" International conference on machine learning. PMLR, 2022.   \n[2] Ansari, Abdul Fatir, et al. \"Neural continuous-discrete state space models for irregularly-sampled time series.\" International Conference on Machine Learning. PMLR, 2023.   \n[3] Oh, YongKyung, Dongyoung Lim, and Sungil Kim. \"Stable Neural Stochastic Differential Equations in Analyzing Irregular Time Series Data.\" arXiv preprint arXiv:2402.14989 (2024).    \n\nI am happy to update my score, if my concerns are adequately addressed.\n\nQUESTIONS:\n- Can the authors clarify by what they mean by \"input sequences must be sampled at the same times, (ii) the sequence length must be fixed\"? These do not seem to be limitations of transformers, especially (ii). \n\nSee weaknesses for other questions.\n\nLIMITATIONS:\nThere is a brief discussion on limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Rough Transformer (RFormer), an extention of the Transformer architecture towards operating on continuous-time representations of time-series data. RFormer employs a novel technique called multi-view signature attention, which performs attention on path signatures pre-computed from input data offline, thereby capturing both local and global dependencies across observations. Experiments on various real-world time-series datasets shows that RFormer enjoys superior predictive performance as well as computational efficiency compared to previous methods, while being robust to changes in sequence length and irregular sampling.\n\nSTRENGTHS:\n- [S1] **Good novelty.** To the best of my knowledge, incorporating rough path theory to time-series representation learning is a novel approach, and would be of great interest to the machine learning community. \n\n- [S2] **Great empirical performance.** Experiments on a wide variety of real-world datasets show large improvements in both accuracy and efficiency, demonstrating strong utility of proposed multi-view signatures in time-series modeling.\n\nWEAKNESSES:\n- [W1] **Questionable motivation of synthetic frequency classification experiments.** The first experimental section tests RFormer on two synthetic datasets with which the task is to classify input time-series based on their ground-truth frequencies. While L233-234 mentions the second setup in particular is designed towards testing the long-range reasoning capabilities of RFormer, but it is unclear whether this indeed the case. For the second synthetic dataset, in particular, how can identifying the frequency be a proxy for long-range reasoning when the frequency $\\omega_0$ is used for $t < t_0$ only? The results on Figure 2 showing that methods that are \"tailor-made for long-range time series modeling\" (L254) such as Neural-CDE and Neural-RDE underperforming significantly also indicates that the designed task is not really representative of long-range reasoning.\n\\\n\\\nMore interesting questions to ask could be: What makes RFormer sample-efficient vs. vanilla Transformer particularly on the Sinusoidal dataset and not so much on the Long Sinusoidal dataset? What makes RFormer more robust to changes in sampling frequency compared to Neural-CDE and Neural-RDE? Table 1 of [A] shows Neural-CDE is also quite robust to dropped data, but is this characteristic not emergent for Sinusoidal and Long Sinusoidal datasets?\n\n- [W2] **Missing analysis on interpolation methods.** By default, RFormer uses piecewise-linear paths for computing path signatures, but as mentioned in L140, it seems any continuous-time interpolation can be used. As such, it would be interesting to discuss (1) whether any other interpolation techniques can be deployed efficiently similarly to piecewise-linear paths and (2) if they lead to any boosts in predictive accuracy, but these discussions are missing in the current draft (i.e., is the currently used piecewise-linear interpolation \"pareto-optimal\" under performance-efficiency trade-offs?).\n\n[A] Kidger et al., Neural Controlled Differential Equations for Irregular Time Series. NeurIPS 2020.\n\nQUESTIONS:\n- [Q1] **Large overlaps in representation space.** Basec on the multi-view signature formulation in Figure 1 and Equation 8, it seems the earlier observations would be covered by a large number of input tokens. As shown in the right plots in Figure 3, this would result in large \"representational overlap\" similarly to the oversmoothing phenomenon in graph representation learning [B]. Considering this, could restricting the global view to a few previous points rather than all previous points be a viable option? or does the theoretical and empirical robustness of RFormer to variable lengths and irregular sampling require that all previous points be covered in the global view?\n\n- [Q2] **Discussion on input length of MLP and Transformer.** L108 states that the input length $L$ of the MLP and Transformer is fixed by assumption, but is this true? Sequences with different lengths can be processed in a single batch via padding.\n\n[B] Rusch et al., A Survey on Oversmoothing in Graph Neural Networks. arXiv 2023.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations in Appendix B.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-29"}