Rough Transformers: Lightweight Continuous-Time Sequence Modelling with Path Signatures

Authors: fernando  moreno-pino
álvaro  arroyo
harrison  waldon
xiaowen  dong
álvaro  cartea
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 16:40:10

Abstract: Time-series data in real-world settings typically exhibit long-range dependencies and are observed at non-uniform intervals. In these settings, traditional sequence-based recurrent models struggle. To overcome this, researchers often replace recurrent models with Neural ODE-based architectures to account for irregularly sampled data and use  Transformer-based architectures to account for long-range dependencies. Despite the success of these two approaches, both incur very high computational costs for input sequences of even moderate length. To address this challenge, we introduce the Rough Transformer, a variation of the Transformer model that operates on continuous-time representations of input sequences and incurs significantly lower computational costs. In particular, we propose multi-view signature attention, which uses path signatures to augment vanilla attention and to capture both local and global (multi-scale) dependencies in the input data, while remaining robust to changes in the sequence length and sampling frequency and yielding improved spatial processing. We find that, on a variety of time-series-related tasks, Rough Transformers consistently outperform their vanilla attention counterparts while obtaining the representational benefits of Neural ODE-based models, all at a fraction of the computational time and memory resources.

Main paper: 
 introduction: 
 Real-world sequential data in areas such as healthcare [68] , finance [38] , and biology [30] often are irregularly sampled, of variable length, and exhibit long-range dependencies. Furthermore, these data, which may be drawn from financial limit order books [10] or EEG readings [89] , are often sampled at high frequency, yielding long sequences of data. Hence, many popular machine learning models struggle to model real-world sequential data, due to input dimension inflexibility, memory constraints, and computational bottlenecks. Rather than treating these data as discrete sequences, effective theoretical models often assume data are generated from some underlying continuous-time process [56, 69] . Hence, there is an increased interest in developing machine learning methods that use continuous-time representations to analyze sequential data.One recent approach to modelling continuous-time data involves the development of continuous-time analogues of standard deep learning models, such as Neural ODEs [14] and Neural CDEs [48] , which extend ResNets [39] and RNNs [32] , respectively, to continuous-time settings. Instead of processing discrete data directly, these models operate on a latent continuous-time representation of input sequences. This approach is successful in continuous-time modelling tasks where standard deep recurrent models fail. In particular, extensions of vanilla Neural ODEs to the time-series setting [73, 48] succeed in various domains such as adaptive uncertainty quantification [62] , counterfactual inference [83] , or generative modelling [9] .In many practical settings, such as financial market volatility [22, 57] or heart rate fluctuations [37] , continuous-time data also exhibit long-range dependencies. That is, data from the distant past may impact the system's current behavior. Deep recurrent models struggle in this setting due to vanishing gradients, whereas continuous-time analogues of these models have been shown to address this difficulty [49] . Several recent works [55, 61] also successfully extract long-range dependencies from sequential data with Transformers [90] , which learn temporal dependencies of a tokenized representation of input sequences. Extracting such temporal dependencies requires a positional encoding of input data, because the attention mechanism is permutation invariant, which projects data into some latent space. The parallelizable nature of the Transformer allows for rapid training and evaluation on sequences of moderate length and it contributes to its success in fields such as natural language processing (NLP). The continuous-time path is irregularly sampled at points marked with a red x. The local and global signatures of a linear interpolation of these points are computed and concatenated to form the multi-view signature. The multi-view signature transform consists of L multi-view signatures.While the above approaches succeed in certain settings, several limitations hinder their wider applications. On the one hand, Neural ODEs and their analogues [48, 73] bear substantial computational costs when modelling long sequences of high dimension; see [60] . On the other hand, Transformers operate on discrete-time representations of input sequences, whose relative ordering is represented by the positional encoding. This representation may inhibit their expressivity in continuoustime data modelling tasks [95] . Moreover, Transformer-based models suffer from a number of difficulties, including (i) input sequences must be sampled at the same times, (ii) the sequence length must be fixed, and (iii) the computational cost scales quadratically in the length of the input sequence. These difficulties severely limit the application of Transformers to continuous-time data modelling.Contributions 1) We introduce Rough Transformers, a variant of the Transformer architecture amenable to the processing of continuous-time signals, which can be easily integrated into existing code-bases. The Rough Transformer is built upon the path signature from Rough Path Theory [54] . We define a novel, multi-scale transformation which projects discrete input data to a continuous-time path and compresses the input data with minimal information loss. Moreover, this transformation is an efficient feature representation of continuous-time paths, because linear functionals of path signatures approximate continuous functions of paths arbitrarily well (see Theorem A.2 in Appendix A).2) We introduce the multi-view attention mechanism to extract both local and global dependencies of very long time-series efficiently. This mechanism operates directly on continuoustime representations of data without the need for expensive numerical solvers or constraints on the smoothness of the data stream. Moreover, the multi-view attention mechanism is provably robust to irregularly sampled data.3) We carry out extensive experimentation on long and irregularly sampled time-series data. In particular, we show that Rough Transformers (i) improve the learning dynamics of the Transformer, making it more sample-efficient and allowing it to achieve better out-of-sample results, (ii) reduce the training cost by a factor of up to 25× when compared with vanilla Transformers and more when compared with Neural ODE based architectures, (iii) maintain similar performance when data are irregularly sampled, where traditional recurrent-based models suffer a substantial decrease in performance [73] , and (iv) yield improved spatial processing, accounting for relationships between different temporal channels without having to pre-define a specific inter-channel relation structure. 
background and methodology: 
 Problem Formulation. In many real-world scenarios, sequential data are time-series sampled from some underlying continuous-time process, so datasets consist of long, irregularly sampled sequences of varied lengths. In these settings, the problem of sequence modelling is described as follows. Let C(R + ; R d ) = {g : R + → R d | g continuous}, and consider X ∈ C(R + ; R d ) which we call a continuous-time path. A time-series of length L with sampling timesT X = {t i } L i=1 ⊂ R + is defined as X = ((t 1 , X 1 ), ..., (t L , X L )), where X i = X(t i ) ∈ R d . Now, define a continuous function on paths f : C(R + ; R d ) → R k . Next define a dataset D = (X i , f ( X i )) N i=1. We seek to approximate the function f from the set D for some downstream task. Importantly, we do not assume that T X = T Y for all X, Y ∈ D, so that D may be irregularly sampled.Sequence Modelling with Transformers. Transformers are used extensively as a baseline architecture to approximate functions of discrete-time sequential data and are successfully applied to settings when input sequences are fixed in length, relatively short, and sampled at regular intervals. First, the Transformer projects input time series X ∈ R L×d to a high-dimensional space X → T (X) ∈ R L×d ′ for d ′ >> d using some linear positional encoding T : R L×d → R L×d ′ . Next, a latent representation of the encoded sequence is learned by a multi-headed self-attention mechanism which splits T (X) into H distinct query, key, and value sequences:Q h = T (X)W Q h , K h = T (X)W K h , V h = T (X)W V h , respectively, with h = 1, ..., H and weight matrices W Q h , W K h , W V h ∈ R d ′ ×d ′ .The multi-head self-attention calculation for each head is given byEQUATIONand the latent representation is projected to the output space R k using a multi-layer perceptron (MLP).The input length L of the MLP and the Transformer is fixed by assumption. To evaluate the Transformer on a time-series X with |T X | ̸ = n, one must perform some transformation (interpolation, extrapolation, etc.) which may degrade the performance of the model. Furthermore, the memory and time complexity of the Transformer is of order O(L 2 d), which presents a substantial difficulty in modelling long sequences.Rough Path Signatures. Broadly, the difficulties faced by the Transformer in modelling timeseries stem from time-series being sampled from underlying continuous-time objects, while the attention mechanism underpinning the Transformer is designed to model discrete sequences. To address these difficulties, Rough Transformers augment standard Transformers by lifting the input time-series to the space of continuous-time functions and performing the self-attention calculation in this infinite-dimensional space. To achieve this, we use the path signature from Rough Path Theory.For a continuous-time path X ∈ C 1 b (R + ; R d ) and times s, t ∈ R + , the path signature of X from s to t, denoted S( X) s,t , is defined as follows. First, letI d = {(i 1 , ..., i p ) : i j ∈ {1, ..., n} ∀ j and p ∈ N} (2)denote the set of all d-multi-indices and I n d = {I ∈ I d : |I| = n}. Next, set S( X) 0 s,t := 1 and for anyEQUATIONwhere ˙ X j = d X j /dt. Abusing notation, define level n of the signature asEQUATIONand define the signature as the infinite sequence S( X) n s,t = (S( X) 0 s,t , S( X) 1 s,t , ..., S( X) n s,t , ...) .Finally, define the truncation of the signature S( X) ≤n s,t = (S( X) 0 s,t , ..., S( X) n s,t ), where S( X) n s,t can be interpreted as an element of the extended tensor algebra of R d :EQUATIONAnalogously, we say that S( X) ≤n s,t ∈ T ((R d )) ≤n . A central property of the signature is that is invariant with respect to time-reparameterization [54] . That is, let γ : [0, T ] → [0, T ] be surjective, continuous, and non-decreasing. Then we haveEQUATIONwhich will be crucial to demonstrate the Rough Transformer's robustness to irregularly sampled data.In contrast to wavelets or Fourier transforms, which parameterize paths on a functional basis, the signature provides a basis for functions of continuous paths. Hence, the path signature is well-suited to sequence modelling tasks in which one seeks to learn a function of the underlying functional. For a more rigorous presentation of signatures and a description of additional properties, see Appendix A and Lyons et al. [54] .Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Yes, the limitations ae discussed in Appendix B.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
rough transformers: 
 Now, we construct the Rough Transformer, a Transformer-based architecture that operates on continuous-time sequential data by means of the path signature.Let D be a dataset of irregularly sampled time-series. To project a discretized time-series X ∈ D to a continuous-time object, let X denote the piecewise-linear interpolation of X. 2 Next, for t k ∈ T , define the multi-view signatureEQUATIONIn what follows, we refer to the components S( X) 0,t k , S( X) t k-1 ,t k as global and local, respectively; see Figure 1 . Intuitively, one can interpret the global component as an efficient representation of long-term information (see Theorem A.2 in Appendix A), and the local component as a type of convolutional filter that is invariant to the sampling rate of the signal. Now, define the multi-view signature transform M (X) = (M (X) 1 , ..., M (X)L) , and denote by M (X) ≤n the truncated signature for a truncation level n. Next, define the multi-view attention mechanism, which uses the multi-view signature transform to extend the standard attention mechanism to the space of continuous functions [54] . First, fix a truncation level n ∈ N, and letd ∈ N be such that M (X) ≤n k ∈ R d. For h = 1, ..., H let W Q, K, Ṽ h ∈ R d× d′for some d′ ∈ N, and letEQUATIONThen, the attention calculation is given byEQUATION)Notice that the attention calculation is similar to (1), however, we stress that the multi-view attention is built on continuous-time objects, the signatures, while the standard attention mechanism acts on discrete objects. The multi-view signature provides a compressed representation of the time series, minimizing the computational costs associated to quadratic scaling without excessive loss of representational capacity, see Appendix F. Transformers. This enables both faster wall-clock training time and the ability to process long input sequences which would otherwise yield out-of-memory errors for the vanilla Transformer, see Figure 2 . Moreover, the multi-view attention mechanism does not require backpropagation through the signature calculation and can be computed offline. This is significantly more computationally efficient compared with the complexity of computing signatures batch-wise in every training step. Finally, the signature of piecewise-linear paths can be computed explicitly, see Appendix A, and there are a number of Python packages devoted to optimized signature calculation [46, 71] .Variable Length and Irregular Sampling. The multi-view signature transform underpinning Rough Transformers is evaluated by constructing a continuous-time interpolation of input data and computing a series of iterated integrals of this interpolation. The bounds of these integrals are a fixed set of time points, meaning that the sequence length of the multi-view attention mechanism is fixed and independent of the sequence length of input samples. Furthermore, the following proposition shows that the output of the Rough Transformer for two (possibly irregular) samplings of the same path is similar. Proposition 3.1. Let T be a Rough Transformer. Suppose X : [0, T ] → R d is a continuous-time process, and let γ : [0, T ] → [0, T ] denote a time-reparameterization. Suppose X and X ′ are samplings of X and X • γ, respectively. Then T(X) ≈ T(X ′ ).Proof. By (7) , S( X) s,t = S( X • γ) s,t for all s, t ∈ [0, T ]. Hence, one has M (X 1 ) ≈ M (X 2 ). Finally, T(X 1 ) ≈ T(X 2 ) because the attention mechanism and final MLP are both continuous.Hence, the Rough Transformer is robust to irregular sampling. In many tasks, the sampling times convey important information about the time-series. In these settings, one may augment the input time-series with its sampling times, that is, write X = ((t 0 , X 0 ), ..., (t L , X L )).Spatial Processing. While an interpolation of input data could be sampled to make vanilla Transformers independent of the length of the input sequence, important locality information could be lost, see Appendix F.2. Instead, Rough Transformers summarize spatial interactions between channels by means of the multi-view signature transform. One may notice that in (5) , the dimension of the signature grows exponentially in the level of the signature n. In particular, whenX i ∈ R d , |S( X) ≤n 0,t | = d(d n -1) d-1= O(d n ), so the multi-view attention calculation is of order O( L2 d n ). In many practical time-series modelling problems, however, the value of d is not very large. The signature terms also decay factorially in the signature level n (see Proposition A.3 in Appendix A), so in practice, one may take the value of n to be small without sacrificing performance. The majority of computational savings result from the reduction of the sequence length to L, and in practice, we take L << L.When the dimension d is large, there are three possible remedies to maintain computational efficiency. First, instead of computing the signature in M (X) k = (S(X) 0,t k , S( Xt k-1 ,t k ), one may compute the log-signature, which is a compressed version of the signature [70] . When the dimension is large enough such that the log-signature is computationally infeasible, one may instead compute the univariate signatures of features coupled with the time channel. That is, consider X ∈ C([0, T ]; R d ), with X(t) = ( X 1 (t), ..., X d (t)). Denote the time-added function X i (t) := (t, X i (t)). Then we define the univariate multi-view signatureEQUATIONThe attention mechanism in this case is constructed as before. Fixing the maximum signature depth to be some value n * , one sees that the number of features in the univariate multi-view signature is approximately 2 n * d. In practice we find that n * ≤ 5 provides sufficient performance, so the order of the attention calculation is O(C L2 d) for C ≤ 2 n * . Finally, one may use randomized signatures to reduce dimension by using a Johnson-Lindenstrauss-type projection to a low-dimensional latent space and computing the signature in this space, as in [23, 21] .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes]• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experiments: 
 In this section, we present empirical results for the effectiveness of the Rough Transformer, hereafter denoted RFormer, on a variety of time-series-related tasks. Experimental and hyperparameter details regarding the implementation of the method are in Appendices C and D. We consider long multivariate time-series as our main experimental setting because we expect signatures to perform best in this scenario. Additional experimentation on long-range reasoning tasks on image-based datasets is left for future work, as these would likely require additional inductive biases.To benchmark RFormer, we consider both discrete-time and continuous-time models. In particular, we include as main baselines traditional RNN models (GRU [17] ), ODE-based methods designed for sequential data (Neural-CDE [48] ), as well as ODE-based methods explicitly designed for long time-series (Neural-RDE [60] ). 3 Furthermore, we compare against a vanilla Transformer [90] which is the RFormer backbone. Finally, we present comparisons with a recent continuous-time Transformer model, ContiFormer [15] , to highlight the computational efficiency gap between RFormer and similar continuous-time models. We note that the first two tasks focus on evaluating the performance improvement of RFormer over the Transformer baseline. For other long-range tasks, we include comparisons to recent state-space models [33, 65, 85] . In the irregular sampling regime, we benchmark against state-of-the-art models tailored to that setting [64, 82] . See Appendix B for additional discussion on related models and more details about our experimental choices. Frequency Classification. Our first experiment is based on a set of synthetically generated time series from continuous paths of the formEQUATIONwhere g(t) is a non-linear trend component, ν and η are two noise terms, and ω is the frequency. Here, the task of the model is to classify the time-series according to its frequency ω. We consider 1000 samples in 100 classes with ω evenly distributed from 10 to 500. Each time-series is regularly sampled with 2000 times-steps on the interval [0, 1]. This synthetic experiment is similar to others in recent work on time-series modelling [52, 93, 58] .We include an additional experiment in which we alter the signal in ( 12) so its frequency is ω 0 for t < t 0 and ω 1 afterward, where the task is to classify the sinusoid based on the first frequency. We call this dataset the "long sinusoidal" dataset. This extension of the original experiment aims to test the ability of the model to perform long-range reasoning effectively. Note that for this task, we also add ODE-RNN [73] to the previously mentioned baselines. HR dataset. Next, we consider the Heart Rate dataset from the TSR archive [87] , originally sourced from Beth Israel Deaconess Medical Center (BIDMC). This dataset consists of time-series sampled from patient ECG readings, and each model is tasked to perform a regression by forecasting the patient's heart rate (HR) at the sample's conclusion. The data, sampled at 125Hz, consists of three-channel time-series (including time), each spanning 4000 time steps. We used the L2 loss metric to assess the performance. Table 1 shows the results, where ⋄ denotes the results from Morrill et al. [60] and † our reproduction.The sequences in the HR dataset are sufficiently short to remain within memory when running the Transformer model. The baseline Transformer model improves over GRU, and ODE-RNN, however, it is less competitive when compared with Neural-RDE, suggesting that the Transformer is not particularly well-suited for this type of task. However, the RFormer model improves over the baseline Transformer by 67%. Across all tasks, we see significant improvements in efficiency as a consequence of the signature computation. We elaborate on this in more detail in the following subsection.Long Time Series Classification. We now evaluate the performance of RFormer on five long time series classification tasks from the UEA time series classification archive [5] . A summary of these datasets is provided in Table 13 in Appendix E. As previously done in [60] , the original train and test datasets are merged and then randomly divided into new train, validation, and test sets, following a 70/15/15 split. The resulting performance metrics are summarized in Table 2 . 4In this setting, we see that RFormer generally matches or slightly outperforms the continuous-time and SSM baselines. Due to the scaling problems of ContiFormer with respect to sequence length, we were unable to run this baseline within GPU memory constraints in most cases, and thus no results are reported (see Appendix G.2 for efficiency comparisons between models). In contrast, RFormer can cheaply train on the same device (see Section 4.2 for details) due to its ability to take advantage of the parallel nature of GPU processing and compress the original time series. This is especially noticeable when compared to continuous-time models (Neural-CDE, Neural-RDE, LogCDE), which are sometimes orders of magnitude slower than our model and consistently report lower or similar results. Additional experimental details can be found in Appendix G, as well as some experiments on hyperparameter sensitivity. Here, we focus on the computational gains of the model when compared with vanilla Transformers and methods that require numerical ODE solvers.Attention-based architectures are highly parallelizable on modern GPUs, as opposed to traditional RNN models which require sequential updating. However, vanilla attention experiences a bottleneck in memory and time complexity as the sequence length L grows. As covered above in Section 3, variations of the signature transform allow the model to operate on a reduced sequence length L without increasing the dimensionality in a way that would become problematic for the model. This allows us to bypass the quadratic complexity of the model without resorting to sparsity techniques commonly used in the literature [28, 52] . Tables 1-3 show that RFormer is competitive when modelling datasets with extremely long sequences without an explosion in the memory requirements. RFormer exploits the parallelism of the attention mechanism to significantly accelerate training time, as the length of the input sequence is decreased substantially. In particular, we observe speedups of 1.4× to 26.11× with respect to standard attention, and higher when compared with all methods requiring numerical solutions to ODEs. The computational efficiency gains of RFormer are attained due to the signature transform reducing the length of the timeseries with minimal information loss. The effectiveness of this transformation can be seen from the ablation study carried out in Appendix F. This contrasts with NRDEs [60] , which augment NCDEs with local signatures of input data, and find that smaller windows often perform better. Furthermore, NRDEs do not experience the same computational gains as RFormer because they must perform many costly ODE integration steps. In Figure 2 , we showcase the improvements in computational efficiency of RFormer compared to vanilla Transformers [90] , continuous-time Transformers [15] , and other continuous-time RNNs [48, 60] when processing sequences from L = 100 samples up to L = 10K. As seen, RFormer is significantly more efficient than its continuous-time and vanilla counterparts, even when performing the signature computation online, which involves computing the signatures for each batch during training, resulting in significant redundant computation. When signatures are precomputed just once before training, the computational time of each epoch remains constant across input all sequence lengths including L = 10K (see the exact signature computation times for different datasets in Table 4 ). We also stress the fact that RFormer also scales gracefully for extremely long sequences (up to L = 250K) with both online and offline computation of the signatures, as shown in Appendix G. Finally, we highlight that ContiFormer has a sample complexity of O(L 2 d 2 S), where S represents the normalized number of function evaluations of the numerical ODE solver, which makes ContiFormer orders of magnitude more computationally intensive when compared to RFormer and prevents the model from running on sequences longer than 500 points without running out of memory (see device details in Appendix C).So far, we mainly focused on the efficiency and inductive bias afforded to the model through the use of signatures. However, a key element of RFormer is that it can naturally deal with irregularly sampled sequences without expensive numerical ODE solvers. This property follows from the fact that signatures are invariant to time reparameterization, see Proposition 3.1. In this subsection, we empirically test this property by training the model on the same datasets but randomly dropping a percentage of the data points. This test intends to find if the model is able to learn continuous-time representations of the original input time-series. The results can be found in Table 5 . We find that RFormer consistently results in the best performance, with a small performance drop when compared to the full dataset. Importantly, this property is achieved in conjunction with the efficiency gains afforded to the model and without the use of expensive numerical ODE solvers. 5 Finally, we perform an additional set of experiments on the 15 univariate classification datasets from the UEA time series classification archive and compare our model with recent state-of-the-art models for irregular time series [64, 82] . Across the board, we find that our model is both faster and more accurate than the continuous-time benchmark despite having a discrete-time Transformer backbone, as shown in Figure 4 . For more details and more exhaustive experimentation on random data drops, see Appendix G. 5 Reasons for improved model performanceIn this final section, we provide explanations for the superior inductive bias of the RFormer model compared to its vanilla Transformer counterpart, despite its lower computational cost. 
spatial processing: 
 First, we highlight that a key reason the model achieves significant compression benefits in the tasks considered is its ability to jointly account for temporal and spatial interactions through the self-attention mechanism and signature terms, respectively. In particular, we believe that for certain datasets, the relationships between different channels of the time series may hold more importance than the temporal information itself, which can often be redundant. This is exemplified in the Eigenworms dataset, which experiences a 20% performance drop when employing univariate signatures, but is able to achieve state-of-the-art performance with a 600× compression rate in the temporal dimension when signatures are applied across all channels, as shown in Figure 6 . To this end, we draw parallels between the use of signatures and the field of temporal graph processing, where the use of the signature over all channels can be seen as a fully connected graph, capturing information from all channels, and the univariate signature would correspond to a graph with only self-connections between the nodes, as depicted in Figure 5 . In our view, this hints towards the idea of using sparse graph learning techniques [19, 25] to reduce the explosion in signature terms while retaining the ability to perform effective spatial processing. To empirically test these claims, we design a synthetic experiment using a 2-channel time series. Each channel contains a signal of the form sin(ω i t + ν i ), i = 1, 2, where ω i and ν i are randomly sampled from the interval [0, 2π]. For half of the dataset, the last 1% of temporal samples in the second channel are set to match the frequency of the first channel. The task is to classify whether the samples in this final interval are of the same frequency. As shown in Figure 5 , RFormer demonstrates greater sample efficiency and achieves higher test accuracy compared to its vanilla Transformer counterpart, highlighting the effectiveness of signatures in spatial processing. In addition to the benefits of higher-order signature terms, we empirically observe that even using level-one signature terms resulted in performance improvements when compared to processing sequences without any transformation. We believe that the reduction in input signal length, achieved without significant information loss through the signature transform is another important factor in the improved inductive bias of RFormer. This finding aligns with the concurrent work of [6] , which highlights some of the drawbacks of decoder-only Transformers for long sequences in terms of both oversquashing and representational collapse. To measure the degree of coarsening in the sequence, we find that interpreting the temporal sequence as a path graph and using ideas from the oversmoothing literature [78] serves as a good way to measure the similarity of the representations being fed to the Transformer. In particular, we compute the Dirichlet Energy [77] , defined in this case asE(X) = 1 N N i=1 ||X i -X i-1 || 2 ofthe temporal sequence resulting from taking increasing window sizes of the global signature. An example of this is shown in Figure 6 for the Eigenworms dataset, where we compared different numbers of windows (from 2 to 18k). Interestingly, we found that the "elbow" of the Dirichlet energy corresponded to 30 windows in this dataset, which we found empirically to be one of the the most performant settings. This hints at the idea of the Dirichlet energy being used for signature hyperparameter tuning as well. 
conclusion: 
 We introduced the Rough Transformer, a variant of the original Transformer that allows the processing of discrete-time series as continuous-time signals through the use of multi-view signature attention. Empirical comparisons showed that Rough Transformers outperform vanilla Transformers and continuous-time models on a variety of time-series tasks and are robust to the sampling rate of the signal. Finally, we showed that RFormer provides significant speedups in training time compared to regular attention and ODE-based methods, without the need for major architectural modifications or sparsity constraints. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This paper does not involve the release of data or models that have a high risk for misuse. However, as mentioned in our impact statement included in Appendix B, we acknowledge potential misuses of our advancements in time series analysis and advocate for ethical application and regulatory oversight. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: The owners of the datasets and packages used are acknowledged in the code and in the manuscript. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: This work is unlikely to result in any harmful societal repercussions. Its primary potential lies in its ability to enhance and advance existing data modelling and machine learning methods.First, we recall that the path is uniquely determined by its signature, which motivates its use as a feature map.Proposition A.1. Given a path X : [0, T ] → R d , then the map P : [0, T ] → R 1+d where P (t) = (t, X(t)) is uniquely determined by it's signature S(P ) 0,T .The proof can be found in Hambly and Lyons [35] .For Rough Transformers, several features of path signatures are important. First, linear functionals on path signatures possess universal approximation properties for continuous functionals. Theorem A.2. Fix T > 0, and letK ⊂ C 1 b ([0, T ]; R d ). Let f : K → R be continuous with respect to the sup-norm topology on C 1 b ([0, T ]; R d ).Then for any ϵ > 0, there exists a linear functional ℓ such that |f (X) -⟨ℓ, S(X) 0,T ⟩| ≤ ϵ ,for any X ∈ K, where X denotes the time-added augmentation of X.For a proof of A.2, see Arribas [2] . Even though Theorem A.2 guarantees that linear functionals are sufficient for universal approximation, linear models are not always sufficient in practice. This motivates the development of nonlinear models built upon the path signature which efficiently extract path behavior.The second feature is that the terms of the path signature decay factorially, as described by the following proposition.Proposition A.3. Given X ∈ C 1 b ([0, T ]; R d ), for any s, t ∈ [0, T ],we have that for anyEQUATIONFor a proof of Proposition A.3, see [54] . Hence, the number of terms in the signature grows exponentially in the level of the signature, but the tail of the signature is well-behaved, so only a few levels in a truncated signature are necessary to adequately approximate continuous functionals.A.1 Signatures of Piecewise Linear Paths.In the Rough Transformer, we use linear interpolation of input time-series to get a continuous-time representation of the data. As mentioned in Section 3, the signature computation in this case is particularly simple.EQUATIONThen the signature of X k is given explicitly byEQUATIONwhere ⊗ denotes the tensor product. Let X k * X k+1 denote the concatenation of X k and X k+1 . That is,EQUATIONThe signature of the concatenation X k * X k+1 is given by Chen's relation, whose proof is in [54] . To state this result, we first note that S( X) n s,t can be interpreted as an element of the extended tensor algebra of R d :EQUATION)Long-Range Sequence modelling. A highly relevant line of research related to enhancing recurrent neural networks' capability to capture long-term dependencies involves the development of various models. These include Unitary RNNs [1] , Orthogonal RNNs [40] , expRNNs [51] , chronoLSTM [86] , antisymmetric RNNs [12] , Lipschitz RNNs [27] , coRNNs [74] , unicoRNNs [75] , LEMs [76] , waveRNN [45] , Linear Recurrent Units [65] , and Structured State Space Models [34, 33] . While we utilize many benchmarks and synthetic tasks from these works to test our model, it is important to note that our work is not intended to compete with the state-of-the-art in these tasks. Therefore, we do not directly compare our model with the models mentioned above. Instead, this paper seeks to show that the baseline Transformer architecture can benefit from the use of signatures by (i) becoming more computationally efficient, (ii) being invariant to the sampling rate of the signal, and (iii) having a good inductive bias for temporal and spatial processing. Furthermore, we highlight that RFormer brings alternative benefits, such as the ability to perform spatial processing effectively, which is a setting in which long-range sequence models typically struggle.Efficient Attention Variants. There are several efficient self-attention variants that have emerged over the years, including Sparse Transformer [16] , Longformer [7] , Linear Transformers [44] , BigBird [94] , Performer [18] , or Diffuser [28] . In our setting, we highlight that a central part of this paper is to showcase how signatures significantly reduce the computational requirements of vanilla attention and empirically demonstrate that this also results in improved learning dynamics and invariance to the sampling frequency of the signal. Given the large efficiency gains that we observed with this approach when employed on vanilla attention, we did not consider that further experimentation on other forms of "approximate" attention was needed. Since most variants of attention seek to make the operation more efficient through several approximations (e.g., linearization or sparsification techniques), we believe that a first attempt at showcasing the power of multi-view signatures on vanilla attention is already significant. However, other variants of attention (such as the ones outlined before) could be added on top of the signature representations to obtain even better efficiency gains.Limitations and Future Work. While we found RFormer to be very performant in our experiments, much of this performance gain relies on heavy hyperparameter tuning, especially when it comes to the choice of window sizes and signature level. However, this could be handled using Neural Architecture Search (NAS) techniques, such as those employed in [79] . Furthermore, despite the computational gains we achieve for low-dimensional sequences, additional work would be required to scale this method to larger dimensions. We should also note that the experiments and results presented in this paper are constrained by the relatively small scale of the models studied.All experiments are conducted on an NVIDIA GeForce RTX 3090 GPU with 24,564 MiB of memory, utilizing CUDA version 12.3. Hyperparameters used to produce the results in Table 2 are reported in Tables 6 . The timings presented in all tables are obtained by executing each model independently for each dataset and averaging the resulting times across 100 epochs. To prevent excessive growth in signature terms, we use the univariate signature in LOB datasets. As an alternative, one could employ randomized signatures [21] or low-rank approximations [11, 13] .This section collects the validation of Step and Depth for the Neural-RDE model. Optimal values are selected for evaluation on test-set. Early-stopping is used with the same criteria as [60] . Table 13 summarises the long temporal modeling datasets from the UEA time series classification archive [5] used in Section 4. In this section, we ablate the use of the multi-view signature transform over both global and local transformations of the input signal. The results for the sinusoidal datasets are shown in Figure 7 . In most cases, the use of both local and global components improves the performance of RFormer. This choice, however, can be seen as a hyperparameter and will be dataset-dependent. One of the main points of the paper is that the shorter representation of the time-series endowed by the signatures helps to significantly reduce the computational cost of the self-attention operation with minimal information loss (and with improved performance in many of the experiments). By equation ( 16), one sees that the first level of the signature of a linear function is the difference between its endpoints. Hence, using multi-view attention with signature level one operates on the increments of piecewise-linear interpolated data, which corresponds to naive downsampling. To test that higher levels of the signature provide improvements in performance, we compare the result of using the signature on the datasets tested in Table 14 below. There is a significant performance gain in considering higher levels of the signature because one can capture the higher-order interactions between the different time-series.Furthermore, we conduct a new set of experiments in which we dropped 30% and 70% of the dataset for RFormer. Note that even with a 70% drop rate in the EigenWorms dataset, the vanilla Transformer fails to run due to memory limitations. Therefore, to provide results for the Transformer model on the EigenWorms dataset, we conduct experiments with an 85% drop rate. This comparison highlights the performance gap between the vanilla Transformer and our proposed model under these conditions, with the RFormer model yielding superior results. All results are computed across five seeds and are summarized in the tables and figure below. We conduct additional experiments to compare the runtime of Rough Transformers with other models.In this experiment, we use the synthetic sinusoidal dataset considered in our paper and compute the runtime per epoch for varying sequence lengths. We demonstrate results for two variants of RFormer: "online", which corresponds to computing the signatures of each batch during training (resulting in significant redundant computation), and "offline", which corresponds to computing the signatures in one go at the beginning of training. We include a recent RNN-based model as a basis for comparison with high-performing RNN baselines. In addition to the models discussed in Section 4, we introduce Continuous Recurrent Units (CRU) [82] as a new baseline. See Table 21 for a summary of the results. Thus, the time needed to compute the signature is inconsequential when compared with the time required to train standard models on the full or even downsampled datasets, since this step has to be carried out only once. To put this into context with an example, we note that it takes 4s to compute the signature representations for the HR dataset (which is about half the time it takes for the Vanilla Transformer to go through one epoch) and results in a 26× increase in computational speed for RFormer when compared to the vanilla Transformer. To showcase that this is the case for not only sequences of moderate length but also extremely long sequences, we also carry out the following experiment where we compute the signature representation for the sine dataset, with a progressively increasing number of datapoints. As seen in Table 23 , this does not cause an explosion in computational time. Also, to provide some context of the performance of ContiFormer compared with our method (and not only results on complexity and training times), we run the model on the sinusoidal classification task for signals of length L = 100 and L = 250. Due to the slow running time of the ContiFormer model, we did not consider sequence lengths of L > 250. We evaluate the ContiFormer model using one head. However, given the subpar results we obtain, we also test it with four heads, using the hyperparameters originally used in the paper for their irregularly sampled time series classification experiments. By contrast, all variations of RFormer tested in this paper for this experiment employ only one head, but reported significantly better results. Time-to-cancellation of limit orders. We use limit order book data provided by LOBSTER [43] and test each model's capacity to calculate the time-to-cancellation of 1000 limit orders of the AMZN ticker over a single trading day. We employ context windows of 1000 time-steps and 20000 time-steps in the past to make the prediction. Raw limit order book data consist of many uninformative samples [4] , so one expects a longer context window would provide more information to make the prediction. However, this cannot be guaranteed given the non-stationarity and low SNR of financial environments [see 3, 4, 81] . Limit order book updates are not regularly sampled, even though they are obtained at a high sampling rate. Since this is a dataset with a higher number of features, we employ the univariate multi-view signature transform to avoid excessive growth with signature level. Table 25 shows that a longer context yields improved performance in this task, highlighting the importance of efficiently processing long time-series in practical applications. RNN-based models (GRU, ODE-RNN) achieve decent performance with both medium and long context windows, despite their problems with vanishing and exploding gradients. In this task, Neural-RDE (see Appendix D for hyperparameters validation details) performs poorly. This is likely because Neural-RDE computes the signature over all channels, which is ineffective in this context. The Transformer and RFormer models report the best performance. Also, the computational time and the performance of the model improve with univariate signatures in this context as well.Finally, to assess the model's ability to understand the dynamics of the time series, we have carried out an additional experiment of forecasting the next-step intraday volatility of an Apple Limit Order Book. The results are shown in Table 26 . Justification: Yes, they are an accurate reflection of the paper's contributions and scope.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Any continuous-time interpolation of X can be used, e.g., splines. However, the signature computation of piecewise-linear paths is particularly fast; see Appendix A.We only benchmark Neural-CDE models in settings where time series are of relatively short length, due to the computational demands of this model for longer sequences.We note that baseline results for this task were taken from[91].We use our own reproduction to test the performance of all models in irregularly sampled datasets. Random dropping requires window sizes larger than 2 because signatures cannot be computed over a single point. The best step size was chosen in accordance with performance on the validation dataset, see Appendix D.For a preliminary version of this paper, we also direct the reader to[59]. 


Decision: This paper proposes a Transformer-based architecture for continuous-time signals, using multi-view attention on path signatures (from rough path theory) to capture both local and global dependencies. The authors effectively addressed the reviewers' concerns during the author-reviewer discussion phase. After the discussion, the majority of reviewers support the acceptance of this paper. AC agrees with the reviewers' positive opinions: this paper presents a novel approach by incorporating rough path theory into time-series representation learning, provides theoretical support for robustness, and empirically demonstrates the effectiveness of the proposed architecture. Although concerns from one negative reviewer remain, AC thinks the author rebuttal well addressed the concerns. For example, the authors added new comparisons with recent baselines as the reviewer suggested.

Therefore, AC recommends acceptance and highly encourages the authors to incorporate all experimental results (e.g., new datasets, new baseline models) and analyses from the author-reviewer discussion into the final manuscript, as they would further strengthen the contributions of this paper.

Reviews: review: paper_summary: paper_summary: The paper proposes Rough Transformer (RFormer), an extention of the Transformer architecture towards operating on continuous-time representations of time-series data. RFormer employs a novel technique called multi-view signature attention, which performs attention on path signatures pre-computed from input data offline, thereby capturing both local and global dependencies across observations. Experiments on various real-world time-series datasets shows that RFormer enjoys superior predictive performance as well as computational efficiency compared to previous methods, while being robust to changes in sequence length and irregular sampling.
strength_weakness: strength_weakness: - [W1] **Questionable motivation of synthetic frequency classification experiments.** The first experimental section tests RFormer on two synthetic datasets with which the task is to classify input time-series based on their ground-truth frequencies. While L233-234 mentions the second setup in particular is designed towards testing the long-range reasoning capabilities of RFormer, but it is unclear whether this indeed the case. For the second synthetic dataset, in particular, how can identifying the frequency be a proxy for long-range reasoning when the frequency $\omega_0$ is used for $t < t_0$ only? The results on Figure 2 showing that methods that are "tailor-made for long-range time series modeling" (L254) such as Neural-CDE and Neural-RDE underperforming significantly also indicates that the designed task is not really representative of long-range reasoning.
\
\
More interesting questions to ask could be: What makes RFormer sample-efficient vs. vanilla Transformer particularly on the Sinusoidal dataset and not so much on the Long Sinusoidal dataset? What makes RFormer more robust to changes in sampling frequency compared to Neural-CDE and Neural-RDE? Table 1 of [A] shows Neural-CDE is also quite robust to dropped data, but is this characteristic not emergent for Sinusoidal and Long Sinusoidal datasets?

- [W2] **Missing analysis on interpolation methods.** By default, RFormer uses piecewise-linear paths for computing path signatures, but as mentioned in L140, it seems any continuous-time interpolation can be used. As such, it would be interesting to discuss (1) whether any other interpolation techniques can be deployed efficiently similarly to piecewise-linear paths and (2) if they lead to any boosts in predictive accuracy, but these discussions are missing in the current draft (i.e., is the currently used piecewise-linear interpolation "pareto-optimal" under performance-efficiency trade-offs?).

[A] Kidger et al., Neural Controlled Differential Equations for Irregular Time Series. NeurIPS 2020.
questions: questions: - [Q1] **Large overlaps in representation space.** Basec on the multi-view signature formulation in Figure 1 and Equation 8, it seems the earlier observations would be covered by a large number of input tokens. As shown in the right plots in Figure 3, this would result in large "representational overlap" similarly to the oversmoothing phenomenon in graph representation learning [B]. Considering this, could restricting the global view to a few previous points rather than all previous points be a viable option? or does the theoretical and empirical robustness of RFormer to variable lengths and irregular sampling require that all previous points be covered in the global view?

- [Q2] **Discussion on input length of MLP and Transformer.** L108 states that the input length $L$ of the MLP and Transformer is fixed by assumption, but is this true? Sequences with different lengths can be processed in a single batch via padding.

[B] Rusch et al., A Survey on Oversmoothing in Graph Neural Networks. arXiv 2023.
limitations: limitations: The authors have adequately addressed the limitations in Appendix B.
paper_summary: paper_summary: The paper proposes Rough Transformers, an attention-based model for long continuous-time signals. The model utilizes ideas from rough path theory to extract path signatures from the continuous-time signal (obtained by interpolation of the original signal). Two types of signatures are extracted: global and local. The global signature extracts long-term information from the signal whereas the local signature extracts local information. Self-attention is then used on this "multi-view" signature. Experiments on classification and regression tasks show improved performance over existing model architectures.
strength_weakness: strength_weakness: - The technical contribution is limited. In such scenarios, the empirical analysis needs to be sufficiently strong.
- The empirical analysis has been conducted on a few toyish datasets. While the results are definitely promising, more experimental support is needed to validate the model.
    - Although the model is motivated from a continuous-time and irregularly-sampled data perspective, the actual investigation of these settings is limited. As per my understanding, all experiments under 4.1 have been conducted on regularly sampled time series (please correct me, if I am wrong). I find it surprising that simple RNN-based methods do not perform well in these settings. If this is indeed the case, some simple CNN-based method should be studied. Recent models based on state space layers (e.g., S4, Mamba) can also be explored as baselines.
   - OOM for important baselines is not really helpful to draw any conclusions. To highlight efficiency of RFormer, please conducted experiments where you increased the context length or other parameters to show where baselines run OOM and how do they perform before that.
   - More experiments are needed, particularly for the forecasting task to understand how well the model understands the dynamics. 
   - I took a brief look at the code and it looks like hyperparameter tuning was conducted for RFormer. Was such tuning also performed for the baselines? If no (which is a valid response), how did you selected the baseline parameters? How sensitive is the model to different hyperparameters? 
- The discussion on related work needs to be moved to the main text and improved. Please contrast RFormer with the related works, particularly the ones that are closely related such as NRDE and ContiFormer. Discussion on some closely related works [1, 2, 3] is missing. Ideally there should also be a comparison with at least of these methods (e.g., CRU).

[1] Schirmer, Mona, et al. "Modeling irregular time series with continuous recurrent units." International conference on machine learning. PMLR, 2022.   
[2] Ansari, Abdul Fatir, et al. "Neural continuous-discrete state space models for irregularly-sampled time series." International Conference on Machine Learning. PMLR, 2023.   
[3] Oh, YongKyung, Dongyoung Lim, and Sungil Kim. "Stable Neural Stochastic Differential Equations in Analyzing Irregular Time Series Data." arXiv preprint arXiv:2402.14989 (2024).    

I am happy to update my score, if my concerns are adequately addressed.
questions: questions: - Can the authors clarify by what they mean by "input sequences must be sampled at the same times, (ii) the sequence length must be fixed"? These do not seem to be limitations of transformers, especially (ii). 

See weaknesses for other questions.
limitations: limitations: There is a brief discussion on limitations.
paper_summary: paper_summary: This paper introduces the Rough Transformer, a variant of the original Transformer that allows the processing of discrete-time series as continuous-time signals through the use of multi-view signature attention. Empirical comparisons shows that Rough Transformers outperform vanilla Transformers and continuous-time models on a variety of time-series tasks and are robust to the sampling rate of the signal.
strength_weakness: strength_weakness: - Empirical evaluations are limited, casting doubts on the true potential of the proposed architecture
- The tasks considered are rather simple, and it is not clear whether the proposed architecture will give favorable tradeoffs between accuracy and efficiency in the more challenging tasks (see below) 
- Missing evaluations on time series forecasting tasks (only classification and regression tasks are considered)
- Missing comparisons with recent RNN models (such as https://arxiv.org/abs/2110.04744, https://arxiv.org/abs/2212.00228), Transformer models (e.g., those studied in https://arxiv.org/abs/2011.04006), State Space models (https://arxiv.org/abs/2111.00396 and the more recent variants such as Mamba: https://arxiv.org/abs/2312.00752) and other sequence models (https://arxiv.org/abs/2305.01638, https://arxiv.org/abs/2209.10655) 
- Missing ablation studies on the components of the proposed architecture, particularly the role of the global and local components in the multi-view signature, truncation level n, etc.
- Missing related work; e.g., the papers mentioned above, https://link.springer.com/article/10.1007/s40304-017-0103-z and https://arxiv.org/abs/1710.10121 for continuous-time DL models, https://arxiv.org/abs/2006.12070, https://arxiv.org/abs/2102.04877 for continuous-time RNN models, https://dl.acm.org/doi/abs/10.5555/3546258.3546305 for using path signatures to understand continuous-time RNNs
questions: questions: - I was wondering how effective is the proposed model in autoregressive generative tasks and time series forecasting (see https://arxiv.org/abs/2311.04147)? These are natural tasks in the domain of sequence modeling 
- While the proposed method can improve the modified transformer model in sequence modeling, why would the method be attractive for practitioners when they could just use more sophisticated models like SSMs for sequence modeling?
limitations: limitations: Yes

Comments: title: Thank you for raising the overall and confidence scores
Reply to Reviewer
comment: Thank you for your response and your very helpful comments, which have greatly improved our paper. We are glad that we could address your concerns and appreciate your decision to raise both the overall and confidence scores. Thank you again!
Thank you authors for your time and commitment in preparing the rebuttal. All of my concerns have been addressed, and thus I increase my rating to 7.
Thank you again for your thorough and constructive review, recommending acceptance, and raising the score!
Thank you for your thorough responses. Many of my concerns have been addressed, so I am raising my score to 6. Looking forward to the final version of the paper. :)