Year: 2022

Abstract:
The problem of processing very long time-series data (e.g., a length of more than 10,000) is a long-standing research problem in machine learning. Recently, one breakthrough, called neural rough differential equations (NRDEs), has been proposed and has shown that it is able to process such data. Their main concept is to use the log-signature transform, which is known to be more efficient than the Fourier transform for irregular long time-series, to convert a very long timeseries sample into a relatively shorter series of feature vectors. However, the logsignature transform causes non-trivial spatial overheads. To this end, we present the method of LOweR-Dimensional embedding of log-signature (LORD), where we define an NRDE-based autoencoder to implant the higher-depth log-signature knowledge into the lower-depth log-signature. We show that the encoder successfully combines the higher-depth and the lower-depth log-signature knowledge, which greatly stabilizes the training process and increases the model accuracy. In our experiments with benchmark datasets, the improvement ratio by our method is up to 75% in terms of various classification and forecasting evaluation metrics.

1 INTRODUCTION
Time-series data occurs frequently in real-world applications, e.g., stock price forecasting (Ariyo et al., 2014; Siami-Namini et al., 2018; Jhin et al., 2021), traffic forecasting (Reinsel, 2003; Fu, 2011; Bai et al., 2020; Fang et al., 2021), weather forecasting (Shi et al., 2015; Seo et al., 2018; Brouwer et al., 2019; Ren et al., 2021), and so on. However, it is known that very long timeseries data (e.g., a time-series length of more than 10,000) is not straightforward to process with deep learning despite various techniques ranging from recurrent neural networks (RNNs) to neural ordinary/controlled differential equations (NODEs and NCDEs). RNNs are known to be unstable when training with such very long sequences and the maximum length that can be processed by NODEs and NCDEs is more or less the same as that by RNNs (Morrill et al., 2021; Aicher et al., 2020; Trinh et al., 2018; Stoller et al., 2019; Bai et al., 2018). However, one breakthrough has been recently proposed, namely neural rough differential equations (NRDEs).
NRDEs are based on the rough path theory which was established to make sense of the controlled differential equation:
dz(t) = f(z(t))dX(t), (1)
whereX is a continuous control path, and z(t) is a hidden vector at time t. A prevalent example ofX is a (semimartingale) Wiener process, in which case the equation reduces to a stochastic differential equation. In this sense, the rough path theory extends stochastic differential equations beyond the semimartingale environments (Lyons et al., 2004).
One key concept in the rough path theory is the log-signature of a path. It had been proved that the log-signature of a path with bounded variations is unique under mild conditions (Lyons & Xu,
2018; Geng, 2017) and most time-series data that happens in the field of deep learning has bounded variations. Therefore, one can interpret that the log-signature is a unique feature of the path.
Given a N -length time-series sample {xi}Ni=0 annotated with its observation time-points {ti}Ni=0, where t0 = 0, tN = T , and ti < ti+1, NRDEs construct a continuous path X(t), where t ∈ [0, T ], with an interpolation algorithm, where X(ti) = (xi, ti) for ti ∈ {ti}Ni=0. In other words, the path has the same value as the observation (xi, ti), when ti is one of the observation timepoints and otherwise, interpolated values. As shown in Fig. 1, a log-signature (each dotted yellow box in the figure) is calculated every P -length sub-path, and another time-series of log-signatures, denoted {LogSigDri,ri+1(X)} ⌊ TP ⌋−1 i=0 , is created. The log-signature calculation has one important hyperparameter D called depth — the higher the depth is, the more accurately represented each sub-path is (cf. Eq. 4). For instance, the best accuracy score is 0.81 for D = 3 vs. 0.78 for D = 2 in EigenWorms. The sub-path length P and the depth D decides the number and the dimensionality of log-signatures, respectively.
However, one downside is dim(LogSigDri,ri+1(X)) > dim(X), where dim(X) means the dimensionality (or the number of channels) of X . As a matter of fact, dim(LogSigDri,ri+1(X)) grows rapidly w.r.t. dim(X) (Morrill et al., 2021). Since the dimensionality of the input data is, given a dataset, fixed, we need to decrease D to reduce overheads. In general, NRDEs require more parameters to process higher-dimensional log-signatures (as shown in Table 4 where the original NRDE design always requires more parameters for D = 3 in comparison with D = 2.)
To this end, we propose to embed the higher-depth signatures onto a lower-dimensional space to i) decrease the complexity of the main NRDE (the solid blue box in Fig. 1), and ii) increase the easiness of training (cf. Fig. 2) and as a result, its model accuracy as well. Fig. 1 shows two possible approaches where we directly reduce the dimensionality of the higher-depth log-signature in Fig. 1 (a) or we adopt the autoencoder architecture to combine both the higherdepth and the lower-depth signatures in Fig. 1 (b) — the first approach in Fig. 1 (a) is a baseline model, and our proposed model is the second approach in Fig. 1 (b). Autoencoders are frequently used for (unsupervised) dimensionality reduction although there also exist other approaches. Since the higher-depth log-signature should be reconstructed from the embedded vector, one can consider that
the higher-depth log-signature is indirectly embedded into the vector. Moreover, the decoder is discarded after its pre-training so our method does not involve any high-cost computation with the higher-depth log-signature in the main training and inference steps as clarified in Table 1.
Our proposed method, LOweR-Dimensional embedding of log-signature (LORD), adopts an NRDE-based autoencoder to combine the higherdepth and the lower-depth log-signature knowledge, and utilizes the embedded knowledge by the encoder for a downstream machine learning task as shown in Fig. 1 (b). The encoder embeds the lower-depth log-signature from the continuous
path X , and the decoder reconstructs the higher-depth log-signature. This specific design is where our key idea lies in. The encoded lower-depth log-signature will contain both the lower-depth and the higher-depth log-signature knowledge because i) it is an encoding of the lower-depth log-signature and ii) the decoder should recover the higher-depth log-signature from it. After pre-training the autoencoder, we discard the decoder to exclude the higher-dimensional processing as early as possible from our framework. After that, we fix the encoder and train only the main NRDE.
We conduct experiments with six very long time-series benchmark datasets and five baselines including NODE, NCDE, and NRDE-based models. RNN-based models cannot process the very long time-series datasets and we exclude them. Our proposed method significantly outperforms all existing baselines. Our contributions can be summarized as follows:
1. We design an NRDE-based continuous autoencoder to combine the higher-depth and the lower-depth log-signature information. Since the decoder is discarded after being trained, we adopt the asymmetric architecture that the encoder is lightweight and the decoder is heavyweight in terms of the number of parameters.
2. Our proposed method outperforms all baselines by large margins (up to 75% for various evaluation metrics, e.g.,R2) in standard benchmark datasets with much smaller model sizes in comparison with the original NRDE design.
2 RELATED WORK AND PRELIMINARIES
NODEs NODEs (Chen et al., 2018) use the following equation:
z(T ) = z(0) + ∫ T 0 f(z(t), t; θf )dt, (2)
which means z(T ) is solely defined by the initial value z(0) — the entire evolving path from z(0) to z(T ) is defined by the initial value in ODEs. NODEs are not considered as a continuous analogue to RNNs but to residual networks (Chen et al., 2018).
NCDEs Let {xi}Ni=0 be an N -length time-series sample and {ti}Ni=0 be its observation timepoints. NCDEs are different from NODEs in that they use the following equation:
z(T ) = z(0) + ∫ T 0 f(z(t); θf )dX(t) = z(0) + ∫ T 0 f(z(t); θf ) dX(t) dt dt, (3)
where X(t) is a continuous interpolated path from {(xi, ti)}Ni=0, where tN = T . In this regards, NCDEs can also be considered as a continuous analogue to RNNs. The adjoint sensitivity method can be used to decrease the space complexity of gradient calculation instead of backpropagation (Kidger et al., 2020).
However, NCDEs has an inherent disadvantage that they are weak at processing very long timeseries samples, because NCDEs directly process the very long time-series samples.
Signature Transform Let [ri, ri+1] be a time duration, where ri < ri+1. The signature of the time-series sample is defined as follows:
Si1,...,ikri,ri+1 (X) =
∫ · · · ∫ ri<t1<...<tk<ri+1 k∏ j=1 dXij dt (tj)dtj ,
SigDri,ri+1(X) = ( {Siri,ri+1(X)} dim(X) i=1 , {S i,j ri,ri+1(X)} dim(X) i,j=1 , . . . , {S i1,...,iD ri,ri+1 (X)} dim(X) i1,...,iD=1 ) ,
(4)
However, the signature has redundancy, e.g., S1,2a,b (X) + S 2,1 a,b (X) = S 1 a,b(X)S 2 a,b(X) where we can know a value if knowing three others. The log-signature LogSigDri,ri+1(X) is then created after dropping the redundancy from the signature. In this work, we use this log-signature definition in default to design our method. By creating a log-signature for every P -length sub-path, the entire sequence length can be reduced — moreover, the log-signature is a unique feature of the subpath (Lyons & Xu, 2018; Geng, 2017). For complexity reasons, we typically use the log-signature of depth D ≤ 3 (Morrill et al., 2021).
NRDEs Owing to the log-signature transform of time-series, NRDEs (Morrill et al., 2021) are defined as follows:
g(X, t) = LogSigDri,ri+1(X)
ri+1 − ri for t ∈ [ri, ri+1),
z(T ) = z(0) + ∫ T 0 f(z(t); θf )g(X, t)dt,
(5)
where LogSigDri,ri+1(X) means the log-signature created from the path X within the interval [ri, ri+1). LogSigDri,ri+1 (X)
ri+1−ri is a piecewise approximation of the time-derivative of the log-signature in the short interval [ri, ri+1). D means the depth of the log-signature. Once we define the subpath length P , the intervals {[ri, ri+1)} ⌊ TP ⌋−1 i=0 are decided, i.e., P = ri+1 − ri for all i. Then, the
time-series of LogSigDri,ri+1(X) constitutes {LogSig D ri,ri+1(X)} ⌊ TP ⌋−1 i=0 in our notation.
NRDEs use Eq. 5 to derive z(T ) from z(0), which can be considered as a continuous analogue to RNNs since it continuously reads the time-derivative of the log-signature. Therefore, z(T ) is defined by the initial value z(0) and the sequence of the time-derivative of the log-signature. We can also use the adjoint sensitivity method to train NRDEs.
Long Sequence Time-series Input (LSTI) The problem of processing very long time-series data is a long standing research problem. There are three ways to solve the LSTI problem. First, it reduces the sequence through truncating/summarizing/sampling from a very long input sequence. However, this method may lose information affecting the prediction accuracy. The second is to give the gradient transformation. As the sequence becomes longer, the gradient vanishing problem occurs. Therefore, in (Aicher et al., 2020), the model is trained using only the gradient of the last step. It also solves the problem using auxiliary losses (Trinh et al., 2018). Finally, CNNs (Stoller et al., 2019; Bai et al., 2018) were used to solve the LSTI problem. The convolutional filter captures long term dependencies, but the receptive fields increase exponentially, breaking the sequence.
3 PROPOSED METHOD
We describe our proposed method, LORD-NRDE. We first clarify the motivation of our work and then describe our proposed model design.
Motivation It is known that NRDEs are a generalization of NCDEs (Morrill et al., 2021, Section 3.2). However, one problem in utilizing NRDEs in real-world environments is that dim(LogSigDri,ri+1(X)) is a rapidly growing function of dim(X) (Morrill et al., 2021, Section A).
Algorithm 1: How to train LORD-NRDE Input: Training data Dtrain, Validating data Dval, Maximum
iteration numbers max iterAE and max iterTASK 1 Initialize the parameters of the encoder (i.e., θf , θϕe ), the
decoder (i.e., θo, θϕs ), and the main NRDE (i.e., θg , θϕz , θϕoutput ).; /* Pre-training of the autoencoder */ 2 for max iterAE iterations do 3 Train the encoder and the decoder using LAE ; /* Main-training of LORD-NRDE */ 4 for max iterTASK iterations do 5 Train the main NRDE using LTASK ; 6 Validate the best main NRDE parameters with Dval; 7 return the encoder and the main NRDE parameters;
This rapid blow-up of dimensionality causes two problems : i) it hinders us from applying NRDEs to high-dimensional time-series data, and ii) it makes the training process complicated since the hidden representation of the large input should be made for a downstream task. Therefore, we propose to embed the log-signature onto a lower dimensional space so that the training process becomes more tractable and enhance the applicability of NRDEs to high-dimensional data. After the embedding, in other words, the dimensionality of embedded vector is typically smaller than that of the log-signature with depth D2 in our setting but it has knowledge of the log-signature with depth D2 > D1.
Overall Architecture We describe our proposed method, LORD-NRDE, which consists of three NRDEs: i) an encoder NRDE, ii) a decoder NRDE and iii) a main NRDE to derive z(T ) from z(0). The role of each NRDE is as follows:
1. The encoder NRDE continuously embeds the log-signature with depth D1 onto another space. We use e(t) to denote the vector embedded from the log-signature at time t.
2. The decoder NRDE reconstructs the log-signature with depth D2 > D1 from e(t). 3. The main NRDE reads e(t) to evolve z(t) and derive z(T ). There is an output layer which
reads z(T ) and makes inference.
We note that dim(LogSigD2ri,ri+1(X)) > dim(e(t)), where t ∈ [0, T ]. We require that the encoder produces embedding vectors that contain both the log-signature knowledge with D1 and D2 depth.
LORD-NRDE To this end, we propose the following method:
z(T ) = z(0)+ ∫ T 0 g(z(t); θg) de(t) dt dt, (6)
e(T ) = e(0)+ ∫ T 0 f(e(t); θf ) LogSigD1ri,ri+1(X) ri+1 − ri dt, for t ∈ [ri, ri+1), (7)
s(T ) = s(0)+ ∫ T 0 o(s(t); θo) de(t) dt dt, (8)
where dim(e(t)) < dim(s(t)) = dim(LogSigD2ri,ri+1(X)), where t ∈ [ri, ri+1), with D2 > D1.
The term de(t) can be considered as an embedding of the log-signature with depth D1 at time t, and e(t) as an embedding of X(t) constructed by the log-signature with depth D1. Similarly, ds(t) is a reconstructed log-signature with depth D2 from the embedding, and s(t) as X(t) reconstructed by the log-signature with depth D2. Since e(0) and s(0) mean the initial values, we set ϕe(X(0); θϕe), ϕs(e(0); θϕs), where ϕ means a transformation function.
Therefore, e(t) and s(t) constitute an autoencoder based on NDREs as shown in Fig. 3. In our design, we use the asymmetrical setting for them, where the encoder RDE is lightweight and the decoder RDE is heavyweight in terms of the number of parameters, considering the applicability of
our design to real-world environments. Since the decoder NRDE is discarded after its pre-training, the heavyweight setting causes no problems afterwards — our method requires more resources for the pre-training process though.
There are three functions g, f , and o in Eqs. 6 to 8 and their definitions are as follows:
g(z(t); θg) = FCoutg,Ng+1(ψ(FChg,Ng (...η(FChg,1(z(t))...))), (9)
f(e(t); θf ) = FCoutf ,Nf+1(ψ(FChf ,Nf (...η(FChf ,1(e(t))...))), (10)
o(s(t); θo) = FCouto,No+1(ψ(FCho,No(...η(FCho,1(s(t))...))), (11)
where FCh is a fully-connected (FC) layer whose output size is h. Ng, Nf , No means the number of FC layers. ψ and η are the hyperbolic tangent and rectified linear unit activations, respectively.
Training Method To train the NRDE-based autoencoder, we use the following loss:
Lrecon =
∑M i=0 ∥(s(ri+1)− s(ri))− LogSigD2ri,ri+1(X)∥ 2 2
M , (12)
LAE = Lrecon + cAE(∥θf∥22 + ∥θo∥22 + ∥θϕe∥22 + ∥θϕs∥22) + ce∥e∥22, (13)
where M = ⌊TP ⌋ − 1, and cAE is a coefficient of the L2 regularization terms. ce also regularizes the scale of the learned embedding. In addition to LAE , we also have a task loss LTASK which is the standard cross-entropy (CE) loss or the mean squared loss (MSE) loss. From z(T ), we have an output layer, which is typically a fully-connected layer with an appropriate final activation function such as softmax, to produce a prediction ŷ. θϕoutput denotes the parameters of the output layer. Using the ground-truth information y, we define the task loss LTASK . Therefore, the final loss can be written as follows:
LTASK = CE(y, ŷ) + cTASK(∥θg∥22 + ∥θϕz∥22 + ∥θϕoutput∥22), (14)
where cTASK is a coefficient of the L2 regularization terms, and CE(y, ŷ) means a cross-entropy loss — we assume classification but it can be accordingly changed for other tasks.
To implement, we define the following augmented ODE:
d
dt [ z(t) e(t) s(t) ] =  g(z(t); θg)f(e(t); θf ) LogSigD1ri,ri+1 (X) ri+1−ri f(e(t); θf ) LogSigD1ri,ri+1 (X) ri+1−ri
o(s(t); θo)f(e(t); θf ) LogSigD1ri,ri+1 (X)
ri+1−ri
, for t ∈ [ri, ri+1), (15)
where we replace de(t)dt with f(e(t); θf ) LogSigD1ri,ri+1 (X)
ri+1−ri (as defined in Eq. 7), and the initial values are defined as follows: [
z(0) e(0) s(0)
] = [ ϕz(X(0)); θϕz) ϕe(X(0)); θϕe) ϕs(e(0); θϕs) ] , (16)
where ϕz and ϕe are transformation functions to produce the initial values from the initial observation X(0), and ϕs is a transformation function to produce the initial reconstructed value from the initial embedded vector e(0). We use a fully-connected layer for each of them.
Alg. 1 shows the training algorithm. In Line 3, we pre-train the autoencoder for max iterAE iterations. After discarding the decoder and fixing the encoder, we then train the main NRDE in Line 5 for max iterTASK iterations. Using Dval, we validate and choose the best parameters.
4 EXPERIMENTAL EVALUATIONS
We describe our experimental environments and results. The mean and variance of 5 different seeds are reported for model evaluation. We refer to Appendix A for reproducibility.
4.1 EXPERIMENTAL ENVIRONMENTS
Datasets We use six real-word dataset which all contain very long time-series samples. There are 3 classification datasets in the University of East Anglia (UEA) repository (Tan & Webb): EigenWorms, CounterMovementJump, and SelfRegulationSCP2, and 3 forecasting datasets in Beth Israel Deaconess Medical Centre (BIDMC) which come from the TSR archive (Tan & Webb): BIDMCHR, BIDMCRR, BIDMCSpO2. We refer to Appendix B for detail of datasets.
Baselines There are three types of baselines in our Experiments, ranging from NODE to NCDE and NRDE-based baseline models. ODE-RNN is one of the state-of-the-art NODE models in processing time-series. Following the suggestion in (Morrill et al., 2021), we merge P observations into one merged observation and feed it into ODE-RNN, in which case ODE-RNN is able to process much longer time-series. The dimensionality of the merged observation is P times larger than that of the original observation or equivalently, the length of the merged time-series is P times shorter than that of the original one. NCDE is the original NCDE design in (Kidger et al., 2020). Attentive NCDE (ANCDE) is an extension of NCDE by adding an attention mechanism into it, which significantly outperforms NCDE (Jhin et al., 2021). For the NRDE-type baselines, we consider i) the original NRDE design (Morrill et al., 2021), and ii) the one in Fig. 1 (a) which directly embeds the higherdepth log-signature into a lower-dimensional space, each of which is called NRDE and DE-NRDE, respectively. For those NRDE-based models, we clarify its one important hyperparameter, depth, as part of its name, e.g., NRDE2 means NRDE with D = 2. One important point is that NRDE is a generalization of NCDE. Therefore, NRDE1 is theoretically identical to NCDE, if using the linear interpolation method, and for this reason, we set D > 1 for NRDE.
Hyperparameters When using NRDEs, there is one crucial hyperparameter, the log-signature depthD. We use two depth settings which were used in (Morrill et al., 2021). For our LORD-NRDE, there are two depths, D1 and D2, where D1 < D2. LORDD1→D2 means that LORD-NRDE’s decoder recovers theD2-depth log-signature from theD1-depth log-signature. The number of layers in the encoder, decoder and main NRDE, Ng , Nf , and No of Eqs. 9 to 11, are in {2, 3}. The hidden sizes, hg , hf , and ho of Eqs. 9 to 11, are in {32, 64, 128, 192}. The coefficients of theL2 regularizers in Eqs. 13 and 14 are in {1× 10−5, 1× 10−6}. The coefficient of the embedding regularizer, ce in Eq. 13 is in {0, 1, 10}. The max iteration numbers, max iterAE and max iterTASK in Alg. 1, are in {400, 500, 1000, 1500, 2000}. The learning rate of the pre-training and main-training is 1×10−3. We also set dim(e(t)) = dim(LogSigD1ri,ri+1(X)), where t ∈ [ri, ri+1).
We also conduct experiments by setting the sub-path length P to 4, 8, 32, 64, 128, 256, or 512 observations. In other words, we create one log-signature for every P input observation for NCDE and NRDE-based models. For ODE-RNN, as described earlier, we simply concatenate P observations into one observation. The final time T is large in our experiments, e.g., T > 10, 000, and the number of log-signatures is ⌊TP ⌋. We test both the adjoint sensitivity method and the backpropagation through the solver.
Evaluation Methods We reuse the very long time-series classification and forecasting evaluation methods of (Morrill et al., 2021) and extend the methods by adding more datasets and more evaluation metrics. We use accuracy, macro F1, and ROCAUC for binary classification; accuracy, macro/weighted F1, and ROCAUC for multi-class classification; and R2, explained variance, mean squared error (MSE), and mean absolute error (MAE) for forecasting — we list the complete results in Appendix D after introducing key results in the main manuscript. We also show the number of parameters for each model — for our LORD, we exclude the parameter numbers of the decoder since it is discarded after the pre-training. We train and test each model 5 times with different seeds. If the mean of score is the same, the smaller standard deviation is better.
4.2 EXPERIMENTAL RESULTS
Summary of Experimental Results Since our main result tables have many items, we quickly summarize their highlights in Tables 2 and 3. To calculate the improvement in evaluation metrics, we use the ratio of improvement over NRDE2 for each metric averaged over all the sub-path lengths and all datasets, e.g., we calculate LORD’s ROCAUC−NRDE2’s ROCAUCNRDE2’s ROCAUC and NRDE2’s MSE−LORD’s MSE NRDE2’s MSE
for all classification cases and average them. The ratio of the number of parameters is calculated similarly.
Table 3: Highlights in Forecasting
Method R2 MSE #Params
ODE-RNN 44% 41% -36% NCDE -45% -55% -30% ANCDE -34% -45% -52% NRDE2 0% 0% 0% NRDE3 0% -5% 82% DE-NRDE2 39% 35% -20% DE-NRDE3 62% 59% 31% LORD1→2 53% 47% -65% LORD1→3 51% 46% -59% LORD2→3 75% 72% -39%
Overall, LORD outperforms other baselines with smaller numbers of parameters. LORDD1→D2 has a larger model size, excluding its decoder, than NRDED1 whereas it has a much smaller model size than NRDED2 . However, the performance of LORDD1→D2 is similar to or better than that of NRDED2 , which proves the efficacy of our method. Using the encoder-decoder structure, the high complexity of processing log-signatures can be reduced and it makes our model smaller and more amenable to train.
In many cases for LORD, the adjoint method and the backpropagation method are comparable in terms of model accuracy. Interestingly, we found that the backpropagation through the Euler method is fast enough with a neglectable sacrifice of accuracy for several cases.
Detailed Experimental Results Table 4 show detailed results for some selected evaluation metrics — full tables with all metrics are in Appendix D. One point is that many best outcomes are made with moderate P settings. For EigenWorms, ODE-RNN can’t achieve good scores for all P settings. Our proposed model, LORD, achieves the best scores. In particular, LORD1→3 with P = 32 has a much smaller number of parameters, compared with other baselines that have similar performance. LORD2→3 also significantly outperforms NRDE3 for both accuracy and mode size. For CounterMovementJump and SelfRegulationSCP2, ODE-RNN is better than other NRDE and NCDE-based baselines. However, LORD marks the best scores in all cases.
In all BIDMC experiments, LORD shows outstanding performance. LORD2→3 shows the best performance in almost all cases. Compared with NCDE and NRDE, DE-NRDE has 40 ∼ 60% improvements and LORD has 50 ∼ 70% improvements. However, LORD’s model size is reduced by 60% whereas DE-NRDE’s model size is reduced by 20%. Therefore, our proposed method is a better embedding method for log-signatures. Unlike EigenWorms, the time-series length in this dataset is rather short. For that reason, ODE-RNN performs better. Overall, DE-NRDE3 with P = 128 has the best performance, among the baseline models. This means that the training difficulty by the large logsignature size can be alleviated by the direct embedding. However, LORD outperforms DE-NRDE.
Additional Results and Visualization Fig. 4 visually compares the three log-signatures and we note that the embedded signature of de(t) has the characteristics of both D1 = 1 and D2 = 3. We refer to our supplementary material for other visualization, detailed results, sensitivity analyses, and reproducibility information.
5 LIMITATIONS AND CONCLUSIONS
The log-signature transform of NRDEs is suitable to irregular and long time-series. However, the log-signature transform requires larger memory footprints and to this end, we presented
an autoencoder-based method to embed their higher-dimensional log-signature into a lowerdimensional space, called LORD. Our method is carefully devised to eradicate the higherdimensional computation as early as right after the pre-training of the autoencoder. In the standard benchmark experiments of very long time-series, our proposed method significantly outperforms existing methods and shows relatively smaller model sizes in terms of the number of parameters. Nonetheless, it is still in its early phase to enhance NRDEs and there exist a couple of items to be studied further, e.g., processing long time-series with many channels.
6 ETHICS STATEMENT
Time-series classification and forecasting is closely related to our daily life, ranging from stock price forecasting, weather forecasting to patient mortality classification and so on. Therefore, our enhanced technology has many benefits to our society. It is hard to say that deep neural networks processing time-series have ethical problems.
7 REPRODUCIBILITY STATEMENT
We include detailed reproducibility information in Appendix A. In particular, we clarify the neural network architectures we used and their best hyperparameter sets for all experiments in our paper. Our code is available in https://github.com/leejaehoon2016/LORD.
8 ACKNOWLEDGEMENTS
Noseong Park is the corresponding author. This work was supported by the Yonsei University Research Fund of 2021, and the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korean government (MSIT) (No. 2020-0-01361, Artificial Intelligence Graduate School Program (Yonsei University), and No. 2021-0-00155, Context and Activity Analysis-based Solution for Safe Childcare).
REFERENCES Christopher Aicher, Nicholas J Foti, and Emily B Fox. Adaptively truncating backpropagation
through time to control gradient bias. In Uncertainty in Artificial Intelligence, pp. 799–808. PMLR, 2020.
Adebiyi A. Ariyo, Adewumi O. Adewumi, and Charles K. Ayo. Stock price prediction using the arima model. 2014.
Lei Bai, Lina Yao, Can Li, Xianzhi Wang, and Can Wang. Adaptive graph convolutional recurrent network for traffic forecasting. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), NeurIPS, volume 33, pp. 17804–17815, 2020.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Convolutional sequence modeling revisited. 2018.
Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. Gru-ode-bayes: Continuous modeling of sporadically-observed time series. In NeurIPS, 2019.
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In NeurIPS. 2018.
Zheng Fang, Qingqing Long, Guojie Song, and Kunqing Xie. Spatial-temporal graph ode networks for traffic flow forecasting. arXiv preprint arXiv:2106.12931, 2021.
Tak-chung Fu. A review on time series data mining. Engineering Applications of Artificial Intelligence, 24(1):164–181, 2011.
Xi Geng. Reconstruction for the signature of a rough path. Proceedings of the London Mathematical Society, 114(3):495–526, 2017.
Sheo Yon Jhin, Heejoo Shin, Seoyoung Hong, Minju Jo, Solhee Park, and Noseong Park. Attentive neural controlled differential equations for time-series classification and forecasting. In ICDM, 2021.
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. In NeurIPS, 2020.
Terry Lyons and Weijun Xu. Inverting the signature of a path, 2018.
Terry Lyons, M. Caruana, and T. Lévy. Differential Equations Driven by Rough Paths. Springer, 2004. École D’Eté de Probabilités de Saint-Flour XXXIV - 2004.
James Morrill, Cristopher Salvi, Patrick Kidger, James Foster, and Terry Lyons. Neural rough differential equations for long time series. 2021.
Gregory C Reinsel. Elements of multivariate time series analysis. Springer Science & Business Media, 2003.
Xiaoli Ren, Xiaoyong Li, Kaijun Ren, Junqiang Song, Zichen Xu, Kefeng Deng, and Xiang Wang. Deep learning-based weather prediction: A survey. Big Data Research, 23, 2021.
Sungyong Seo, Arash Mohegh, George Ban-Weiss, and Yan Liu. Automatically inferring data quality for spatiotemporal forecasting. In International Conference on Learning Representations, 2018.
Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional lstm network: A machine learning approach for precipitation nowcasting. NeurIPS, 28, 2015.
Sima Siami-Namini, Neda Tavakoli, and Akbar Siami Namin. A comparison of arima and lstm in forecasting time series. 2018.
Daniel Stoller, Mi Tian, Sebastian Ewert, and Simon Dixon. Seq-u-net: A one-dimensional causal u-net for efficient sequence modelling. arXiv preprint arXiv:1911.06393, 2019.
Bagnall A. Bergmeir C. Keogh E. Petitjean F. Tan, C. W. and G. I. Monash Webb. Uea classification/regression datasets. http://www.timeseriesregression.org/, http://www. timeseriesclassification.com/.
Trieu Trinh, Andrew Dai, Thang Luong, and Quoc Le. Learning longer-term dependencies in rnns with auxiliary losses. In International Conference on Machine Learning, pp. 4965–4974. PMLR, 2018.
Table 7: The best hyperparameter in ANCDE
Data P HiddenPath Size #Hidden Layers Attention Channel Size
EigenWorms 4 128 3 64 32 128 3 64
128 64 2 32
CounterMovementJump 64 128 2 128 128 256 3 128 512 64 2 256
SelfRegulationSCP2
32 256 3 128 64 64 2 128 256 128 2 128
BIDMC32HR 8 128 2 64
128 128 2 64 512 128 2 64
BIDMC32RR 8 128 2 64
128 128 2 64 512 128 3 64
BIDMC32SpO2 8 128 2 64
128 128 2 64 512 128 2 64
Table 8: The best hyperparameter in DE-NRDE
Data P Compression Ratio #Hidden Layers Hidden Size
D = 2 3 2 3 2 3
EigenWorms 4 0.5 0.5 1 1 128 128
32 0.5 0.5 1 1 128 128 128 0.7 0.7 1 1 64 128
CounterMovementJump 64 0.3 0.3 1 2 128 128 128 0.3 0.5 2 2 128 64 512 0.7 0.3 1 1 64 128
SelfRegulationSCP2 32 0.3 0.3 1 2 64 128 64 0.3 0.3 2 2 64 64 256 0.3 0.3 2 2 64 128
BIDMC32HR 8 0.7 0.7 2 2 64 128
128 0.7 0.7 1 1 128 128 512 0.7 0.7 1 1 64 128
BIDMC32RR 8 0.5 0.5 1 1 128 128
128 0.7 0.7 1 1 128 128 512 0.7 0.7 1 1 64 64
BIDMC32SpO2 8 0.5 0.5 1 1 128 128
128 0.7 0.7 1 1 64 64 512 0.7 0.7 1 1 64 64
A BEST HYPERPARAMETERS
Our software and hardware environments are as follows: UBUNTU 18.04 LTS, PYTHON 3.7.10, PYTORCH 1.8.1, CUDA 11.4, and NVIDIA Driver 470.42.01, i9 CPU, and NVIDIA RTX A6000.
A.1 BASELINE
There are 5 baselines in our experiments. Each model has its own hyerparameters. ODE-RNN method has a hyperparameter of hidden path size. NCDE and NRDE have also the same hyperparameter, and in addition, the size of hidden dimension and the number of hidden layers in their CDE and RDE functions. The Attentive NCDE (ANCDE) has all of those hyperparameters and in addition, the attention channel size. In DE-NRDE, there are addition hyperparameters related to embedding, which are the embedding compression ratio, the size of hidden dimension, and the number of hidden layers in the embedding layer.
In (Morrill et al., 2021), there are three models ,ODE-RNN, NCDE and NRDE, and four datasets, EigenWorms, BIDMC32HR, BIDMC32RR, and BIDMC32SpO2. For these combinations, we set the hyperparameters as reported in (Morrill et al., 2021). In other cases, we find the best hyperparameters in the following range. All kinds of hidden size in ODE-RNN, NCDE, ANCDE, and NRDE are in {32, 64, 128, 256}. The number of hidden layers in those is in {2,3}. In DE-NRDE, the embedding compression ratio, the hidden size, and the number of hidden layers in the embedding layer are in {0.3, 0.5, 0.7}, {64, 128}, and {1,2}, respectively. Tables 5 to 8 show the best hyperparameters of the baselines.
A.2 LORD-NRDE
Table 9 shows the best hyperparameter configuration of our method in each dataset.
EigenWorms has a time-series length of 17,984 and a channel of 7 which contains the movement data of roundworms. The goal is to classify each worm among 5 worm types. CounterMovementJump has 4,250 length and 4 channels which means accelerations data of each 3D-axis. Using accelerations data, the type of jump is predicted among 3 types. The object of SelfRegulationSCP2 is to classify whether the subject moves the computer’s cursor up or down, using 8 channels of the EEG data. Its time-series length is 1,153.
For forecasting, three Beth Israel Deaconess Medical Centre (BIDMC) datasets are used. Using the PPG and ECG information, each task is to predict a person’s heart rate (HR), respiratory rate (RR), or oxygen saturation (SpO2), respectively. The time-series length is 4,000.
C VISUALIZATION
We show other PCA based visualizations of the log-signatures. Using PCA, we extract the distributions of the most important principle components of {LogSigD1ri,ri+1(X)} ⌊ TP ⌋−1 i=0 , {LogSigD2ri,ri+1(X)} ⌊ TP ⌋−1 i=0 , and de(t). If two paths have similar distributions on those components, the information contained by the two paths is similar. In Fig. 5, the distribution of de(t) is somewhere in between the distributions of {LogSigD2ri,ri+1(X)} ⌊ TP ⌋−1 i=0 and {LogSigD1ri,ri+1(X)} ⌊ TP ⌋−1 i=0 . From the encoder-decoder learning, de(t) can learn the information of {LogSigD2ri,ri+1(X)} ⌊ TP ⌋−1 i=0 .
ANCDE 4 0.815±0.046 0.770±0.065 0.810±0.046 0.957±0.012 Not Applicable 92726 32 0.774±0.033 0.727±0.060 0.770±0.033 0.925±0.021 Not Applicable 92726 128 0.497±0.064 0.456±0.074 0.501±0.063 0.705±0.038 Not Applicable 22806 NRDE2 4 0.744±0.044 0.655±0.076 0.725±0.059 0.915±0.032 Not Applicable 64933 32 0.779±0.029 0.713±0.044 0.774±0.027 0.934±0.011 Not Applicable 64933 128 0.728±0.088 0.661±0.106 0.726±0.092 0.908±0.037 Not Applicable 64933 NRDE3 4 0.718±0.083 0.616±0.127 0.695±0.101 0.914±0.023 Not Applicable 297893 32 0.810±0.043 0.787±0.051 0.814±0.044 0.911±0.020 Not Applicable 297893 128 0.610±0.117 0.544±0.142 0.600±0.120 0.862±0.063 Not Applicable 297893 DE-NRDE2 4 0.436±0.048 0.230±0.073 0.303±0.066 0.538±0.045 Not Applicable 41331
32 0.728±0.074 0.631±0.141 0.711±0.095 0.918±0.026 Not Applicable 41331 128 0.769±0.068 0.723±0.078 0.770±0.070 0.939±0.016 Not Applicable 49304
DE-NRDE3 4 0.421±0.074 0.206±0.096 0.281±0.091 0.583±0.086 Not Applicable 179371
32 0.626±0.047 0.528±0.038 0.585±0.054 0.878±0.036 Not Applicable 179371 128 0.841±0.042 0.810±0.029 0.840±0.038 0.960±0.022 Not Applicable 241223
LORD1→2 4 0.795±0.051 0.744±0.051 0.787±0.053 0.948±0.009 24299 38973 32 0.841±0.053 0.841±0.058 0.843±0.051 0.957±0.016 24299 38973 128 0.759±0.064 0.748±0.082 0.760±0.061 0.900±0.009 24299 34813 LORD1→3 4 0.774±0.088 0.733±0.122 0.759±0.101 0.944±0.023 86907 34813 32 0.862±0.053 0.855±0.058 0.861±0.054 0.963±0.021 86907 34813 128 0.744±0.081 0.716±0.087 0.737±0.076 0.892±0.044 86907 38973 LORD2→3 4 0.821±0.054 0.783±0.062 0.815±0.056 0.960±0.011 278679 132465 32 0.841±0.021 0.793±0.053 0.835±0.027 0.963±0.008 278679 132465 128 0.841±0.038 0.823±0.055 0.835±0.049 0.949±0.023 278679 128305
ANCDE 64 0.355±0.041 0.342±0.034 0.342±0.035 0.526±0.033 Not Applicable 135852 128 0.467±0.060 0.457±0.065 0.457±0.065 0.604±0.031 Not Applicable 252332 512 0.458±0.073 0.428±0.102 0.428±0.102 0.588±0.074 Not Applicable 285740 NRDE2 64 0.396±0.116 0.391±0.118 0.391±0.118 0.560±0.089 Not Applicable 107907 128 0.396±0.076 0.379±0.079 0.379±0.078 0.513±0.052 Not Applicable 189059 512 0.396±0.091 0.388±0.092 0.388±0.092 0.546±0.076 Not Applicable 50435 NRDE3 64 0.465±0.088 0.462±0.093 0.463±0.093 0.616±0.062 Not Applicable 529411 128 0.409±0.084 0.399±0.081 0.400±0.081 0.573±0.070 Not Applicable 521859 512 0.465±0.055 0.453±0.050 0.453±0.051 0.618±0.045 Not Applicable 2107395
DE-NRDE2 64 0.398±0.075 0.390±0.076 0.389±0.075 0.573±0.076 Not Applicable 51910 128 0.384±0.058 0.334±0.071 0.334±0.071 0.567±0.054 Not Applicable 90886 512 0.375±0.054 0.359±0.064 0.359±0.064 0.525±0.055 Not Applicable 39114 DE-NRDE3 64 0.409±0.072 0.373±0.073 0.373±0.074 0.568±0.068 Not Applicable 204300 128 0.474±0.038 0.437±0.053 0.437±0.054 0.626±0.030 Not Applicable 279378 512 0.425±0.069 0.396±0.076 0.398±0.076 0.571±0.059 Not Applicable 730892
LORD1→2 64 0.589±0.038 0.580±0.041 0.580±0.041 0.717±0.023 3262 87191 128 0.566±0.061 0.563±0.066 0.563±0.066 0.676±0.058 12158 51063 512 0.501±0.080 0.495±0.082 0.495±0.082 0.638±0.053 7998 65399 LORD1→3 64 0.555±0.041 0.551±0.042 0.550±0.042 0.688±0.038 7282 43127 128 0.528±0.029 0.523±0.027 0.523±0.027 0.656±0.024 53746 137879 512 0.508±0.033 0.499±0.028 0.499±0.028 0.637±0.020 8338 88439 LORD2→3 64 0.396±0.049 0.383±0.052 0.383±0.052 0.540±0.061 77158 113265 128 0.396±0.078 0.384±0.083 0.385±0.083 0.542±0.068 27110 146737 512 0.400±0.044 0.374±0.022 0.375±0.022 0.537±0.047 31270 36369
D FULL EXPERIMENTAL RESULTS
In Tables 10 to 15, all experimental results are shown. #Params(D) and #Params(R) denote the number of parameters in the decoder part and in the rest parts except the decoder, respectively.
Table 16: Sensitivity of max iterAE in EigenWorms (P = 128)
Method max iterAE Accuracy Macro F1 Weighted F1 ROCAUC
LORD1→2
0 0.656±0.094 0.588±0.107 0.649±0.100 0.847±0.056 100 0.713±0.042 0.716±0.058 0.713±0.046 0.862±0.023 300 0.744±0.048 0.727±0.059 0.744±0.051 0.893±0.029 500 0.733±0.105 0.725±0.104 0.738±0.111 0.889±0.035 700 0.774±0.049 0.773±0.070 0.777±0.056 0.908±0.026 900 0.723±0.091 0.702±0.092 0.722±0.083 0.891±0.054
LORD1→3
0 0.600±0.140 0.513±0.215 0.561±0.188 0.820±0.068 100 0.713±0.056 0.636±0.139 0.694±0.085 0.884±0.030 300 0.733±0.069 0.725±0.053 0.730±0.063 0.868±0.036 500 0.733±0.039 0.722±0.054 0.730±0.046 0.872±0.035 700 0.744±0.031 0.730±0.043 0.741±0.034 0.872±0.021 900 0.744±0.048 0.724±0.040 0.740±0.053 0.884±0.015
LORD2→3
0 0.477±0.059 0.338±0.105 0.420±0.078 0.764±0.051 100 0.667±0.091 0.591±0.127 0.642±0.106 0.888±0.044 300 0.790±0.042 0.754±0.060 0.783±0.050 0.927±0.017 500 0.826±0.071 0.789±0.113 0.825±0.072 0.951±0.029 700 0.846±0.101 0.816±0.126 0.847±0.103 0.928±0.046 900 0.856±0.067 0.833±0.087 0.857±0.067 0.949±0.027
E SENSITIVITY EXPERIMENTS
Fig. 6 and Table 16 show that in general, the performance of LORD is improved asmax iterAE gets larger in EigenWorms (P = 128). Our encoder-decoder structure can successfully integrate the lower-depth log-signature and the higher-depth log-signature information into the embedding vector e, and the well-integrated information helps the model perform better. In general, the lower-depth and the higher-depth log-signature information are blended better as the encoder-decoder structure is more trained.
F TRAIN LOSS
Fig. 7 visualizes several training cases for our method and the original NRDE design. In general, our method shows much better stability across the entire training period.
G END-TO-END TRAINING
LORD has two distinguished characteristics in its training process. First, LORD uses a two-phase training strategy, a pre-training and a main training processes. Second, the training process of LORD is not end-to-end. In other words, the decoder is not used and the encoder is fixed in the main training phase. In this section, we test with various end-to-end-training settings for LORD.
There are three possible configurations for the end-to-end-training. The first configuration is training the encoder (rather than fixing it) with LTASK in the main training phase, which is denoted as FineTuning. The second one is Co-Train where both LAE and LTASK are used for training the encoder, the decoder and the main NRDE — recall that in FineTuning, we use only LTASK
We experiment with EigenWorms (P = 128). Table 16 shows the results. Among the three endto-end models, Co-Train shows the best performance. Co-Train(w.o. pre) has the worst performance in general. These results prove that the pre-training makes the main training easier.
In this section, we experiment with short time-series datasets. We use CharacterTrajectory and LiveFuelMoistureContent from (Tan & Webb). The object of CharacterTrajectory is to classify 20 characters using the trajectory information of writing on a tablet. Its length is 182. LiveFuelMoistureContent consists of daily reflectance data at 7 spectral bands for predicting moisture rate in vegetation. Its length is 365. In Tables 18 to 22, we summarize hyperparameters. Because of the short time-series lengths, we test P = 1 for all except NRDE-based models.
In Table 23, NRDE achieves the best scores among baseline models and LORD achieves the best scores among all methods. In Table 24, DE-NRDE2 is the best. Except DE-NRDE2, ODE-RNN is the best. However, LORD also has comparable scores. From these results, NRDE and NRDE-based models are empirically useful for short-time series as well.
Table 23: CharacterTrajectory
Method P Accuracy Macro F1 Weighted F1 ROCAUC #Params(D) #Params(R)
ODE-RNN 1 0.716±0.036 0.697±0.041 0.712±0.038 0.969±0.007 Not Applicable 27570 4 0.950±0.012 0.947±0.013 0.950±0.012 0.998±0.001 Not Applicable 29106 16 0.968±0.004 0.967±0.005 0.968±0.004 0.999±0.000∗ Not Applicable 103218 32 0.962±0.009 0.959±0.010 0.962±0.009 0.997±0.001 Not Applicable 17650
NCDE 1 0.954±0.004 0.952±0.004 0.954±0.004 0.998±0.000 Not Applicable 149844 4 0.951±0.005 0.948±0.005 0.951±0.005 0.998±0.000 Not Applicable 233620 16 0.905±0.020 0.901±0.020 0.905±0.020 0.994±0.002 Not Applicable 93588 32 0.829±0.012 0.818±0.013 0.828±0.012 0.983±0.002 Not Applicable 93588
ANCDE 1 0.959±0.007 0.958±0.007 0.959±0.007 0.998±0.000 Not Applicable 669757 4 0.952±0.008 0.950±0.008 0.952±0.008 0.998±0.000 Not Applicable 286845 16 0.897±0.009 0.893±0.011 0.896±0.010 0.994±0.001 Not Applicable 538173 32 0.777±0.020 0.762±0.020 0.774±0.021 0.981±0.001 Not Applicable 103293 NRDE2 4 0.970±0.004 0.968±0.005 0.970±0.004 0.999±0.000∗ Not Applicable 795924 16 0.951±0.008 0.949±0.009 0.951±0.008 0.998±0.001 Not Applicable 193428 32 0.950±0.007 0.948±0.008 0.950±0.007 0.998±0.000 Not Applicable 98836 NRDE3 4 0.966±0.006 0.965±0.006 0.966±0.006 0.999±0.000∗ Not Applicable 274132 16 0.960±0.006 0.960±0.006 0.961±0.006 0.999±0.000∗ Not Applicable 531604 32 0.961±0.002 0.959±0.002 0.961±0.002 0.999±0.000∗ Not Applicable 1088916 DE-NRDE2 4 0.898±0.077 0.890±0.084 0.897±0.078 0.994±0.007 Not Applicable 599707 16 0.907±0.013 0.900±0.012 0.906±0.012 0.996±0.002 Not Applicable 112281 32 0.897±0.018 0.890±0.018 0.896±0.017 0.995±0.001 Not Applicable 76187 DE-NRDE3 4 0.933±0.031 0.926±0.035 0.932±0.032 0.997±0.001 Not Applicable 105885 16 0.947±0.014 0.943±0.015 0.947±0.014 0.998±0.000 Not Applicable 286883 32 0.913±0.013 0.907±0.014 0.912±0.013 0.996±0.001 Not Applicable 617891 LORD1→2 4 0.977±0.002∗ 0.976±0.003∗ 0.977±0.002∗ 0.999±0.000∗ 12158 235880 16 0.955±0.010 0.953±0.010 0.955±0.010 0.998±0.001 12158 96232 32 0.829±0.069 0.817±0.077 0.826±0.074 0.985±0.011 40126 261928 LORD1→3 4 0.976±0.005 0.975±0.006 0.976±0.005 0.999±0.000∗ 53746 98568 16 0.959±0.006 0.957±0.006 0.959±0.006 0.998±0.001 53746 98568 32 0.810±0.069 0.796±0.074 0.808±0.071 0.983±0.013 53746 63560 LORD2→3 4 0.971±0.008 0.969±0.009 0.971±0.008 0.999±0.000∗ 77158 372418 16 0.971±0.003 0.969±0.003 0.971±0.003 0.999±0.000∗ 218086 391650 32 0.964±0.008 0.963±0.008 0.964±0.008 0.999±0.000∗ 77158 372418
NCDE 1 -0.009±0.037 -0.008±0.037 1.102±0.040 0.787±0.010 Not Applicable 42241 4 -0.004±0.020 -0.003±0.020 1.098±0.022 0.785±0.007 Not Applicable 91521 16 -0.043±0.031 -0.043±0.031 1.140±0.034 0.794±0.011 Not Applicable 42241 32 0.009±0.029 0.009±0.029 1.083±0.032 0.779±0.005 Not Applicable 660481
ANCDE 1 -0.025±0.078 -0.024±0.079 1.120±0.086 0.785±0.009 Not Applicable 101714 4 0.014±0.007 0.015±0.007 1.077±0.007 0.781±0.003 Not Applicable 241938 16 0.003±0.030 0.004±0.030 1.089±0.033 0.781±0.010 Not Applicable 93394 32 0.006±0.017 0.006±0.017 1.087±0.019 0.784±0.009 Not Applicable 177746 NRDE2 4 -0.010±0.043 -0.009±0.043 1.103±0.047 0.785±0.004 Not Applicable 1284353 16 -0.373±0.274 -0.373±0.274 1.501±0.299 0.824±0.020 Not Applicable 2502657 32 -0.151±0.166 -0.151±0.166 1.258±0.182 0.816±0.023 Not Applicable 1240833 NRDE3 4 -0.480±0.311 -0.480±0.311 1.618±0.340 0.830±0.012 Not Applicable 1710977 16 -1.308±2.265 -1.307±2.264 2.522±2.475 0.863±0.089 Not Applicable 3438465 32 -2.183±4.622 -2.182±4.621 3.478±5.051 0.896±0.174 Not Applicable 857601 DE-NRDE2 4 0.040±0.007 0.040±0.007 1.049±0.007 0.770±0.005 Not Applicable 930650 16 0.042±0.006 0.042±0.006 1.047±0.007 0.768±0.005 Not Applicable 799243 32 0.043±0.005∗ 0.043±0.005∗ 1.046±0.006∗ 0.767±0.006∗ Not Applicable 902042 DE-NRDE3 4 -0.778±1.599 -0.776±1.599 1.943±1.747 0.796±0.039 Not Applicable 1256207 16 -0.037±0.046 -0.036±0.046 1.133±0.051 0.790±0.010 Not Applicable 1103486 32 -0.657±0.831 -0.651±0.822 1.810±0.908 0.829±0.045 Not Applicable 457191 LORD1→2 4 0.017±0.015 0.018±0.014 1.074±0.016 0.784±0.011 76684 120937 16 0.019±0.007 0.020±0.008 1.072±0.008 0.781±0.007 216844 441481 32 0.013±0.007 0.014±0.007 1.078±0.007 0.784±0.002 216844 441481 LORD1→3 4 0.018±0.010 0.018±0.011 1.074±0.011 0.782±0.006 612148 441481 16 0.002±0.009 0.003±0.009 1.091±0.010 0.790±0.003 136180 70153 32 0.005±0.011 0.005±0.010 1.087±0.012 0.788±0.007 278452 99529 LORD2→3 4 -0.025±0.037 -0.025±0.038 1.120±0.041 0.793±0.004 2081028 648629 16 -0.015±0.027 -0.014±0.027 1.109±0.030 0.793±0.004 1016196 957557 32 0.008±0.002 0.008±0.002 1.085±0.003 0.788±0.002 508356 540181

Review summaries:
The authors propose to increase the efficiency of neural rough differential equations by encoding the log-signature into a lower dimensional space that is predictive of higher order signatures during a pre-training stage.  This enables the authors to work with lower dimensional log-signatures during the main training stages. The method performs competitively on a range of long time-series tasks.  

The manuscript proposes a new method of training neural RDEs with a pretrained autoencoder to reduce overhead of the log-signature transform when dealing with long time-series. The neural RDE is trained on the image of a pretrained autoencoder as a way to reduce complexity and memory usage.  The methods is compared to several different baselines, including neural CDE and ODE methods. These are tested on 6 real world datasets for both classification and forecasting.

The paper seeks to improve upon the recently introduced "Neural Rough Differential Equation" (NRDE) model by using an autoencoder to reduce the dimensionality of the input time series. The proposed autoencoder embeds "log-signature" information into a lower-dimensional space and can be trained prior to the main NRDE model. This approach is novel, easily understood, and intuitively should outperform previous log-signature based models. The authors demonstrate that it gives significant improvements on long time series problems through several experiments.