LORD: Lower-Dimensional Embedding of Log-Signature in Neural Rough Differential Equations
Authors: JAEHOON LEE
Jinsung Jeon
Sheo yon Jhin
Jihyeon Hyeong
Jayoung Kim
Minju Jo
Kook Seungji
Noseong Park
Venue: ICLR 2022 Poster (2022)

Abstract: The problem of processing very long time-series data (e.g., a length of more than 10,000) is a long-standing research problem in machine learning. Recently, one breakthrough, called neural rough differential equations (NRDEs), has been proposed and has shown that it is able to process such data. Their main concept is to use the log-signature transform, which is known to be more efficient than the Fourier transform for irregular long time-series, to convert a very long time-series sample into a relatively shorter series of feature vectors. However, the log-signature transform causes non-trivial spatial overheads. To this end, we present the method of LOweR-Dimensional embedding of log-signature (LORD), where we define an NRDE-based autoencoder to implant the higher-depth log-signature knowledge into the lower-depth log-signature. We show that the encoder successfully combines the higher-depth and the lower-depth log-signature knowledge, which greatly stabilizes the training process and increases the model accuracy. In our experiments with benchmark datasets, the improvement ratio by our method is up to 75\% in terms of various classification and forecasting evaluation metrics.

Published as a conference paper at ICLR 2022
LORD: LOWER-DIMENSIONAL EMBEDDING OF LOG- SIGNATURE IN NEURAL ROUGH DIFFERENTIAL EQUA-
TIONS
Jachoon Lee, Jinsung Jeon, Sheoyon Jhin, Jihyeon Hyeong, Jayoung Kim, Minju Jo, Seungji Kook, and Noseong Park
Yonsei University
Seoul, South Korea
{jaehoon ee, ]jsjjs0902, sheoyonj, heejii, jayoung, kim, alflsow112, 2t 2021321393 noseong/@yonsei. oore ac. .r.
ABSTRACT
The problem of processing very long time-series data (e.g., a length of more than 10,000) is long-standing research problem in machine learning. Recently, breakthrough, called neural rough differential equations (NRDEs), has been pro- posed and has shown that it is able to process such data. Their main concept is to use the log-signature transform, which is known to be more efficient than the Fourier transform for irregular long time-series, series sample into a relatively shorter series of feature vectors. However, the log- signature transform causes non-trivial spatial overheads. To this end, we present the method of LOweR DoweR-Dimentt Dimensional embedding of log-signature (LORD), where we define an NRDE-based autoencoder to implant the higher-depth log- signature knowledge into the lower-depth log-signature. We show that the encoder success- fully combines the higher-depth and the lower-depth log-signature knowledge, which greatly stabilizes the training process and increases the model accuracy. In our experiments with benchmark datasets, the improvement ratio by our method is up to 75% in terms of various classification and forecasting evaluation metrics.
convert a very long time-
INTRODUCTION
Time-series data occurs frequently in real-world applications, e.g., stock price forecasting (Ariyo
al., 2014, Siami-Namini et al., 2018 Jhin et al. [021], traffic forecasting (Reinsel 2003 Ful
2011
Bai et al. 2020 Fang et al., 2021). weather forecasting (Shi et al. 2015 Seo et al. 2018
Brouwer et al., 2019 Ren et al. 2021). and on. However, it is known that very long time- series data (e.g.. a time series length of more than 10,000) is not straightforward to process with deep learning despite various techniques ranging from recurrent neural networks (RNNs) to neural ordinary/controlled differential equations (NODEs and NCDEs). RNNs are known to be unstable when training with such very long sequences and the maximum length that can be processed by NODEs and NCDEs is more less the same that by RNNs (Morrill et al. 2021 Aicher et al. 2020 Trinh et al. 2018; Stoller et al.) 22199; Bai et al. 2018). However, one breakthrough has been recently proposed, namely neural rough differential equations (NRDEs).
NRDEs are based on the rough path theory which was established to make sense of the controlled differential equation:
dz((t (((())dX(().
where X a continuous control path, and z(t) is hidden vector at time f. A prevalent example X (semimartingale) Wiener process, in which case the equation reduces stochastic differential equation. In this sense, the rough path theory extends stochastic differential equations beyond the semimartingale environments (Lyons et al. 2004).
One key concept in the rough path theory is the log-signatue of a path. It had been proved that the log-signature of path with bounded variations is unique under mild conditions (Lyons & Xu

---

Published as a conference paper at ICLR 2022
Higher-depth
(00)
Resporse
200)
Respoesee
Heavyweigtt Decodor (will be discanded alle: Imining)
Nearal RDE
Recevery
NRDE
RDD
NRDE
Reduction
Dimensionality Reduction
Key Design
Lightwiggt Enooder
Encoding
signature summary
Highar-dapth
LogSiof 2 (x)3
for fealure) for each Howwwer the logaligratnn aire grows list wirl. the origi input size and the depth.
Lower-dopth
LoySigh
(X0)
Path
Path
The encoder comb nos the lower- cepth and the higher-depth log- agnnstr information
1o
(a) Directly reducing the dimensionality
(b) Combining both log signature information (our method)
Figure 1: Two possible approaches to reduce the dimensionality of the higher-depth log- log-signatur. The dark gray means a processing in a higher-dimensional space and the light gray means that in a lower-dimensional space. Our embedding method in (b) is better than the baseline in (a) in that i) the higher-ding disber--imensional processing is deferred to the decoder which will be discarded after pre-training the autoencoder, and ii) our method does not involve any gigen--dimeniionll processing during the main training and inference processes. We pre-train the autoencoder and in the main training step, discard the decoder, fix the encoder, and train only the main NRDE for downstream task (ef. In this way, we can exclude the higher-dimen dimensional processing early as possible.
Table
2018 Geng 2017) and most time-series data that happens in the field of deep learning has bounded variations. Therefore, can interpret that the log-signature is unique feature of the path.
Given a .-llngth time-series sample {x, LN annotated with its observation time-points { where to = 0, IN = T. and t; 1,-1, NRDEs construct continuous path X(). where & E [0, with an interpolation algorithm, where Xf,,) = (x,, t2) for t, E {/ In other words, the path has the same value the observation (x,, t2), when t; is one of the observation time- points and otherwise, interpolated values. As shown in Fig. [ a log-signature (each dotted yellow box in the figure) is calculated every P-length sub-path, and another time-series of log-signatures,
denoted LLogSig...... (x) 1 is created. The log-signature calculation has important hyperparameter D called depth- the higher the depth is, the more accurately represented each sub-path is (ef. E.. For instance, the best accuracy score is 0.81 for D vs. 0.78 for D = in EigenWorms. The sub-path length P and the depth D decides the number and the dimensionality of log-signatures, respectively.
However, one downside is dim dimgLogSigg,.r... (X)) dim(X), where dim(() means the dimen- sionality (or the number of channels) of X. As a matter of fact, dim(LogSigh.. (X)) grows rapidly w.r.t dim(X) (Morrill et al. 2021). Since the dimensionality of the input data is, given a dataset, fixed, we need to decrease 17 reduce overheads. In general, NRDEs require more param- eters to process higher-dimensional log-signatures (as shown in Table 4 where the original NRDE design always requires more parameters for D in comparison with
2.)
To this end, we propose to embed the higher-depth signatures onto lower-dimensional space to i) decrease the complexity of the main NRDE (the solid blue box in Fig. D and ii) increase the easiness of training (cf. Fig. ( and as a result, its model accuracy as well. Fig. I shows two possible approaches where we directly reduce the dimensionality of the higher-depth log-signature in Fig. (T) or we adopt the autoencoder architecture to combine both the higher- depth and the lower-depth signatures in Fig. [ (b) the first ap- proach in Fig. (a) is baseline model, and our proposed model is the second approach in Fig. [b). Autoencoders are frequently used for (unsupervised) dimensionality reduction although there also ist other approaches. Since the higher-depth log-signature should be reconstructed from the embedded vector, one consider that
BIDMC32RRIP== 128)
NADE, LORD,
Train Logg
Epoch
Figure 2: The loss curve of LORD is stable. Other figures are in Appendix
2

---

Published as a conference paper at ICLR 2022
the higher-depth log-signature is indirectly embedded into the vector. Moreover, the decoder is discarded after its pre-trainn pre-training so our method does not involve any high-cost computation with the higher-depth log-signature in the main training and inference steps clarified in Table !
Our proposed method, LOweR- LOwcR-Dimensione Dimensional em- bedding of log-signature (LORD), adopts an NRDE-based autoencoder to combine the higher- depth and the lower-depth log-signature knowl- edge, and utilizes the embedded knowledge by the encoder for downstream machine learning task as shown in Fig. (b). The encoder embeds the lower-depth log-signature from the continuous path X, and the decoder reconstructs the higher-depth log-signature. This specific design is where our key idea lies in. The encoded lower-depth ong-sinnature will contain both the lower-depth and the higher-depth log-signature knowledge because it is encoding of the lower-depth log-signature and ii) the decoder should recover the higher-depth log-signature from it. After pre-training the au- toencoder, discard the decoder to exclude the nigher-dimensiona processing as early possible from our framework. After that, we fix the encoder and train only the main NRDE.
Table I: Two phase training in our method
Pre-training
Main training
Decoder
Training
Discarded
Encoder
Training
Fixed
Main NRDE
Not Applicable
Training
We conduct experiments with six very long time-series benchmark datasets and five baselines in- cluding NODE, NCDE, and NRDE-based models. RNN-based models cannot process the very long time-series datasets and exclude them. Our proposed method significantly outperforms all existing baselines, Our contributions can be summarized as follows:
1. We design an NRDE-basee continuous autoencoder to combine the higher-depth and the lower-depth log-signature information. Since decoder discarded after being trained, we adopt the asymmetric architecture that the encoder is lightweight and the decoder is heavyweight in terms of the number of parameters.
2. Our proposed method outperforms all baselines by large margins (up to 75% for various evaluation metrics, e.g., (²2) ) in standard benchmark datasets with much smaller model sizes in comparison with the original NRDE design.
Our code is available httttt::/ggthhhi http:/tubb.comm//eeeeeeotoooo//o 6/1ORD
RELATED WORK AND PRELIMINARIES
NODEs NODEs Chen et al. 2018 use the following equation:
z =(0)= L'" (1
which means ((T) is solely defined by the initial value z the entire evolving path from z0)) to (T) is defined by the initial value in ODEs. NODEs are not considered as a continuous analogue to RNNs but to residual networks (Chen et al. 2018).
NCDEs Let [x:) be an N-length time-series sample and (4 be its observation time- points. NCDEs are different from NODEs in that they use the following equation:
(T) = x(0) + S' /(a(0))))xx(0==>>) =()) L' s(x(1):0)) dx(1) di,
(3)
di
where Xtt is a continuous interpolated path from {(x;, 44) where in = T. In this re- gards, NCDEs also be considered continuous analogue RNNs. The adjoint sensitivity method can be used to decrease the space complexity of gradient calculation instead of backpropa- gation (Kidger et al. 2020).
However, NCDEs has an inherent disadvantage that they are weak at processing very long time- series samples, because NCDEs directly process the very long time-series samples.
3

---

Published as a conference paper at ICLR 2022
Signature Transform Let (r,, r++1l be time duration, where n, M+ The signature of the time-series sample is defined as follows:
dx (tz)dtj,
Shirii (X)
dt
G9 Et. Crifl
(X) (15 (1))))4((2))) (x) rutru (X))ii((X)
dim(x)
{S
dimn(X)
(X)
dim(X)
(4)
However, the signature has redundancy, e.g., (2) ( S (2)( where can know a value knowing three others. The log-signature LogSigh.. (X) is then created after dropping the redundancy from the signature. In this work, we use this log-signature defini- tion in default to design
method. By creating log-signature for every P-length sub-path, the entire sequence length can be reduced moreover, the log-signature is a unique feature of the sub- path (Lyons & Xu 2018 Geng 2017). For complexity reasons, typically the log-signature of depth Σ (Morrill et al. 2021).
NRDES Owing to the log-signature transform of time-series, NRDEs (Morill et al. 2021 are defined as follows:
9(X1)) LogSig (X)
for l (1a,)41))
ri+1 - ri
(5)
(TT = z(0) + L'' f(2(t)))))((Xx)))t
where LogSigh, D
r3: (X) means the log-signature created from the path X within the interval
LogSigh rees (X)
is piecewise approximation of the time-derivative of the log-signature in the short interval (r, D means the depth of the log-signature. Once we define the sub-
[r,, re41).
1:
path length P. the intervals ((r)) FJ-1 are decided, i.c., P = Ts+1 - r for all i. Then, the 1
time-series of LogSig (X) constitutes (LogSigh.ree (x) in our notation.
NRDEs use Eq. to derive z(T) from z(0), which can be considered as a continuous analogue to RNNs since it continuously reads the time-derivative of the log-signature. Therefore, z(T)) is defined by the initial value z0) and the sequence of the time-derivative of the log-signature. We can also use the adjoint sensitivity method train NRDEs.
Long Sequence Time-series Input (LSTI) The problem of processing very long time-series data is long standing research problem. There are three ways to solve the LSTI problem. First, reduces the sequence through runcaiig/uumnarizng/ammpii However, this method may lose information affecting the prediction accuracy. The second to give the gradient transformation. As the sequence becomes longer, the gradient vanishing problem occurs. Therefore, in (Aicher et al.) (2200). the model is trained using only the gradient of the last step. It also solves the problem using auxiliary losses (Trinh et al., 2018). Finally, CNNs (Stoller et al. 2019 Bai et al. 2018) were used to solve the LSTI problem. The convolutional filter captures long term dependencies, but receptive fields increase exponentially, breaking the sequence.
from very long input sequence.
3 PROPOSED METHOD
We describe our proposed method, LORD-NRDE. We first clarify the motivation of our work and then describe our proposed model design.
Motivation It is known that NRDEs are generalization of NCDEs (Morril et al. 2021 Section 3.2). However, one problem in utilizing NRDEs in real-world environments is that dim(LogSion (X)) is rapidly growing function of dim(X) (Morrill et al. 2021 Section A).
4

---

Published as a conference paper at ICLR 2022
Reconstructed log-signarures with a highier-dpth Higher-depth
Algorithm 1:
How to train LORD-NRDE
Input:
Training data Dirain. Validating data Deat, Maximum iteration numbers max iter AE and maz sterTAsk Initialize the parameters of the encoder (i.c., oy, )e the decoder (i.e., O,, Ox and the main NRDE (i.e., 0g. Oz.. Poutput): )::
Autoenooder Loss LAB
Herryweight (Will be discarded after training)
1* Pre-training of the autoencoder
Lightweight Enooder
2 for mar iler AFE ilerations do
Train the encoder and the decoder using LAw LORD-NRDE
Lewendeplh
Kain-training
Log.si signatures with lower-depth
for max minz_iterTAte ASK iterations do
5
Train the main NRDE using LTASK: Validate the best main NRDE parameters with Deati
Figure 3: The proposed NRDE- based autoencoder
7 return the encoder and the main NRDE parameters;
This rapid up of dimensionality causes two problems: i) it hinders us from applying NRDEs to nigh-dimensional time-series data, and ii) it makes the training process complicated since the hidden representation of the large input should be made for a downstream task. Therefore, we propose to embed the log-signature onto lower dimensional space that the training process becomes more tractable and enhance the applicability of NRDEs to high-dimensional data. After the embedding, in other words, the dimensionality of embedded vector is typically smaller than that of the log-signature with depth D2 in our setting but it has knowledge of the signatre with depth D2 > D1.
Overall Architecture We describe our proposed method, LORD-NRDE, which consists of three NRDEs: i) encoder NRDE, ii) decoder NRDE and iii) main NRDE to derive z(T") from z(0). The role each NRDE is as follows:
1. The encoder NRDE continuously embeds the log-signature with depth D1 onto another space. We use e(t) to denote the vector embedded from the log signature at time t.
2. The decoder NRDE reconstructs the log-signature with depth D2 > D2 from e(t).
3. The main NRDE reads e(t) evolve ztt) and derive (TT). There an output layer which reads (T) and makes inference.
We note that dim GigggggiggPi D2 produces embedding vectors that contain both the log-signature knowledge with D1 and D2 depth.
(X)) > (em)) where E [o, T]. We require that the encoder
LORD-NRDE
To this end, we propose the following method:
de(l) dt e(T) = e(0)+ {' s(e(();0)) LogSig (X),
z(T) z0)+ (' g(x(t): 0y)
dt,
(6)
rit1 rs dl, for t E [rame)))
s(T) s(0)+ (" 0() de(t) dt,
where dim()(t)) dim(s(t)) = dim Sigg (X)), where e (14)) with D2 > D1-
The term de(t) can be considered as an embedding of the log-signature with depth Dj at time i, and e(t) as embedding of (t) constructed by the log-signature with depth D,. Similarly, ds(t) is reconstructed log-signature with depth D2 from the embedding, and s(t) as (t) reconstructed by the log log-signature with depth Dz. Since c(0) and s(0) mean the initial values, set de (X As.), dz (c(c((); 0, where & means transformation function.
Therefore, e(t) and s(t) constitute an autoencoder based NDREs as shown in Fig. In design, we use the asymmetrical setting for them, where the encoder RDE is lightweight and the decoder RDE is heavyweight in terms of the number of parameters, considering the applicability of
5

---

Published as a conference paper at ICLR 2022
design to real-world environments. Since the decoder NRDE is discarded after its pre- tre-training, the heavyweight setting causes no problems afterwards our method requires more resources for the pre-training process though.
There are three functions 9. f. and o in Eqs. 8 and their definitions are as follows:
g(z(t):0g) = NN (bb(F (..., (, [()(();0)) = N (-. ( (e()))... o(s(t); 0,) = FCoutt,,N (bb (...)(FC.... )..)),,
(9)
(10)
(11)
where FC₁, is a cully-connected (FC) layer whose output size is h. Ng. N,, No means the number of layers, v and 1) are the hyperbolic tangent and rectified linear unit activations, respectively.
Training Method To train the NRDE-based autoencoder, we use the following loss:
E [[(((r+))) - s(r.)) - Log (X)I
Trecon
(12)
M
LAE Treeom + CAEE (10,11 + 110.112 + 110...1... 112 10 (12) + cellel12.
(13)
and CAF is a coefficient of the L2 regularization terms. ce also regularizes the scale of the learned embedding. In addition to LaE, we also have task loss LTASK which is the standard cross-entropy (CE) loss or the mean squared loss (MSE) loss. From zTT). we have an output layer, which is typically a fully-connected layer with an appropriate final activation function such as softmax, to produce prediction ÿ. Opoctput denotes the parameters the output layer. Using the ground-truth truth information y, define the task loss LTASK. Therefore, the final loss can be written as follows:
where M = [
LTASK = CE v) crASK (118 + 110 12 100 112),
(14)
where CTASK is coefficient of the L2 regularization terms, and CFE 9) means cross-entropy loss - we assume classification but can be accordingly changed for other tasks.
To implement, define the following augmented ODE:
LogSig S (X) (X)
d ((z(()),,)(((((t).)) 2.
ri+1
dt (t)
f(e(t);0,)
(X)
for t. E [r,,))))
(15)
LogSigh (X)
o(s(t):)))((e(t))
r4t1
where we replace de(t) de with ((e());
LogSig (X)
(as defined in Eq. 7 and the initial values
ret1 T4
defined as follows:
(0)) c(0) de (x Pox
(x Oc..)
(16)
[s(0)
(e())) 0
where dg and de transformation functions to produce the initial values from the initial observa- tion (0), and de is a transformation function to produce the initial reconstructed value from the initial embedded vector e(0). We a fully-connected layer for each of them.
Alg. shows the training algorithm. In Line we pre-train the autoencoder for mar mar_iler AE itera- tions. After discarding the decoder and fixing the encoder, we then train the main NRDE in Line 5 for max iterT ASK iterations. Using Deat. we validate and choose the best parameters.
EXPERIMENTAL EVALUATIONS
We describe our experimental environments and results. The mean and variance of 5 different seeds reported for model evaluation. We refer Appendix for reproducibility.

---

Published as a conference paper at ICLR 2022
4.1 EXPERIMENTAL ENVIRONMENTS
Datasets We use six real-word dataset which all contain very long time-series samples. There classification datasets in the University of East Anglia (UEA) repository (Tan & Webb): EigenWorms rms,
and Se Sel fRegulllll at atgggllltninSP...... SnSCP2, and 3 forecasting datasets in Beth Israel Deaconess Medical Centre (BIDMC) which come from the TSR archive (Tan & Webb): BIDMCHR, BIDMCRR, BIRR,,IDMME BIDMCSpO2. We refer to Appendix B for detail of datasets.
reovement
Baselines There are three types of baselines in our Experiments, ranging from NODE NCDE and NRDE- based baseline models. ODE-RNN is one of the sfate-of-the-22 the-art NODE models in pro- cessing time-series. Following the suggestion in (Morrill et al. (02I). we merge P observations into one merged observation and feed it into ODE-RNN, in which case CDE-RNN is able to process much longer time-series. The dimensionality of the merged observation is P times larger than that of the original observation or equivalently, the length of the merged time-series is P times shorter than that of the original one. NCDE is the original NCDE design in (Kidger et al. 2020). Attentive NCDE (ANCDE) is an extension of NCDE by adding outperforms NCDE (Jhin et al. 2021). For the NRDE-type baselines, we consider i) the original NRDE design (Morrill et al. 2021), and ii) the one in Fig. (a) which directly embeds the higher- depth log-signature into a lower-dimensional space, each of which is called NRDE and DE-NRD respectively. For those NRDE-based models, we clarify its one important hyperparameter, depth, as part of its name, c.g., NRDE₂ means NRDE with D = 2. One important point is that NRDE is a generalization of NCDE. Therefore, NRDE, is theoretically identical NCDE, if using the linear interpolation method, and for this reason, we set D > 1 for NRDE.
attention mechanism into it, which significantly
Hyperparameters When using NRDEs, there is one crucial hyperparameter, the log-signature depth D. We use two depth settings which were used in (Morrill et al. 202T). For our LORD-NRDE, there are two depths. D1 and D2. where D1 D2- LORD D, D, means that LORD-NRDE's decoder recovers the D2-depth log-signature from the Di-depth log-signature. The number of layers in the encoder, decoder and main NRDE, Ng. Ny. ; and N₂, of Eqs. to I1 are in (2, The hidden sizes, h f,, and ho of Eqs. to 11 are in [32. 64, 128, 192). The coefficiett of the L2 regularizers in
in Eqs. 13 and 14 are in {1 X 10
x 10 10-6). The coefficient of the embedding regularizer. Eq. 13 1s in {0. I, 10). The max iteration numbers, max itErAE AE and mar iferTASK in Alg. 1 are in 500, 1000, 1500, 2000]. The learning rate of the pre-training and main-training is 1 x 10 We also set di)(((())) = dimm( Sig (X)), where 1 E [ri, 1.
We also conduct experiments by setting the sub-path length Pt 4, 8. 32, 64. 128, 256, or 512 ob- servations. In other words, we create one bignaigae for every P input observation for NCDE and NRDE-based models. For ODE-RNN, as described earlier, we simply concatenate P observations into one observation. The final time T is large in our experiments. e.g., T> 10, 000, and the num- ber of log-signatures [ We test both the adjoint sensitivity method and the backpropagation through the solver.
Evaluation Methods We reuse the very long time-series classification and forecasting evalua- tion methods of (Morrill et al. 2021) and extend the methods by adding more datasets and more evaluation metrics. We use accuracy, macro F1, and ROCAUC for binary classification; accuracy, macro/weighted F1, and ROCAUC for multi-class classification; and R². explained variance, mean squared error (MSE), and mean absolute error (MAE) for forecasting - we list the complete results in Appendix D after introducing key results in the main manuseript. We also show the number of parameters for each model - for our LORD, we exclude the parameter numbers of the decoder since is discarded after the pre-training. We train and test each model 5 times with different seeds. If the mean of score is the same, the smaller standard deviation is better.
4.2 EXPERIMENTAL RESULTS
Summary of Experimental Results Since our main result tables have many items, we quickly summarize their highlights in Tables 2 and 3 To calculate the improvement in evaluation metrics, we use the ratio of improvement over NRDEa for each metric averaged over all the sub-path lengths and all datasets, e.g., we calculate LORD's ROCAUC-- NRDE,'s 's ROCAUC classification cases and average them. The ratio of the number of parameters is calculated similarly.
ROCAUC and NRDE2's MSR MSF-TORR' MSE for all KRD2,'s
7

---

Published as a conference paper at ICLR 2022
Table 2: Highlights in Classification
Table 3: Highlights in Forecasting
Method
Acc.
Mac.FI
ROCAUC
#Params
Method
Sf² MSE
#lParams
ODE-RAN
-9%
-19%
-11%
-20%
OUE-RNN
44%
41
-36%
NCDE
4%
-8%
-5%
64%
NCDE
45%
-55%
-30%
AMODE
0%
0%
15%
ANCDE
-34%
-52%
BRDE2
0%
0%
0%
0%
NRGEg
0%
0%
0%
NRDEg
NR
82%
DE-NRGEg
.4%
-7%
-3%
48%
DE NRC E2
39%
35%
-20%
DR-NRRFg
DE-NRDEE
62%
59%
31%
L0RD1-+2
16%
23%
11%
-55%
LORD
53%
47%
-5%
1-RR0=+3
15%
20%
10%
-46%
TORDD 1a
51%
46%
-59%
LORD2-+3
9%
4%
17%
LORD2-+3
-39%
Overall, LORD outperforms other baselines with smaller numbers of parameters. LORD D has a larger model size, excluding its decoder, than NRDED, whereas it has a much smaller model size than NRDE D2- However, the performance of L0 similar to better than that of NRDED,,, which proves the efficacy of our method. Using the encoder-decoder structure, the high complexity of processing log-signatures can be reduced and makes our model smaller and more amenable to train.
In many cases for LORD, the adjoint method and the backpropagation method are comparable in terms of model accuracy. Interestingly, we found that the backpropagation through the Fuler method fast enough with neglectable sacrifice of accuracy for several cases.
Detailed Experimental Results Table show detailed results for some selected evaluation met- rics full tables with all metrics are in Appendix [ One point is that many best outcomes are made with moderate P settings. For gig rms, ODE-RNN can't achieve good scores for all P settings. Our proposed model, LORD, achieves the best scores. In particular, LORD]-3 with P = 32 has much smaller number of parameters, compared with other baselines that have similar performance. LORD2-3 also significantly outperforms NRDE3 for both accuracy and mode size. For CounterMovement
and Sel fRegulate annannno2,,,... DNE-RNN is better than other NRDE and base- baselines. However, LORD marks the best scores in all cases.
In all IMC experiments, LORD shows outstanding performance. LORD2-43 shows the best perfor- mance in almost all cases. Compared with NCDE and NRDE, DE-NRDE has 40 ~ 60% improvements and LORD has 50 ~ 70% improvements. However, LORD'S model size is reduced by 60% whereas DE DE-NRDE's model size is reduced by 20%. Therefore, proposed method is better embedding method for log-signatures. Unlike EigenWorms, the time-series length in this dataset is rather performs better. Overall, DE-NRDE, with P = 128 has the best
short. For that reason, performance, among the baseline models. This means that the training difficulty by the large log- signature size can be alleviated by the direct embedding. However, LORD outperforms DE-NRDE.
BIDMC32HR 1, 3 8)
Additional Results and Visualization Fig. 4 visually compares the three log-signatures and we note that the embedded signature of dc(t) has the characteristics of both D2 refer to our supplementary material for other visualization, detailed results, sensitivity analyses, and reproducibility information.
P: Cagsig dolti
and D2 3. We
5 LIMITATIONS AND CONCLUSIONS
The log-signature transform of NRDEs is suitable to irregular and long time-series. However, the log-signature transform re- quires larger memory footprints and to this end, presented an antoencoder-based method to embed their higher-dimensional log-signature into a lower- dimensional space, called LORD. Our method is carefully devised to eradicate the higher- dimensional computation as early as right after the pre-training of the autoencoder. the standard benchmark experiments of very long time-series, existing methods and shows relatively smaller model sizes in terms of the number of parameters. Nonetheless, it is still in its early phase to enhance NRDES and there exist couple of items to be studied further, e.g., processing long time-series with many channels.
Figure 4: PCA-based visual- ization of log-signatures
proposed method significantly outperforms
8

---

Published as a conference paper at ICLR 2022
Table 4: Detailed experimental results. The best results for each is in boldface and for all P settings additionally with asterisk. Moderate configurations of P are needed to see the best results in many cases.
Accurucy
Macro Fl 32
@Params 32
32
128
128
128
ORE-RNN
0.35+0.01
0.35+0.03
0.40+0.04
0.00
0.15#0.07
0.25+0.11
2.9 10
5.4
x
10*
1.4 x 10
NCDE
0.68+0.10 10
0.76+0.05
0.46++0.0
0.61
0.69=0.07
0.41+0.07
2.1 x 10
2.1
x
10
2.1 x 10
ANCDE
0.81=0.05
0.77+0.03
0.50+0.06
0.77+0.07
0.73+0.06
0.4620.07
9.3 x 10
9.3
10
2.3 10
NRDE2
0.74=0.04
0.7840.03
0.7340.09
0.6640.08
0.7120.04
0.6640.11
6.5 x 10
6.5
10
6 10
0,7249.08
0.8120.04
0.6110.12 12
0.6240.13
0.7920.05
0.54g0.14
3.0 10
3.0
10
3.0
10
DE-N
0.44g00.5
0.7340.07
0.7740.07
0.2340.07
0.6320.14
0.7240.08
4.1 10
4.1
10
4.9 10"
RRCN
0.42+0.07
0.6340.05
0.84+0.04
0,21+0.10
0.53+0.04
0.81+0.03
1.8 x 10
1.8
10
2.4 10
50RD
0.80+0.05
0.84+0.05
0.76+0.06
0,74+0.05
0.84#0.06
0.75+0.08
3.9 x 10'
3.9
10
3.5 10"
LORD1 3+3
0.77=0.09
0.86=0.05*
0.74 08
0.73+0.12
0.85+0.06
0.72+0.09
3.5 x 10
3.5
10
3.9
10
50RMg x33
0.82+0.05
0.84+0.02
0.8420004
0.78+0.06
0.79+0.05
0.82+0.06
1.3 10
1.3
10
1.3 x 10P
P = 64
128
512
64
128
512
64
128
512
Counte wwuneeetioon countertovveeeee sump
ODE RNN
0.47=0.02
0.47=0.09
0.45+0.00 03
0.45#0.05
0.46#0.09
0.44=0.04
2.5 x 10
2.1
x 10
1.4% x 10
NCDE
0.35=0.06
0.40#0006
0.41#0005
0.32+0.09
0.3520.05
0.38:0.06
5.8 10
2.5
10
1.7
10
ANCDE
0.35g0.04
0.4710.06
0.4610.007
0.3420.03
0.4620.07
0.43a0.10
1.4 10
2.5
10
2.9 10
SRCEg
0.40
0.40a0.08
0.40=0.09
0.39
0.38
0.39.0.09
1.1 x 10
1.9
10
5.0
10
NRR
0.47+00.99
0.41+0.08
0.47+0.06
0,46+40.99
0.40=0.08
0.45.0.05
5.3 x 10
5.2
10
2.1 10
BR
0.40=0.07
0. 0.38+0.06
0.38+0.05
0.39+0.08
0,33#0.07
0.36+0.06
5.2
9.1
10
3.9 x 10"
DE NRRDa
0.41=0.07
0.47+0.04
0.42+0.07
0,37=0.07
0,44+0.05
0,40+0.08
2.0 10
2.8
10
7.3 x 10
1ORD1
0.59=0.04"
0.57=00.6
0. 1.50+0.08
0.58
0.56
0.49+0.08
8.7 10
5.1
104
6.5
10a
5ORD
0.56+0.04
0.53+0.03
0.51+0.03
0.55+0.04
0.52+0.03
0.50=0.03
4.3 x 10
1.4
10
8.8 10
LORD2-43 3
0.40=0.05
0.40#0.08
0.40+0004
0.38+0.05
0,38=0.08
0.37=0.02
1.1 x 10
1.5 x
10
3.6 x 10
P' = 32
64
256
32
64
256
32
64
256
Sel F.ngRRggll.o. LnSCCP2
ODE-RNN
0.55
0.58g00.01
0.56.20.04
0.54200.06
0.4620.05
0.4820.04
5.7 x 10'
4.1 x 10
1.4 x 10
NCDD
0.59.0.07
0.52+0.08
0.54+0.06
0.47+0.12
0.4620.16
0.54g0.09
4.2 x 10
2.2 x
10
3.2 x 10
ANCDE
0.60=0.06
0.54+0.04
0.49+0005
0,55+0.05
0.49:0.05
0.43+0.07
3.9 10
x 10
x 10
NRCEa
0.52+0.11
0.56+40.09
0.52+0.11
0.00
0.45=0.06
0.50+0.07
3.2 10
6.2
10
6.2 10
NRCEa
0.54=0.05
0.56+0.06
0.5 0.50+0.07
0,56++0005
0.50=0.12
0,48=0:07
3.4 10
1.7
x
10
3.4
x 10
DE NRDD22
0.59 10
0.55200006
0.53+0.08
0.53
0,5020.08
0. 0.5020.08
1.1 X 10
2.0
10
2.0 10
DE-NRDEa
0.55+0.11 11
0.51+0.08
0.51+0.11
0,57+0.07
0.54#0.06
0.55+0.03
1.1 X 10
5.4
x
10
1.1 x 10
LORD 1-+2
0.57=0.10 10
0.3850.12
0.3320008
0.57
0.61=10.06
0.51=0.08
10
144
10"
7.5 X 10"
LORD 1
0.5720.06
0.62=0.10*
0.53+0.10
0.5020.06
0.5920.11
0.5620.04
1.4 10
1.2
10
1.6 X 10
_ORU2-33
0.6120.06
0.61 10
0.60220077
0.55.0.09
0.5620.07
0.53a0.06
2.2 x 10
5.3
x 10
2.6 x 16*
R²
MSB
#Params 128
P 8
128
512
128
512
512
ODE-RNK
0.57+0.29
0.92+0.01
0.81+0.02
0.41
0.07=0.01
0.1020.02
4 x 10
1.5 10
5,2 x 10
NCDE
0.39:0.04
0.23+0004
0.19
0,59:0.04
0.7420.04
0.78:0.04
5.0 x 10a
5.0
10
5.0 10
BTDMC 32HR
ANCDE
NRCE2
0.44=0.03
0.63+0.04
0.17+0.06
0.67+0.07
0. 0.64+0.07
0.2610001
0.54:0003
0.36+0.04
0.32+0.07
0,8020.06
0.71=0.01
0.34+0.07
4.7 x 10
7.5 10
7.5
4.7
10a
10
7.5 10a
4.7
10a
NRCEa
0.65=0008
0. 0.5800.1 13
0.40+0.13
0,34+0.08
0.40=0.13
0,58#0.13
1.4 x 10
1.4
10
1.4 10
DE NBDD2
0.82:0.08
0.7620005
0.6422002
0,1720.07
0,2320.05
0.3510.02
6.3 x 10
6.0
10
3.9
10"
0.89g0.05
0.93400.2
0.8140.09
0.11
0.0720.02
0.18a0.00 08
1.2 x 10
1.0
10
1.0
10
LORD -+2
0.98:0.01
0.94
0.442:0.04
0.02240.01
0.0620:01
0.54g0.04
2.7
3.4
10
3.4 10
SORD1 13
0. 0.98=
0.9240.01
0.45+0.03
0.0240.01
0.07+0.01
0.53+0.03
2.7 10
6.5
10
5.5
10
5ORD2-33
0, 0.98=0.01
0.95=0.01
0.85=0.04
0.02=0.01
0.04
0.15#0.04
3.6 10
5.2
10
5.4 10
ODR-RNN
0.54+0.20
0.72+0.01
0.65+0.02
0,45+0. 19
0.27+0.01
0.34+0.02
5.4 x 10
1.2 10
3.4 10"
NCDE
0.21+0.03
0.34+0.05
0.24+0.00 03
0.7640.03
0.64#0.05
0.74 03
8.7 x 10
8.7
10
8.7 10
ANCDE
0.30=0.06
0.39
0.27+0003 03
0.68#0.06
0.5920.09
0.70+0.03
4.7 x 10
4.7
10a
5.6
10a
BTMMC32RR
NRCE2
0.2720.04
0.52x0.08
0.50+0.16
0,7020.04
0.4620.08
0,4810.16
1.2 x 10a
1.2
10a
1.2 10a
0.3420.04
0.6540.11
0.42+0.13
0.6420.03
0.34g0.13
0.56G0.13
2.2 x 10a
2.2
10
2.2 10a
EE-NRCB2
0.64+5001
0.5840.06
0.55140.0
0.34#0.01
0.4020.06
0.4340.06
8.8 x 10
1.0
10"
1.0 10"
DR-NRDF
0.70+0.06
0.70+0.02
0.57+0.05
0,29+0.05
0.29+0.02
0.41+0.05
10
1.6
10
1.6 10a
1
0.81=0.02
0.6640.01
0.45+0.04 04
0.18+0.02
0,33+0.01
0.53+0.04
2.7
2.7 10"
4.0 10"
=0RD)-33
0.80=0.02
0.67+0.02
0.45=00033
0,20++0.22
0.32+0.01
0.53+0.03
2.7 x 10a
4.5
10a
4.5
10a
1.RR2
0.80+0.01
0.74+0.01
0.66+40011
0.1900.01
0.25+0.01
0.33+0.01
3.6 x 10
x 10
1.0 x 10
ODE RNN
0.55=0.11
0.90=0.03
0.75 02
0.4620.11
0.10=0.03
0.26:0.03
4 x 10
1.5 10a
5.2 x 10a
MCDE
0.24+0.07
0.21+0.07
0.33+0.04 04
0.79+0.07
0.81#0.08
0.69+0.05
8.7 10
8.7
10
8.7 x 10a
ANCDE
000000.03
0.33+0.05
0.37+0.03
0.71
0.70=0.05
0.65#0.03
4.7 x 10
4.7
x 10a
4.7 x 10a
81086323p0
NRDE2
0.22+0008
0.4725. 10
0.61 18
0,81
0.5520.10
0.4120.19
1.2 x 10a
1.2
10
1.2 10a
NRDDE
0.13g0.09
0.6830.11 15
0.6010.15
0.9020.09
0.33g0.15
0.42a0.19
2.2 x 10
2.2
10
2,2 10
DE-N8DD
0.85g0001
0.74+40.5
0.5910.04
0.1520.01
0,27
0.4340.04
1.0
10"
1.0 10"
DE-NRCE
0.90=0.08
0.93+0.03
0.76+0.07
00
0.08=0.03
0.25#0.07
1.4 10
1.6
10
1.6
10
.ORD
0.97+0.00
0.91+0.01
0. 0.58+0.05
0.03+0.00
0.09=0.01
0.43+0.08
2.7 x 10
3.4
10a
6.2 10
3ORD1-43
0.98=0.01"
0. 0.89:0.03
0.5 0.52+0.04
0.0220.01
0.12=0.03
0.49#0.05
2.7 x 10
2.7
10
3.8
10a
20RD2 +3
0.98:2001"
0.9420.01
0,83:0.03
0.02+0.01
0.0620.01
0.18:0.03
3.6 x 10
5.6
10a
6.9 10

---

Published as a conference paper at ICLR 2022
6 ACKNOWLEDGEMENTS
Noseong Park is the corresponding author. This work was supported by the Yonsei University Re- search Fund of 2021, and the Institute of Information & Communications Technology Planning & Evaluation (ITTP) grant funded by the Korean government (MSIT) (No. 2020-0-01361, Artificial Intelligence Graduate School Program (Yonsei University), and No. 2021-0-00155, Context and Activity Analysis-basdd Solution for Safe Childcare).
REFERENCES
Christopher Aicher, Nicholas J Foti, and Emily Fox. Adaptively truncating backpropagation through time to control gradient bias. In Uncertainty in Artificial Intelligence, pp. 799-808. PMLR, 2020.
Adebiyi A. Ariyo, Adewumi O. Adewumi, and Charles K. Ayo. Stock price prediction using the arima model. 2014.
Lei Bai, Lina Yao, Can Li, Xianzhi Wang, and Can Wang. Adaptive graph convolutional recurrent network for traffic forecasting. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), NeurIPS, volume 33, pp. 17804-1781111 2020.
Shaojie Bai, Zico Kolter, and Vladlen Koltun. Convolutional sequence modeling revisited. 2018.
Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. Gru-ode-bayes: Continuous modeling of sporadically-observed time series. In NeurlPS, 2019.
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In NeurIPS. 2018
Zheng Fang, Qingqing Long, Guojie Song, and Kunqing Xie. Spatial-temporal graph ode networks for traffic flow forecasting. arXiv preprint arXiv:2106.1290 2021.
Tak-chung Fu. A review on time series data mining. Engineering Applications of Artificial Intelli- gence, 24(1):164-18) 181, 2011.
Xi Geng. Reconstruction for the signature of a rough path. Proceedings of the London Mathematical Soccitty 114(3):95-522, 2017.
Sheo Yon Jhin, Heejoo Shin, Scoyoung Hong, Minju Jo, Solhee Park, and Noseong Park. Attentive neural controlled differential equations for time-series classification and forecasting. In ICDM, 2021.
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equa- tions for irregular time series. In NeurIPS, 2020.
Terry Lyons and Weijun Xu. Inverting the signature of path, 2018
Terry Lyons, M. Caruana, and T. Lévy. Differential Equations Driven by Rough Paths. Springer, 2004. École D'E de Probabilités de Saint-Flour XXXIV 2004.
James Morrill, Cristopher Salvi, Patrick Kidger, James Foster, and Terry Lyons. Neural rough differential equations for long time series. 2021.
Gregory C Reinsel. Elements of multivariate time series analysis. Springer Science Business Media, 2003.
Xiaoli Ren, Xiaoyong Li, Kaijun Ren, Junqiang Song, Zichen Xu, Kefeng Deng, and Xiang Wang. Deep learning-basee weather prediction: A survey. Big Data Research, 23, 2021.
Sungyong Seo, Arash Mohegh, George Ban Beiss,, and Yan Liu. Automatically inferring data qual- ity for spatiotemporal forecasting. In International Conference on Learning Representations, 2018.
10

---

Published as a conference paper at ICLR 2022
Xingjian Shi, Zhourong Chen, Hao Wang. Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional Istm network: A machine learning approach for precipitation nowcasting, NeurIPS, 28, 2015.
Sima Siami-Namini, Neda Tavakoli, and Akbar Siami Namin. A comparison of arima and Istm in forecasting time series. 2018.
Daniel Stoller, Mi Tian, Sebastian Ewert, and Simon Dixon. Seq-u-net: A one-dimensional causal u-net for efficient sequence modelling. arXiv preprint arXiv:1911.06399 2019.
Bagnall A. Bergmeir C. Keogh E. Petitjean F. Tam, C. W. and G. Monash Webb, Uea classifica-
tion/regression datasets. http://www hwtp::/wwwwww.......mm wwweeeeeeereeeeeeeee..eeeeeeeeee wtt.://aaaeieeeeerrrr.o...ooo.o.
http://www.
Trieu Trinh, Andrew Dai, Thang Luong. and Quoc Le. Learning longer-term dependencies in mns
with auxiliary losses. In International Conference on Machine Learning, pp. 4965-4974, PMLR, 2018.

---

Published as a conference paper at ICLR 2022
Table 5: The best hyperparameter in ODE-RNN
Table 6: The best hyperparameter in NCDE and NRDE
Hidden Path
CDE Function's is #Hidden
Hidden
Data
Hidden
Layers
Data
P
Path Size
D
2
3
2
3
Movement.Juug
512
128
64
256
64
64
Movemant dump
Councer-
512
128
128
64
256
64
64
256
128
256
128
128
64
64
256
128
64
3
3.2
3
3
3
3
Regulation8CP
Self-
256
32
64
128
64
ReguiationSCe2
Sell
256
64
256
64
256
256
128
128
64
256
128
64
64
256
64
3
2
2
Table 8: The best hyperparameter in DE-NRDE
Table 7: The best hyperparameter in ANCDE
Hidden Path Size
#Hidden Layers
Attention
Compression
#Hidden
Hidden
Data
Channel Size
Data
Ratio
Layers
Sizee
=
2
128
0.5
0.5
128
128
RigenWorms
32
128
3
64
BigenWorms
0.5
128
128
128
128
32
128
0.7
0,7
1
64
128
Counter
128
256
3
128
Counter
0.3
128
128
Movement.
128
0.3
0.5
2
128
64
NovemsatJJup
512
0.7
0.3
64
128
Salf-
256
128
128
Regalation8CP2
128
Self-
64
0.3
0,3 2
64
64
128
RegulaLionSC22
256
0.3
0.3
128
128
2
64
0.7
2
64
128
BIDMC32HIR
128
BIDNC321RR
128
0.7
0.7 1
128
128
512
128
64
128
128
2
64
RIDMC32RR
0.5
0.5
128
128
512
128
3
64
RTOMC32RR
128
0.7
0.7
128
128
512
64
64
128
64
0.5
0,5
128
128
6TDMC32SpO2
128
128
64
512
128
2
64
BIDMC323P02
0,7
512
0.7
0,7
64
64
A BEST HYPERPARAMETERS
Our software and hardware environments are as follows: UBUNTU 18.04 LTS, PYTHON 3.7.10, PYTORCH 1.8.1, CUDA 11.4, and NVIDIA Driver 470.42.01, 19 CPU, and NVIDIA RTX A6000.
A.1 BASELINE
There are 5 baselines in our experiments. Each model has its own hyerparameters. ODE RNN method has hyperparameter of hidden path size. NCDE and NRDE have also the same hyperparam- eter, and in addition, the size of hidden dimension and the number of hidden layers in their CDE and RDE functions. The Attentive NCDE (ANCDE) has all of those hyperparameters and in addition, the attention channel size. In DE-NRDE, there are addition hyperparameters related to embedding, which are the embedding compression ratio, the size of hidden dimension, and the number of hidden layers in the embedding layer.
In (Morrill et al. 202I), there are three models
NCDE and NRDE, and four datasets, EigenWorms, B BIDMC32HR, BIDMC32RR, and BIDMC32SpO0 For these combinations, we set the hyperparameters as reported in (Morrill et al. 2021). In other cases, we find the best hyperparam- eters in the following range. All kinds of hidden size in ODE-RNI NCDE, ANCDE, and NRDE are in (32, 64, 128, 256). The number of hidden layers in those is in (2.3). In DE DE-NRDE, the embedding compression ratio, the hidden size, and the number of hidden lavers in the embedding layer are in (0.3, 0.5, 0.7). (64, 128), and (1,2), respectively. Tables to show the best hyperparameters of the baselines.
A.2 LORD-NRDE
Table 9 shows the best hyperparameter configuration of our method in each dataset.
12

---

Published as a conference paper at ICLR 2022
Table 9: The best hyperparameter in LORD
Method
Data
P
N
N
No
b
hy
ho
CTASK
AE
maraatr
TASK
3
64
64
64
1
X 10
1 10
400
400
NigenNorms
32
3
3
64
64
64
1
x 10
x 10
2000
400
128
3
64
x 10
1 10
2000
400
10
10
Courter-
Movement Jump
128
3
64
64
x
500
400
512
2
64
128
1 1
x
1500
400
32
3
32
32
32
1 X 10
1 10
1000
400
Salf-
DegulationSCP
3
128
128
128
X
10
1 10
500
400
1ORD1-+2
128
X
3
64
64
10
1x
2000
RTDMC32HR
128
32
64
10
X 10
1000
500
512
3
32
64
32
1 X 10
X 10
1000
500
3
64
64
64
1 X 10
1 10
400
2000
BIDMC32RR
128
3
64
64
1 10
1 10
1000
500
10-5
x 10-3
1000
3
64
64
10
10
400
2000
BIDMC329p02
128
3
32
64
32
1 X 10
1 10-
1000
500
512
3
64
64
64
1
X 10
x 10
1000
500
3
64
64
64
1
X 10
1 10
400
400
RigenNorms
3
64
X 10
1 10
2000
400
32
64
10
500
400
Counter
128
3
128
128
128
1 X 10
1 10
500
400
Movement
512
3
32
128
32
1
X 10
1 10-6
500
400
32
2
32
128
32
1
X 10
1 10
500
400
Sell
RegulationSCP2
32
128
1
1
1500
400
1ORD
3
128
3
128
64
128
10
x 10
400
2000
DIDMC32IIR
128
3
32
64
32
1 X 10
1 10
1000
500
512
3
32
64
32
1 10
1 10-
1000
500
3
64
1 10-3
x 10-3
400
2000
BIGMC32RR
10-3
10-3
3
1x
1
500
64
64
1 10
1 10
400
2000
SIDM:32Sp02
128
3
64
64
64
1 x 10
1 10-5
1000
500
512
3
32
64
1
x 10
1 10-5
1000
500
3
64
64
1 X 10
1 10
400
400
NigenNorms
x
2000
128
64
64
1x
1x
2000
400
3
128
128
128
1 10
1 10-6
1500
400
128
2
64
128
64
1 X 10-6
1 10-6
1000
400
Movement Jump
512
3
64
32
64
1
10
1 10-6
1000
400
Self-
3
32
1
x
1x 10
500
400
RegulationSCP2
128
1ORDa >8
256
2
32
128
1 10
1 10-6
500
400
3
64
64
64
1
X 10
1 10
400
2000
128
3
32
64
32
1
10
1 10
1000
500
512
3
64
64
1
X 10-3
1 10-3
1000
500
BICMC32RR
128
3
64
192
x 10
10
1000
500
512
3
128
64
128
1
X
1 10
1000
500
3
64
64
64
1
X 10
1
10
400
2000
BTNМC328р02
128
3
64
64
x
10
x 10
1000
500
3
128
x
x 10
1000
B DATASET
EigenWorms has time-series length 17,984 and a channel which contains the movement data of roundworms. 'he goal is to classify each worm among 5 worm types. MrunterrovementJJme has 4,250 length and 4 channels which means accelerations data of each 3D-axis. Using accelerations data, the type of jump is predicted among types. The object of Sel www.eeeetioonnae.. 2 is to classify whether the subject moves the computer's s cursor up or down, using 8 channels of the EEG data. Its time-series length is 1,153.
13

---

Published as a conference paper at ICLR 2022
For forecasting, three Beth Israel Deaconess Medical Centre (BIDMC) datasets used. Using the PPG and ECG information, each task is to predict a person's heart rate (HR), respiratory rate (RR), or oxygen saturation (SpO2), respectively. The time-series length is 4,000.
C VISUALIZATION
We show other PCA based visualizations of the log-signatures. Using PCA, we extract the distributions of the most important principle components of (LogSigh. D. (x)) )=0
1
L (x)) and de(t). two paths have similar distributions on those components, the information contained by the two paths is similar. In Fig. the distribution of de(t) is some-
where in between the distributions of {LogSigh P2 71 (x)) == and {LogSigrite Pr
(X)) 1 1
1
From the encoder-decoder learning, de(t) can learn the information of (LogSight.... (X) (x)
ElgenWommsDD 1, 2.P 128)
EganWWormjDD, D. 3. 128)
ElgenWomsiDp
2.D, 3.
EigenWormsDD
D2 2.P 32]
3
Coptigh
Loptigh deşi
Loptigh
topiy'
deti
ec vauy
(a) Eigen Worms
(b) Eigen Worms
(c) Eigen Worms
(d) Eigen Worms
BIDMC32HR(D,
BIDMC32HRIO, 1.D, 3
BIDMC32HRID, 1,D,
BIDMC32HRID, 2,0, =))
Loptigh
Logttg
Logfigh
Logtig
Logdig
(e) BIDMC32HR
(I) BIDMC32HR
BIDMC32HR
(h) BIDMC32HR
BIDMC =1.0, -2
BIDMC32RRJD -1 =10,, 2
BIDMES2RRID, -,
BIOMES2RR(D, 2.0 -,
Logrigh
Loptigh
Lnptigh
Inply
delt
Aopiip
de
BIDMC32RR
(i) BIDMC32RR
(k) BIDMC32RR
(I) BIDMC32RR
OD 1.0, - -
BDDC325502ID, = 1.0 = -
BIDMC32SRO2ID, 2.0, 0,++ -
BIDM - 2,0, -
Leptigh
topip'
Loptig
Coptiph
zoptp ded0
desi
dest
Legay Ln dul
(m) BIDMC32SpO2
(n) BIDMC32SpO2
(o) BIDMC32SpO2
(p) BIDMC32SpO2
Figure 5: Visualization of Py (X)
(X))
L and de(t) in
Eigenforms and BIDMC.
14

---

Published as a conference paper at ICLR 2022
Table 10: EigenWorms
Method
P
Accuracy
Macro
Weighted F1
ROCAUC
#Params(D)
#Params(R)
4
0.354x0.011
0.10740.002
0.19340.004
0.511=0.054
Not Applicable
28707
GDE-RNN
32
0.34940.029
0.00000.065
0.23240.064
0.439=0.027
Nat Applicable
53795
128
0.39540.043
0.249%0.110
0.20420.095
0.51720.068
Applicable
139811
0.67740.099
0.610=0.122
0.66520.10
0.11300.018
Not Applicable
21253
NCDS
32
0.75910.047
0.69020.065
0.75640.052
0.90720.016
Not Applicable
21253
128
0.45640.046
0.415+0.066
0.58400051
0.678=0.055
Not Applicable
21253
4
0.81520.046
0.77020.065
0.81010.046
0.95720.012
Not Applicable
92726
ANCIE
0.774x0.033
0.727=0.060
0.770#0.033
0.925=0.021
Applicable
92726
128
0.49720.064
0.436=0.074
0.501#0.063
0.705=0.038
Not Applicable
22806
0.74480.044
0.65560:076
0.725
0.915=0.032
Nor Applicable
64933
0,779x0.029
0.713+0.044
0.774+0.027
0.934-0.011
Applicable
64933
128
0.72840.088
1661100116
0.726
0.908=0.037
Nat Applicable
64933
0,71840.083
0.61G=0.127
0.695#0.101
0.91420.023
Not Applicable
297893
NRDEE
32
0.81010.043
0,78740.051
0.81410.044
0.911=0.020
Not Applicable
297893
128
0.610#0.117
0,544z0.144
0.600 120
0.862=0.063
Not Applicable
297893
0.43640.048
0.301+0.066
0.538=0.045
Applicable
41331
DE NRDE2
32
0.728+0.074
0,63 141
0,711
0.918=0.026
Not Applicable
41331
128
0.76910.068
0.723=0.078
0.770+0.070
0.939=0.016
Not Applicable
49304
0.421
0,206+0 096
0.281+0.091
0.583=0.086
Not Applicable
179371
DE-MRDES
32
0.626*0.047
0.528+0.038
0.585#0.054
0.878=0.036
Not Applicable
179371
128
0.841+0.042
0.810=0.029
0.840+0.038
0.960=0.022
Applicable
241223
4
0.79540.051
0.744+0:051
0.787+0.053
0.948=0.009
24299
38973
IORC
32
0.841+0.053
0.841=0.058
0.843+0.051
0.957=0.016
24299
38973
128
0.75910.064
0.74840.082
0.76010.061
0.900=0.009
24299
34813
0.77440.088
0.73360 122
0.75950.10 10]
0.944.0.023
86907
34813
1
0.862#0.053
0.855=0.058
0.861+0.055
0.963.0.00
34813
128
0.744x0.081
0.716x0.087
0,73740.076
0.892=0.044
86907
38973
0.82140.054
0.783+0.062
0.815+0.056
0.960=0.011
278679
132465
TORD
32
0.841+0.021
0,793+0.053
0.835+0.027
0.963=0.008
278679
132465
128
0.841#0.038
0.823=0.055
0.835+0.049
0.949=0.023
278679
128305
Table 11: CounterNovveenttuung
Method
P
Accuracy
Macro Fl
Weighted F1
ROCAUC
#Params(D)
AParams(R)
0.47430.023
0.453=0.027
0.45520.026
0.589=0:018
Applicable
24737
ODE CDE-RNNI
128
0,47010.093
0,45820.094
0,45910.094
0.617=0.055
Not Applicable
213537
512
0.499000322
0.43620.037
0.43640.039
0.593=0.031
Not Applicable
139425
0.351#0.057
0.31720.091
0.31840.091
0:473:0:024
Applicable
38371
NCD2
128
0.40410.064
0.350.0.050
0.35240.050
0.520=0.007
Not Applicable
25475
512
0,41140.047
0.379:0.064
0.378+0.065
0.496=0.026
Applicable
46723
64
0.35510.041
0,34240.034
0.34210.035
0.52620.033
Not Applicable
135852
ANCDE
128
0.467+0.060
1457=00.05
0.457 065
0.604:0.031
Not Applicable
252332
0.45810.073
0.42840.1 102
0,588=0.074
Applicable
285740
64
0.39660.116
0,391=0.115
0.391 188
0.560=0.089
Not Applicable
107907
NBDE2
128
0.39610.076
0.379=0.079
0.379+0.078
0.513=0.052
Not Applicable
189059
512
0.39640.091
0,38840.092
0.38810.092
0.54620.076
Not Applicable
50435
64
0.46540.088
0.462s0.093
0.46340.093
0.616=0.062
Not Applicable
$29411
NRDFA
128
0.40940.084
0.399:0.081
0.400+0.081
0.573=0.070
Applicable
512
0.465+0.055
0.453=0.050
0,453#0.051
051
0.618=0.045
Not Applicable
2107395
64
0.39840.075
0.390+0.076
0.38940.075 075
0.573=0.076
Nat Applicable
$1910
128
0.384+0.058
0,334+0.071
0.334+0.071
0.567=0.054
Not Applicable
90886
$12
0.37540.054
0.35910.064
0.35920.064
0.525=0.055
Not Applicable
39114
64
0,40910.072
0.37340.073
0.373400074
0.56820.068
Not Applicable
2043000
OR-MRDES
128
0,474x0.038
0.437+0.053
0.43740.054
0.626=0.030
Nat Applicable
279378
512
0.425+0.069
0.396+0.076
0.39840.07 076
0.571=0.059
Not Applicable
730892
64
0.589H0.038
0.580=0:041
0.580+0.04) 0441
0.717=0.023
3262
87191
1ORD
128
0.566+0.061
0,563e0.066
0.563#0.00 066
0.676=0.058
12158
51063
0,50120.080
0.49520.082
0.49520.082
0.63820.053
65399
64
0.555H0.04]
0,551=0.042
0.550H0.042
0.688=0.038
7282
43127
LORC
128
0.52840.029
0.523#0.027
0.52340.027
0.65620.02
53746
137879
512
0.508+0.033
0.499:0.025 028
0.49440.028
0.637=0.020
8338
88439
64
0.39640.049
0.383+0.052
0.38340.052
0.540=0.061
77158
113266
T.RC +3
128
0.39640.078
0.384=0.083
0.385+0.083
0.542=0.068
27110
146737
512
0,400+0.044
0,374+0.022
0,375#0.022
0.537=0.047
31270
36369
D FULL EXPERIMENTAL RESULTS
In Tables 10 to 15
all experimental results are shown. #Params(D) and #Params(R) denote the number parameters in the decoder part and in the rest parts except the decoder, respectively.
15

---

Published as a conference paper at ICLR 2022
Table 12: Sel wrl.flReaaatonSCP
Method
Accuracy
F1
ROCAUC
#Params(D)
#Params(K)
0,55120.054
0.54050.064
0.60340.041
Not Applicable
57375
UDE-KNN
0.57940.012
0.462s0.051
0.564:0.019
Not Applicable
40991
0.55840.045
0.480z0.039
0.533=0.055
Not Applicable
139295
32
0.586+0.071
0.471+0.122
0.577=0.130
Not Applicable
42241
NCDE
0.51640.079
0,459=0.156
0.587=0.129
Not Applicable
214657
0.544100.58 256
0.537x0.087
0.57820.086
Not Applicable
316161
0.60420.006
0.552:0.048
0.64620.041
Not Applicable
391698
0.54420.039
0.49020.048
Applicable
134034
256
0,488+0.05
0.427+0.067
0.461=0.073
Not Applicable
208914
0.51940.11 10
0.513+0.058
0.572=0.092
Not Applicable
322689
0.561+0.090
0,449+0.066
0.575=0.071
Not Applicable
622209
256
0.51640. 110
0.498:0.071
0.56420.096
Not Applicable
622209
0.537+0.052
0.560=0.050
Applicable
0.55840.059
0.504x0.118
0.58640.107
Not Applicable
1710977
256
0.498+0.066
0.483+0.074
0.543=0.079
Not Applicable
3438465
0.52540.130
Applicable
DR-NRDF2
64
+0.51+0.063
0.496+0.081
0.566=0.073
Not Applicable
196747
0.530+0.078
0,501+0.078
0.538=0.088
Not Applicable
0.551+0.100 109
0.570=0.074
0.655=0.076
Not Applicable
1092158
DE-NRDES
64
0.50510.076
0.53910.059
0.56610.110
Not Applicable
542462
0.512+0.100
0.547=0:031
Applicable
32
0.570 105
0.56610.058
0.652=0.059
14572
24361
TORD )2
0.582+0.119
0.61340.057
256
0,530+0.00 081
0.512+0.08|
0.536=0.087
76684
74857
0.572+0.058
0.501+0.056
0.584+0.03
70132
138761
LORD
0.61 8+0.099
0,591 109
0.659=0 094
11888
123241
256
0.526430.10
0,558+0.038
0.607=0.06
278452
161161
0.60730.059
0,55240.087
0.64520.093
260580
216405
L-3D
0.607#0.095
0.555#0.072
0.619:0.060
999684
$34037
256
0.59640.075
0.529+0.055
0.623+0.064
259524
254549
Table 13: BIDMC32HR
Method
n
Explained Variance
MSE
MAE
#Params(D)
MParams(R)
8
0.56Pa0.290 290
0.570=0.299 290
0.41520.279
0.45720.172
Not Applicable
3871
GDE-RNN
128
0.924+0.011
0.925=0.011
0.07340.011
0.167=0.009
Not Applicable
15391
0,182:0.024
Applicable
8
0.38820.042
0.38820.022
0,59020.041
0.552:0.023
Not Applicable
49921
NCDE
128
0.22740.042
0.231=0.046
0.74520.040
0.638=0.017
Not Applicable
49921
512
1.191
0.192=0.041
0.779*0.039
0.654=0.018
Not Applicable
4992
8
0.43720.031
0.440=0.033
0,34210.030
0.52720.019
Not Applicable
47194
0.79620.060
0.666:0.019
Applicable
47194
512
0.26420.011
0.270=0.012
0.70920.011
0.628=0.016
Not Applicable
47194
0.630H0.044
0.631=0.044
0.356E0.049
0:55200.000
Nor Applicable
74689
0.66$20.072
0.665=0.072
0.32320.069
0.40920.047
Applicable
74689
512
0.642a0.071
0.443-0.071
0.34420.069
0.41920.054
Not Applicable
74689
0.650H0.022
0.650=0.882
0.33
0.430
Not Applicable
140737
128
0.58240.131
0.583201131
0.40240.12
0.45820.085
Not Applicable
1407377
512
0.395#0. 131
0.396=0.13
0,582 126
0.535=0.039
Not Applicable
140737
0.82240.077
0.822-0.077
0.171+0.074
Applicable
DE NRDD2
128
0.758+0.051
0.759=0.022
0,233+0.00 049
0.332=0.045
Not Applicable
59589
$12
0.64010.024
0.643=0.023
0,34620.022 023
0.412=0.016
Not Applicable
58885
0. (149
0,88==0.049
0.1 048
0.222=0.050
Not Applicable
T19050
DE-NRDEa
128
0.92740.016
0.928=0.016
0.070=0.016
0.176=0.024
Not Applicable
102538
0.810+0.085
0.810=0.086
0.183+0.00 082
0.299=0.080
Applicable
8
006
0.979=0.006
0.020 008
0.084=0.014
10285
27277
1ORD1-+2
128
0.93740.009
0.937=0.009
0.061
0.119=0.005
3277
34189
512
0.44120.039
0.443=0.041
0.53820.038
0.47920.033
3277
34189
8
01982400005
0.92-00000
0.018
0.07920.008
12645
27277
1ORD
0.92220.012
0.923=0.013
0.07510.012
0.12922.013
65293
512
0.45040.031
0.452=0.029
0.530+0.030
0.47420.007
4613
54925
0.97840.006
0.979=0.006
0.021+0.000
0.078=0.009
15471
35563
TORD
128
0.954+0.006
0.954=0.006
0.045+0.006
0.121=0.007
6095
51915
512
0.84840.043
0.849=0.043
0.146t0.041
0.216:0.028
15471
54283
16

---

Published as a conference paper at ICLR 2022
Table 14: BIDMC32RR
Method
P
R
Explained Variance
MSR
MAF
#Parama(D)
HParams(R)
0,53620. 195
0.540=0.192
0.44750.188
0.49520.124
Applicable
53791
OUE-ANN
128
0.721a0.010
0.722=0.011
0.26920.010
0.36020.006
Not Applicable
122911
0.652=00017
0.337+0:018
Applicable
8
0.215a0.028
0.21520.028
0.75740.027
0.663a0.010
Not Applicable
86913
NCDE
128
0.337=0.053
0.339=0.053
0.63920.051
0.59520.015
Not Applicable
86913
512
0:23640.026
0.237=0.026
0.736+0.025
0.626=0.008
Nor Applicable
86913
8
0.29820.062
0.302=0.044
0,67750.000
0.625=0.033
Not Applicable
47194
ANCDE
128
0,38620.097
0.387=0.098
0.592+0.094
0.57620.036
Not Applicable
47194
512
0.27040.028
0.271=00228
0.70420:027
0.61920.010
Not Applicable
55514
8
0,27420.037
0.74200.077
0.70020:035
0.63720.019
Not Applicable
123969
NRDEE
0.46220.079
0.500:0.035
Applicable
512
0.49840.162
0.498=0.162
0.484 156
0.512=0.075
Not Applicable
123969
0.33740.035
0.338-0.035
0.639+0.034
0.600=0.019
Nor Applicable
222785
128
0.646#0. 138
0.647=0.138
0,341+0.133
0,403=0.087
Not Applicable
222785
512
0.424=0.130
0.424=0.130
0.55650. 126
0.53720.071
Not Applicable
222783
0.644H0.013
0.647:00122
013
0.388=0.017
Applicable
88196
DE NRDE2
128
0.58420.061
0.586=0.006
0.401=0.059
0.440=0.036
Not Applicable
100677
512
0.552+0.063
0.554=0.061
0.432+0.00 061
0,474=0.033
Nor Applicable
99973
0.290=0.053
Applicable
DE-MRDF3
128
0.703+0.017
0,70=-00017
0.287+0.016
0.352=0.015
Not Applicable
164106
512
0.57420.053
0.575=0.054
10.051
0.454=0.025
Not Applicable
162570
8
0. 80820.021
000000022
0,18550.020
0.272:0.011
10285
27277
10RD
128
0.66220.007
0.661=0.007
0.32620.006
0.38620.012
10285
27277
512
0.44810.044
0.4
0,532x0.042
0.52240.012
10285
39757
8
0.79820.017
0.799000117
0.195E0:016
0:27720.014
12645
27277
1ORD
128
00601
0.672=0.015
0.318+00015
0.376=0.012
12645
44973
512
00051
0.453=0.023
0.530=0.025
0.520=0.012
12645
44973
0.800+0.007
0.801=0.007
0.192+0.000 007
0.281=0.008
15471
35563
10RD2-43
128
0.74420.011
0.744=0.011
0.247+0.010
0.315=0.010
15471
136459
512
0.66220.015
0.66420.014
0.32620.014
0.38620.011
46511
103531
Table 15: 3IDMC32SPO2
Method
nt.
Explained Variance
MSE
MAE
#Params(D)
[Params(R)
8
0.551#0.
0.552=0.100 108
0,46450.112
0.487=0.082
Not Applicable
3871
ODE-RNI
128
0.89920.025
0.901=0.023
0.10420.026
0.225=0.035
Not Applicable
15391
0.74920.024
0,25910.025
0.36520.021
Applicable
52255
8
0.24020.071
0.245:0.067
0.78520.073
0.68620.034
Not Applicable
86913
NCDE
128
0.21520.074
0.217=0.072
0.81120.077
0.694200.335
Not Applicable
86913
512
0.328+0.045
0.332=0.045
0.69420.046
0.639=0.057
Not Applicable
86913
8
+0.014
0.313=0.035
0.712+0.035
0.662=0.014
Not Applicable
47194
ANODE
128
0.327+0.049
0.327=0.049
0.696#0.055
0.631=0.017
Applicable
47194
512
0.37240.033
0.37420.034
0.64820.034
0.59220.013
Not Applicable
47194
0,21840.078
0219=00078
0.80740.080 080
0.69420.036
Nor Applicable
123969
NRDD2
0.47023.100 100
0.471=0.100
0.54750.10 103
0.55920.048
Applicable
123969
512
0.607+0.180 180
0.607=0.180
0.406+0 186
0.455=0.123
Not Applicable
123969
0.131#0.091
0.134
0.898 094
0.720=0.027
Nat Applicable
222785
128
0.681+0.147
0.681=0.147
0.330+0. 151
0.410=0.126
Not Applicable
222785
512
0.596#0 181
0.596=0.18
0.417 187
0.46==0.139
Not Applicable
222785
0.854+0.013
0.151+0.013
0.252=0.015
Applicable
DE NRDD2
128
0.740=0.047
0.740=0.045
0,269 049
0.313=0.039
Not Applicable
99973
512
0.586+0.042
0.586=0.042
0.428+0.04 043
0.453=00033
Not Applicable
99973
0.903200079
0303200079
0.10050.088
0.19220.0M0
Applicable
139144
DE-NRDEg
128
0.926400.26
0.927=0.026
0.076+0.027
0.164:0.024
Not Applicable
162570
0.759+0.067
0.357=0.061
Applicable
8
0.975+0.003
0.976=0.002
0.025+0 003
0.096=0.009
10285
27277
10RD1-42
128
0.909+0.009
0.910=0.009
0.094
0.172:0.013
3277
34189
512
0.58320.047
0.583=0.047
0.43120.048
0.43520.024
10285
61549
8
0.981200005
0.98120.004
0.020 005
0.08620..006
12645
27277
1ORD
0.186=0.029
27277
512
0.521
0.522-0.044
0.495+0.04 046
0.459=0.024
4613
38349
0.98140.005
0.981=0.005
0.019 005
0.088=0.007
15471
35563
TORD
128
0.940+0.007
0.940=0.007
0.06240.007
145=0.004
15471
56395
512
0.829+0.027
0.830=0.027
0.17740.028
0.236:0.016
46511
69323
17

---

Published as a conference paper at ICLR 2022
Table 16: Sensitivity of mar iter AE in EigenWorms (P 128)
Method
mar.atera A
Accuracy
Macro
Weighted F1
ROCAUC
0.656x00.044
0.588a0.107
0.649s0.10 100
0.847a0.056
100
0.713+0.042
0.716=0:058
0.713+0.046
0.862#0.023
0.744:0.048
0.727s0.059
0.744a0.051
0.89340.029
LORD1-22
500
0.733+0. 105
0.725=0 104
0.738 111
0.889=0.035
0.774=0.049
0.773+0.070
0.777+0.056
0.908=0.026
900
0.723=0.091
0.702=0.092
0.722=0.085
0.891=0.054
0. 0.60060. 140
0.513=0:215
0.561 188
0.820=0.068
100
0.713+0.056
0.636=0.133
0.694+0.085
0.88420.030
300
0.733+0.069
0.725=0.053
0,73010.063
0.86820.036
L0RD1-+3
0.733+0.039
0.730=0.04g 0.722=0.054
0.872=0.055
700
0.74450.031
0.730=0.043
0,741=0.034
0.872=0.021
900
0,74420.048
0.724x0.040
0.740+0.053
0.88420.015
0.338s0.10 0,477=0.059
0,420=0.078
0.76420.051
100
0.667a0.091
0.591=0.127
0.642a0. 106
0.88820.044
0.790=0.042
0.754=0.060
0.783+0.050
0.927=0.017
TORDg
500
0.826a0.071
0.789=0.113
0.825a0.022
0.95120.029
700
0,84640. 101
0.816#0.126
0.84740. 103
0.928:0.046
0.856+0.067
0.833=0.067
0.85740.067
0.949=0.027
E SENSITIVITY EXPERIMENTS
Fig. aand Table 16 that in general, the performance of LORD is improved as max jiter AB gets larger in EigenWorms (P 128). Our mncoder-decoder structure successfully integrate the lower-depth log-signature and the higher-depth log-signature information into the embedding vector e, and the well-integrated information helps the model perform better. In general, the lower-depth and the higher-depth log-signature information are blended better as the encoder-decoder structure more trained.
2.
0.15
0.50
0.75
mac
niC))
0.05
6:55
24
100
300
516
750
100
0 350
360
503
700
910
156
310
503
703
M03
max, /tee
max 1t0Cx
max, Rtfs
(a) LORD1-2
(b) LORD1-3
(c) LORD2-+3
Figure 6: Sensitivity to mar_iterA
F TRAIN LOSS
Fig. 7 visualizes several training cases for our method and the original NRDE design. In general, our method shows much better stability across the entire training period.
G END-TO-END TRAINING
LORD has two distinguished characteristics its training process. First, LORD uses two-phase training strategy, pre-training and a main training processes. Second, the training process of LORD is not end-to-end. In other words, the decoder is not used and the encoder is fixed in the main training phase. In this section, we test with various end-to-end-training settings for LORD.
There are three possible configurations for the end-to-end- aad--t--edd-raiinng.... The first configuration is training the encoder (rather than fixing it) with LTASK in the main training phase, which is denoted as FineTuning, The second one is Co-Train where both LA and LTASK are used for training the encoder, the decoder and the main NRDE recall that in F FineTuning we only Lr ASK
18

---

Published as a conference paper at ICLR 2022
BIDMC32HRIP-
BIDMC32HR(P 128)
BIDMC32HR(P 512)
NRDE,
NADE,
NADE,
LORD
LORD
LORD
Train LOSS
Train Loos
Train
2010
1502
2502
Epoch
Fpoch
(a)
(b)
(c)
BIDMC32RR/P
BIDMC32ARIP 128)
BIDMC32RR/P 512)
NRDE: LORD
NADE,
NRDF,
LORD, +2
LORD
Train Loss
Train Loss
Train Loos
sie
1.60
3500
56c
1033
1508
1000 3503
Epoch
Epoch
Epoch
(d)
(e)
Figure 7: Train loss of NRDE3 and LORD2
Table 17: Comparison between LORD and its various end-to-end training variations in EigenWorms (P 128)
Method
Accuracy
Macro Fi
Weighted
ROCAUC
0.759+0.064
0.748=0.062
0.760+0.061
0.900=0.000 0.09
PineTuring
0,74420.075
0.72620.086
0,740+0.077
0.882+0042
LORD
Co-Train
0.71840:021
0.68620.022
0.71520.014
0.86520.030
Co-Train (w. pro)
0.67920.035
0.650=0.031
0.669:0.029
0.832=0.013
TORD
0.744+0.081
0.716=0.087
0.737+0.076
0.89240.044
FineTuning
0.69240.055
0.649:0.049
0.680+0.069
0.86520.018
LORD: +3
Co Train
0.74420.036
0.72320.059
0,74120.037
0.86820.02
Co-Train (M. pre)
0,692400069
0.67420.083
0,69420.064
0.839200066
LORD
0.841=000338
0.82320.055
0.835-20.049
0,949=0.023
Funiiinin
0.731+0.061
0.687=0.075
0.714+0.062
0.919+0.005
10RD2-3
Train
0.80840.061
0.772=0.074
0.802+0.071
0.94820.021
Co-Trsin (M.0. o. pre)
0.58110.112
0.49920.145
0.55720.117
0.80120.065
train the encoder. The last method is Co-Train o. pre) which is the same as Co-Train without any pre-training process. The same hyperparameters are used for all end-to-end models.
We experiment with EigenWorms (P 128). Table [o]sows the results. Among the three end- to-end models, Co-Train shows the best performance. Co-Train (w. pre) has the worst performance in general. These results prove that the pre-training makes the main training easier.
Table 18: The best hyperparameter in ODE-RNN
Table 19; The best hyperparameter in NCDE and NRDE
CDE Function's HHideen
Hidden Path Size
Data
P
Path Size
Hidden Size
I Layers
Data
3
128
1
64
256
Charactez-
4
128
Character-
4
128
256
64
256
256
128
2
Trajectory
16
256
Trajectory
16
256
256
128
64
256
3
2
32
64
32
256
128
128
64
64
128
2
2
128
LiveFuel-
Liveruel-
4
64
128
128
128
256
64
3
2
Noi.sturaContant
16
64
MolatureContent
16
64
256
64
256
256
2
32
32
256
256
64
256
128
64
19

---

Published as a conference paper at ICLR 2022
Table 20: The best hyperparameter in ANCDE
Table 21: The best hyperparameter in DE-NRDE
Data
Hididen Path Size
Allidden Layers
Atlention
Compression
Hiidden
Hidden
Channel Size
Layers
256
256
D
==2
2
3
3
Character-
4
0.7
0.3
128
Trajectory
16
256
3
64
Character-
16
0.5
0.5
128
64
Trajectory
32
0.7
0.5
2
128
128
128
128
LiveFuel-
128
128
Leveuul-
Moistureeontent
128
MoistureContent
16
0.3
0.3
2
64
64
32
0.7
64
32
256
3
64
Table 22: The best hyperparameter in LORD
Method
N
N₂
hy
hg
CAE
CTASK
AE TASK
moz_iter
64
10 -8
10-8
Trajactory
Character-
3
64
128
10
1 10
1000
400
L0XD1-12
32
3
3
128
256
128
1 x 10
1 1x100
1000
400
4
3
3
128
128
128
1 10
1 1 10-
2000
400
Liveruel-
Meiscuretonten:
3
256
256
256
1 10
1 1x100-6
2000
400
1500
10
1000
Trajectory
Character-
128
128
10
10-
1000
400
L030
32
3
128
64
128
1 X 10
1 10-9
1000
400
4
3
3
256
256
256
1 X 10
1 10-
2000
400
Liveruel-
Moiscuretonten:
3
64
64
1 X 10
1 10-6
2000
400
128
128
1 X
1500
400
128
Character-
Jeaject
256
256
256
10
10
1000
400
L-3D2-43
32
3
3
128
256
128
1 X 10
1 10
1000
400
Leveeell-
3
3
256
64
256
1 X 10
1 10
2000
400
Mcis.ureContent
3
128
256
128
X 10
10
2000
400
1
X
X
1500
H ADDITIONAL EXPERIMENTS WITH SHORT TIME-SERIES
In this section, we experiment with short time-series datasets. We use ChazacterTrajectory and LiveFuee ue MMissuureooneen
from (Tan & Webb). The object of Character- Trajectory is classify 20 characters using the trajectory information of writing on tablet.
Its length is 182. Li vevuelelll 1o MeistureContent consists of daily reflectance data at spectral bands for predicting moisture rate in vegetation. length is 365. T8blesI81us 22 we summarize hyperparameters. Because of the short time-series lengths, we test P for all except NRDE-based models.
In Table 23 NRDE achieves the best scores among baseline models and LORD achieves the best scores among all methods. In Table 24 DE- DR-NRDE2 the best. Except DE DE-NRDE2, ODE-RNN is the best. However, LORD also has comparable scores. From these results, NRDE and NRDE-based models are empirically useful for short-time series as well.
20

---

Published as a conference paper at ICLR 2022
Table 23: CharacterTrajectory
Method
P
Accuracy
Macro
Weighted
ROCAUC
#Params(D)
#Params(
0.71620.036
0.69720.041
0.712:0.038
0.969 007
Not Applicable
27570
0.950=0.012
0.947=0.013
0,950=0.012
0,998+0.001
Not Applicable
29106
ODE RNN 16
0.96810.000
0.967:0.005
0.96820.004
0.99920.000
Not Applicable
103218
0.96240.009
0.959
0.962s0.009
0,99740.001
Applicable
0,95420.004
0.952a0.004
0.95420.004
0,99850.000
Not Applicable
149844
a.951
0.94820.005
0.951200005
0.998 000
Not Applicable
233620
NCDE
16 0.905+0.020
0.901#0.020
0,905=0.020
0.994x0.002
Not Applicable
93588
0.829+0.012 32
0.818#0.013
0,828=0.012
0.983+0.00 002
Not Applicable
93588
0.959=0.007 007
0095800007
0,95920.007
0.99840.00 000
Applicable
669757
4
0,952+0.008 008
0.950=0.008
0.952=0.008
0,998s0.000
Not Applicable
286845
16
0.89740.009
0.893
0.89610.010
0.994 001
Not Applicable
538173
0.777+0.020 32
0.762+0.020
0.774=0.021
0,981+0.001
Not Applicable
103293
4
0.970:20.000
0.96820.005
0.970
0.99920.000
Not Applicable
795924
NRDE2
16
0.951 008
0.949=0.009
0.95
0,998
Not Applicable
193428
32
0.95040.007
0.94800.008
0.950=0.007
0,9981000000
Not Applicable
98836
4
0.906:000006
0096550.006
0.96020.006
0.99920.000
Not Applicable
274132
NRDE
0.96020.006
0.96020.006
0.961=0.006
0.99920.000
Applicable
531604
32
0.961 40.002
0.959a0.002
0.961=0.002
0.99920.000
Nox Applicable
1088916
0.89820.077
0.89020.084
0.89200078
0.394
Not Applicable
599707
NRDE&
0.907
0.900=00022
0.906=0.012
0,996+0.00
Not Applicable
112281
32
0.897=0.018
0.890200018
0.89620.017
0,995 001
Not Applicable
76187
0.933+0.031
0.926=0.035
0.932=00032
0.99740.00
Applicable
DE NRCEg
16
0,947=0.014
0.943=0.015
0.947=0.014
0,998
Not Applicable
286883
0.91340.013 32
0.90720.014
0.91220.013
0,99640.000 0011
Not Applicable
617891
0.977+40.002
0.97
0.99920.000
235880
T-RDD--2
16
0.955=0.010
0.953#0.010
0.955=0.010
0.998+0.001
12158
96232
0.829 32
0.817=0.077
0.836=0.074
0,985+0.00
40126
261928
4
0.976 005
0.975#0.006
0.976:0.005
0.99920.000
53746
98568
LORD
16
0.959 006
0.957=0.006
0.959=0.005
0.998+0 001
53746
98568
0.81020.069
0.79620.077
0.80820.071
0,983
53746
63560
4
0.901
0.96920.009
0.912000088
0.99920.000
77158
572418
L0RD2-3
3
16
0.971
0.969=0.003
0.971=0.003
0.99920.000
218086
391650
32
0.96420.008
0.963=0.008
0.964=0.008
0.999=0.000
77158
372418
Table 24: Li Live MooiuueeMMMiiio stwrrronerennnn
Method
r
12.
Explained Variance
MSE
MAE
AParam(())
iParams(R)
0.02660.000
0.026:0.005
1.06420.005
0.775:0.001
Not Applicable
25631
0.02910.00 003
0.02910.003
1062200.03
0.77420.002
Applicable
10271
ODE-ANA
16
0.03240.0m 003
0.033+0.003
1058400.00
0.775a0:004
Not Applicable
16415
0.031+0.004
0.031+0.004
1059=00.04
0.77440.003
Not Applicahle
-0.00920.057
0.008820.007
1.00240.040
0.787a0.010
Not Applicable
42241
0.0044a..020
0.003+0.021 020
109840.022
0.785+0.007
Not Applicahle
91521
KCDF
16
-0.043=0.081
-0.43++0.031
140+00.044
0.794#0.011
Not Applicable
42241
0.00910.029 32
0.009=0.029
1083+0.032
0.779+0.005
Not Applicable
660481
20.02500077
0.02420.079
1120200086
0.78520.009
Not Applicable
101714
4
0.014x00.007
0.015
107740.007
0.781
Not Applicable
241938
16
0.003+0.030
0.004+0.030
1889+0.033
0.781#0.010
Not Applicahle
93394
0.006+0.017
0.006:00017
0887900099
0.784+0.009
Not Applicable
177746
4
-0.010=0.043
0.009+0.043
100=0.047
0.785+0.004
Not Applicahle
1284353
INDD2
16
-0.37330.274
0.37340.274
1.5910.2299
0.82420.020
Not Applicable
2502657
32
-0.15 #0.16
+0. 166
1,25840.
0.81640.023
Not Applicable
1240833
4
40.40000,111
4.402003111
1.61810.340
D. 0.830200012
Not Applicable
1710977
NRDEE
1,30042..64
2.522+2.475
86340.0089
Not Applicable
32
-2. 183+4.622
2. 1822446621
3.478+5.051
0. 1786640.74
Not Applicahle
857601
0.04010.007
0.040=0.007
1.04920.000
0.770:0.005
Not Applicable
930650
XRDE2
0.04220.006
0.042:0.006
1.047:0.007
0.768:0.005
Not Applicable
799243
32
0.043=0.005
0.04340.005
1.04620.006
0.76710.006
Not Applicable
902042
10.77851.599
0..7GE1.559
1,94321.747
0.79690.039
Not Applicable
1256207
16
-0.03720.046
0.0362000446
1.13340.051
0.79010.010
Not Applicable
1103486
32
8.65570.831
00.822
1810+0.908
D. 0.829+0.045
Not Applicahle
457191
0.01710:015
0.01820:014
L074400016
0.78420:011
120937
1ORD1-+2
16
1.019+0.007
0.020=0.00R
1.72200.00 008
0.781
216844
441481
32
0.01340.007
0.01420.007
107840.007 007
0.78420.002
216844
441481
0.01840.010
0.018:0.011
107420.011
0.762000006
612148
44148]
LORD
16
0.00240 009
0.00
1091100010
0.790#0.003
136180
70153
0.005$0.011
20005
1.87400012
0.788
278452
4
0.02520.057
0.025200038
1.12041:041
0.79340.004
2081028
648629
16
5+0.022
0.01444..07 027
1009-0.030
0.793+0 004
1016196
957557
0.008+0.002
0.008=0.002
1,08540.003
0.788+0.002
506356
$40181
21


Reviews:
[{"review_id": "yxIb3c8kiqO", "reviewer": "Reviewer_GYq6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I very much like the paper, particularly the main idea of using a pretrained signature-based autoencoder for time series. I believe this is novel and a natural improvement over standard techniques involving higher-depth (log-)signatures, which suffer from a \"curse of dimensionality\" with respect to depth. Whilst the proposed signature-based autoencoder does fit neatly with the NRDE model, I think that it is interesting for time series problems/models more generally.\n\nThe authors demonstrate that their methodology significantly (and consistently) outperforms pre-existing continuous-time models for long time series problems. These experiments are very thorough and compare against several baselines (ODE-RNN, NCDE, NRDE and the recently proposed Attentive NCDEs of [Jhin et al. 2021](https://arxiv.org/abs/2109.01876)) across 6 datasets.\n\nOverall, I believe that this is a strong paper and the proposed methodology would be of great interest to the community. I could not find any substantial weaknesses, and thus only have the following minor points / typos to discuss:\n\n### Minor points\n\n* Autoencoders (and other dimensionality reduction techniques) are commonly used in machine learning. I believe this should be discussed in the introduction.\n\n* I would say that the prevalent example of a control path X is a Wiener process coupled with time (i.e. dX_t = d(t, W_t)), which gives a stochastic differential equation. More generally, it should be clearer that time points $(t_i)$ can be included as an extra channel of the data $(x_i)$ when constructing the path $X$.\n\n* In Section 2, I think NRDEs should be introduced after NCDEs and adjoint backpropagation should be mentioned (since one of the advantages of Neural ODE models is that they can be trained in a memory-efficient manner using an adjoint equation). Likewise, it is worth mentioning whether adjoint backpropagation is used in the \"Training Method\" and \"Experimental Evaluation\" sections.\n\n* As I understand it, the autoencoder takes $\\big(LogSig_{r_i, r_{i+1}}^{D_1}\\big)$ as input and is trained using $\\big(LogSig_{t_i, t_{i+1}}^{D_2}\\big)$ where $D_1 < D_2$. So if the partition used to compute each $LogSig_{t_i, t_{i+1}}^{D_2}$ is finer than $(r_i)$, then some information in $\\big(LogSig_{t_i, t_{i+1}}^{D_2}\\big)$ may not be contained in $\\big(LogSig_{r_i, r_{i+1}}^{D_1}\\big)$. Thus, the claim that \"the decoder should recover the higher-depth log-signature\" on page 3 could be further discussed.\n\n* I am not a fan of the notation $LogSig_{r_i, r_{i+1}}(X(t))$ on pages 5 and 6, since the log-signature transform takes a continuous path as input and $X(t)$ is a vector. Hence I prefer the notation $LogSig_{r_i, r_{i+1}}(X)$, as used in the original NRDE paper.\n\n* $r_0, r_1$ is used instead of $r_i, r_{i+1}$ on pages 5 and 6. Moreover, I think $r_i, r_{i+1}$ should be detailed in Equations (8) and (16) in a similar manner to Equation (3) – that is, it should be clear that $t\\in[r_i, r_{i+1})$.\n\n* NRDE$_1$ is only theoretically identical to NCDE when piecewise linear interpolation is used to construct $X$ (page 7). For example, if the NCDE uses natural cubic splines, then the methods are slightly different. \n\n* I like the authors use of tables on both pages 8 and 9. Perhaps Tables 2 and 3 could use bold text to highlight which models give the best improvements? Similarly, Table 4 could use bold for \"best in column\" and bold text + asterisk for \"best over all models\". (this comment is also relevant to the tables in the appendix)\n\n### Typos\n\n* I would say \"continuous control path\" instead of \"control continuous path\" (page 1)\n* \"calculated\" instead of \"calculate\" (page 2)\n* \"Pre-training\" instead of \"Pre-trianing\" (page 3)\n* \"create\" instead of \"crate\" (page 7)\n* The use of the word \"Appendix\" (pages 2, 6, 7, 8 and 10). For example, I believe \"In Appendix\" should be either \"in the appendix\" or followed by a letter – such as \"In Appendix A\"\n* \"sensitivity experiments\" seems to have a lower case \"s\" (page 18 in appendix)\n* \"the encoder-decoder structure is more trained\" (page 18 in appendix)", "summary_of_the_paper": "The paper seeks to improve upon the recently introduced \"Neural Rough Differential Equation\" (NRDE) model by using an autoencoder to reduce the dimensionality of the input time series. The proposed autoencoder embeds \"log-signature\" information into a lower-dimensional space and can be trained prior to the main NRDE model. This approach is novel, easily understood, and intuitively should outperform previous log-signature based models. The authors demonstrate that it gives significant improvements on long time series problems through several experiments.", "summary_of_the_review": "The LORD-NRDE model proposed by the paper is novel, intuitive and shows impressive performance across a wide varying of experiments. In particular, I find the use of a signature-based autoencoder for time series particularly interesting (in its own right). For these reasons, I would recommend the paper for acceptance into ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "yav3RWuXsiY", "reviewer": "Reviewer_RtnZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "__Strengths:__\n \n- The idea of pretraining the learned features using an encoder/decoder architecture is interesting and a good contribution to the field even though the principle, relying on autoencoders for pretraining is not uncommon. That said, in this case the autoencoder is not balanced (i.e. it is not reconstructing the original input).  \n- Experiments are very thorough covering a wide range of applications and tasks. The empirical study is a useful contribution to this stream of research. \n- The reported performance is good, validating the idea. \n\n_Question:_\n\nHave the authors considered end-to-end training? If no, why not and if yes, what were the reasons?\n\n__Weaknesses:__\n\nThe writing is not very clear in some places. For example, the authors state that the “log-signature transform provides a basis itself” citing Kidger et al. 2019. \n - It is not mentioned what the log signature is a basis of.\n - I cannot find any such statement in the source. The only relevant statement I can find in Kidger et. al. relates to the signature (and its universal non-linearity) __not__ the log-signature. \nCan the authors clarify what they mean? \n\n_Minor:_\n\nThe first sentence is a bit awkward. It is just a long list of 15(!) citations (more than 50% of all sources), mostly from the last two years, to underscore the relevance of time series data. \nFirst, time series analysis is a much older area, so the recency bias is odd. Second, a lot of the sources are unpublished. I feel like the references should be better motivated if they need to stay.  \n\n__Recommendation__:\n\nIn principle, the paper is a good contribution. However, the writing of the paper is sometimes unclear and I am reluctant to recommend acceptance so long as there are (mathematical) statements that I cannot verify. That said, I believe the authors can fix this.\n", "summary_of_the_paper": "The authors propose to increase the efficiency of neural rough differential equations by encoding the log-signature into a lower dimensional space that is predictive of higher order signatures during a pre-training stage. \nThis enables the authors to work with lower dimensional log-signatures during the main training stages. The method performs competitively on a range of long time-series tasks. \n", "summary_of_the_review": "The paper proposes an interesting extension to neural rough differential equations which shows good results in practice and is in principle a reasonable contribution to ICLR. However, the writing lacks detail in some instances (see comments) to an extent that the claims cannot be verified. \nFor this reason, I will stop short of recommending accepting at this point\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "shKGMPOAI1t", "reviewer": "Reviewer_ncTs", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The manuscript is well presented and clear, and the details on experiments are thorough in the main text and would be easy to reproduce, the appendix is also clear and to the point. \n\nThe problem of reducing spatial overhead in signature methods is an important one and is in need of more study, and I believe the manuscript is a fine addition the the literature. I would also like to point the authors to the following paper:\nhttps://openreview.net/forum?id=dx4b7lm8jMM\nwhich tackles the same problem by a different method.\n\nIn the appendix it looks like the autoencoder is trained for as many or many more iterations than the NRDE for a large share of the tasks,  but I can't find any mention on how it affects the training time.\n\nThe proposed model is specifically made for long time-series, but not all the baseline methods are. In the interest of fairness it would be nice if there was some mention (in the appendix) of how well the model compares to the baselines on some shorter time-series too.\n\nSignature and neural CDE methods in ML are up-and-coming fields and the method proposed here is an interesting take on dimensionality reduction for these methods which performs very well compared to the baselines chosen. I think the manuscript is of high quality and believe it should be accepted.\n", "summary_of_the_paper": "The manuscript proposes a new method of training neural RDEs with a pretrained autoencoder to reduce overhead of the log-signature transform when dealing with long time-series. The neural RDE is trained on the image of a pretrained autoencoder as a way to reduce complexity and memory usage.\n\nThe methods is compared to several different baselines, including neural CDE and ODE methods. These are tested on 6 real world datasets for both classification and forecasting.", "summary_of_the_review": "The ideas presented in the manuscript are novel and well presented, and the method seems to drastically cut down on overhead, and will be useful for both classification and forecasting.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "e0uY8nc-Hrw", "reviewer": "Reviewer_yePm", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Discussion\n\n**Main ideas** From an efficiency standpoint, I really like the idea of using a pretrained lightweight encoder / heavyweight decoder. Amortising costs like this is an eminently sensible thing to do.\n\nI would note that I do not think this idea is entirely novel. For any supervised problem learning $x \\mapsto y$, then it is very standard to perform dimensionality reduction (either linearly via PCA or nonlinearly via an autoencoder) on $x$ prior to training. The proposed technique is an instance of this idea.\n\nMoreover I note that the encoder is evaluated on just the low-dimensional log-signatures, so that in principle the information contained in the higher-dimensional terms of the log-signatures has already been lost. Furthermore (in principle) the composed \"main NRDE\"-encoder could be learnt by a single NRDE as originally introduced.\n\nDespite these (substantial) concerns, I am broadly in favour of this paper. The context in which these techniques is applied (long time series, rough differential equations, log-signatures) is already highly nontrivial, and the gains demonstrated are substantial. Moreover a great many papers are written by combining two existing techniques (here, NRDEs and autoencoders), and identifying profitable such combinations is very much worthwhile.\n\n**Clarity** One concern I do have is the clarity of the presentation. I've collected some specific thoughts below, but I believe the quality of the writing can be improved in general, as the mathematical constructions used are generally not too clear. (I would suggest that the authors replicate this by sitting down with a colleague unfamiliar with this work, and have them highlight the points at which the constructions used are no longer clear.)\n\n**Experiments** The experiments are very thorough, including all possible NCDE-derived benchmarks (at least that I am aware of), including ODE-RNNs, NCDEs, NRDEs, the attentive NCDEs of Jhin et al. 2021, and even a \"naive\" alternative to their own work. They consider six different datasets across both classification and regression, and both point-predictions and forecasting. They cover a wide variety of hyperparameters and several evaluation metrics.\n\nI particularly appreciated the use of summary tables (Tables 2 and 3) as a readable summary of results. (I would suggest perhaps bolding the best value in each column?)\n\n**Backpropagation** One important detail that I did not see discussed is the manner of backpropagation. Was this done via optimise-then-discretise (sometimes called just \"the adjoint method\") or via discretise-then-optimise (\"through the solver\")? (Or indeed via any other technique?) This can affect results quite dramatically. DtO is preferred if possible but Morrill et al. 2021 report requiring OtD due to memory constraints.\n\n**The inclusion of time** I do want to highlight one mathematical typo, that is subtle but very important. On page 2, the interpolated path $X$ is claimed to satisfy $X(t_i) = x_i$, rather than (as is correct) $X(t_i) = (t_i, x_i)$.\n\n- Suppose the contrary, and that only $X(t_i) = x_i$. Consider training an NCDE/NRDE just to output the timespan $t_n - t_1$ of a time series $((t_1, x_1), \\ldots, (t_n, x_n))$. If all $x_i = 0$ then any reasonable interpolation scheme will produce $X \\equiv 0$ and so  $z(t) = z(t_1) + \\int_{t_1}^t f(z(s)) d X(s) = z(t_1)$ for all $t$. This is certainly not capable of calculating the timespan $t_n - t_1$.\n- In contrast if time is present in the control ($X(t_i) = (t_i, x_i)$)  then we can take $f$ to be a matrix $[[1, 0], [0, 0]]$ so that $z(t) = z(t_1) + \\int_{t_1}^t f(z(s))dX(s) = \\int_{t_1}^t d s = z(t_1) + t - t_1$. (And then take the initial value network for $z(t_1)$ to be identically zero.)\n- The universal approximation theorem for CDEs (Kidger et al. 2020, Appendix B) assures us that nothing else is missed, and that by including time in the control we are not losing any information unnecessarily.\n\n## Specific points / typos\n\n- Page 2: \"calculate\" -> \"calculated\"\n- Page 2: in mathematical writing writing e.g. \"298K\" should be avoided in favour of \"$2.98 \\times 10^5$\".\n  - Page 2: the connection between number of parameters and dimensionality $D$ is not clear. Nothing stops an end user from applying a small model (with few parameters) to a high-dimensional dataset, after all.\n- Page 2: Equation (2) is referenced alongside the fact that \"dim(s) grows exponential wrt. dim(x)\". However dim(s) refers to the dimensionality of the log-signature, whilst equation (2) is an expression for the signature. (I don't actually know the asymptotics for the log-signature -- in particular whether or not its exponential -- but it is certainly true that its gets quite large quite quickly.)\n- Page 2: The notations dim(s) and dim(x) aren't defined.\n- Table 1: \"trianing\" -> \"training\"\n- Page 3: I think saying that a baseline is outperformed by \"75%\" isn't clear at all. 75% according to which metric, for one thing?\n- Page 3: I believe it should be made clear that $\\{x_i\\}_{i=0}^N$ are a sample _along_ a time series, and not different samples down the batch dimension.\n- Page 3: \"e.g.\" (and similar abbreviations like \"i.e.\") are generally discouraged in academic writing (in favour of using the words \"for example\").\n- Page 3: to be honest, I'd suggest avoiding comparison to the Fourier transform. At first glance these looks like they might be related -- they're both integrals of a path -- but that's about where the commonalities end. They're really very different things.\n- Equation (2): $r_i$, $r_{i+1}$ aren't defined.\n- Page 4: You have now introduced two notations for the same thing, namely $LogSig$ and $s_j$. I would suggest avoiding two different notations for the same object, for clarity's sake. If you absolutely must then make sure to link them with an explicit equality (and not merely the word \"constitutes\"), as again this maximises clarity.\n- Page 4: I would suggest re-ordering the concepts that are introduced: NODEs, NCDEs, (Log-)signatures, NRDEs, LSTI. This helps to build up concepts in the order that will probably be most familiar to the reader.\n- Page 4: NCDEs: I would note that NCDEs conversely have advantages that NRDEs do not. In particular NCDEs can use high-order numerical ODE solvers. (NRDEs are really just about choosing a particular differential equation solver -- namely a low-order one that happens to utilise higher terms in the log-signature.)\n- Algorithm 1: I don't see that including this actually adds anything.\n- Page 5: the constraint $dim(e(t)) = dim(logsig(...))$ doesn't actually seem to actually be necessary. Any low dimensionality would suffice.\n- Equations (10)--(12): I note that a tanh is used before the final fully-connected layer. This was something of an oddity in Morrill et al. 2021; the NCDE formulation of Kidger et al. 2020 placed the tanh *after* the final fully-connected layer instead. As far as I know there's no reason to prefer one over the other, although the tanh-after formulation is more common. I would be curious if the authors have any observations on which is more effective?", "summary_of_the_paper": "This paper suggests hybridising neural rough differential equations (NRDEs) with nonlinear dimensionality reduction techniques (autoencoders). This is an entirely reasonable approach, and by doing so the authors demonstrate impressive improvements across long time series benchmarks.", "summary_of_the_review": "The main idea is sound and the empirical justification is impressively thorough. I have some concerns about clarity/novelty but these are not sufficient to detract from my overall positive impression of this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel method for training neural rough differential equations, a recent model for processing very long time-series data. The method involves a lower-dimensional embedding of the log-signature, which is obtained via pretrained autoencoder to reduce overhead. The results show significant and consistent improvements over previous methods on long time-series data.

Overall, the reviewers and I all agree that this paper offers a novel and impactful contribution leading to significant improvements over previous state-of-the-art methods for training neural rough differential equations. I recommend acceptance.
Decision: Accept (Poster)