Neural Lad: A Neural Latent Dynamics Framework for Times Series Modeling
Authors: Ting Li
Jianguo Li
Zhanxing Zhu
Venue: NeurIPS 2023 poster (2023)
Keywords: Neural CDE
Time-series forecasting
Latent Dynamic

Abstract: Neural ordinary differential equation (Neural ODE) is an elegant yet powerful framework to learn the temporal dynamics for time series modeling.
However, we observe that existing Neural ODE forecasting models suffer from two disadvantages:
i) controlling the latent states only through the linear transformation over the local change of the observed signals may be inadequate;
ii) lacking the ability to capture the inherent periodical property in time series forecasting tasks;
To overcome the two issues, 
we introduce a new neural ODE framework called \textbf{Neural Lad}, a \textbf{Neural} \textbf{La}tent \textbf{d}ynamics model in which the latent representations evolve with an ODE enhanced by the change of observed signal and seasonality-trend characterization. We incorporate the local change of input signal into the latent dynamics in an attention-based manner and design a residual architecture over basis expansion to depict the periodicity in the underlying dynamics. To accommodate the multivariate time series forecasting, we extend the  Neural Lad through learning an adaptive relationship between multiple time series.        
Experiments demonstrate that our model can achieve better or comparable performance against existing neural ODE families and transformer variants in various datasets. Remarkably, the empirical superiority of Neural Lad is consistent across  short and long-horizon forecasting for both univariate, multivariate and even irregular sampled time series.

Neural Lad: A Neural Latent Dynamies Framework for Times Series Modeling
Ting Li Ant Group, Beijing, China 1t3170680antgroup. com
Jianguo Li Ant Group, Beijing, China 1ijg zero@antgroup.ccooooo com
Zhanxing Zhu P
Changping National Lab & Peking University, Beijing, China zhanxing. zhu8pku. edu. cn
Abstract
Neural ordinary differential equation (Neural ODE) is an elegant yet powerful framework to learn the temporal dynamics for time series modeling. However, we observe that existing Neural ODE forecasting models suffer from two disadvan- tages: i) controlling the latent states only through the linear transformation over the local change of the observed signals may be inadequate; ii) lacking the ability to capture the inherent periodical property in time series forecasting tasks; To over- come the two issues, introduce new neural ODE framework called Neural Lad, a Neural Latent dynamics model, in which the latent representations evolve with ODE enhanced by the change of observed signal and seasonality-trend characterization. We incorporate the local change of input signal into the latent dynamics in attention-basee manner and design residual architecture over basis expansion to depict the periodicity in the underlying dynamics. To accommodate the multivariate time series forecasting, we extend the Neural Lad by learning an adaptive relationship between multiple time series. Experiments demonstrate that our model can achieve better comparable performance against existing neural ODE families and transformer variants in various datasets. Remarkably, the empirical superiority of Neural Lad is consistent across short and morioorizon forecasting for both univariate and multivariate irregularly sampled time series.
Introduction
Achieving accurate time series forecasting has been a long-standing challenge for decades in various applications, such traffic flow prediction (36), weather forecasts (8) management of energy consumption ①. economic analysis [10), etc. With their remarkable representation power, deep learning-based approaches are capable of modeling the complex dynamics in time series, and thus dominate in forecasting tasks recently. These methods range from recurrent neural network (RNN
24 28)), family of neural ordinary differential equations [3 27 13) Transformer variants 29 40 5
19
to basis-expansion-bare based models (c.g. N-Beats
(3)).
Our design particularly follows the family of neural ODEs, since it is elegant as well powerful framework to learn the temporal dynamics for time series modeling. Unfortunately, existing variants of neural ODEs, e.g. vanilla neural ODE B. ODE-RNN (77, neural CDEs [3] 22 do not sufficiently characterize the local change of obverved signal and ignore inherent seasonality-trene altributes in time series forecasting tasks. To this end, introduce Neural Lad, where the latent states' dynamics evolve with sophisticatedly designed neural ODE.
member the Neural ODE family,
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

DOBSSbver
Scasendity--rcn innafornaion
outt)
llidken state
ODESoler
Amereon Nenwork
Puts
Paln
f1
Neural ODE
Neural CDE
Neural Lad
Figure 1: The sketch of Neural ODE, Neural CDE and our proposed Neural Lad for constructing hidden states from observations. Latent ODE: simply modifying hidden states each observed data point; Neural CDE: taking the signal path into consideration which makes the hidden state continues; Neural Lad: enhancing the Neural CDE with transformation over input change of signal and time-dependent dynamics.
has the following characteristics tailored for forecasting challenging univariate and multivariate time series.
We incorporate the change of interpolated observations into the latent ODE in an attention- based design to learn more flexible and expressive hidden states.
The latent dynamics function is time-dependent, and formulated as a decomposable form w.r.t. the hidden state, the input signal and time.
To accommodate the multivariate time series forecasting, we extend the Neural Lad through learning an adaptive relationship between multiple time series.
The comparison between our proposed Neural Lad and other related members of Neural ODEs is described in Figure. [ The main difference is regarding how to construct the dynamics for the hidden states. Compared with Neural CDE, we employ a memory based neural network to model how the local change of the interpolated signal path influences the latent dynamics, Further, the latent dynamic transformation particularly tailored for
function is weighted with time-dependent seasonality-trent complex time series. After obtained the enhanced latent dynamics, the same as other variants of Neural ODEs, an ODESolver used to obtain the integral hidden states. For multivariate time-series forecasting, we additionally embed an adaptive graph convolution function to encode the hidden states by incorporating the correlation with other time series (see Section 3). We then demonstrate the empirical superiority of Neural Lad across short and long-horizon forecasting for both univariate and multivariate time series data.
Related Work
Transformer variants for time series. Inspired by the success of transformer-basee networks in CV were proposed to capture the long term temporal dependency in time-series forecasting. Specifically, Informer (40) is an efficient network with ProbSparse attention and generative decoder. Autoformer B4 implemented an decomposition transformer network with an auto-correlation mechanism. FEDformer [41) proposed to capture the global profile of time series with seasonality-tres decomposition. However, transformer-bases models consider the temporal correlation with self-attention mechanisms, where the computational complexity is high particularly when the horizon is long. Although some works like PyraFormer 21 took effort accelerate the attention operations, the performance degraded due to the computation-acooe -accurrcccccccyyy trade-off.
and NLP domains, series of transformer variants 40 34
Linear networks for time series. Different from variants of transformer, works 20
used various types of linear blocks extract features. N-beats 23 proposed to stack multi-layer linear blocks and employed ceasonal-trent basis expansion in each layer to enhance forecasting performance, which more time and memory efficient than transformers. SCINet (20) designed a wwwnsample-conoolv---neee network to capture temporal correlation. Besides, (37] rethought the problem of long-horizon time-series forecasting and found that simple one-layer fully-connected network could achieve
architecture including multiple sample convolutions and interaction

---

comparable even better prediction performance than transformer variants. However, the linear block- based networks are sensitive to data distribution and pyram-pasamettrs according our experiments. preventing them from generalizing to more scenarios.
Neural ODE family. To address the non-uniformly sampled time-series forecasting problem, neural ODE families (26), 6 [3] were introduced to model time series with continuous latent dynamics. ODE-RNN and latent ODE [26] generated interpretable hidden states through the neural ordinary differential equation. Neural CDE [3] interpolated the irregularly sampled observations into continuous path that were then considered in latent dynamics. STG-NCDE [ extended the Neural CDE from univariate to multivariate time-series with graph convolution to capture spatial correlation. Although Neural ODE families are more interpretable and stable than transformers and linear networks, existing Neural ODE models suffer from the ability to characterize the local change of the observed signal and the inherent periodicity-trent attributes in time series forecasting tasks.
2 Methodology
Let (==)T denote the time series, we consider time series prediction problem as
Futt = Ge(z:-Lat) + E,
(1)
where H is the length of horizon for prediction, L is the length of lookback window of historical
time series, is the residual of the prediction, and Ge(() (-) is parameterized neural network that will be elaborated later. Thus the loss function to learn the parameters Q could be the mean square error or other types,
min {<(0) | GG(
To fully capture the intrinsic characteristics of time series data, we propose adopt continuous dynamical system with hidden states 2t to model the entire times series, described with following neural ordinary differential equation,
dz, = (2.--,, 2g = <(xo); It [(x),
dt
where <(-) and ((-) are parameterized neural network that model the relationship 20 To and + 2t, respectively; F()) specifies the dynamics of the hidden state and is also neural network to be learned. We call as Neural Latent Dynamics and its design is critical for modeling the time series which will be presented later.
According to the formulation provided in 3). the solution to this ODE is determined by the initial condition at 2g and the latent dynamics along the integration path, i.c.
= =g + L.
s)ds, & E [0,4]
This neural ODE aims to learn the hidden states all times for time series forecasting, which can be estimated using a numerical ODE solver; and the next step prediction can be obtained by Ezx)
21, .1.... Ze ODESolver(F: (F, zo, (la,...:)) ())
Generally, the latent dynamics F-) can be any complex form with sufficient capacity to model the sequential data, In this work, we propose a crucial design over the choice of latent dynamics function F(-) particularly tailored for time series, i.c. F(z,, To:t, t) is decomposable w.r.t. the hidden state zt, the input signal Tozt and time f, named as neural decomposable latent dynamics. The decomposed F()) in our paper is as:
F(z,, To.r, t)
(6)
where the vector field function fz() expands the hidden state 2t, (t) explicitly models the time-dependency with periodical and trend property, and gr is an attention based network to describe how the change of observed signal influences the latent dynamics. The specification of these designs will be elaborated in Sec. [2.1 and 2.2
We emphasize that our model choice over the latent dynamics F(-) is drastically different from other neural ODE family members which only fo(:;) was considered, as shown in Figure The entire model can be thought continuous analogue of recurrent neural networks with layerwise adaptivity.
3

---

5.5
5.5
50-
0.
0.0
5.2
5-
50
100
150
200
250
50
100 150
220
252
50
100
150
200
290
Figure 2:
Visualization of attention weights. Left: attention weight; Middle: observed time series;
Right: the weights of the 83rd components along time in log scale.
Here the decomposability assumption allows to maintain simple yet effective design over the latent dynamics without loss of expressivity, and its effectiveness will be verified empirically.
We highlight that the explicit dependence on time / could both increase the expressivity and inter- pretability of time series modeling within the neural ODE framework, particularly for the complex time series with strong periodicity,
experimentally illustrated in Section +
To achieve more compact model, the fz(ze; f) is designed network. In the following, we will specify two designs that aim to explicitly describe the how Jo (To:a) and hww (f) depends on the input signal 2o:t and time f, respectively.
simple multiple-layer residual neural
2.1 Attention-based network for modeling the change of input signal
The change of input signal x can directly influence the local dynamics of the hidden states. To incorporate the input signal into the latent dynamics f in differentiable manner, we have rely a continuous input signal. However, the observed times series is either regular or irregular sampled sequential data ( thus, we first need to conduct continuous approximation over the discrete time series. Inspired by the idea of neural controlled differential equation, cubic spline could be natural alternative due its minimum regularity for handling certain edge cases. Concretely, let X denote the cubic spline with knots / = o,..., such that X, = (xe, () E Rd=-1
With the continuous approximation X, typically called "signal path", the dynamic change of input signal could be depicted by the derivative dx/ dt, and the component go (Zoz) in the decomposed latent dynamics Eq (6) is specified as
[XX dt
(x(x)))= =
E Rd
where 9u R2 > RR in our paper is a trainable attention based network to describe how the change of input signal influences the latent dynamics, inspired by the idea of memory networks (30) and matching networks (31)
Concretely, let M Idd., x(d, be the memory matrix to be learned for the path gradient, /X/dt. shared for the entire time series. This matrix aims to memorize representative patterns of the local change along the signal path, stored as dn rows of M that can used further pattern matching. Then, for each time f, an attention-based transformation is employed for weighting the patterns stored in the memory matrix M to obtain the final expressive control signal,
dX dt
= MT softmax M dx dt.
De
(
Our memory-basee based modeling over the local change of the signal path is different from that of Neural CDE [13,, where only the path gradient d'X/d for single time point was used. The introduction of the memory matrix allows the model to combine different patterns of local changes, which could also help to provide more discriminative features particularly for the segments with abrupt changes.
We visualize the attention weights so softmar(A) [ dX Figure 2, where memory size dm = 128 and the length of time series is 250. We can observe that around f = 150 with obvious changes, the learned attention weights are extremely sparse and distinguishing from other segments.
dr. along one time series in Weather datasct in
2.2 Seasonality-trend transformation-ben based latent dynamics
In most of existing neural ODEs, the dynamics function only depends on 2t, not explicitly on the time t. However, in the scenario of time series, particularly for those with complex periodic and

---

trending characteristics, only relying on &n and 2o:t, F(-) might not be sufficient for capturing the hidden seasonal and trendy attributes. To this end, we propose to force the latent dynamic function to explicitly depend on the time I, as described by the decomposed formulation Eq (6). which forms non-autonomous ODE the theory of dynamical system.
With the decomposable latent dynamics, we propose to design the time-dependency function Aww (t) characterize the intrinsic seasonality and trend property based on basis expansion. It is the combination of a periodic function ( parameterized by w<() to model the seasonal characteristics
and polynomial function (p) parameterized by w(p) to model the trend features:
h (
(t)
ii (cos(2*t)at) Tt) + sin(2
(9)
0 =
m
(P) (t)
(10)
1=0
where (1) takes the form of trigonometric basis expansion with pre-specified frequencies (1, 1/2, //4,,,.1/2m,.t m, and my are the number of seasonal and polynomial basis, respec- tively. Then, the time-dependency factor (1) for the latent dynamics accounts the sum of the two terms,
(t) ( (t) ( (t) Then, multiply the vector field output (z)) with the time- sime-dependency function h (t), and define it as the 1-th basic block,
(11)
B(z ,4) = Aww (z)
(12)
We stack multiple layers of the basic block residual manner that is easy to train,
(0)
(1) Ztg Zz 24
B(z()) t). ...,,,,, 2 = 2. 2 B(2 (L-1) .t.
(13)
Therefore, by taking the time seasonality and trend into consideration, the decomposed formulation of F(.) becomes,
dxX
dxX
(Z,, Foot, 1) B(z,, l)g.
Iom 1m (1)So(z))))
(14)
dt
dt
where {0, w. v) are parameters to be learned for the decomposed functions l((), f [zz, 1) and g(auz)) respectively. Finally, the solution of hidden state 24 can be derived as follows, which is amenable to typical neural ODE solver.
24 = 20 + +/ hew (s)
dX
ds,
(15)
ds
where fo(z)) E Rd (dd+t) to accommodate the dimensionality of gu (dX/dt), specified simple residual network.
Multivariate Neural Lad
The multivariate time series forecasting typically aim to model the temporal and spatial relationship between multiple time seriese simultaneously to enhance the prediction performance. Representative models include STGCN (36). AGCRN (2) and NRI [14] among which AGCRN is the most effective and efficient approach. Inspired by AGCRN, we adaptively optimize the graph structure and learn the spatial correlation with generalized graph convolution operation jointly.
We start with the graph convolution time series for each time slice
15 for capturing the spatial dependency between multivariate
(16) where 7, E xxd= represents the 1-th slice of hidden states the N time series, and 6 E Nxds collects the projected states into matrix, D E NxN
AD
bb,
XN is the degree matrix, A gd. xds and b are
5

---

the trainable weight and offset, A E RNNN is the adjacency matrix, and + - 3-AAD- is the low-order Chebyshev matrix. Therefore, [q. is the graph convolution on the hidden states to capture the correlation between multiple time series (i.e. nodes).
To make the adjaceney matrix A more flexible, learnable embedding matrix E E R is designed where each row represents the embedding for each time series and de is the dimension of node embedding. Thus, EE" could measure the similarity between these nodes. Then the normalized adjacency matrix D à AD- à is approximated by the following nonlinear transformation.
D - = ((o(EE))
(17)
where a the ReLU activation function and ò is the softmax operation. We now specify generalized version of the graph convolution in Eq. (T6) improve adaptivity and expressivity. The spatial embedding s for all the times series are defined as follows,
$ = (((EE)))))) Z,A E Rxd.
(18)
where 7 + (000(((FFF))) is the adaptive Chebyshev matrix inspired by AGCRN 22. and we let the dimension of the embedding of Φ, the same as that the latent state 2, for simplicity, i.c. dz. Thus, for i-th time series, the spatial component corresponds to the i-th row of the embedding matrix 4
9 (x) :1
(19)
where 7 = {A, F} denotes the collection of parameters to learned. Then, for each time series, the hidden state 2n incorporating both temporal dynamics and spatial correlation is modeled as:
24 20 S. Now (Z.))
dx ds
ds.
(20)
The (0) is the multi-variate extension of Eq. 15} with the additional spatial component (Z,).
In the encoder stage, we evaluate the time-dependent dynamics h fo())) transformation of the signal path go (dX/dt) then obtain the for all the time steps by ODESolver. In the decoder stage, we generate the hidden states and predicted outputs in a step-by-step or one-shot manner through {(2r).
Discussions
We now summarize several benefits of Neural Lad for times series modeling as follows.
is the first neural ODE framework for time series which the dynamic function explicitly depends the hidden state zt, input Tn and the time L. It can seamlessly fit into the training procedure of neural ODE family and the model may be trained with memory- efficient baased backpropagation even across observations.
2. The explicit dependence on the time & through the decomposable latent dynamics B(z, t) (t) allows our neural ODE to adaptively reveal the periodicity of complex time series. Besides, the gu de learns the weight of the control signal. To extend multi- variate time series, we design the function es, (2x) to learn the latent dynamics from spatial correlation. This can also be verified from experimental part in Section 4
3. When considering the relationship between input and hidden states, we employ the contin- uous approximation of observed discrete time series to facilitate the integration over the continuous. Thus, for the original time series either regularly irregularly sampled, they can be fit into our model for training and inference.
Experiments
We conduct experiments on both synthetic and real-world datasets on univariate, multivariate and irregularly sampled time series forecasting tasks to demonstrate the empirical performance of our proposed approach.

---

Table 1: Comparison between Neural Lad and STG-NCDE synthetic dataset.
HORIZON
STG-NCDE
ONLY A
ONLY 9n
NEURAL LAD
MAR
RMSE
MAE
RMSE
MAE
RMSE
MAE
RMSE
12
2.31
3.85
1.69
3.21
1.44
2.62
1.27
2.38
48
2.55
4.52
1.61
2.79
1.48
2.59
1.47
2.54
96
3.37
5.29
2.28
3.82
2.07
3.57
1.77
3.12
10
VVVVVV
-10
-10
-10
-20
250
500
750
1000
250
500
750
1000
50
100
150
200
50
100
150
200
Figure 3: Left two panels: examples of generated time series. Right two panels: prediction results. Blue: the input signal and ground truth. Green: STG-NCDE Red: Neural Lad.
4.1 Regularly sampled time series prediction: synthetic data
We first implement the multivariate Neural Lad on synthetic dataset consisting of 30 periodic time series with various frequencies and linearly increasing amplitudes. The code for generating synthetic time series follows that of Neural ODE (27). Concretely, 30 time series each with 2304 time steps are generated according to the functions Zt,t ag,t sin (2 + Φ) + n,,t, where time series, @ is the phase, 7 denotes the standard Gaussian noise. The amplitude and frequency ascends linearly with time t, i.c. a,, a,,o + each time series the initial parameters are sampled from uniform distribution, di.0 U[2 10] and bi.o r U (0, 10) to increase the diversity.
the index of each
-amin t and b,, 6,,0 + bman -bmin t, where for Emao Emin
Imoo Emoa-Amit
Two examples of generated time series are shown in the left two panels in Figure 3 We can easily observe that these time series exhibit changing frequencies and amplitudes along time.
Performance. We compare strong baseline STG-NCDE 6 with our proposed model on this synthetic dataset with prediction horizon {12, 48, 96]. The dimension of hidden states, is searched from {8, 16, 32, 64) and we find the best setting is 16. The results are shown in Table. It can be observed that Neural Lad outperforms STG-NCDE on both short and long-term forecasting. Moreover, our method improves it by 45% on the short-term horizon and by 47% on long-term horizon, both of which are large margins. We plot the predicted time series with horizon 90 by STG-NCDE and Neural Lad in the right two panels of Figure. It can be observed that Neural Lad can fit well the overall shape the time series.
Ablation study. We also conduct an ablation study by experimenting Neural Lad only with one component, periodicity (t) or iitention-based transformation of control signal (d /t),, as shown in Table | We find that both of the two components play important roles for enhancing performance. Moreover, the attention-ben based transformation 9, contributes slightly more for this task.
4.2 Irregularly sampled time series classification: PhysioNet sepsis
One advantage of the neural ODE family, including Neural Lad, is that it naturally adapts to the case of irregularly sampled partially observed data. In this part, we apply our approach to the PhysioNet 2019 challenge sepsis prediction [25], including 40,335 time series of variable length from the patients in ICU. Measurements are around 39-dimensional features with an hourly resolution, most of which are missing and only 10.3% of values are observed. The task is binary classification problem to predict whether sepsis is developed during the patents' stay. We follow the experimental settings of [33] where the cases with or without observational intensity
both considered.
Since the datasets is highly imbalanced (5% positive rate), AUC is evaluated for different approaches, as reported in Table 2 We can observe that Neural Lad outperforms other alternatives when considering the observational intensity with low memory usage. Moreover, Neural Lad increases the memory less than 6% on both two classification tasks.

---

Table 2: Comparison between Neural Lad and other neural ODE variants on irregularly sampled dataset, PhysioNet sepsis classification.
TEST AUC
MEMORY USAGE(MB)
MODEL
INTENSITY
No INTENSITY
INTENSITY
No INTENSITY
GRU-ODE
0.852
0.771
454
273
GRU-A;
0.878
0.840
837
826
GRU-D
0.871
0.850
889
878
ODE-RNN
0.874
0.833
696
686
NEURAL CDE
0.880
0.776
244
122
NEURAL LAD
0.897
0.812
Table 3: Univariate datasets: the long horizon forecasting performance. Bold font represents the best accuracy while the underlined means the second best.
ETTml
ETTm2
ETTh2
Weather
Dataset
Horizon
MSE
MAE
MSE
MAE
MSE
МАΕ
MSE
MAE
96
1.214
0.665
0.266
0.328
0.432
0.422
0.259
0.254
192
1.261
0.609
0.340
0.371
0.534
0.473
0.309
0.292
Repeat
336
1.283
0.707
0.412
0.410
0.591
0.508
0.377
0.338
720
1.319
0.729
0.521
0.465
0.588
0.517
0.465
0.394
96
0.672
0.571
0.365
0,453
3.755
1.525
0.300
0.384
192
0.795
0.669
0.533
0,563
5.602
1,931
0.598
0,544
Informer
40
336
1.212
0.871
0.887
1,201
4.721
1.835
0.578
0.523
720
1.166
0.823
3.379
1.338
3.647
1.625
1.059
0.741
avg
0.961
0.734
1.410
0.810
4.431
1.729
0.634
0.548
96
0.505
0.475
0.255
0.339
0.358
0.397
0.266
0,336
192
0.553
0.496
0.281
0.340
0.456
0.452
0.307
0.367
Autoformer
34
336
0.621
0.537
0.339
0.372
0.482
0.486
0.359
0.395
720
0.671
0.561
0.433
0.432
0.515
0.511
0.419
0.428
avg
0.558
0.517
0.327
0.371
0.450
0.459
0.338
0.382
96
0.379
0.419
0.203
0.287
0.346
0.388
0.217
0.296
192
0.426
0.441
0.269
0.328
0.429
0.439
0.276
0.336
FEDformer 4
336
0.445
0.459
0.325
0.366
0.496
0.487
0.339
0,380
720
0.543
0.490
0.421
0.415
0.463
0.474
0.403
0.428
avg
0.448
0,452
0.305
0.349
0.437
0.449
0.309
0,360
96
0.320
0.373
288
0.404
0.427
Crossformer 39
336
0.495
0.515
720
0.526
0.542
96
0.179
0.275
0.161
0.229
192
0.307
0.376
0.220
0.281
MICN
32
336
0.325
0.388
0.278
0.331
720
0.502
0.490
0.311
0.356
avg
0.328
0.382
0.243
0.299
96
0.479
0.419
0.190
0.270
0.328
0.375
0.204
0.238
192
0.384
0.387
0.259
0.314
0.430
0.445
0.263
0.311
STG-NCDE
336
0.420
0.413
0.303
0.339
0.451
0.458
0.273
0.312
720
0.492
0.461
2.115
1.029
0.994
0.689
0.356
0.347
avg
0.444
0.420
0.717
0.488
0.551
0.492
0.274
0.302
DLincar 37
avg
0.403
0.407
0.350
0.401
0.559
0.515
0.265
0.317
LighTTS 38
avg
0.435
0.437
0.409
0,436
0.602
0.543
0.261
0.312
TimesNet
3
avg
0.400
0.406
0.291
0.333
0.414
0.427
0.259
0.287
96
0.337
0.359
0.172
0.258
0.275
0.326
0.162
0.221
192
0.356
0.373
0.242
0.305
0.367
0.391
0.214
0.268
Neural Lad
336
0.396
0.406
0.282
0.333
0.410
0,422
0.265
0.303
720
0.462
0.448
0.404
0.400
0.418
0.446
0.323
0.345
avg
0.388
0.396
0.275
0.324
0.368
0.396
0.241
0.284
4.3 Real-world univaraite and mutlivariate time series prediction
Baseline methods. For long-horizon univariate time series forecasting, we compare Neural Lad with
SCINet, Neural CDE, DLinear [37]. LighTTS [38), TimesNet [33]. MICN [32), Transformer variants

---

Table 4: The multivariate forecasting performance of Neural Lad PEMS datasets.
PEMSD3
PEMSD4
PEMSD7
PEMSD8
Model
MAE
RMSE
MAE
RMSE
MAE
RMSE
MAE
RMSE
GraphWaveNet
19.12
32.77
24.89
38.66
26.39
41.50
18.28
30.05
MSTGCN
19.54
31.93
23.96
37.21
29.00
43.73
19.00
29.13
DCRNN
17.99
30.31
21.22
33.44
25.22
38.61
16.82
26.36
STGCN
17.55
30.42
21.16
34.89
25.33
39.34
17.50
27.09
ASTGCN
17.34
29.56
22.93
35.22
24.01
37.87
18.25
28.06
AGCRN
15.98
28.25
19.83
32.26
22.37
36.55
15.95
25.22
ST-GODE
16.50
27.84
20.84
32.82
22.59
37.54
16.81
25.97
Z-GCNETs
16.64
28.15
19.50
31.61
21.77
35.17
15.76
25.11
ST-WA
15.17
26.63
19.06
31.02
20.74
34.05
15.41
24.62
DSTAGNN
15.57
27.21
19.30
31.46
21.42
34,51
15.67
24.77
SCI-Net
14.98
24.08
19.27
31.27
21.19
34.03
15.72
24.76
STG-NCDE
15.57
27.09
19.21
31.09
20.53
33.84
15.45
24.81
Neural Lad (Ours)
14.67
24.59
18.98
30.94
20.21
33.78
15.28
24.66
including Informer 440. Autoformer [34], Fedformer [41). Crossformer [39] For multivariate time
series considering spatial correlation, we compare our method against other graph-based deep
learning approaches, including STGCN (36), ASTGCN (T.. MSTGCN [12, DCRNN [8] Graph
Wavenet [35], AGCRN [2). ST-WA (7 DSTAGNN 16) Z-GCNETs 4. SCI-Net 20) ST-GODE 9
and STG-NCDE (6.
Univariate time series. Table 3 shows the performance comparison on long-term time-series forecasting. For fair competition, only consider the temporal latent dynamics ignoring the spatial correlation in this experiment. Specifically, we generate the hidden state with (q.T5) instead of 20 We can observe that our proposed method outperforms all the transformer baselines by a large margin. For the model FEDformer that performs better than all other transformer variants (Autoformer, Informer, and Pyraformer). Neural Lad outperforms it on all datasets. For linear baselines, we improved the TimesNet [33 and DLinear [37 on all dataset with the average of different horizons.
The paper Crossformer
39 conduct experiments on different horizons [24, 48, 168), therefore we take it as "_" if there are no results published in the origin paper. In the same way. for dataset (E7Tm2 and ETTh2) without performance published in paper MICN [32] we fill with "_" From the table 3 we can find that although Crossformer performs better at the short horizon, it fails on long horizon prediction. For TimesNet [33], Neural Lad outperform on average of four horizons on all dataset.
Multivariate time series. Table 4 shows that the proposed Neural Lad achieves state-of-the-as performance on the four PEMS datasets MAE metric, and also outperforms most baselines on MAPE and MSE metrics. For the baseline SCINet, although the proposed Neural Lad slightly reduces prediction error than which on PEMSD3 and PEMSD4 dataset, our method reduces MAE by 4.6% on PEMSD7 dataset and by 2.8% on PEMSD8 dataset. For neural ODE family baselines, even though STG-NCDE performs better than ST-GODE by a large margin, the proposed method improves STG-NCDE by 5.8%, 1. 1.2%, 1.6%, 1.1% on the four PEMS datasets, respectively. Besides, our method performs slightly better than ST-WA. Overall, Neural Lad achieves the best performance for multivariate time-series forecasting in Neural ODE family.
5 Conclusion
In this work, we have introduced new variant to Neural ODE family, Neural Lad, particularly tailored for accurate time series modeling and forecasting. We conduct extensive experiments both univariate, multivariate and irregularly sampled time series datasets, Neural Lad has achieved remarkable forecasting performance in all these scenarios. Extending materials about experiment settings, computation cost, and visualizations are presented in Appendix.
In the future, we plan to improve the training efficiency of Neural Lad when considering long look- back window. This might tie with how to conduct a faster autonomous ODE solver. Another direction is to propose more architecture designs for the involved neural networks in latent dynamics model.

---

References
[1] Kadir Amasyali and Nora El-Gohary. A review of data-driven building energy consumption prediction studies. Renewable and Sustainable Energy Reviews, 81:1192-1205, 2018.
[2] Lei Bai, Lina Yao, Can Li, Xianzhi Wang, and Can Wang. Adaptive graph convolutional recurrent network for traffic forecasting. In In proceedings of 34th Conference on Neural Information Processing Systems, 2020.
[3] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
[4] Yuzhou Chen, Ignacio Segovia-Dominguee and Yulia Gel. Z-genets: Time zigzags at graph convolutional networks for time series forecasting. In Proc. 38th International Conference on Machine Learning (ICML 2021), 2021.
[5] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. 2019.
[6] Jeongwhan Choi, Hwangyong Choi, Jechyun Hwang, and Noseong Park. Graph neural con- trolled differential equations for traffic forecasting. In AAAI, 2022.
[7] Razvan-Gabriel Cirstea, Bin Yang, Chenjuan Guo, Tung Kieu, and Shirui Pan. Towards spatio-temporal aware traffic time series forecasting. ICDE 2022, 2022.
[8] Lasse Espeholt, Shreya Agrawal, Casper Sønderby, Manoj Kumar, Jonathan Heek, Carla Bromberg, Cenk Gazen, Rob Carver, Marcin Andrychowicz, Jason Hickey, al. Deep learning for twelve hour precipitation forecasts. Nature communications, 13(1):1-10, 2022.
[9] Zheng Fang, Qingqing Long, Guojie Song, and Kunqing Xie. Spatial-temporal graph ode networks for traffic flow forecasting. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 364-373, 2021.
[10] Philippe Goulet Coulombe, Maxime Leroux, Dalibor Stevanovic, and Stéphane Surprenant. How is machine learning useful for macroeconomic forecasting? Journal of Applied Econometrics, 37(5):920-966 90422099962220 2022.
[11] Shengnan Guo, Youfang Lin, Ning Feng, Chao Song, and Huaiyu Wan. Attention based spatial- temporal graph convolutional networks for traffic flow forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 922-929. 929,999998 2019.
[12] Ziyu Jia, Youfang Lin, Jing Wang. Xiaojun Ning, Yuanlai Hc, Ronghao Zhou, Yuhan Zhou, and Li-wei H. Lehman. Multi-view temiial-emmpral graph convolutional networks with do- main generalization for sleep stage classification. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 19:1977-1986, 2021.
[13] Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems, 15:6696-6707,0 2020.
[14] Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. arXiv preprint 1802.04677 2018.
[15] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference Learning Representations, 2017.
[16] Shiyong Lan, Yitong Ma, Weikang Huang, Wenwu Wang, Hongyu Yang, and Pyang Li. Dstagnn: Dynamic spatial-temporal aware graph neural network for traffic flow forecasting. In Proc. 39th International Conference on Machine Learning (ICMI. 2022), 2022.
[17] Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
10

---

[18] Yaguang Li. Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural net- work: Data-driven traffic forecasting, In International Conference on Learning Representations (ICLR 18), 2018.
[19] Bryan Lim, Sercan O. Arik, Nicolas Loeff, and Tomas Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting. 2020.
[20] Minhao Liu, Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma, and Qiang Xu. Scinet: Time series modeling and forecasting with sample convolution and interaction. Thirty-sixth Conference Neural Information Processing Systems (NeurIPS), 2022, 2022.
[21] Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting, In International Conference on Learning Representations, 2022.
[22] James Morrill, Cristopher Salvi, Patrick Kidger, and James Foster. Neural rough differential equations for long time series. In International Conference on Machine Learning, pages 7829-7838. PMLR, 2021.
[23] Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. In International Conference on Learning Representations, 2020.
[24] Syama Sundar Rangapuram, Matthias Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. desa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018
[25] Matthew A Reyna, Chris Josef, Salman Seyedi, Russell Jeter, Supreeth P Shashikumar, M Bran- don Westover, Ashish Sharma, Shamim Nemati, and Gari D Clifford. Early prediction of sepsis from clinical data: the physionet/computing in cardiology challenge 2019. In 2019 Computing in Cardiology (CinC), pages Page-1. IEEE, 2019.
[26] Yulia Rubanova, Ricky T. Q. Chen, and David K Duvenaud. Latent ordinary differential equations for irregularl-sampleed time series. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[27] Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud, Latent ordinary differential equations for prregularly-samplee time series. Advances in neural information processing systems, 32, 2019.
[28) David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3):1181-111, 2020.
[29] Jaak Simm, Adam Arany, Edward De Brouwer, and Yves Moreau. Expressive graph informer networks. 2020.
[30] Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. End-to-end memory networks. Advances in neural information processing systems, 28, 2015.
[31] Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
[32] Huiqiang Wang, Jian Peng, Feihu Huang, Jince Wang, Junhui Chen, and Yifei Xiao. Mien: Multi-scale local and global context modeling for long-term series forecasting. 2023.
[33] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long, Timesnet: Temporal 2d-variation modeling for general time series analysis. In International Conference on Learning Representations, 2023.
[34] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: transformers with Auto-Correlation for long-term series forecasting. In Advances in Neural Information Processing Systems, 2021.
Decomposition
11

---

[35] Zonghan Wu, Shirui Pan, Guodong Long. Jing Jiang, and Chengqi Zhang. Graph wavenet for deep maatialtemmoral graph modeling. In Proceedings of the 28th International Joint Conference Artificial Intelligence, IJCAI11, page 1907-1913. AAAI Press, 2019.
[36] Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting. In IJCAI, 2018.
[37] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? 2023.
[38] Tianping Zhang, Yizhuo Zhang, Wei Cao, Jiang Bian, Xiaohan Yi, Shun Zheng, and Jian Li. Less is more: Fast multivariate time series forecasting with light sampling-oriente mip structures, 2022.
[39] Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In International Conference on Learning Representa- tions, 2023.
[40] Haoyi Zhou, Shanghang Zhang, Jieqi Peng. Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In The Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Virtual Conference, volume 35, pages 11 1100-11115.,.111111111111111--11 AAAI Press, 2021.
[41] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Proc. 39th International Conference on Machine Learning (ICML 2022), 2022.
12


Reviews:
[{"review_id": "py6Hi8jzB0", "reviewer": "Reviewer_gsAa", "summary": "This paper propose a novel neural ordinary differential equation framework for time series modeling. The main contribution is in the design of latent dynamic function $F(\\cdot)$ which can be decomposed into hidden state dynamics $f_\\theta(z_t)$, time-dependency with periodical and trend property $h_w(t) $, and attention-based network to model signal to latent dynamics effect $g_v(x_0:t)$. \nAlso, they showed that the model can be extend to a multivariate time series forecasting. The proposed design is theoretically sound and authors proved them to be effective on synthetic and real data. ", "questions": "- Synthetic data seems to have only trend dynamics. It would be nice to see the result from seasonal(periodic) dynamics\n", "limitations": "N/A", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written and the contribution is well stated. \n- The proposed design is compared thoroughly with different types of baseline in various synthetic/real data and outperforms baseline in most cases.", "weaknesses": "- Analysis on the computational complexity/cost is missing", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer's recognition on the novelty and contribution of our work. We address the concerns in the following. \n\n\n**Weaknesses Part**:\n\n**Q**: Analysis on the computational complexity/cost is missing\n\n**Reply**: We add the complexity analysis and computational time comparison with other approaches, as shown in Figure 1 in the attached pdf file. We run all experiments on a Tesla a100-80g GPU. The training time of Neural Lad for the toy dataset is about 8s per epoch, the forecasting time is 0.4s for every validation iteration (0.4/265=0.0015s).  For large real-world traffic datasets (such as PEMAS03, PEMAS04), the training time is 2-3 minutes per epoch. We also visualize the training process on Toy dataset in Figure 1 .\nWe can observe that the NeuralLad converges faster than STG-NCDE, so it achieves  better performance much earlier than the baseline. \n\n\n**Questions Part**:\n\n**Q**: Synthetic data seems to have only trend dynamics. It would be nice to see the result from seasonal(periodic) dynamics\n\n**Reply**:\nThe generative formula of the synthetic data is $x_{i,t}= \\sin(2\\pi \n b_{i,t} t + \\phi) + n_{i, t}$, **including both periodicity and trend dynamics**,  where the changing of frequency $b_{i,t}$ and amplitude $a_{i,t}$ represent the seasonality and trend,  respectively. "}, {"review_id": "o5mtoWQLkN", "reviewer": "Reviewer_LTR9", "summary": "This paper presents a new framework for modeling time series using a controlled latent neural-ODE-based dynamics model. The proposed latent dynamics function uses a special factorized structure, which effectively disentangles the influences of time (via a periodic basis expansion to capture periodic patterns), the current latent state, and the input signal's history (leveraging an attention-based architecture). Both uni- and multivariate versions of the model are presented. Numerical validations are conducted across a variety of tasks and benchmarked against a number of neural ODE and transformer variants.", "questions": "* Can you comment on the cost aspects (i.e. training, inference and memory footprints) of the Neural Lad models? I am curious how it compares to the baselines tested in the paper.\n* The notation across Equations (13-14) is slightly unclear to me. How does $B(z_t, t)$ in equation (14) relate to the $B$ operations in Equation (13)?\n", "limitations": "Not much discussion on the limitations.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The paper is generally well written and easy to follow.\n* The numerical validation is quite extensive, spanning over tasks of different natures and (short- and long-term) prediction regimes. The proposed method is compared against a wide range of variants in the neural ODE and transformer model families.\n", "weaknesses": "* The idea itself is not the most novel, involving a simple factorization and model architectures drawing cues from popular ones already existing in the literature. However, this is largely compensated by the exhaustive numerical validations where consistent improvements are observed.\n* Discussion on the computation costs and potential limitations is missing.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer's recognition and valuable comments on the contribution of our work. We address the concerns in the following.\n\n**Weakness Part:**\n\n**Q:** \"The idea itself is not the most novel, involving a simple factorization and model architectures drawing cues from popular ones already existing in the literature. However, this is largely compensated by the exhaustive numerical validations where consistent improvements are observed.\" \n\n**Reply:** We admit that our model is not a completely new model, since it is a new member of neural ODE family. However, **the idea of constructing a decomposable design for latent dynamics for time series modeling is novel and effective** according to our knowledge over literature. We emphasize that our model choice over the latent dynamics F(·) is drastically different from othe neural ODE family members in which only $f_θ(z_t)$  was considered, as shown in Figure 1. The entire model can be thought as a continuous analogue of recurrent neural networks with layerwise adaptivity. Here the decomposability assumption allows us to maintain a simple yet effective design over the latent dynamics without loss of expressivity, and its effectiveness has been verified empirically. \n \nCompared to Neural CDE (Kidger et al 2020,) and STG-NCDE (choi et al 2022), one contribution is to model the explicit time dependence on time $t$ with the decomposable time dynamics. In Figure 5 of Appendix, we can observe that learned linear weights and convolution weights are more sparse than Neural CDE by considering the seasonal and trend time dynamics, which indicates that the proposed component can capture the hidden dynamics better so it is not necessary to fit the future with more parameters.\nThe other contribution is to use a memory network on the control gradient to capture the local change of the observed signals, which can improve the performance with a large margin (shown in the Table 1), and model the sudden change of input signals (visualized in Figure 2).\n\n**Q:** \"Discussion on the computation costs and potential limitations is missing.\"\n\n**Reply:** We add the complexity analysis and computational time comparison with other approaches in Figure 1 in the attached pdf file. We run all experiments on a Tesla a100-80g GPU. The training time of Neural Lad for the toy dataset is about 8s per epoch, the forecasting time is 0.4s for every validation iteration (0.4/265=0.0015s).  For large real-world traffic datasets (such as PEMAS03, PEMAS04), the training time is 2-3 minutes per epoch. We visualize the training process on Toy dataset in Figure 1. We can observe that the NeuralLad converges faster than STG-NCDE, so it achieves better performance earlier than baselines. \n\n\n**Questions Part:**\n\n**Q:** Can you comment on the cost aspects (i.e. training, inference and memory footprints) of the Neural Lad models? I am curious how it compares to the baselines tested in the paper.\n\n**Reply:** Compared to its main baseline neural CDE, our model does not increase too much overhead; remarkably, NeuralLad converges faster than STG-NCDE, so it achieves better performance earlier than the baseline, as shown by  Figure 1 in the attached pdf file. \n\n**Q:** \"The notation across Equations (13-14) is slightly unclear to me. How does $B(z_t, t)$ in equation (14) relate to the  $B$ operations in Equation (13)?\"\n\n**Reply:**  The basis expansion component $B(z_t, t)$ in eq.(14) is a multi-layer stacked residual network as shown in eq.(13),  where the $L$ is the number of layers of the basis network. In addition, the structure of $B(z_t, t)$ is shown in Fig. 1.(a) in Appendix. \n\n"}, {"review_id": "RbIO4kQgdc", "reviewer": "Reviewer_GHEt", "summary": "This paper addresses the problem of characterizing the local change of observed signals and ignoring inherent periodical property in time series forecasting tasks. A new neural ODE-based framework is proposed with 1) a decomposable latent space for time-dependent dynamics and 2) an attention-based design for local changes in observation. The framework is further extended to multivariate settings using graph-based networks to adaptively learn the spatial correlation. Experiments have been presented on both univariate and multivariate settings to demonstrate the improved forecasting performance of the proposed model.", "questions": "Please check the weaknesses mentioned above.", "limitations": "N.A.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The presentation of the dynamic model is clear and well-structured. The attention-based networks handle the local change of the signal and the time-dependency function characterizes the seasonal and trend properties of the signal.\n2. Empirical results show improved forecasting performance of the proposed model on both synthetic and real-world datasets.", "weaknesses": "1. I feel that the approach is somewhat incremental from the perspective of the methodology in that it is an extension of the previous works (e.g. Kidger et al 2020, Choi et al 2022), in combination with a decomposable form of latent dynamics and attention-based feature extractor. Could the author elaborate more on how the proposed method differs from the previous works?\n2. Experimental settings could be more elaborated: 1) The motivation of choosing baseline models in Table 2, Table 3, and Table 4, and how the proposed model improved could be more detailed; 2) The author should also provide more information about datasets used in the two univariate and multivariate settings in terms of their local property or seasonal features. I also think the ablation study is not sufficient: 1) There should be evidence to prove the benefits of using attention-based networks for local changes; 2) The detail of improvements in spatial relationships should also be provided.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer's recognition and valuable comments on the contribution of our work. We address the concerns in the following.\n\n**Weakness Part:** \n\n**Q:** \"I feel that the approach is somewhat incremental from the perspective of the methodology in that it is an extension of the previous works (e.g. Kidger et al 2020, Choi et al 2022), in combination with a decomposable form of latent dynamics and attention-based feature extractor. Could the author elaborate more on how the proposed method differs from the previous works?\" \n\n**Reply:** As shown in the reply to reviewer LTR9, the Neural Lad is a new member of neural ODE family. The idea of constructing a decomposable design for latent dynamics for time series modeling is novel and effective according to our knowledge over literature. We emphasize that our model choice over the latent dynamics F(·) is drastically different from othe neural ODE family members in which only $f_θ(z_t)$ was considered, as shown in Figure 1. The entire model can be thought as a continuous analogue of recurrent neural networks with layerwise adaptivity. Here the decomposability assumption allows us to maintain a simple yet effective design over the latent dynamics without loss of expressivity, and its effectiveness has been verified empirically.\n\nCompared to Neural CDE (Kidger et al 2020,) and STG-NCDE (choi et al 2022), one contribution is to model the explicit time dependence on time $t$ with the decomposable time dynamics. In Figure 5 of Appendix, we can observe that learned linear weights and convolution weights are more sparse than Neural CDE by considering the seasonal and trend time dynamics, which indicates that the proposed component can capture the hidden dynamics better so it is not necessary to fit the future with more parameters.\nThe other contribution is to use a memory network on the control gradient to capture the local change of the observed signals, which can improve the performance with a large margin (shown in the Table 1), and model the sudden change of input signals (visualized in Figure 2).\n\n\n\n**Q:** \"Experimental settings could be more elaborated....\" \n\n**Reply:** \n1) The principle of selecting baselines for comprison is that we identify recently works that have competitive and even state-of-the-art prediction performance. Concretely, in Table 2, the PhysionNet sepsis classification task, we choose the same baselines as Neural CDE. \nIn Table 3, for the univariate time-series forecasting, we choose three kinds of models including the widely-used transformer models (Autoformer, Fedformer), the light-weighted linear network (LightTS, DLinear) and Neural CDE network (STG-NCDE). \nIn Table4, for the multi-variate time-series datasets, we choose the graph time-series model such as STGCN, AGCRN, DSTAGNN and STG-NCDE as baselines.\n\n2) Regarding the details of used datasets for testing, we will follow your advice and  add more description to make them more self-contained in the revision. Thank you for your valuable suggestions. \n\n3) Regarding the ablation study:from the performance perspective, we show the benefits of attention-based network in Table 1. Specifically, the $g_v$ in Table 1 is the memory network component, we can observe that the MAE degrades from $2.31$ to $1.44$ on horizon $12$ and from $3.37$ to $2.07$ on horizon $96$ by only adding $g_v$ to STG-NCDE, which demonstrates the benefits of memory network.  From the explanation perspective, we visualize the time-series and attention weight of the memory network in Figure 2 in Appendix, we can observe that the learned weights are extremely sparse and distinguish from others when the time-series has sudden changes. \n\nThe benefit of considering spatial relationship has already been verified by STG-NCDE  that can improve the performance of Neural CDE with a large margin on the traffic dataset, for example, STG-NCDE improves Neural CDE from $20.44$ to $15.57$ on PEMS3, and from $26.31$ to $19.21$ on PEMS4. Neural Lad can be seen as a non-trivial extension over STG-NCDE, which also enjoys the advantage of considering spatial relationship, as shown in Table 2. \n\n"}, {"review_id": "jv44s9wHoz", "reviewer": "Reviewer_85MJ", "summary": "The authors propose a new neural ODE framework, Neural Lad (Neural Latent dynamics model), that decomposes the differential function in three components: a NN differential function, an attention-based network,  a time-dependency function and a graph convolution network for spatial correlations. They evaluate the method on short and long-horizon forecasting tasks for both univariate and multivariate time series data. The proposed method outperforms or performs on par with the existing Neural ODE, time-series transformer, and graph NN based models. \n\n\n", "questions": "Eq.6 The differential equation F() is decomposed in 3 components, differential function, attention-network, time-dependency network. This function, F() is used in the ODESolver that iteratively class this function for dt increments. For every increment, the model goes through some number of blocks (L), where the output of the differential function is scaled by the time-dependency function. Could you please clarify what is the output of $h_w(t)$, and why would you scale the state-dependent dynamics by it? \n\nDo you have some additional results where it can be seen that this multiplication identifies seasonal trends as seen in the data?  \n\nIs the extrapolation (forecasting) autoregressive or you have to pre-define the output sequence length? \n\nFor Eq.8 from self-attention perspective we have keys, queries in values. In relation to your work, I would identify the values as $\\frac{dX}{dt}$ rather than M. Why do you apply the softmax weights to the memory matrix rather than the signal path, which you subsequently use to update the vector from the differential equation?\n\n\nWhat is the number of parameters, complexity, compute time of the proposed method? As cubic spline is an approximation method as well as the time expansion results in additional residual connections, how does it compare to existing NODE models and the transformer architectures? \n\nExperiments.\nFor all experiments, in the final version, I would like to ask the authors to also report the std across multiple runs of the model. \n\nIt is unclear how many timepoints were taken as input to for example to compute the attention-based network output? \n\nFor simulated data, are there more visualization that showcase that the model has captured the linear trend in the data? \nSimilarly, for the ablations on the simulated data when the $h_w$ component is removed, does the model then fail to capture the linear trend? \nIs there a clear pattern that the $g_v$ function is beneficial for time-series that have sudden changes, like weather forecasting? \n\nAs the authors have changed the RHS of the differential equation, is there a theoretical proof that $f_\\theta(z_t)$ still learns the correct underlying dynamics? What are the necessary/sufficient conditions under which this is the case? \n\n\n\n", "limitations": "The authors have briefly touched upon the training efficiency of Neural Lad, however, I would recommend the authors to include a more thorough discussion on this, especially as  in the related work section this is mentioned as the major draw back for transformer based models, however, it is not explicitly discussed for the model at hand. This is information is also missing in the supplementary material. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: \nThe work is a novel combination of known techniques and it is clear how it compares to existing work, for example,  Neural CDEs. \nThe related work section is well organized and all the related methods have been addressed, providing an adequate overview of the related work for the reader. \n\nClarity: \nThe submission is very clearly written. Therefore it is easy to follow the proposed method.  \n\n\nSignificance: \nThe authors provide a novel NODE based method for long-horizon forecasting \n\n", "weaknesses": "Quality: \n\nThe authors introduce a function $h_w(t)$ which should extract seasonality trend. However, neither mathematically nor experimentally, it isn't clear whether this function really achieves this. Traditionally trend is extracted as the mean value of the data, while seasonal part would be the remaining time series data, when trend is removed. From the introduced equations, where the output of the differential function is scaled by the time-dependency factor is at the moment unclear how this is achieved. \n\nSimilarly, the authors introduce an attention-based network to model change of the input signal, but the obtained weight is multiplied with the 'memory matrix' rather than the input signal itself, thus it is unclear, what information does this memory matrix has learned that reflects the input signal. This matrix is also not visualized. \n\nClarity:\n\nFor eq. (1) and eq (2) I would recommend the authors to update the prediction to $\\hat{x_{t:t+H}}$. So then in eq. (2) L2 loss is between the true observations and the predictions by the network. \n\nLine 92, the relationship for the fist function $\\xi$ seems to be in the wrong direction, given whats defined in eq. (3). \n\nFor the experiments section there are crucial points missing: how many datapoints used as input, dataset details, etc. This information should be provided in the supplementary material. \n\nUnder the hyperparameter section there are also missing details on what type of solver was used, solver parameters, what type of optimizer, etc. \n\nSection 2.2. the block residual architecture is unclear from the text, I would recommend moving the figure from the appendix to main text. \n\nSignificance: \nThe results show that the proposed method indeed achieves an improved performance compared to the existing models. But I would like that the authors would have provided some further analysis, results on the different decomposition components, to strengthen their claims of the purpose of the different functions. Moreover, from a theoretical standpoint it is unclear under which conditions $f_\\theta(z_t)$ still remains the correct underlying differential equation. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer's recognition and valuable comments on the contribution of our work. We address the concerns in the following.\n\n**Weakness Part:**\n\n**Quality:**\n\n**Q:** The mechanism of time-dependent component\n \n**R**: The effectiveness of the designed time-dependent function $h_w(t)$ could be verified from various aspects. \n\n**i)** Empirically, according to our ablation study, Table.1 shows that using $h_w$ for extracting seasonality and trend  could improve the prediction performance. \n\n**ii)** From the perspective of underlying mechanism, typically seasonality is  a regular, cyclical, recurring fluctuation. Therefore,  we propose to constrain the seasonality component of $h_w(t)$ to belong to the class of periodic functions, and the natural choice for the basis is the Fourier series. The trend is that most of the time it is  a slowly varying function. In order to mimic this behaviour we propose to constrain to be a polynomial of small degrees, a function slowly varying across forecast window. Our modeling over seasonality and trend is reasonable and different from the traditional one  using mean of time series and its residual. \n\n**iii)** With this careful design, the learned latent dynamics of $z_t$ could learn more sophisticated details than that without such design, as demonstrated in Figure 4 in Appendix. \n\n\n**Q:** The mechanism of attention-based network\n\n**Reply:** Our attention-based modeling over the change of the observed signal is inspired by memory networks [30] and matching networks [31]. The memory matrix $M$ should be learned,  and can be seen as a memory bank, where each row represents a particular pattern describing one aspect of input signal's local change.  The introduction of  the memory matrix allows the model to combine different patterns of local changes, which could also help to provide more discriminative features particularly for the segments with abrupt changes. In Fig. 2 of Appendix, we visualize the memory weight and the input signals, we can observe the learned weights are obviously distinguished from others at sudden changed points. \n\n\n**Clarity:**\n\n**Q:** clarity of eq. (1) and (2)\n\n**Reply:** We would follow your advice for the notation, where actually the prediction $\\hat{x}_{t:t+H} = G_{\\Theta} (x_{t-L:t})$. \n\n**Q:** the wrong direction in eq. (3).\n\n**Reply:**  We apologize for the typo here.  The last equation of (3) should be $x_{t:t+H} = \\xi(z_t)$, denoting the decoding process. We use the learned dynamics $z_t$ to predict the following $H$ steps, as illustrated by Figure 1 in Appendix. \n\n\n**Q:**  Significance of the proposed components \n\n**Reply:** From the performance point of view, we present an ablation study in Table 1, from which we can observe both the time-dependent $h_w(t)$ and memory-based component for input signals improve performance.\nAlso according to the visualization, we can see that the learned weights are more sparse than Neural CDE in Fig. 5, indicating Neural Lad tends to avoid overfitting.  \nFrom a theoretical standpoint, we think the time-dependent component $h_w(t)$  makes the model learn the weights of basis expansion instead of direct fitting, therefore the network weights are sparse than neural CDE.\nMoreover, CDE only deals with the case when the hidden dynamic is linear to the change of input signal, which may fail when the control signal is non-linear.  Therefore we propose an attention-based network to model the local change of observation signals. \n\n**Questions Part**: \n\n**Q:** \n\"...Could you please clarify what is the output of $h_w(t)$, and why would you scale the state-dependent dynamics by it?\"\n\n**Reply:**  The output of  $h_w(t)$  is a scalar, indicating the instant strength of periodicity and trend. This scaling is one modeling choice of incorporating the periodicity and trend into the dynamics of latent z. \n\n**Q:** \n\"Do you have some additional results where it can be seen that this multiplication identifies seasonal trends as seen in the data?\"\n\n**Reply:** Empirically, according to our ablation study for synthetic datasets with very strong peoridicity and trend property, the experiment results in the Table.1 shows that using $h_w$  could improve the prediction performance. Additionally, with the employment of this scaling, the learned latent state $z$ shows more fine-grained details, as plotted in Fig. 4 of Appendix.\n\n\n**Q:** \nRunning time of Neural Lad.\n\n**Reply:**  We add the complexity analysis and computational time comparison with other approaches, as shown in Figure 1 in the attached pdf file. \nWe can observe that the NeuralLad converges faster than STG-NCDE, so it achieves better performance earlier than baselines. \n\nFor the prediction accuracy comparison with existing neural ODE and transformer models, we show them in Table 1-4 for different tasks in the main paper. \n\n**Q:**  \"Experiments. For all experiments, in the final version, I would like to ask the authors to also report the std across multiple runs of the model.\" \n\n**Reply:** For the current version, as we fix the random seed the same as the baseline models, the experiment results are fixed when running multiple times for both uni-variates and multi-variate time-series forecasting tasks. Indeed, we will also report the std across multiple runs for different random seeds. \n\n**Q:**  Visualization of toy data\n\n**Reply:** In the right two panels of Figure 3 in main paper, we show that without consideration of trend and seasonality, the neural CDE model underestimates the rising trend near the peak and thus cannot capture the true waveform accurately. \n\n**Q:**\nwhether neural lad can learn the correct dynamics? \n\n**Reply:** It is straightforward to show that our model can still learn the correct underlying dynamics, since our Neural Lad is a non-trivial generalization of original neural CDE that has already been proved to be able to sufficiently expressive. \n\nThe details on the datasets and hyperparameter settings are described in Appendix"}]

Author rebuttal:
We thank all the reviewers' recognition and valuable comments on our work. We have carefully responsed the concerns raised for each reviewers, including clarification of the novelty, adding more exprimental results. We attached the additional experimental results  as a pdf file for further check. 

Meta-review: This paper proposes a specific functional form for the dynamics function in a neural ODE. The proposal function has three components: a component to model the time-dependency (e.g. seasonal trends), a function to model the effect from the previous history on the dynamics (parameterized via a attention network) and a state dependent function. There is some novelty put into the intelligent design of the function h to model seasonal trends using a trignometric basis expansion with pre-specific frequencies. Overall, this is a simple idea and much of the novelty of this paper lies in the intelligent design for the neural architecture of the dynamics function. Overall the reviewers found the work clear, accessible and even if incremental at times. The empirical evaluation is very thorough. The model is carefully studied in a variety of benchmark datasets and compared to an extensive suite of related work. Overall this is a strong empirical paper and I'm inclined to accept.
Decision: Accept (poster)