Title: Neural Lad: A Neural Latent Dynamics Framework for Times Series Modeling
Venue: NeurIPS 2023
Accepted: yes
Keywords: Neural CDE

Time-series forecasting

Latent Dynamic
Abstract: Neural ordinary differential equation (Neural ODE) is an elegant yet powerful framework to learn the temporal dynamics for time series modeling.
However, we observe that existing Neural ODE forecasting models suffer from two disadvantages:
i) controlling the latent states only through the linear transformation over the local change of the observed signals may be inadequate;
ii) lacking the ability to capture the inherent periodical property in time series forecasting tasks;
To overcome the two issues, 
we introduce a new neural ODE framework called \textbf{Neural Lad}, a \textbf{Neural} \textbf{La}tent \textbf{d}ynamics model in which the latent representations evolve with an ODE enhanced by the change of observed signal and seasonality-trend characterization. We incorporate the local change of input signal into the latent dynamics in an attention-based manner and design a residual architecture over basis expansion to depict the periodicity in the underlying dynamics. To accommodate the multivariate time series forecasting, we extend the  Neural Lad through learning an adaptive relationship between multiple time series.        
Experiments demonstrate that our model can achieve better or comparable performance against existing neural ODE families and transformer variants in various datasets. Remarkably, the empirical superiority of Neural Lad is consistent across  short and long-horizon forecasting for both univariate, multivariate and even irregular sampled time series.

Neural Lad: A Neural Latent Dynamics Framework for Times Series Modeling

Abstract

Neural ordinary differential equation (Neural ODE) is an elegant yet powerful framework to learn the temporal dynamics for time series modeling. However, we observe that existing Neural ODE forecasting models suffer from two disadvantages: i) controlling the latent states only through the linear transformation over the local change of the observed signals may be inadequate; ii) lacking the ability to capture the inherent periodical property in time series forecasting tasks; To overcome the two issues, we introduce a new neural ODE framework called Neural Lad , a Neural La tent d ynamics model, in which the latent representations evolve with an ODE enhanced by the change of observed signal and seasonality-trend characterization. We incorporate the local change of input signal into the latent dynamics in an attention-based manner and design a residual architecture over basis expansion to depict the periodicity in the underlying dynamics. To accommodate the multivariate time series forecasting, we extend the Neural Lad by learning an adaptive relationship between multiple time series. Experiments demonstrate that our model can achieve better or comparable performance against existing neural ODE families and transformer variants in various datasets. Remarkably, the empirical superiority of Neural Lad is consistent across short and long-horizon forecasting for both univariate and multivariate irregularly sampled time series.

1 Introduction

Achieving accurate time series forecasting has been a long-standing challenge for decades in various applications, such as traffic flow prediction [36], weather forecasts [8], management of energy consumption [1], economic analysis [10], etc. With their remarkable representation power, deep learning-based approaches are capable of modeling the complex dynamics in time series, and thus dominate in forecasting tasks recently. These methods range from recurrent neural network (RNN [24, 28]), family of neural ordinary differential equations [3, 27, 13], Transformer variants [29, 40, 5, 19, 17, 34, 41] to basis-expansion-based models (e.g. N-Beats [23]).

Our design particularly follows the family of neural ODEs, since it is an elegant as well as powerful framework to learn the temporal dynamics for time series modeling. Unfortunately, existing variants of neural ODEs, e.g. vanilla neural ODE [3], ODE-RNN [27], neural CDEs [13, 22] do not sufficiently characterize the local change of observed signal and ignore inherent seasonality-trend attributes in time series forecasting tasks. To this end, we introduce a new member to the Neural ODE family, Neural Lad , where the latent states' dynamics evolve with a sophisticatedly designed neural ODE. It

ODESolver

Seasonality-trend transformation

hw(t)fe(zt)

t1

t3

n

Hidden state

ODESolver

个个个个个

ODESolver

Path

g(dx/dt)

Attention-basedNetwork

Path

x2

X3

X1

X2

V

t1

t2

t3

t3

NeuralODE

Neural CDE

NeuralLad

has the following characteristics tailored for forecasting challenging univariate and multivariate time series.

We incorporate the change of interpolated observations into the latent ODE in an attentionbased design to learn more flexible and expressive hidden states.

The latent dynamics function is time-dependent, and formulated as a decomposable form w.r.t. the hidden state, the input signal and time.

To accommodate the multivariate time series forecasting, we extend the Neural Lad through learning an adaptive relationship between multiple time series.

The comparison between our proposed Neural Lad and other related members of Neural ODEs is described in Figure. 1. The main difference is regarding how to construct the dynamics for the hidden states. Compared with Neural CDE, we employ a memory-based neural network to model how the local change of the interpolated signal path influences the latent dynamics. Further, the latent dynamic function is weighted with time-dependent seasonality-trend transformation particularly tailored for complex time series. After obtained the enhanced latent dynamics, the same as other variants of Neural ODEs, an ODESolver is used to obtain the integral hidden states. For multivariate time-series forecasting, we additionally embed an adaptive graph convolution function to encode the hidden states by incorporating the correlation with other time series (see Section 3). We then demonstrate the empirical superiority of Neural Lad across short and long-horizon forecasting for both univariate and multivariate time series data.

Related Work

Transformer variants for time series. Inspired by the success of transformer-based networks in CV and NLP domains, a series of transformer variants [40, 34, 41] were proposed to capture the long term temporal dependency in time-series forecasting. Specifically, Informer [40] is an efficient network with ProbSparse attention and generative decoder. Autoformer [34] implemented an decomposition transformer network with an auto-correlation mechanism. FEDformer [41] proposed to capture the global profile of time series with seasonality-trend decomposition. However, transformer-based models consider the temporal correlation with self-attention mechanisms, where the computational complexity is high particularly when the horizon is long. Although some works like PyraFormer [21] took effort to accelerate the attention operations, the performance was degraded due to the computation-accuracy trade-off.

Linear networks for time series. Different from variants of transformer, works [20, 23, 37] used various types of linear blocks to extract features. N-beats [23] proposed to stack multi-layer linear blocks and employed seasonal-trend basis expansion in each layer to enhance forecasting performance, which is more time and memory efficient than transformers. SCINet [20] designed a downsample-convolve-interact architecture including multiple sample convolutions and interaction network to capture temporal correlation. Besides, [37] rethought the problem of long-horizon time-series forecasting and found that a simple one-layer fully-connected network could achieve

comparable even better prediction performance than transformer variants. However, the linear blockbased networks are sensitive to data distribution and hyper-parameters according to our experiments, preventing them from generalizing to more scenarios.

Neural ODE family. To address the non-uniformly sampled time-series forecasting problem, neural ODE families [26, 6, 13] were introduced to model time series with continuous latent dynamics. ODE-RNN and latent ODE [26] generated interpretable hidden states through the neural ordinary differential equation. Neural CDE [13] interpolated the irregularly sampled observations into a continuous path that were then considered in latent dynamics. STG-NCDE [6] extended the Neural CDE from univariate to multivariate time-series with graph convolution to capture spatial correlation. Although Neural ODE families are more interpretable and stable than transformers and linear networks, existing Neural ODE models suffer from the ability to characterize the local change of the observed signal and the inherent periodicity-trend attributes in time series forecasting tasks.

2 Methodology

Let { x t } T t =0 denote the time series, we consider time series prediction problem as

where H is the length of horizon for prediction, L is the length of lookback window of historical time series, ϵ is the residual of the prediction, and G Θ ( · ) is parameterized neural network that will be elaborated later. Thus the loss function to learn the parameters Θ could be the mean square error or other types,

To fully capture the intrinsic characteristics of time series data, we propose to adopt a continuous dynamical system with hidden states z t to model the entire times series, described with following neural ordinary differential equation,

where ζ ( · ) and ξ ( · ) are parameterized neural network that model the relationship z 0 → x 0 and z t → x t , respectively; F ( · ) specifies the dynamics of the hidden state and is also a neural network to be learned. We call it as Neural Latent Dynamics and its design is critical for modeling the time series which will be presented later.

According to the formulation provided in Eq.(3), the solution to this ODE is determined by the initial condition at z 0 and the latent dynamics along the integration path, i.e.

This neural ODE aims to learn the hidden states at all times for time series forecasting, which can be estimated using a numerical ODE solver; and the next step prediction can be obtained by x t = ξ ( z t ) .

Generally, the latent dynamics F ( · ) can be any complex form with sufficient capacity to model the sequential data. In this work, we propose a crucial design over the choice of latent dynamics function F ( · ) particularly tailored for time series, i.e. F ( z t , x 0: t , t ) is decomposable w.r.t. the hidden state z t , the input signal x 0: t and time t , named as neural decomposable latent dynamics. The decomposed F ( · ) in our paper is as:

where the vector field function f θ ( z t ) expands the hidden state z t , h w ( t ) explicitly models the time-dependency with periodical and trend property, and g v is an attention-based network to describe how the change of observed signal influences the latent dynamics. The specification of these designs will be elaborated in Sec. 2.1 and 2.2.

We emphasize that our model choice over the latent dynamics F ( · ) is drastically different from other neural ODE family members in which only f θ ( z t ) was considered, as shown in Figure 1. The entire model can be thought as a continuous analogue of recurrent neural networks with layerwise adaptivity.

11

5.8

1.0

5.6

50

0.5

5.4

100

0.0

5.2

0.5

5.0

0

50

100

150

200

250

0

50

100

150

200

250

0

50

100

150

200

250

Here the decomposability assumption allows us to maintain a simple yet effective design over the latent dynamics without loss of expressivity, and its effectiveness will be verified empirically.

We highlight that the explicit dependence on time t could both increase the expressivity and interpretability of time series modeling within the neural ODE framework, particularly for the complex time series with strong periodicity, as we experimentally illustrated in Section 4.

To achieve a more compact model, the f θ ( z t , t ) is designed as a simple multiple-layer residual neural network. In the following, we will specify two designs that aim to explicitly describe the how g v ( x 0: t ) and h w ( t ) depends on the input signal x 0: t and time t , respectively.

2.1 Attention-based network for modeling the change of input signal

The change of input signal x can directly influence the local dynamics of the hidden states. To incorporate the input signal into the latent dynamics f in a differentiable manner, we have to rely on a continuous input signal. However, the observed times series is either regular or irregular sampled sequential data { x t } T t =0 , thus, we first need to conduct a continuous approximation over the discrete time series. Inspired by the idea of neural controlled differential equation, cubic spline could be a natural alternative due to its minimum regularity for handling certain edge cases. Concretely, let X denote the cubic spline with knots t = 0 , . . . , T such that X t = ( x t , t ) ∈ R d x +1 .

With the continuous approximation X , typically called 'signal path', the dynamic change of input signal could be depicted by the derivative dX/dt , and the component g v ( x 0: t ) in the decomposed latent dynamics Eq (6) is specified as

where g v : R d x +1 → R d x +1 in our paper is a trainable attention-based network to describe how the change of input signal influences the latent dynamics, inspired by the idea of memory networks [30] and matching networks [31].

Concretely, let M ∈ R d m × ( d x +1) be the memory matrix to be learned for the path gradient, dX/dt , shared for the entire time series. This matrix aims to memorize representative patterns of the local change along the signal path, stored as d m rows of M that can be used further pattern matching. Then, for each time t , an attention-based transformation is employed for weighting the patterns stored in the memory matrix M to obtain the final expressive control signal,

Our memory-based modeling over the local change of the signal path is different from that of Neural CDE [13], where only the path gradient dX/dt for a single time point was used. The introduction of the memory matrix allows the model to combine different patterns of local changes, which could also help to provide more discriminative features particularly for the segments with abrupt changes.

We visualize the attention weights softmax ( M dX dt ) along one time series in Weather dataset in Figure 2, where memory size d m = 128 and the length of time series is 250. We can observe that around t = 150 with obvious changes, the learned attention weights are extremely sparse and distinguishing from other segments.

2.2 Seasonality-trend transformation-based latent dynamics

In most of existing neural ODEs, the dynamics function only depends on z t , not explicitly on the time t . However, in the scenario of time series, particularly for those with complex periodic and

trending characteristics, only relying on z t and x 0: t , F ( · ) might not be sufficient for capturing the hidden seasonal and trendy attributes. To this end, we propose to force the latent dynamic function to explicitly depend on the time t , as described by the decomposed formulation Eq (6), which forms a non-autonomous ODE in the theory of dynamical system.

With the decomposable latent dynamics, we propose to design the time-dependency function h w ( t ) to characterize the intrinsic seasonality and trend property based on basis expansion. It is the combination of a periodic function h ( s ) w parameterized by w ( s ) to model the seasonal characteristics and a polynomial function h ( p ) w parameterized by w ( p ) to model the trend features:

where h ( s ) w ( t ) takes the form of trigonometric basis expansion with pre-specified frequencies { 1 , 1 / 2 , 1 / 4 , ... 1 / 2 m s +1 } , m s and m p are the number of seasonal and polynomial basis, respectively. Then, the time-dependency factor h w ( t ) for the latent dynamics accounts to the sum of the two terms,

Then, we multiply the vector field output f θ ( z t ) with the time-dependency function h w ( t ) , and define it as the l -th basic block,

We stack multiple layers of the basic block in a residual manner that is easy to train,

Therefore, by taking the time seasonality and trend into consideration, the decomposed formulation of F ( · ) becomes,

where { θ, w, v } are parameters to be learned for the decomposed functions h ( t ) , f ( z t , t ) and g ( x 0: t ) , respectively. Finally, the solution of hidden state z t can be derived as follows, which is amenable to typical neural ODE solver,

where f θ ( z t ) ∈ R d z × ( d x +1) to accommodate the dimensionality of g v ( dX/dt ) , specified as a simple residual network.

3 Multivariate Neural Lad

The multivariate time series forecasting typically aim to model the temporal and spatial relationship between multiple time seriese simultaneously to enhance the prediction performance. Representative models include STGCN [36], AGCRN [2] and NRI [14], among which AGCRN is the most effective and efficient approach. Inspired by AGCRN, we adaptively optimize the graph structure and learn the spatial correlation with a generalized graph convolution operation jointly.

We start with the graph convolution [15] for capturing the spatial dependency between multivariate time series for each time slice t ,

where Z t ∈ R N × d z represents the t -th slice of hidden states for the N time series, and Φ t ∈ R N × d Φ collects the projected states into a matrix, D ∈ R N × N is the degree matrix, Λ ∈ R d z × d Φ and b are

the trainable weight and offset, A ∈ R N × N is the adjacency matrix, and I + D -1 2 AD -1 2 is the low-order Chebyshev matrix. Therefore, Eq.(16) is the graph convolution on the hidden states to capture the correlation between multiple time series (i.e. nodes).

To make the adjacency matrix A more flexible, a learnable embedding matrix E ∈ R N × d e is designed where each row represents the embedding for each time series and d e is the dimension of node embedding. Thus, EE T could measure the similarity between these nodes. Then the normalized adjacency matrix D -1 2 AD -1 2 is approximated by the following nonlinear transformation,

where σ is the ReLU activation function and ϕ is the softmax operation. We now specify a generalized version of the graph convolution in Eq. (16) to improve adaptivity and expressivity. The spatial embedding s for all the N times series are defined as follows,

where I + ϕ ( σ ( EE ⊤ )) is the adaptive Chebyshev matrix inspired by AGCRN [2], and we let the dimension of the embedding of Φ t is the same as that of the latent state z t for simplicity, i.e. d z . Thus, for i -th time series, the spatial component corresponds to the i -th row of the embedding matrix Φ t

where γ = { Λ , E } denotes the collection of parameters to be learned. Then, for each time series, the hidden state z t incorporating both temporal dynamics and spatial correlation is modeled as:

The Eq.(20) is the multi-variate extension of Eq.(15) with the additional spatial component ϕ γ ( Z t ) .

In the encoder stage, we evaluate the time-dependent dynamics h w ( t ) , f θ ( z t ) , transformation of the signal path g v ( dX/dt ) , then obtain the z t for all the time steps by ODESolver. In the decoder stage, we generate the hidden states and predicted outputs in a step-by-step or one-shot manner through ξ ( z T ) .

Discussions We now summarize several benefits of Neural Lad for times series modeling as follows.

It is the first neural ODE framework for time series in which the dynamic function explicitly depends on the hidden state z t , input x 0: t and the time t . It can seamlessly fit into the training procedure of neural ODE family and the model may be trained with memoryefficient adjoint-based backpropagation even across observations.

The explicit dependence on the time t through the decomposable latent dynamics B ( z t , t ) = h w ( t ) f θ ( z t ) allows our neural ODE to adaptively reveal the periodicity of complex time series. Besides, the g v ( dX dt ) learns the weight of the control signal. To extend to multivariate time series, we design the function ψ γ ( z t ) to learn the latent dynamics from spatial correlation. This can also be verified from experimental part in Section 4.

When considering the relationship between input and hidden states, we employ the continuous approximation of observed discrete time series to facilitate the integration over the continuous. Thus, for the original time series either regularly or irregularly sampled, they can be fit into our model for training and inference.

4 Experiments

We conduct experiments on both synthetic and real-world datasets on univariate, multivariate and irregularly sampled time series forecasting tasks to demonstrate the empirical performance of our proposed approach.

0

250

500

750

1000

10

0

10

0

250

500

750

1000

10

0

10

0

50

100

150

200

10

0

10

0

50

100

150

200

20

0

20

4.1 Regularly sampled time series prediction: synthetic data

We first implement the multivariate Neural Lad on a synthetic dataset consisting of 30 periodic time series with various frequencies and linearly increasing amplitudes. The code for generating synthetic time series follows that of Neural ODE [27]. Concretely, 30 time series each with 2304 time steps are generated according to the functions x i,t = a i,t sin(2 πb i,t t + ϕ ) + n i,t , where i is the index of each time series, ϕ is the phase, n i,t denotes the standard Gaussian noise. The amplitude and frequency ascends linearly with time t , i.e. a i,t = a i, 0 + a max -a min t max -t min t and b i,t = b i, 0 + b max -b min t max -t min t , where for each time series the initial parameters are sampled from uniform distribution, a i, 0 ∼ U [2 , 10] and b i, 0 ∼ U [0 , 10] to increase the diversity.

Two examples of generated time series are shown in the left two panels in Figure.3. We can easily observe that these time series exhibit changing frequencies and amplitudes along time.

Performance. We compare a strong baseline STG-NCDE [6] with our proposed model on this synthetic dataset with prediction horizon { 12 , 48 , 96 } . The dimension of hidden states, is searched from { 8 , 16 , 32 , 64 } and we find the best setting is 16 . The results are shown in Table. 1. It can be observed that Neural Lad outperforms STG-NCDE on both short and long-term forecasting. Moreover, our method improves it by 45% on the short-term horizon and by 47% on long-term horizon, both of which are large margins. We plot the predicted time series with horizon 90 by STG-NCDE and Neural Lad in the right two panels of Figure. 3. It can be observed that Neural Lad can fit well the overall shape of the time series.

Ablation study. We also conduct an ablation study by experimenting Neural Lad only with one component, periodicity h w ( t ) or attention-based transformation of control signal g v ( dX/dt ) , as shown in Table 1. We find that both of the two components play important roles for enhancing performance. Moreover, the attention-based transformation g v contributes slightly more for this task.

4.2 Irregularly sampled time series classification: PhysioNet sepsis

One advantage of the neural ODE family, including Neural Lad, is that it naturally adapts to the case of irregularly sampled partially observed data. In this part, we apply our approach to the PhysioNet 2019 challenge on sepsis prediction [25], including 40,335 time series of variable length from the patients in ICU. Measurements are around 39-dimensional features with an hourly resolution, most of which are missing and only 10.3% of values are observed. The task is a binary classification problem to predict whether sepsis is developed during the patents' stay. We follow the experimental settings of [13], where the cases with or without observational intensity are both considered.

Since the datasets is highly imbalanced (5% positive rate), AUC is evaluated for different approaches, as reported in Table 2. We can observe that Neural Lad outperforms other alternatives when considering the observational intensity with low memory usage. Moreover, Neural Lad increases the memory less than 6% on both two classification tasks.

4.3 Real-world univaraite and mutlivariate time series prediction

Baseline methods. For long-horizon univariate time series forecasting, we compare Neural Lad with SCINet, Neural CDE, DLinear [37], LightTS [38], TimesNet [33], MICN [32], Transformer variants

including Informer [40], Autoformer [34], Fedformer [41], Crossformer [39]. For multivariate time series considering spatial correlation, we compare our method against other graph-based deep learning approaches, including STGCN [36], ASTGCN [11], MSTGCN [12], DCRNN [18], Graph Wavenet [35], AGCRN [2], ST-WA [7], DSTAGNN [16], Z-GCNETs [4], SCI-Net [20], ST-GODE [9] and STG-NCDE [6].

Univariate time series. Table 3 shows the performance comparison on long-term time-series forecasting. For fair competition, we only consider the temporal latent dynamics ignoring the spatial correlation in this experiment. Specifically, we generate the hidden state with Eq.(15) instead of Eq.(20). We can observe that our proposed method outperforms all the transformer baselines by a large margin. For the model FEDformer that performs better than all other transformer variants (Autoformer, Informer, and Pyraformer), Neural Lad outperforms it on all datasets. For two linear baselines, we improved the TimesNet [33] and DLinear [37]on all dataset with the average of different horizons. The paper Crossformer [39] conduct experiments on different horizons [24 , 48 , 168] , therefore we take it as "-" if there are no results published in the origin paper. In the same way, for dataset ( ETTm2 and ETTh2 ) without performance published in paper MICN [32], we fill it with "-". From the table 3, we can find that although Crossformer performs better at the short horizon, it fails on long horizon prediction. For TimesNet [33], Neural Lad outperform it on average of four horizons on all dataset.

Multivariate time series. Table 4 shows that the proposed Neural Lad achieves state-of-the-art performance on the four PEMS datasets on MAE metric, and also outperforms most baselines on MAPEand MSE metrics. For the baseline SCINet, although the proposed Neural Lad slightly reduces prediction error than which on PEMSD3 and PEMSD4 dataset, our method reduces MAE by 4 . 6% on PEMSD7 dataset and by 2 . 8% on PEMSD8 dataset. For neural ODE family baselines, even though STG-NCDE performs better than ST-GODE by a large margin, the proposed method improves STG-NCDE by 5 . 8% , 1 . 2% , 1 . 6% , 1 . 1% on the four PEMS datasets, respectively. Besides, our method performs slightly better than ST-WA. Overall, Neural Lad achieves the best performance for multivariate time-series forecasting in Neural ODE family.

5 Conclusion

In this work, we have introduced a new variant to Neural ODE family, Neural Lad, particularly tailored for accurate time series modeling and forecasting. We conduct extensive experiments on both univariate, multivariate and irregularly sampled time series datasets, Neural Lad has achieved remarkable forecasting performance in all these scenarios. Extending materials about experiment settings, computation cost, and visualizations are presented in Appendix.

In the future, we plan to improve the training efficiency of Neural Lad when considering long lookback window. This might tie with how to conduct a faster autonomous ODE solver. Another direction is to propose more architecture designs for the involved neural networks in latent dynamics model.

References

Kadir Amasyali and Nora M El-Gohary. A review of data-driven building energy consumption prediction studies. Renewable and Sustainable Energy Reviews , 81:1192-1205, 2018.

Lei Bai, Lina Yao, Can Li, Xianzhi Wang, and Can Wang. Adaptive graph convolutional recurrent network for traffic forecasting. In In proceedings of 34th Conference on Neural Information Processing Systems , 2020.

Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems , 31, 2018.

Yuzhou Chen, Ignacio Segovia-Dominguez, and Yulia R. Gel. Z-gcnets: Time zigzags at graph convolutional networks for time series forecasting. In Proc. 38th International Conference on Machine Learning (ICML 2021) , 2021.

Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. 2019.

Jeongwhan Choi, Hwangyong Choi, Jeehyun Hwang, and Noseong Park. Graph neural controlled differential equations for traffic forecasting. In AAAI , 2022.

Razvan-Gabriel Cirstea, Bin Yang, Chenjuan Guo, Tung Kieu, and Shirui Pan. Towards spatio-temporal aware traffic time series forecasting. ICDE 2022 , 2022.

Lasse Espeholt, Shreya Agrawal, Casper Sønderby, Manoj Kumar, Jonathan Heek, Carla Bromberg, Cenk Gazen, Rob Carver, Marcin Andrychowicz, Jason Hickey, et al. Deep learning for twelve hour precipitation forecasts. Nature communications , 13(1):1-10, 2022.

Zheng Fang, Qingqing Long, Guojie Song, and Kunqing Xie. Spatial-temporal graph ode networks for traffic flow forecasting. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 364-373, 2021.

Philippe Goulet Coulombe, Maxime Leroux, Dalibor Stevanovic, and Stéphane Surprenant. How is machine learning useful for macroeconomic forecasting? Journal of Applied Econometrics , 37(5):920-964, 2022.

Shengnan Guo, Youfang Lin, Ning Feng, Chao Song, and Huaiyu Wan. Attention based spatialtemporal graph convolutional networks for traffic flow forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 922-929, 2019.

Ziyu Jia, Youfang Lin, Jing Wang, Xiaojun Ning, Yuanlai He, Ronghao Zhou, Yuhan Zhou, and Li-wei H. Lehman. Multi-view spatial-temporal graph convolutional networks with domain generalization for sleep stage classification. IEEE Transactions on Neural Systems and Rehabilitation Engineering , 29:1977-1986, 2021.

Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems , 33:6696-6707, 2020.

Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. arXiv preprint arXiv:1802.04687 , 2018.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations , 2017.

Shiyong Lan, Yitong Ma, Weikang Huang, Wenwu Wang, Hongyu Yang, and Pyang Li. Dstagnn: Dynamic spatial-temporal aware graph neural network for traffic flow forecasting. In Proc. 39th International Conference on Machine Learning (ICML 2022) , 2022.

Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. In International Conference on Learning Representations (ICLR '18) , 2018.

Bryan Lim, Sercan O. Arik, Nicolas Loeff, and Tomas Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting. 2020.

Minhao Liu, Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma, and Qiang Xu. Scinet: Time series modeling and forecasting with sample convolution and interaction. Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS), 2022 , 2022.

Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International Conference on Learning Representations , 2022.

James Morrill, Cristopher Salvi, Patrick Kidger, and James Foster. Neural rough differential equations for long time series. In International Conference on Machine Learning , pages 7829-7838. PMLR, 2021.

Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. In International Conference on Learning Representations , 2020.

Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

Matthew A Reyna, Chris Josef, Salman Seyedi, Russell Jeter, Supreeth P Shashikumar, M Brandon Westover, Ashish Sharma, Shamim Nemati, and Gari D Clifford. Early prediction of sepsis from clinical data: the physionet/computing in cardiology challenge 2019. In 2019 Computing in Cardiology (CinC) , pages Page-1. IEEE, 2019.

Yulia Rubanova, Ricky T. Q. Chen, and David K Duvenaud. Latent ordinary differential equations for irregularly-sampled time series. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for irregularly-sampled time series. Advances in neural information processing systems , 32, 2019.

David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting , 36(3):1181-1191, 2020.

Jaak Simm, Adam Arany, Edward De Brouwer, and Yves Moreau. Expressive graph informer networks. 2020.

Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. End-to-end memory networks. Advances in neural information processing systems , 28, 2015.

Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems , 29, 2016.

Huiqiang Wang, Jian Peng, Feihu Huang, Jince Wang, Junhui Chen, and Yifei Xiao. Micn: Multi-scale local and global context modeling for long-term series forecasting. 2023.

Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In International Conference on Learning Representations , 2023.

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with Auto-Correlation for long-term series forecasting. In Advances in Neural Information Processing Systems , 2021.

Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chengqi Zhang. Graph wavenet for deep spatial-temporal graph modeling. In Proceedings of the 28th International Joint Conference on Artificial Intelligence , IJCAI'19, page 1907-1913. AAAI Press, 2019.

Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting. In IJCAI , 2018.

Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? 2023.

Tianping Zhang, Yizhuo Zhang, Wei Cao, Jiang Bian, Xiaohan Yi, Shun Zheng, and Jian Li. Less is more: Fast multivariate time series forecasting with light sampling-oriented mlp structures, 2022.

Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In International Conference on Learning Representations , 2023.

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In The Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Virtual Conference , volume 35, pages 11106-11115. AAAI Press, 2021.

Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Proc. 39th International Conference on Machine Learning (ICML 2022) , 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new neural ODE framework, Neural Lad (Neural Latent dynamics model), that decomposes the differential function in three components: a NN differential function, an attention-based network,  a time-dependency function and a graph convolution network for spatial correlations. They evaluate the method on short and long-horizon forecasting tasks for both univariate and multivariate time series data. The proposed method outperforms or performs on par with the existing Neural ODE, time-series transformer, and graph NN based models.\n\nSTRENGTHS:\nOriginality: \nThe work is a novel combination of known techniques and it is clear how it compares to existing work, for example,  Neural CDEs. \nThe related work section is well organized and all the related methods have been addressed, providing an adequate overview of the related work for the reader. \n\nClarity: \nThe submission is very clearly written. Therefore it is easy to follow the proposed method.  \n\n\nSignificance: \nThe authors provide a novel NODE based method for long-horizon forecasting\n\nWEAKNESSES:\nQuality: \n\nThe authors introduce a function $h_w(t)$ which should extract seasonality trend. However, neither mathematically nor experimentally, it isn't clear whether this function really achieves this. Traditionally trend is extracted as the mean value of the data, while seasonal part would be the remaining time series data, when trend is removed. From the introduced equations, where the output of the differential function is scaled by the time-dependency factor is at the moment unclear how this is achieved. \n\nSimilarly, the authors introduce an attention-based network to model change of the input signal, but the obtained weight is multiplied with the 'memory matrix' rather than the input signal itself, thus it is unclear, what information does this memory matrix has learned that reflects the input signal. This matrix is also not visualized. \n\nClarity:\n\nFor eq. (1) and eq (2) I would recommend the authors to update the prediction to $\\hat{x_{t:t+H}}$. So then in eq. (2) L2 loss is between the true observations and the predictions by the network. \n\nLine 92, the relationship for the fist function $\\xi$ seems to be in the wrong direction, given whats defined in eq. (3). \n\nFor the experiments section there are crucial points missing: how many datapoints used as input, dataset details, etc. This information should be provided in the supplementary material. \n\nUnder the hyperparameter section there are also missing details on what type of solver was used, solver parameters, what type of optimizer, etc. \n\nSection 2.2. the block residual architecture is unclear from the text, I would recommend moving the figure from the appendix to main text. \n\nSignificance: \nThe results show that the proposed method indeed achieves an improved performance compared to the existing models. But I would like that the authors would have provided some further analysis, results on the different decomposition components, to strengthen their claims of the purpose of the different functions. Moreover, from a theoretical standpoint it is unclear under which conditions $f_\\theta(z_t)$ still remains the correct underlying differential equation.\n\nQUESTIONS:\nEq.6 The differential equation F() is decomposed in 3 components, differential function, attention-network, time-dependency network. This function, F() is used in the ODESolver that iteratively class this function for dt increments. For every increment, the model goes through some number of blocks (L), where the output of the differential function is scaled by the time-dependency function. Could you please clarify what is the output of $h_w(t)$, and why would you scale the state-dependent dynamics by it? \n\nDo you have some additional results where it can be seen that this multiplication identifies seasonal trends as seen in the data?  \n\nIs the extrapolation (forecasting) autoregressive or you have to pre-define the output sequence length? \n\nFor Eq.8 from self-attention perspective we have keys, queries in values. In relation to your work, I would identify the values as $\\frac{dX}{dt}$ rather than M. Why do you apply the softmax weights to the memory matrix rather than the signal path, which you subsequently use to update the vector from the differential equation?\n\n\nWhat is the number of parameters, complexity, compute time of the proposed method? As cubic spline is an approximation method as well as the time expansion results in additional residual connections, how does it compare to existing NODE models and the transformer architectures? \n\nExperiments.\nFor all experiments, in the final version, I would like to ask the authors to also report the std across multiple runs of the model. \n\nIt is unclear how many timepoints were taken as input to for example to compute the attention-based network output? \n\nFor simulated data, are there more visualization that showcase that the model has captured the linear trend in the data? \nSimilarly, for the ablations on the simulated data when the $h_w$ component is removed, does the model then fail to capture the linear trend? \nIs there a clear pattern that the $g_v$ function is beneficial for time-series that have sudden changes, like weather forecasting? \n\nAs the authors have changed the RHS of the differential equation, is there a theoretical proof that $f_\\theta(z_t)$ still learns the correct underlying dynamics? What are the necessary/sufficient conditions under which this is the case?\n\nLIMITATIONS:\nThe authors have briefly touched upon the training efficiency of Neural Lad, however, I would recommend the authors to include a more thorough discussion on this, especially as  in the related work section this is mentioned as the major draw back for transformer based models, however, it is not explicitly discussed for the model at hand. This is information is also missing in the supplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of characterizing the local change of observed signals and ignoring inherent periodical property in time series forecasting tasks. A new neural ODE-based framework is proposed with 1) a decomposable latent space for time-dependent dynamics and 2) an attention-based design for local changes in observation. The framework is further extended to multivariate settings using graph-based networks to adaptively learn the spatial correlation. Experiments have been presented on both univariate and multivariate settings to demonstrate the improved forecasting performance of the proposed model.\n\nSTRENGTHS:\n1. The presentation of the dynamic model is clear and well-structured. The attention-based networks handle the local change of the signal and the time-dependency function characterizes the seasonal and trend properties of the signal.\n2. Empirical results show improved forecasting performance of the proposed model on both synthetic and real-world datasets.\n\nWEAKNESSES:\n1. I feel that the approach is somewhat incremental from the perspective of the methodology in that it is an extension of the previous works (e.g. Kidger et al 2020, Choi et al 2022), in combination with a decomposable form of latent dynamics and attention-based feature extractor. Could the author elaborate more on how the proposed method differs from the previous works?\n2. Experimental settings could be more elaborated: 1) The motivation of choosing baseline models in Table 2, Table 3, and Table 4, and how the proposed model improved could be more detailed; 2) The author should also provide more information about datasets used in the two univariate and multivariate settings in terms of their local property or seasonal features. I also think the ablation study is not sufficient: 1) There should be evidence to prove the benefits of using attention-based networks for local changes; 2) The detail of improvements in spatial relationships should also be provided.\n\nQUESTIONS:\nPlease check the weaknesses mentioned above.\n\nLIMITATIONS:\nN.A.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper presents a new framework for modeling time series using a controlled latent neural-ODE-based dynamics model. The proposed latent dynamics function uses a special factorized structure, which effectively disentangles the influences of time (via a periodic basis expansion to capture periodic patterns), the current latent state, and the input signal's history (leveraging an attention-based architecture). Both uni- and multivariate versions of the model are presented. Numerical validations are conducted across a variety of tasks and benchmarked against a number of neural ODE and transformer variants.

STRENGTHS:
* The paper is generally well written and easy to follow.
* The numerical validation is quite extensive, spanning over tasks of different natures and (short- and long-term) prediction regimes. The proposed method is compared against a wide range of variants in the neural ODE and transformer model families.

WEAKNESSES:
* The idea itself is not the most novel, involving a simple factorization and model architectures drawing cues from popular ones already existing in the literature. However, this is largely compensated by the exhaustive numerical validations where consistent improvements are observed.
* Discussion on the computation costs and potential limitations is missing.

QUESTIONS:
* Can you comment on the cost aspects (i.e. training, inference and memory footprints) of the Neural Lad models? I am curious how it compares to the baselines tested in the paper.
* The notation across Equations (13-14) is slightly unclear to me. How does $B(z_t, t)$ in equation (14) relate to the $B$ operations in Equation (13)?

LIMITATIONS:
Not much discussion on the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06

comments: SUMMARY:
This paper propose a novel neural ordinary differential equation framework for time series modeling. The main contribution is in the design of latent dynamic function $F(\cdot)$ which can be decomposed into hidden state dynamics $f_\theta(z_t)$, time-dependency with periodical and trend property $h_w(t) $, and attention-based network to model signal to latent dynamics effect $g_v(x_0:t)$. 
Also, they showed that the model can be extend to a multivariate time series forecasting. The proposed design is theoretically sound and authors proved them to be effective on synthetic and real data.

STRENGTHS:
- The paper is well-written and the contribution is well stated. 
- The proposed design is compared thoroughly with different types of baseline in various synthetic/real data and outperforms baseline in most cases.

WEAKNESSES:
- Analysis on the computational complexity/cost is missing

QUESTIONS:
- Synthetic data seems to have only trend dynamics. It would be nice to see the result from seasonal(periodic) dynamics

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-11