SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ting Li
Ant Group, Beijing, China
lt317068@antgroup.com
Zhanxing Zhu
Changping National Lab & Peking University, Beijing, China
zhanxing.zhu@pku.edu.cn
Neural ordinary differential equation (Neural ODE) is an elegant yet powerful framework to learn the temporal dynamics for time series modeling. However, we observe that existing Neural ODE forecasting models suffer from two disadvantages: i) controlling the latent states only through the linear transformation over the local change of the observed signals may be inadequate; ii) lacking the ability to capture the inherent periodical property in time series forecasting tasks; To overcome the two issues, we introduce a new neural ODE framework called Neural Lad, a Neural Latent dynamics model, in which the latent representations evolve with an ODE enhanced by the change of observed signal and seasonality-trend characterization. We incorporate the local change of input signal into the latent dynamics in an attention-based manner and design a residual architecture over basis expansion to depict the periodicity in the underlying dynamics. To accommodate the multivariate time series forecasting, we extend the Neural Lad by learning an adaptive relationship between multiple time series. Experiments demonstrate that our model can achieve better or comparable performance against existing neural ODE families and transformer variants in various datasets. Remarkably, the empirical superiority of Neural Lad is consistent across short and long-horizon forecasting for both univariate and multivariate irregularly sampled time series.
Achieving accurate time series forecasting has been a long-standing challenge for decades in various applications, such as traffic flow prediction [36], weather forecasts [8], management of energy consumption [1], economic analysis [10], etc. With their remarkable representation power, deep learning-based approaches are capable of modeling the complex dynamics in time series, and thus dominate in forecasting tasks recently. These methods range from recurrent neural network (RNN [24, 28]), family of neural ordinary differential equations [3, 27, 13], Transformer variants [29, 40, 5, 19, 17, 34, 41] to basis-expansion-based models (e.g. N-Beats [23]).
Our design particularly follows the family of neural ODEs, since it is an elegant as well as powerful framework to learn the temporal dynamics for time series modeling. Unfortunately, existing variants of neural ODEs, e.g. vanilla neural ODE [3], ODE-RNN [27], neural CDEs [13, 22] do not sufficiently characterize the local change of observed signal and ignore inherent seasonality-trend attributes in time series forecasting tasks. To this end, we introduce a new member to the Neural ODE family, Neural Lad, where the latent states' dynamics evolve with a sophisticatedly designed neural ODE. It
Figure 1: The sketch of Neural ODE, Neural CDE and our proposed Neural Lad for constructing hidden states from observations. Latent ODE: simply modifying hidden states at each observed data point; Neural CDE: taking the signal path into consideration which makes the hidden state continues; Neural Lad: enhancing the Neural CDE with transformation over input change of signal and time-dependent dynamics.
has the following characteristics tailored for forecasting challenging univariate and multivariate time series.
- We incorporate the change of interpolated observations into the latent ODE in an attention-based design to learn more flexible and expressive hidden states.
- The latent dynamics function is time-dependent, and formulated as a decomposable form w.r.t. the hidden state, the input signal and time.
- To accommodate the multivariate time series forecasting, we extend the Neural Lad through learning an adaptive relationship between multiple time series.
The comparison between our proposed Neural Lad and other related members of Neural ODEs is described in Figure. 1. The main difference is regarding how to construct the dynamics for the hidden states. Compared with Neural CDE, we employ a memory-based neural network to model how the local change of the interpolated signal path influences the latent dynamics. Further, the latent dynamic function is weighted with time-dependent seasonality-trend transformation particularly tailored for complex time series. After obtained the enhanced latent dynamics, the same as other variants of Neural ODEs, an ODESolver is used to obtain the integral hidden states. For multivariate time-series forecasting, we additionally embed an adaptive graph convolution function to encode the hidden states by incorporating the correlation with other time series (see Section 3). We then demonstrate the empirical superiority of Neural Lad across short and long-horizon forecasting for both univariate and multivariate time series data.
Transformer variants for time series. Inspired by the success of transformer-based networks in CV and NLP domains, a series of transformer variants [40, 34, 41] were proposed to capture the long term temporal dependency in time-series forecasting. Specifically, Informer [40] is an efficient network with ProbSparse attention and generative decoder. Autoformer [34] implemented an decomposition transformer network with an auto-correlation mechanism. FEDformer [41] proposed to capture the global profile of time series with seasonality-trend decomposition. However, transformer-based models consider the temporal correlation with self-attention mechanisms, where the computational complexity is high particularly when the horizon is long. Although some works like PyraFormer [21] took effort to accelerate the attention operations, the performance was degraded due to the computation-accuracy trade-off.
Linear networks for time series. Different from variants of transformer, works [20, 23, 37] used various types of linear blocks to extract features. N-beats [23] proposed to stack multi-layer linear blocks and employed seasonal-trend basis expansion in each layer to enhance forecasting performance, which is more time and memory efficient than transformers. SCINet [20] designed a downsample-convolve-interact architecture including multiple sample convolutions and interaction network to capture temporal correlation. Besides, [37] rethought the problem of long-horizon time-series forecasting and found that a simple one-layer fully-connected network could achieve
comparable even better prediction performance than transformer variants. However, the linear block-based networks are sensitive to data distribution and hyper-parameters according to our experiments, preventing them from generalizing to more scenarios.
Neural ODE family. To address the non-uniformly sampled time-series forecasting problem, neural ODE families [26, 6, 13] were introduced to model time series with continuous latent dynamics. ODE-RNN and latent ODE [26] generated interpretable hidden states through the neural ordinary differential equation. Neural CDE [13] interpolated the irregularly sampled observations into a continuous path that were then considered in latent dynamics. STG-NCDE [6] extended the Neural CDE from univariate to multivariate time-series with graph convolution to capture spatial correlation. Although Neural ODE families are more interpretable and stable than transformers and linear networks, existing Neural ODE models suffer from the ability to characterize the local change of the observed signal and the inherent periodicity-trend attributes in time series forecasting tasks.
Let  $\{x_{t}\}_{t = 0}^{T}$  denote the time series, we consider time series prediction problem as
$$
x _ {t: t + H} = G _ {\Theta} \left(x _ {t - L: t}\right) + \epsilon , \tag {1}
$$
where  $H$  is the length of horizon for prediction,  $L$  is the length of lookback window of historical time series,  $\epsilon$  is the residual of the prediction, and  $G_{\Theta}(\cdot)$  is parameterized neural network that will be elaborated later. Thus the loss function to learn the parameters  $\Theta$  could be the mean square error or other types,
$$
\min  _ {\Theta} \left\{\mathcal {L} (\Theta) = \| x _ {t: t + H} - G _ {\Theta} \left(x _ {t - L: t}\right) \| _ {2} ^ {2} \right\} \tag {2}
$$
To fully capture the intrinsic characteristics of time series data, we propose to adopt a continuous dynamical system with hidden states  $z_{t}$  to model the entire times series, described with following neural ordinary differential equation,
$$
\frac {d z _ {t}}{d t} = F \left(z _ {t - 1}, x _ {0: t - 1}, t\right), \quad z _ {0} = \zeta \left(x _ {0}\right), \quad x _ {t} = \xi \left(z _ {t}\right), \tag {3}
$$
where  $\zeta(\cdot)$  and  $\xi(\cdot)$  are parameterized neural network that model the relationship  $z_0 \to x_0$  and  $z_t \to x_t$ , respectively;  $F(\cdot)$  specifies the dynamics of the hidden state and is also a neural network to be learned. We call it as Neural Latent Dynamics and its design is critical for modeling the time series which will be presented later.
According to the formulation provided in Eq.(3), the solution to this ODE is determined by the initial condition at  $z_0$  and the latent dynamics along the integration path, i.e.
$$
z _ {t} = z _ {0} + \int_ {0} ^ {t} F \left(z _ {s}, x _ {0: s}, s\right) d s, s \in [ 0, t ] \tag {4}
$$
This neural ODE aims to learn the hidden states at all times for time series forecasting, which can be estimated using a numerical ODE solver; and the next step prediction can be obtained by  $x_{t} = \xi (z_{t})$ .
$$
z _ {1}, \dots , z _ {t} = \operatorname {O D E S o l v e r} \left(F, z _ {0}, \left(t _ {0}, \dots , t\right)\right) \tag {5}
$$
Generally, the latent dynamics  $F(\cdot)$  can be any complex form with sufficient capacity to model the sequential data. In this work, we propose a crucial design over the choice of latent dynamics function  $F(\cdot)$  particularly tailored for time series, i.e.  $F(z_{t},x_{0:t},t)$  is decomposable w.r.t. the hidden state  $z_{t}$ , the input signal  $x_{0:t}$  and time  $t$ , named as neural decomposable latent dynamics. The decomposed  $F(\cdot)$  in our paper is as:
$$
F \left(z _ {t}, x _ {0: t}, t\right) = h _ {w} (t) f _ {\theta} \left(z _ {t}\right) g _ {v} \left(x _ {0: t}\right), \tag {6}
$$
where the vector field function  $f_{\theta}(z_t)$  expands the hidden state  $z_{t}$ ,  $h_w(t)$  explicitly models the time-dependency with periodical and trend property, and  $g_{v}$  is an attention-based network to describe how the change of observed signal influences the latent dynamics. The specification of these designs will be elaborated in Sec. 2.1 and 2.2.
We emphasize that our model choice over the latent dynamics  $F(\cdot)$  is drastically different from other neural ODE family members in which only  $f_{\theta}(z_t)$  was considered, as shown in Figure 1. The entire model can be thought as a continuous analogue of recurrent neural networks with layerwise adaptivity.
Figure 2: Visualization of attention weights. Left: attention weight; Middle: observed time series; Right: the weights of the 83rd components along time in log scale.
Here the decomposability assumption allows us to maintain a simple yet effective design over the latent dynamics without loss of expressivity, and its effectiveness will be verified empirically.
We highlight that the explicit dependence on time  $t$  could both increase the expressivity and interpretability of time series modeling within the neural ODE framework, particularly for the complex time series with strong periodicity, as we experimentally illustrated in Section 4.
To achieve a more compact model, the  $f_{\theta}(z_t,t)$  is designed as a simple multiple-layer residual neural network. In the following, we will specify two designs that aim to explicitly describe the how  $g_v(x_{0:t})$  and  $h_w(t)$  depends on the input signal  $x_{0:t}$  and time  $t$ , respectively.
2.1 Attention-based network for modeling the change of input signal
The change of input signal  $x$  can directly influence the local dynamics of the hidden states. To incorporate the input signal into the latent dynamics  $f$  in a differentiable manner, we have to rely on a continuous input signal. However, the observed times series is either regular or irregular sampled sequential data  $\{x_{t}\}_{t = 0}^{T}$ , thus, we first need to conduct a continuous approximation over the discrete time series. Inspired by the idea of neural controlled differential equation, cubic spline could be a natural alternative due to its minimum regularity for handling certain edge cases. Concretely, let  $X$  denote the cubic spline with knots  $t = 0,\dots ,T$  such that  $X_{t} = (x_{t},t)\in \mathbb{R}^{d_{x} + 1}$ .
With the continuous approximation  $X$ , typically called "signal path", the dynamic change of input signal could be depicted by the derivative  $\frac{dX}{dt}$ , and the component  $g_{v}(x_{0:t})$  in the decomposed latent dynamics Eq (6) is specified as
$$
g _ {v} \left(x _ {0: t}\right) = g _ {v} \left(\frac {d X}{d t}\right) \in \mathbb {R} ^ {d _ {x} + 1}, \tag {7}
$$
where  $g_v: \mathbb{R}^{d_x + 1} \to \mathbb{R}^{d_x + 1}$  in our paper is a trainable attention-based network to describe how the change of input signal influences the latent dynamics, inspired by the idea of memory networks [30] and matching networks [31].
Concretely, let  $M \in \mathbb{R}^{d_m \times (d_x + 1)}$  be the memory matrix to be learned for the path gradient,  $dX / dt$ , shared for the entire time series. This matrix aims to memorize representative patterns of the local change along the signal path, stored as  $d_m$  rows of  $M$  that can be used further pattern matching. Then, for each time  $t$ , an attention-based transformation is employed for weighting the patterns stored in the memory matrix  $M$  to obtain the final expressive control signal,
$$
g _ {v} \left(\frac {d X}{d t}\right) = M ^ {\top} \operatorname {s o f t m a x} \left(M \frac {d X}{d t}\right). \tag {8}
$$
Our memory-based modeling over the local change of the signal path is different from that of Neural CDE [13], where only the path gradient  $dX / dt$  for a single time point was used. The introduction of the memory matrix allows the model to combine different patterns of local changes, which could also help to provide more discriminative features particularly for the segments with abrupt changes.
We visualize the attention weights  $\text{softmax}(M \frac{dX}{dt})$  along one time series in Weather dataset in Figure 2, where memory size  $d_m = 128$  and the length of time series is 250. We can observe that around  $t = 150$  with obvious changes, the learned attention weights are extremely sparse and distinguishing from other segments.
2.2 Seasonality-trend transformation-based latent dynamics
In most of existing neural ODEs, the dynamics function only depends on  $z_{t}$ , not explicitly on the time  $t$ . However, in the scenario of time series, particularly for those with complex periodic and
trending characteristics, only relying on  $z_{t}$  and  $x_{0:t}$ ,  $F(\cdot)$  might not be sufficient for capturing the hidden seasonal and trendy attributes. To this end, we propose to force the latent dynamic function to explicitly depend on the time  $t$ , as described by the decomposed formulation Eq (6), which forms a non-autonomous ODE in the theory of dynamical system.
With the decomposable latent dynamics, we propose to design the time-dependency function  $h_w(t)$  to characterize the intrinsic seasonality and trend property based on basis expansion. It is the combination of a periodic function  $h_w^{(s)}$  parameterized by  $w^{(s)}$  to model the seasonal characteristics and a polynomial function  $h_w^{(p)}$  parameterized by  $w^{(p)}$  to model the trend features:
$$
h _ {w} ^ {(s)} (t) = \sum_ {i = 0} ^ {m _ {s}} w _ {i} ^ {(s)} (\cos (2 ^ {i + 1} \pi t) + \sin (2 ^ {i + 1} \pi t)) \tag {9}
$$
$$
h _ {w} ^ {(p)} (t) = \sum_ {j = 0} ^ {m _ {p}} w _ {j} ^ {(p)} t ^ {j} \tag {10}
$$
where  $h_w^{(s)}(t)$  takes the form of trigonometric basis expansion with pre-specified frequencies  $\{1, 1/2, 1/4, \ldots, 1/2^{m_s + 1}\}$ ,  $m_s$  and  $m_p$  are the number of seasonal and polynomial basis, respectively. Then, the time-dependency factor  $h_w(t)$  for the latent dynamics accounts to the sum of the two terms,
$$
h _ {w} (t) = h _ {w} ^ {(s)} (t) + h _ {w} ^ {(p)} (t) \tag {11}
$$
Then, we multiply the vector field output  $f_{\theta}(z_t)$  with the time-dependency function  $h_w(t)$ , and define it as the  $l$ -th basic block,
$$
B \left(z _ {t} ^ {(l)}, t\right) = h _ {w} (t) f _ {\theta} \left(z _ {t} ^ {(l)}\right) \tag {12}
$$
We stack multiple layers of the basic block in a residual manner that is easy to train,
$$
z _ {t} ^ {(0)} = z _ {t}, z _ {t} ^ {(1)} = z _ {t} ^ {(0)} - B \left(z _ {t} ^ {(0)}, t\right), \dots , z _ {t} ^ {(L)} = z _ {t} ^ {(L - 1)} - B \left(z _ {t} ^ {(L - 1)}, t\right). \tag {13}
$$
Therefore, by taking the time seasonality and trend into consideration, the decomposed formulation of  $F(\cdot)$  becomes,
$$
F _ {\Theta} \left(z _ {t}, x _ {0: t}, t\right) = B \left(z _ {t}, t\right) g _ {v} \left(\frac {d X}{d t}\right) = h _ {w} (t) f _ {\theta} \left(z _ {t}\right) g _ {v} \left(\frac {d X}{d t}\right). \tag {14}
$$
where  $\{\theta, w, v\}$  are parameters to be learned for the decomposed functions  $h(t), f(z_t, t)$  and  $g(x_{0:t})$  respectively. Finally, the solution of hidden state  $z_t$  can be derived as follows, which is amenable to typical neural ODE solver,
$$
z _ {t} = z _ {0} + \int_ {0} ^ {t} h _ {w} (s) f _ {\theta} \left(z _ {s}\right) g _ {v} \left(\frac {d X}{d s}\right) d s, \tag {15}
$$
where  $f_{\theta}(z_t) \in \mathbb{R}^{d_z \times (d_x + 1)}$  to accommodate the dimensionality of  $g_v(dX / dt)$ , specified as a simple residual network.
3 Multivariate Neural Lad
The multivariate time series forecasting typically aim to model the temporal and spatial relationship between multiple time serieses simultaneously to enhance the prediction performance. Representative models include STGCN [36], AGCRN [2] and NRI [14], among which AGCRN is the most effective and efficient approach. Inspired by AGCRN, we adaptively optimize the graph structure and learn the spatial correlation with a generalized graph convolution operation jointly.
We start with the graph convolution [15] for capturing the spatial dependency between multivariate time series for each time slice  $t$ ,
$$
\Phi_ {t} = \left(I + D ^ {- \frac {1}{2}} A D ^ {- \frac {1}{2}}\right) Z _ {t} \Lambda + b, \tag {16}
$$
where  $Z_{t}\in \mathbb{R}^{N\times d_{z}}$  represents the  $t$ -th slice of hidden states for the  $N$  time series, and  $\Phi_t\in \mathbb{R}^{N\times d_\Phi}$  collects the projected states into a matrix,  $D\in \mathbb{R}^{N\times N}$  is the degree matrix,  $\Lambda \in \mathbb{R}^{d_z\times d_\Phi}$  and  $b$  are
the trainable weight and offset,  $A \in \mathbb{R}^{N \times N}$  is the adjacency matrix, and  $I + D^{-\frac{1}{2}}AD^{-\frac{1}{2}}$  is the low-order Chebyshev matrix. Therefore, Eq.(16) is the graph convolution on the hidden states to capture the correlation between multiple time series (i.e. nodes).
To make the adjacency matrix  $A$  more flexible, a learnable embedding matrix  $E \in \mathbb{R}^{N \times d_e}$  is designed where each row represents the embedding for each time series and  $d_e$  is the dimension of node embedding. Thus,  $E E^T$  could measure the similarity between these nodes. Then the normalized adjacency matrix  $D^{-\frac{1}{2}} A D^{-\frac{1}{2}}$  is approximated by the following nonlinear transformation,
$$
D ^ {- \frac {1}{2}} A D ^ {- \frac {1}{2}} = \phi \left(\sigma \left(E E ^ {\top}\right)\right), \tag {17}
$$
where  $\sigma$  is the ReLU activation function and  $\phi$  is the softmax operation. We now specify a generalized version of the graph convolution in Eq. (16) to improve adaptivity and expressivity. The spatial embedding  $s$  for all the  $N$  times series are defined as follows,
$$
\Phi_ {t} = \left(I + \phi \left(\sigma \left(E E ^ {\top}\right)\right) Z _ {t} \Lambda \in \mathbb {R} ^ {N \times d _ {z}} \right. \tag {18}
$$
where  $I + \phi (\sigma (EE^{\top}))$  is the adaptive Chebyshev matrix inspired by AGCRN [2], and we let the dimension of the embedding of  $\Phi_t$  is the same as that of the latent state  $z_{t}$  for simplicity, i.e.  $d_z$ . Thus, for  $i$ -th time series, the spatial component corresponds to the  $i$ -th row of the embedding matrix  $\Phi_t$
$$
\phi_ {\gamma} ^ {(i)} (Z _ {t}) = \Phi_ {t} [ i,: ], \tag {19}
$$
where  $\gamma = \{\Lambda, E\}$  denotes the collection of parameters to be learned. Then, for each time series, the hidden state  $z_{t}$  incorporating both temporal dynamics and spatial correlation is modeled as:
$$
z _ {t} = z _ {0} + \int_ {0} ^ {t} h _ {w} (s) \phi_ {\gamma} \left(Z _ {s}\right) f _ {\theta} \left(z _ {s}\right) g _ {v} \left(\frac {d X}{d s}\right) d s. \tag {20}
$$
The Eq.(20) is the multi-variate extension of Eq.(15) with the additional spatial component  $\phi_{\gamma}(Z_t)$ . In the encoder stage, we evaluate the time-dependent dynamics  $h_w(t)$ ,  $f_{\theta}(z_t)$ , transformation of the signal path  $g_v(dX / dt)$ , then obtain the  $z_t$  for all the time steps by ODESolver. In the decoder stage, we generate the hidden states and predicted outputs in a step-by-step or one-shot manner through  $\xi(z_T)$ .
Discussions We now summarize several benefits of Neural Lad for times series modeling as follows.
1. It is the first neural ODE framework for time series in which the dynamic function explicitly depends on the hidden state  $z_{t}$ , input  $x_{0:t}$  and the time  $t$ . It can seamlessly fit into the training procedure of neural ODE family and the model may be trained with memory-efficient adjoint-based backpropagation even across observations.
2. The explicit dependence on the time  $t$  through the decomposable latent dynamics  $B(z_{t},t) = h_{w}(t)f_{\theta}(z_{t})$  allows our neural ODE to adaptively reveal the periodicity of complex time series. Besides, the  $g_{v}\left(\frac{dX}{dt}\right)$  learns the weight of the control signal. To extend to multivariate time series, we design the function  $\psi_{\gamma}(z_{t})$  to learn the latent dynamics from spatial correlation. This can also be verified from experimental part in Section 4.
3. When considering the relationship between input and hidden states, we employ the continuous approximation of observed discrete time series to facilitate the integration over the continuous. Thus, for the original time series either regularly or irregularly sampled, they can be fit into our model for training and inference.
We conduct experiments on both synthetic and real-world datasets on univariate, multivariate and irregularly sampled time series forecasting tasks to demonstrate the empirical performance of our proposed approach.
Table 1: Comparison between Neural Lad and STG-NCDE on synthetic dataset.
HORIZON	STG-NCDE	ONLYhw	ONLYgv	NEURAL LAD
MAE	RMSE	MAE	RMSE	MAE	RMSE	MAE	RMSE
12	2.31	3.85	1.69	3.21	1.44	2.62	1.27	2.38
48	2.55	4.52	1.61	2.79	1.48	2.59	1.47	2.54
96	3.37	5.29	2.28	3.82	2.07	3.57	1.77	3.12
Figure 3: Left two panels: two examples of generated time series. Right two panels: prediction results. Blue: the input signal and ground truth. Green: STG-NCDE [6]. Red: Neural Lad.
4.1 Regularly sampled time series prediction: synthetic data
We first implement the multivariate Neural Lad on a synthetic dataset consisting of 30 periodic time series with various frequencies and linearly increasing amplitudes. The code for generating synthetic time series follows that of Neural ODE [27]. Concretely, 30 time series each with 2304 time steps are generated according to the functions  $x_{i,t} = a_{i,t}\sin (2\pi b_{i,t}t + \phi) + n_{i,t}$ , where  $i$  is the index of each time series,  $\phi$  is the phase,  $n_{i,t}$  denotes the standard Gaussian noise. The amplitude and frequency ascends linearly with time  $t$ , i.e.  $a_{i,t} = a_{i,0} + \frac{a_{max} - a_{min}}{t_{max} - t_{min}} t$  and  $b_{i,t} = b_{i,0} + \frac{b_{max} - b_{min}}{t_{max} - t_{min}} t$ , where for each time series the initial parameters are sampled from uniform distribution,  $a_{i,0} \sim U[2,10]$  and  $b_{i,0} \sim U[0,10]$  to increase the diversity.