SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yong Liu\* Chenyu Li\* Jianmin Wang, Mingsheng Long
School of Software, BNRist, Tsinghua University, China
{liuyong21, lichenyu20}@mails.tsinghua.edu.cn, {jimwang,mingsheng}@tsinghua.edu.cn
Real-world time series are characterized by intrinsic non-stationarity that poses a principal challenge for deep forecasting models. While previous models suffer from complicated series variations induced by changing temporal distribution, we tackle non-stationary time series with modern Koopman theory that fundamentally considers the underlying time-variant dynamics. Inspired by Koopman theory that portrays complex dynamical systems, we disentangle time-variant and time-invariant components from intricate non-stationary series by Fourier Filter and design Koopman Predictor to advance respective dynamics forward. Technically, we propose Koopa as a novel Koopman forecaster composed of stackable blocks that learn hierarchical dynamics. Koopa seeks measurement functions for Koopman embedding and utilizes Koopman operators as linear portraits of implicit transition. To cope with time-variant dynamics that exhibits strong locality, Koopa calculates context-aware operators in the temporal neighborhood and is able to utilize incoming ground truth to scale up forecast horizon. Besides, by integrating Koopman Predictors into deep residual structure, we ravel out the binding reconstruction loss in previous Koopman forecasters and achieve end-to-end forecasting objective optimization. Compared with the state-of-the-art model, Koopa achieves competitive performance while saving  $77.3\%$  training time and  $76.0\%$  memory. Code is available at this repository: https://github.com/thuml/Koopa.
Time series forecasting has become an essential part of real-world applications, such as weather forecasting, energy consumption, and financial assessment. With numerous available observations, deep learning approaches exhibit superior performance and bring the boom of deep forecasting models. TCNs [4, 43, 47] utilize convolutional kernels and RNNs [12, 22, 37] leverage the recurrent structure to capture underlying temporal patterns. Afterward, attention mechanism [42] becomes the mainstream of sequence modeling and Transformers [31, 48, 53] show great predictive power with the capability of learning point-wise temporal dependencies. And the recent revival of MLPs [32, 51, 52] presents a simple but effective approach to exhibit temporal dependencies by dense weighting.
In spite of elaboratively designed models, it is a fundamental problem for deep models to generalize on varying distribution [1, 25, 33], which is widely reflected in real-world time series because of inherent non-stationarity. Non-stationary time series is characterized by time-variant statistics and temporal dependencies in different periods [2, 14], inducing a huge distribution gap between training and inference and even among each lookback window. While previous methods [16, 28] tailor existing architectural design to attenuate the adverse effect of non-stationarity, few works research on the theoretical basis that can be applied to deal with time-variant temporal patterns naturally.
Figure 1: The measurement function  $g$  maps between non-stationary time series and the nonlinear dynamical system so that the timeline will correspond to a system trajectory. Therefore, time series variations in different periods are reflected as sub-regions of nonlinear dynamics, which can be portrayed and advanced forward in time by linear Koopman operators  $\{\mathcal{K}_1,\mathcal{K}_2,\mathcal{K}_3\}$  respectively.
From another perspective, real-world time series acts like time-variant dynamics [6]. As one of the principal approaches to analyze complex dynamics, Koopman theory [20] provides an enlightenment to transform nonlinear system into measurement function space, which can be described by a linear Koopman operator. Several pilot works accomplish the integration with deep learning approaches by employing autoencoder networks [40] and operator-learning [26, 49]. More importantly, it is supported by Koopman theory that for time-variant dynamics, there exists a coordinate transformation of the system, where localized Koopman operators are valid to describe the whole measurement function space into several subspaces with linearization [23, 36]. Therefore, Koopman-based methods are appropriate to learn non-stationary time series dynamics (Figure 1). Besides, the linearity of measurement function space enables us to utilize spectral analysis to interpret nonlinear systems.
In this paper, we disentangle non-stationary series into time-invariant and time-variant dynamics and propose Kooppa as a novel Koopman forecaster, which is composed of modular Koopman Predictors (KP) to hierarchically describe and advance forward series dynamics. Concretely, we utilize Fourier analysis for dynamics disentangling. And for time-invariant dynamics, the model learns Koopman embedding and linear operators to reveal the implicit transition underlying long-term series. As for the remaining time-variant components that exhibit strong locality, Koopa performs context-aware operator calculation and adaptation within different lookback windows. Besides, Koopman Predictor goes beyond the canonical design of Koopman Autoencoder without the binding reconstruction loss, and we incorporate modular blocks into deep residual architecture [32] to realize end-to-end time series forecasting. Our contributions are summarized as follows:
- From the perspective of modern dynamics Koopman theory, we propose Koopa composed of modular Fourier Filter and Koopman Predictor, which can hierarchically disentangle and exploit time-invariant and time-variant dynamics for time series forecasting.
- Based on the linearity of Koopman operators, the proposed model is able to utilize incoming series and adapt to varying dynamics for scaling up forecast horizon.
- Compared with state-of-the-art methods, our model achieves competitive performance while saving  $77.3\%$  training time and  $76.0\%$  memory averaged from six real-world benchmarks.
2.1 Time Series Forecasting with DNNs
Deep neural networks (DNNs) have made great breakthroughs in time series forecasting. TCN-based models [4, 43, 47] explore hierarchical temporal patterns and adopt shared convolutional kernels with diverse receptive fields. RNN-based models [12, 22, 37] utilize the recurrent structure with memory to reveal the implicit transition over time points. MLP-based models [32, 51, 52] learn point-wise weighting and the impressive performance and efficiency highlight that MLP performs well for modeling simple temporal dependencies. However, their practical applicability may still be constrained on non-stationary time series, which is endowed with time-variant properties and poses challenges for model capacity and efficiency. Unlike previous methods, Koopa fundamentally considers the complicated dynamics underlying time series and implements efficient and interpretable transition learners in both time-variant and time-invariant manners inspired by Koopman theory.
Recently, Transformer-based models have also achieved great success in time series forecasting. Initial attempts [19, 27, 48, 53] renovate the canonical structure and reduce the quadratic complexity
for long-term forecasting. However, recent studies [16, 21] find it a central problem for Transformer and other DNNs to generalize on varying temporal distribution and several works [15, 16, 21, 28] tailor to empower the robustness against shifted distribution. Especially, PatchTST [31] boosts Transformer to the state-of-the-art performance by channel-independence and instance normalization [41] but may lead to unaffordable computational cost when the number of series variate is large. In this paper, our proposed model supported by Koopman theory works naturally for non-stationary time series and achieves the state-of-the-art forecasting performance with remarkable model efficiency.
2.2 Learning Dynamics with Koopman Operator
Koopman theory [20] has emerged over decades as the dominant perspective to analyze modern dynamical systems [6]. Together with Dynamic Mode Decomposition (DMD) [38] as the leading numerical method to approximate the Koopman operator, significant advances have been accomplished in aerodynamics and fluid physics [3, 9, 30]. Recent progress made in Koopman theory is inherently incorporated with deep learning approaches in the data science era. Pilot works [29, 40, 50] leverage data-driven approaches such as Koopman Autoencoder to learn the measurement function and operator simultaneously. PCL [3] further introduces a backward procedure to improve the consistency and stability of the operator. Based on the capability of Koopman operator to advance nonlinear dynamics forward, it is also widely applied to sequence prediction. By means of Koopman spectral analysis, MDKAE [5] disentangles dominant factors underlying sequential data and is competent to forecast with specific factors. K-Forecast [24] utilizes Koopman theory to handle nonlinearity in temporal signals and propose to optimize data-dependent basis for long-term time series forecasting. By leveraging predefined measurement functions, KNF [44] learns Koopman operator and attention map to cope with time series forecasting with changing temporal distribution.
Different from previous Koopman forecasters, we design modular Koopman Predictors to tackle time-variant and time-invariant components with hierarchically learned operators, and renovate Koopman Autoencoder by removing the reconstruction loss to achieve fully predictive training.
3.1 Koopman Theory
A discrete-time dynamical system can be formulated as  $x_{t + 1} = \mathbf{F}(x_t)$ , where  $x_{t}$  denotes the system state and  $\mathbf{F}$  is a vector field describing the dynamics. However, it is challenging to identify the system transition directly on the state because of nonlinearity or noisy data. Instead, Koopman theory [20] hypothesizes the state can be projected into the space of measurement function  $g$ , which can be governed and advanced forward in time by an infinite-dimensional linear operator  $\mathcal{K}$ , such that:
$$
\mathcal {K} \circ g (x _ {t}) = g (\mathbf {F} (x _ {t})) = g (x _ {t + 1}). \tag {1}
$$
Koopman theory provides a bridge between finite-dimensional nonlinear dynamics and infinite-dimensional linear dynamics, where spectral analysis tools can be applied to obtain in-depth analysis.
3.2 Dynamic Mode Decomposition
Dynamic Mode Decomposition (DMD) [38] seeks the best fitted finite-dimensional matrix  $K$  to approximate infinite-dimensional operator  $\mathcal{K}$  by collecting the observed system state (a.k.a. snapshot). Although DMD is the standard numerical method to analyze dynamics, it only works on linear space assumptions, which can be hardly identified without prior knowledge. Therefore, eDMD [45] is proposed to avoid handcrafting measurement functions and harmonious incorporations are made with the learning approach by employing autoencoders, which yields Koopman Autoencoder (KAE). By the universal approximation theorem [13] of deep networks, KAE finds desired Koopman embedding  $g(x_{t})$  with learned measurement function in a data-driven approach.
3.3 Time Series as Dynamics
It is challenging to predict real-world time series because of inherent non-stationarity. But if we zoom in the timeline, we will find the localized time series exhibited weak stationarity. It coincides with Koopman theory to analyze large nonlinear dynamics. That is, the measurement
Figure 2: Left: Koopa structure. By taking the residual of previous block fitted dynamical system, each block learns hierarchical dynamics and Koopa aggregates the forecast of all blocks. Right: For time-invariant forecast, Koopa learns globally shared dynamics from each lookback-forecast window pair. For time-variant forecast, the model calculates localized and segment-wise dynamics.
function space can be divided into several neighborhoods, which are discriminately portrayed by localized linear operators [23]. Therefore, we leverage Koopman-based approaches that tackle large nonlinear dynamics by disentangling time-variant and time-invariant dynamics. Inspired by Wold's Theorem [46] that every covariance-stationary time series  $X_{t}$  can be formally decomposed as:
$$
X _ {t} = \eta_ {t} + \sum_ {j = 0} ^ {\infty} b _ {j} \varepsilon_ {t - j}, \tag {2}
$$
where  $\eta_{t}$  denotes the deterministic component and  $\varepsilon_{t}$  is the stochastic component as the stationary process input of linear filter  $\{b_{j}\}$ , we introduce globally learned and localized linear Koopman operators to exploit respective dynamics underlying different components.
4 Koopa
We propose Koopa composed of stackable Koopa Blocks (Figure 2). Each block is obliged to learn the input dynamics and advance it forward for prediction. Instead of struggling to seek one unified operator that governs the whole measurement function space, each Koopa Block is encouraged to learn operators hierarchically by taking the residual of previous block fitted dynamics as its input.
**Koopa Block** As aforementioned, it is essential to disentangle different dynamics and adopt proper operators for non-stationary series forecasting. The proposed block shown in Figure 2 contains Fourier Filter that utilizes frequency domain statistics to disentangle time-variant and time-invariant components and implements two types of Koopman Predictor (KP) to obtain Koopman embedding respectively. In Time-invariant KP, we set the operator as a model parameter to be globally learned from lookback-forecast windows. In Time-variant KP, analytical operator solutions are calculated locally within the lookback window, with series segments arranged as snapshots. In detail, we formulate the  $b$ -th block input  $X^{(b)}$  as  $[x_1,x_2,\dots ,x_T]^\top \in \mathbb{R}^{T\times C}$ , where  $T$  and  $C$  denote the lookback window length and the variate number. The target is to output a forecast window of length  $H$ . Our proposed Fourier Filter conducts disentanglement at the beginning of each block:
$$
X _ {\text {v a r}} ^ {(b)}, X _ {\text {i n v}} ^ {(b)} = \operatorname {F o u r i e r F i l t e r} \left(X ^ {(b)}\right). \tag {3}
$$
Respective KPs will predict with time-invariant input  $X_{\mathrm{inv}}^{(b)}$  and time-variant input  $X_{\mathrm{var}}^{(b)}$ , and Time-variant KP simultaneously outputs the fitted input  $\hat{X}_{\mathrm{var}}^{(b)}$ :
$$
Y _ {\text {i n v}} ^ {(b)} = \operatorname {T i m e I n v K P} \left(X _ {\text {i n v}} ^ {(b)}\right), \tag {4}
$$
$$
\hat {X} _ {\text {v a r}} ^ {(b)}, Y _ {\text {v a r}} ^ {(b)} = \operatorname {T i m e V a r K P} \left(X _ {\text {v a r}} ^ {(b)}\right).
$$
Figure 3: Left: Time-invariant KP learns Koopman embedding and operator with time-invariant components globally from all windows. Right: Time-variant KP conducts localized operator calculation within lookback window and advances dynamics forward with the obtained operator for predictions.
Unlike KAEs [45, 44] that introduce a loss term for rigorous reconstruction of the lookback-window series, we feed the residual  $X^{(b + 1)}$  as the input of next block for learning a corrective operator. And the model forecast  $Y$  is the sum of predicted components  $Y_{\mathrm{var}}^{(b)}, Y_{\mathrm{inv}}^{(b)}$  gathered from all Koopa Blocks:
$$
X ^ {(b + 1)} = X _ {\text {v a r}} ^ {(b)} - \hat {X} _ {\text {v a r}} ^ {(b)}, Y = \sum \left(Y _ {\text {v a r}} ^ {(b)} + Y _ {\text {i n v}} ^ {(b)}\right). \tag {5}
$$
Fourier Filter To disentangle the series components, we leverage Fourier analysis to find the globally shared and localized frequency spectrums reflected on different periods. Concretely, we precompute the Fast Fourier Transform (FFT) of each lookback window of the training set, calculate the averaged amplitude of each spectrum  $S = \{0,1,\dots ,[T / 2]\}$ , and sort them by corresponding amplitude. We take the top percent of  $\alpha$  as the subset  $\mathcal{G}_{\alpha}\subset S$ , which contains dominant spectrums shared among all lookback windows and exhibits time-invariant dynamics underlying the dataset. And the remaining spectrums are the specific ingredient for varying windows in different periods. Therefore, we divide the spectrums  $S$  into  $\mathcal{G}_{\alpha}$  and its complementary set  $\bar{\mathcal{G}}_{\alpha}$ . During training and inference, FourierFilter(·) conducts the disentanglement of input  $X$  (block superscript omitted) as
$$
X _ {\text {i n v}} = \mathcal {F} ^ {- 1} \big (\operatorname {F i l t e r} \left(\mathcal {G} _ {\alpha}, \mathcal {F} (X)\right) \big),
$$
$$
\begin{array}{l} X _ {\text {i n v}} = \mathcal {F} ^ {- 1} \left(\operatorname {F i l t e r} \left(\bar {\mathcal {G}} _ {\alpha}, \mathcal {F} (X)\right)\right) = X - X _ {\text {i n v}}, \end{array} \tag {6}
$$
where  $\mathcal{F}$  means FFT,  $\mathcal{F}^{-1}$  is its inverse and  $\mathrm{Filter}(\cdot)$  only passes corresponding frequency spectrums with the given set. We validate the disentangling effect of our proposed Fourier Filter in Section 5.2 by calculating the variation degree of temporal dependencies in the disentangled series.
Time-invariant KP Time-invariant KP is designed to portray the globally shared dynamics, which discovers the direct transition from lookback window to forecast window as  $\mathbf{F}:X_{\mathrm{inv}}\mapsto Y_{\mathrm{inv}}$ . Concretely, we introduce a pair of Encoder:  $\mathbb{R}^{T\times C}\mapsto \mathbb{R}^D$  and Decoder:  $\mathbb{R}^D\mapsto \mathbb{R}^{H\times C}$  to learn the common Koopman embedding for the time-invariant components of running window pairs, where  $D$  denotes the embedding dimension. Working on the data-driven measurement function, we introduce the operator  $K_{\mathrm{inv}}\in \mathbb{R}^{D\times D}$  as a learnable parameter in each Time-invariant KP, which regards the embedding of lookback and forecast window  $Z_{\mathrm{back}},Z_{\mathrm{fore}}\in \mathbb{R}^{D}$  as running snapshot pairs. The procedure is shown in Figure 3 and TimeInvKP(·) is formulated as follows:
$$
Z _ {\text {b a c k}} = \operatorname {E n c o d e r} \left(X _ {\text {i n v}}\right), Z _ {\text {f o r e}} = K _ {\text {i n v}} Z _ {\text {b a c k}}, Y _ {\text {i n v}} = \operatorname {D e c o d e r} \left(Z _ {\text {f o r e}}\right). \tag {7}
$$
Time-variant KP As time-variant dynamics changes continuously, we utilize localized snapshots in a window, which constitute a temporal neighborhood more likely to be linearized. To obtain semantic snapshots and reduce iterations, the input  $X_{\mathrm{var}}$  is divided into  $\frac{T}{S}$  segments  $\mathbf{x}_j$  of length  $S$ :
$$
\mathbf {x} _ {j} = \left[ x _ {(j - 1) S + 1}, \dots , x _ {j S} \right] ^ {\top} \in \mathbb {R} ^ {S \times C}, j = 1, 2, \dots , T / S. \tag {8}
$$
We assume  $S$  is divisible by  $T$  and  $H$ ; otherwise, we pad the input or truncate the output to make it compatible. Time-variant KP aims to portray localized dynamics, which is manifested analytically as the segment-wise transition  $\mathbf{F}:\mathbf{x}_t\mapsto \mathbf{x}_{t + 1}$  with observed snapshots. We utilize another pair of Encoder:  $\mathbb{R}^{S\times C}\mapsto \mathbb{R}^D$  to transform each segment into Koopman embedding  $z_{j}$  and Decoder:  $\mathbb{R}^D\mapsto \mathbb{R}^{S\times C}$  to transform the fitted or predicted embedding  $\hat{z}_j$  back to time segments  $\hat{\mathbf{x}}_j$ :
$$
z _ {j} = \operatorname {E n c o d e r} \left(\mathbf {x} _ {j}\right), \hat {\mathbf {x}} _ {j} = \operatorname {D e c o d e r} \left(\hat {z} _ {j}\right). \tag {9}
$$
Given snapshots collection  $Z = [z_{1},\dots ,z_{\frac{T}{S}}]\in \mathbb{R}^{D\times \frac{T}{S}}$ , we leverage eDMD [45] to find the best fitted matrix that advances forward the system. We apply one-step operator approximation as follows:
$$
Z _ {\text {b a c k}} = \left[ z _ {1}, z _ {2}, \dots , z _ {\frac {T}{S} - 1} \right], Z _ {\text {f o r e}} = \left[ z _ {2}, z _ {3}, \dots , z _ {\frac {T}{S}} \right], K _ {\text {v a r}} = Z _ {\text {f o r e}} Z _ {\text {b a c k}} ^ {\dagger}, \tag {10}
$$
where  $Z_{\mathrm{back}}^{\dagger} \in \mathbb{R}^{(\frac{T}{S} - 1) \times D}$  is the Moore-Penrose inverse of lookback window embedding collection. The calculated  $K_{\mathrm{var}} \in \mathbb{R}^{D \times D}$  varies with windows and helps to analyze local temporal variations as a linear system. With the calculated operator, the fitted embedding is formulated as follows:
$$
[ \hat {z} _ {1}, \hat {z} _ {2}, \dots , \hat {z} _ {\frac {T}{S}} ] = [ z _ {1}, K _ {\text {v a r}} z _ {1}, \dots , K _ {\text {v a r}} z _ {\frac {T}{S} - 1} ] = [ z _ {1}, K _ {\text {v a r}} Z _ {\text {b a c k}} ]. \tag {11}
$$
To obtain a prediction of length  $H$ , we iterate operator forwarding to get  $\frac{H}{S}$  predicted embedding:
$$
\hat {z} _ {\frac {T}{S} + t} = \left(K _ {\text {v a r}}\right) ^ {t} z _ {\frac {T}{S}}, t = 1, 2, \dots , H / S. \tag {12}
$$
Finally, we arrange the segments transformed by  $\mathrm{Decoder}(\cdot)$  as the module outputs  $\hat{X}_{\mathrm{var}}, Y_{\mathrm{var}}$ . The whole procedure is shown in Figure 3 and TimeVarKP( $\cdot$ ) can be formulated as Equation 8-13.
$$
\hat {X} _ {\text {v a r}} = \left[ \hat {\mathbf {x}} _ {1}, \dots , \hat {\mathbf {x}} _ {\frac {T}{S}} \right] ^ {\top}, Y _ {\text {v a r}} = \left[ \hat {\mathbf {x}} _ {\frac {T}{S} + 1}, \dots , \hat {\mathbf {x}} _ {\frac {T}{S} + \frac {H}{S}} \right] ^ {\top}. \tag {13}
$$
Forecasting Objective In Koopa, Encoder, Decoder and  $K_{\mathrm{inv}}$  are learnable parameters, while  $K_{\mathrm{var}}$  is calculated on-the-fly. To maintain the Koopman embedding consistency in different blocks, we share Encoder, Decoder in Time-variant and Time-invariant KPs, which are formulated as  $\phi_{\mathrm{var}}$  and  $\phi_{\mathrm{inv}}$  respectively, and use the MSE loss with the ground truth  $Y_{\mathrm{gt}}$  for parameter optimization:
$$
\operatorname {a r g m i n} _ {K _ {\text {i n v}}, \phi_ {\text {v a r}}, \phi_ {\text {i n v}}} \mathcal {L} _ {\mathrm {M S E}} (Y, Y _ {\mathrm {g t}}). \tag {14}
$$
Optimizing by a single forecasting objective based on the assumption that if reconstruction failed, the prediction must also fail. Thus eliminating forecast discrepancy helps for fitting observed dynamics.
Datasets We conduct extensive experiments to evaluate the performance and efficiency of Koopa. For multivariate forecasting, we include six real-world benchmarks used in Autoformer [48]: ECL (UCI), ETT [53], Exchange [22], ILI (CDC), Traffic (PeMS), and Weather (Wetterstation). For univariate forecasting, we evaluate the performance on the well-acknowledged M4 dataset [39], which contains four subsets of periodically collected univariate marketing data. And we follow the data processing and split ratio used in TimesNet [47].
Notably, instead of setting a fixed lookback window length, for every forecast window length  $H$ , we set the length of lookback window  $T = 2H$  as the same with N-BEATS [32], because historical observations are always available in real-world scenarios and it can be beneficial for deep models to leverage more observed data with the increasing forecast horizon.
Baselines We extensively compare Koopa with the state-of-the-art deep forecasting models, including Transformer-based model: Autoformer [48], PatchTST [31]; TCN-based model: TimesNet [47], MICN [43]; MLP-based model: DLinear [51]; Fourier forecaster: FiLM [54], and Koopman forecaster: KNF [44]. We also introduce additional specialized models N-HiTS [7] and N-BEATS [32] for univariate forecasting as competitive baselines. All the baselines we reproduced are implemented based on the original paper or official code. We repeat each experiment three times with different random seeds and report the test MSE/MAE. And we provide detailed code implementation and hyperparameters sensitivity in Appendix C.
5.1 Time Series Forecasting