Koopa: Learning Non-stationary Time Series Dynamics with Koopman Predictors
Authors: Yong Liu
Chenyu Li
Jianmin Wang
Mingsheng Long
Venue: NeurIPS 2023 poster (2023)
Keywords: Time series forecasting
Deep learning
TL;DR: We propose a Koopman-based model for time series forecasting and our model achieves state-of-the-art performance with remarkable efficiency.

Abstract: Real-world time series are characterized by intrinsic non-stationarity that poses a principal challenge for deep forecasting models. While previous models suffer from complicated series variations induced by changing temporal distribution, we tackle non-stationary time series with modern Koopman theory that fundamentally considers the underlying time-variant dynamics. Inspired by Koopman theory of portraying complex dynamical systems, we disentangle time-variant and time-invariant components from intricate non-stationary series by Fourier Filter and design Koopman Predictor to advance respective dynamics forward. Technically, we propose Koopa as a novel Koopman forecaster composed of stackable blocks that learn hierarchical dynamics. Koopa seeks measurement functions for Koopman embedding and utilizes Koopman operators as linear portraits of implicit transition. To cope with time-variant dynamics that exhibits strong locality, Koopa calculates context-aware operators in the temporal neighborhood and is able to utilize incoming ground truth to scale up forecast horizon. Besides, by integrating Koopman Predictors into deep residual structure, we ravel out the binding reconstruction loss in previous Koopman forecasters and achieve end-to-end forecasting objective optimization. Compared with the state-of-the-art model, Koopa achieves competitive performance while saving 77.3% training time and 76.0% memory.

Koopa: Learning Non-stationary Time Series Dynamics with Koopman Predictors
Yong Liu; Chenyu Li; Jianmin Wang, Mingsheng Long School of Software, BNRist, Tsinghua University, China
(liuyong21, lichenyu2000mmaile tsinghua. edu. en, {jimwang,, miinghhengg)tssinggaa...
Abstract
Real-world time series are characterized by intrinsic non-stationarity that poses principal challenge for deep forecasting models. While previous models suffer from complicated series variations induced by changing temporal distribution, we tackle non-stationary time series with modern Koopman theory that fundamentally considers the underlying time-variant dynamics. Inspired by Koopman theory that portrays complex dynamical systems, we disentangle time-variant and time- invariant components from intricate non-stationary series by Fourier Filter and design Koopman Predictor to advance respective dynamics forward. Technically, we propose Koopa as novel Koopman forecaster composed of stackable blocks that learn hicrarchical dynamics. Koopa seeks measurement functions for Koop- man embedding and utilizes Koopman operators linear portraits of implicit transition. To cope with time-variant dynamics that exhibits strong locality, Koopa calculates context-aware operators in the temporal neighborhood and is able to utilize incoming ground truth to scale up forecast horizon. Besides, by integrating Koopman Predictors into deep residual structure, we ravel out the binding recon- struction loss in previous Koopman forecasters and achieve end-to-end forecasting objective optimization. Compared with the state-of-the-an model, Koopa achieves competitive performance while saving 77.3% training time and 76.0% memory. Code is available at this repository: https::/ htt//gitttt githubt..b..... www. Soom/hhul/KKopa
Introduction
Time series forecasting has become essential part of real-world applications, such weather forecasting, energy consumption, and financial assessment. With numerous available observations, deep learning approaches exhibit superior performance and bring the boom of deep forecasting
models. TCNs 43 47
utilize convolutional kernels and RNNs 12 22 37 leverage the recurrent
structure capture underlying temporal patterns. Afterward, attention mechanism 42 becomes the mainstream of sequence modeling and Transformers [31 48 53) show great predictive power with the capability of learning point-wise temporal dependencies. And the recent revival of MLPs B2 51 52
presents simple but effective approach to exhibit temporal dependencies by dense weighting.
In spite of elaboratively designed models, is fundamental problem for deep models to generalize on varying distribution [ 25 33,, which is widely reflected real-world time series because of inherent non-stationarity. Non-stationary time series is characterized by time-variant statistics and temporal dependencies different periods 2 14], inducing a huge distribution gap between training
and inference and even among each lookback window. While previous methods [16] 28 tailor existing architectural design to attenuate the adverse effect of non-stationarity, few works research on the theoretical basis that can be applied to deal with time-varian temporal patterns naturally.
Equal Contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Non-stationary Time series
Nonlinear System
Linear System
Time
Period
Measurement p(()
Parted
Function
Dynamics
Period
Figure 1: The measurement function 9 maps between non -tatioati time series and the nonlinear dynamical system so that the timeline will correspond to a system trajectory. Therefore, time series variations different periods are reflected as sub-regions of nonlinear dynamics, which can be portrayed and advanced forward in time by linear Koopman operators (A1,, K2, Ks)) respectively.
From another perspective, real-world time series acts like time-variani dynamics [6] As one of the principal approaches to analyze complex dynamics, Koopman theory ( provides an enlightenment to transform nonlinear system into measurement function space, which can be described by a linear Koopman operator. Several pilot works accomplish the integration with deep learning approaches
by employing autoencoder networks [40] and operator-learning [26] 49 More importantly, it is
supported by Koopman theory that for viriintrian dynamics, there exists coordinate transformation of the system, where localized Koopman operators are valid to describe the whole measurement
function space into several subspaces with linearization 23 36). Therefore, Koopman- bbased methods are appropriate to learn non-stationary time series dynamics (Figure I Besides, the linearity of measurement function space enables us utilize spectral analysis interpret nonlinear systems.
In this paper, we disentangle non-stationary series into time-invariant and time-variant dynamics and propose Koopa novel Koopman forecaster, which is composed of modular Koopman Predictors (KP) to hierarchically describe and advance forward series dynamics. Concretely, we utilize Fourier analysis for dynamics disentangling. And for time-invariant dynamics, the model learns Koopman embedding and linear operators to reveal the implicit transition underlying long-term series. As for the remaining time-variant components that exhibit strong locality, Koopa performs context-aware operator calculation and adaptation within different lookback windows. Besides, Koopman Predictor goes beyond the canonical design Koopman Autoencoder without the binding reconstruction loss, and we incorporate modular blocks into deep residual architecture 332 to realize end-to-end time series forecasting. Our contributions are summarized as follows:
From the perspective of modern dynamics Koopman theory, we propose Koopa composed of modular Fourier Filter and Koopman Predictor, which hierarchically disentangle and exploit time-invariant and time-variant dynamics for time series forecasting.
Based on the linearity of Koopman operators, the proposed model is able to utilize incoming series and adapt to varying dynamics for scaling up forecast horizon.
Compared with state-of-te fi-thh--rt methods, our model achieves competitive performance while saving 77.3% training time and 76.0% memory averaged from six real-world benchmarks.
Related Work
2.1 Time Series Forecasting with DNNs
Deep neural networks (DNNs) have made great breakthroughs time series forecasting. TCN-based models 4 43 47 explore hierarchical temporal patterns and adopt shared convolutional kernels with diverse receptive fields. RNN-based models [12] 22 37] utilize the recurrent structure with memory to reveal the implicit transition over time points. MLP-based models [32
51 52 learn
point-wise weighting and the impressive performance and efficiency highlight that MLP performs well for modeling simple temporal dependencies. However, their practical applicability may still be constrained on non-stationary time series, which is endowed with time-variant properties and poses challenges for model capacity and efficiency. Unlike previous methods, Koopa fundamentally considers the complicated dynamics underlying time series and implements efficient and interpretable transition learners in both time-variant and imme-invariant manners inspired by Koopman theory.
Recently, Transformer-basee models have also achieved great success in time series forecasting. Initial attempts 19 27 48 53 renovate the canonical structure and reduce the quadratic complexity

---

for long-term forecasting. However, recent studies [16] 21 find it central problem for Transformer
and other DNNs generalize on varying temporal distribution and several works [5 16
21 28 tailor
to empower the robustness against shifted distribution. Especially, PatchTST 31 boosts Transformer to the state-of-the-an performance by -annel-indeeendence and instance normalization 41 but may lead to unaffordable computational cost when the number of series variate is large. In this paper, our proposed model supported by Koopman theory works naturally for non-stationary time series and achieves the state-of-the-art forecasting performance with remarkable model efficiency.
2.2 Learning Dynamics with Koopman Operator
Koopman theory 20 has emerged over decades as the dominant perspective to analyze modern dynamical systems 6 Together with Dynamic Mode Decomposition (DMD) (38 as the leading nu- merical method to approximate the Koopman operator, significant advances have been accomplished
in aerodynamics and fluid physics 9| Recent progress made in Koopman theory is inherently
incorporated with deep learning approaches in the data science era. Pilot works [2]
40 50 leverage
data-driven approaches such as Koopman Autoencoder to learn the measurement function and oper- ator simultaneously. PCL [3] further introduces a backward procedure to improve the consistency and stability of the operator. Based on the capability of Koopman operator to advance nonlinear dynamics forward, it is also widely applied to sequence prediction. By means of Koopman spectral analysis, MDKAE [5] disentangles dominant factors underlying sequential data and is competent to forecast with specific factors. K-Forecast 24 utilizes Koopman theory to handle nonlincarity in temporal signals and propose to optimize data-dependent basis for long-term time series forecasting. By leveraging predefined measurement functions, KNF (44 learns Koopman operator and attention map to cope with time series forecasting with changing temporal distribution.
Different from previous Koopman forecasters, we design modular Koopman Predictors to tackle time-variant and time-invariant components with hierarchically learned operators, and renovate Koopman Autoencoder by removing the reconstruction loss to achieve fully predictive training.
Background
3.1 Koopman Theory
discrete-time dynamical system can be formulated as Fz-1 = F(z)), where Iy denotes the system state and F is vector field describing the dynamics. However, it is challenging identify the system transition directly on the state because of nonlinearity or noisy data. Instead, Koopman theory [20] hypothesizes the state be projected into the space of measurement function g, which can be governed and advanced forward in time by an infinite-dimensional linear operator K,, such that:
Ko g(x,) = ( (z.)) g(x++1).
(1)
Koopman theory provides a bridge between dimensional linear dynamics, where spectral analysis tools can be applied to obtain in-depth analysis.
nonlinear dynamics and infinite-
3.2 Dynamic Mode Decomposition
Dynamic Mode Decomposition (DMD) 38 seeks the best fitted finite- dintteidiinsionall matrix K to approximate infiniteddimensional operator by collecting the observed system state (a.k.a. snapshot). Although DMD is the standard numerical method analyze dynamics, it only works on linear space assumptions, which can be hardly identified without prior knowledge. Therefore, eDMD (45) is proposed to avoid handerafting measurement functions and harmonious incorporations the learning approach by employing autoencoders, which yields Koopman Autoencoder (KAE). By the universal approximation theorem [I3] of deep networks, KAE finds desired Koopman embedding g(x)) with learned measurement function in a data-driven approach.
made with
3.3 Time Series as Dynamies
It is challenging to predict real-world time series because of inherent non-stationarity. But if we zoom in the timeline, we will find the localized time series exhibited weak stationarity. coincides with Koopman theory analyze large nonlinear dynamics. That is, the measurement
3

---

Koopa Block
input x
Lookback Window
Forecast Window
Fourier Filter
Time-invariant forecast
Globally learned dynamics
X
Kine
Time-variant KP
Time-invariant KP
resid residual 7
forecast
Snapshet Pairs (Series window)
X
input
Koopa Block
12
Kiny
Knopa Block
Time-variant forerast
Locally calcuted dynamics
residual
forocast
Kvur
K
K
X
Koopa Block
model
forecast
Koopa
Snapshot Pairs (Series segmment
Figure 2: Left: Koopa structure. By taking the residual of previous block fitted dynamical system. each block learns hierarchical dynamics and Koopa aggregates the forecast of all blocks. Right: For time-invariant forecast, Koopa learns globally shared dynamics from each mookback-freecan window pair. For time-variant forecast, the model calculates localized and segment-wise dynamics.
function space can be divided into several neighborhoods, which are discriminately portrayed by localized linear operators (23). Therefore, we leverage Koopman- approaches that tackle large nonlinear dynamics by disentangling vimi-variant and time-invariant dynamics. Inspired by Wold's Theorem [46] that every covariance-stationay time series X, can be formally decomposed as:
X; = M/E
1=0
where 77, denotes the deterministic component and E, the stochastic component as the stationary process input of linear filter {by}- we introduce globally learned and localized linear Koopman operators exploit respective dynamics underlying different components.
4 Koopa
We propose Koopa composed of stackable Koopa Blocks (Figure D. Each block is obliged to learn the input dynamics and advance it forward for prediction. Instead of struggling to seek one unified operator that governs the whole measurement function space, each Koopa Block is encouraged to learn operators hierarchically by taking the residual of previous block fitted dynamics its input.
As aforementioned, it essential to disentangle different dynamics and adopt proper operators for non-stationary series forecasting. The proposed block shown in Figure 2 contains Fourier Filter that utilizes frequency domain statistics disentangle time-variant and time-invariant components and implements two types of Koopman Predictor (KP) to obtain Koopman embedding respectively. In Time-invariant KP, we set the operator as a model parameter be globally learned from bookback-forecan windows. In Time-variant KP, analytical operator solutions are calculated locally within the lookback window, with series segments arranged as snapshots. In detail, we formulate the b-th block input (b) as [x1, F2, xT]] E RTXC, where T and C' denote the lookback window length and the variate number. The target is to output forecast window of length H. Our proposed Fourier Filter conducts disentanglement at the beginning of each block:
Koopa Block
(b)
(b)
= FourierFilter(XX (x))).
var
inv
Respective KPs will predict with time- invariant input x and time-variant input X and Time-
variant KP simultaneously outputs the fitted input (b)
y(b)
TimeInvKP(XX (x((()
inv
X (b) var
(b) Time MareVarPP(XX) (())..

---

Time-invariant KP
input
Time-variant KP
input
X
calculate oar
Xinx
oooo Encoder
XC2
eDMD
Encoder
Encoder
Encader
o
fore
Kinv
Trnnck
Reur
2g
Kour
Rear
Neyr
O
Decoder
Decoder
Decader
Decoder
Decoder
000
X
x
X
Yuu
X4
inv
forecast
fitted input
forecast
Figure 3: Left: Time-invarian, KP learns Koopman embedding and operator with time-invariant com- ponents globally from all windows. Right: Time-variant KP conducts localized operator calculation within lookback window and advances dynamics forward with the obtained operator for predictions.
that introduce loss term for rigorous reconstruction of the lookhack-window series, we feed the residual ((++1) as the input of next block for learning a corrective operator. And the model forecast Y is the sum of predicted components y-(b) y-()) gathered from all Koopa Blocks:
Unlike KAFS 45
44
inv
x((+1)
x-()
(b) var
Y = Σ (yr(t)
Y-(b)
Fourier Filter To disentangle the series components, we leverage Fourier analysis find the globally shared and localized frequency spectrums reflected on different periods. Concretely, we precompute the Fast Fourier Transform (FFT) of each lookback window the training set, calculate the averaged amplitude of each spectrum S = {0, 1...., [T/2]]. and sort them by corresponding amplitude. We take the top percent of a as the subset Ga C which contains dominant spectrums shared among all lookback windows and exhibits invarian dynamics underlying the dataset. And the remaining spectrums are the specific ingredient for varying windows different periods. Therefore, we divide the spectrums S into G., and its complementary set G.. During training and inference, FourierFilter()) conducts the disentanglement of input (block superscript omitted) as
Xim = F- Filter (g... F(X))) X = == (Ga, F(X))) = Xine
(6)
where F means FFT, F- its inverse and Filter(-) only passes corresponding frequency spectrums with the given set. We validate the disentangling effect of our proposed Fourier Filter in Section 5.2 by calculating the variation degree of temporal dependencies in the disentangled series.
Time-invariant KP Time-invariant KP is designed to portray the globally shared dynamics, which discovers the direct transition from lookback window to forecast window as F Xim H Y Concretely, we introduce a pair of Encoder R2''C'' + RDD and Decoder IRD + & to learn the common Koopman embedding for the time-invariant components of rumning window pairs, where D denotes the embedding dimension. Working on the data-driven measurement function, we introduce the operator KIn D> as a learnable parameter in each Time-invariant KP, which regards the embedding of lookback and forecast window Znack Ztane E ED as running snapshot pairs. The procedure is shown in Figure 3 and TimeInvKP(.) is formulated as follows:
Zrack Encoder( Xinv), (innderrr(ii),)..))))))
Ztore Kin Zrack, Yím (rocoderrZrooe)
(7)
5

---

As vimi-variant dynamics changes continuously, we utilize localized snapshots in a window, which constitute temporal neighborhood more likely to be linearized. To obtain semantic snapshots and reduce iterations, the input X is divided into T g segments x, of length S: , T//..
Time-variant KP
(8) We assume S is divisible by T and H; otherwise, we pad the input or truncate the output to make it compatible. Time-variant KP aims to portray localized dynamies, which is manifested analytically as the segment-wise transition F X, + X,+1 with observed snapshots. We utilize another pair of Encoder RRC + RR to transform each segment into Koopman embedding 2j and Decoder: RDD + RXC transform the fitted or predicted embedding 23 back to time segments Xj:
x = [ 2zs!
2j Excoder(x,), ) x (
(9)
Given snapshots collection Z [ øg]
R Dx 5 leverage eDMD 45
to find the best
fitted matrix that advances forward the system. We apply one-step operator approximation as follows:
Track = [21, 2g 11: Zeor = [22 23....,; zy), Kvar 7 Tack
(10)
where Zback E ( is the Peorosee inverse of lookback window embedding collection. The calculated Kvar E IR DxD varies with windows and helps to analyze local temporal variations as a linear system. With the calculated operator, the fitted embedding is formulated as follows:
[&1, ...,, 2gl = [2), &1 K f [21, Zpack
(11)
To obtain a prediction of length H, we iterate operator forwarding get # 5 predicted embedding:
2 2r
(Kvar) zz. t t= 1,
(12)
Finally, we arrange the segments transformed by Decoder(.) as the module outputs X Yua. The whole procedure is shown in Figure 3 and Time VarKP(.) can be formulated as Equation 8
X [x,....,... &g1", Yvar = -x |xz+1....,,, +
(13)
Forecasting Objective In Koopa, Encoder, Decoder and Kinv are learnable parameters, while Kvrr is calculated on-yhe-fly. To maintain the Koopman embedding consistency in different blocks, share Encoder, Decoder in Time-variant and Time-invariant KPs, which are formulated óvar and Oin respectively, and use the MSE loss with the ground truth Yg: for parameter optimization:
argminK...eor.t.n &mm. LMSE (Y, Yg).-
(14)
Optimizing by single forecasting objective based on the assumption that if reconstruction failed, the prediction must also fail. Thus eliminating forecast discrepancy helps for fitting observed dynamics.
5 Experiments
Datasets We conduct extensive experiments to evaluate the performance and efficiency of Koopa. For multivariate forecasting, we include six real-world benchmarks used in Autoformer [48 ECL (UCI), ETT (3 Exchange [22). IIII (CDC), Traffic (PeMS), and Weather (Wetterstation). For univariate forecasting, we evaluate the performance the well-acknowledged M4 dataset [39]. which contains four subsets of periodically collected univariate marketing data. And we follow the data processing and split ratio used in TimesNet (47. Notably, instead of setting a fixed lookback window length, for every forecast window length H, we set the length of lookback window T = 2H the same with N-BEATS (32] because historical observations are always available in real-world scenarios and can be beneficial for deep models to leverage more observed data with the increasing forecast horizon.
Baselines We extensively compare Koopa with the state-of-the-ard deep forecasting models, includ- ing Transformer-bases model: Autoformer [88, PatchTST [11: TCN-based model: TimesNet [47] MICN 43); Mased model: DLinear [51]; Fourier forecaster: FiLM [54), and Koopman fore-
caster: KNF 44
We also introduce additional specialized models N- ? and N-BEATS [2)
for univariate forecasting as competitive baselines. All the baselines we reproduced are implemented based on the original paper official code. We repeat each experiment three times with different random seeds and report the test MSE/MAF. And we provide detailed code implementation and peerpaaramesess sensitivity in Appendix C

---

5.1 Time Series Forecasting
Forecasting results We list the results in Table 2 with the best in bold and the second underlined. Koopa shows competitive forecasting performance in both multivariate and univariate forecasting. Concretely, Koopa achieves state-of-the-ar performance in more than 70% multivariate settings and consistently outperforms other deep models in the univariate settings.
Notably, Koopa surpasses the state-of-the-arr Koopman- based forecaster KNF by a large margin in real-world time series, which can be attributed to our hierarchical dynamics learning and disentangling mechanism. Also, as the representative of efficient linear models, the performance of DLinear is still subpar in ILI, Traffic and Weather, indicating that nonlinear dynamics underlying the time series poses challenges for model capacity and point-wise weighting may not be appropriate to portray timi-variant dynamics. Besides, compared with painstakingly trained PatchTST with channel- independence mechanism, our model can achieve a close and even better performance with naturally addressed non-stationary properties of real-world time series.
Table 1: Multivariate forecasting results with different forecast lengths H E {24,36, 48,,,66480000 40) for ILI and H {48, 96, 144, 192) for others. We set the lookback length T 2H. Additional results (ETTml, ETTm2, ETTh1) are provided in Appendix D.I
Models
Koopa
PStchTST
TimesNet
DLinear
MICN
KNF
FiLM
Autoformer
Metric MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAB
48/0.130 0.234 0.147 0.246 1.44 0.254 0.158 0.241 0.156 0.271 0.175 0,265 0.197 0.270 0.164 0.272
BCL
96 0.236 0.143 0.241 0.170 0.275 0.153 0.245 0.165 0.277 0.198 0.284 0.238 0.341 0.182 0.289
0.149 0.247 0.145 0.241 0.183 0.287 0.152 0.245 O. 163 0.274 0.204 0.297 0.234 0.338 0.210 0.315
192 192 0.156 0,254 0.147 0.240 0.189 0.291 0.153 0.246 0.171 0.284 0.245 0.321 0.240 0.339 0.221 0.324
08/0.2 0,300 0.223 0.297 0.241 0.319 0.226 0.305 0.260 0.336 0.385 0.376 0.261 0.324 0.355 0,380
ETTh2
96 (1,297 0.349 0.300 0.353 0.325 0.376 0.294 0.351 0.343 0.393 0.433 0.446 0.322 0.372 0.427 0.432
144 0.333
0.381 0.346 0.390 0.374 0.408 0.354 0.397 0.374 0.411 0.441 0.456 0.352 0.397 0.457 0.461
192 0.356 0.393 0.383 0.406 0.394 0.434 0.385 0.418 0.455 0.464 0.528 0.503 0.361 0.410 0.503 0.491
Exchange
48 0.143 0.044 0. 044 0.059 0.1 0.043 0.145 0.117 0.248 0.128 0.271 0.071 0.2 0.125 0.252
96 0.083 0.207 0.085 0.204 O. 120 0.255 0.084 0.220 0.108 0.251 0.294 0.394 0.1 0.245 0.280 0.386
144 0.130
0.261 0.132 0.260 0.206 0.334 0.2 0.253 0.152 0.301 0.597 0.578 0.174 0.306 0.520 0.523
192 084 0.309 0.174 0,300 0.377 0.463 0.178 0.299 0.187 0.331 0.654 0.595 0.241 0.364 0.653 0.592
24
1.621 0.800 2.063 0.881 2.464 1.039 2.624 1.118 4.380 1,558 3.722 1.432 3.590 1.424 2.831 1.085
ILI
36
1.803 0.855 2.178 0.943 2.388 1.007 2.693 1.156 3.314 1.313 3.941 1.448 4,200 1,383 2.801 1.088
48
1.768
0.903 1.916 0.896 2.370 1.040 2.852 1.229 2.457 1.085 3,287 1.377 3.317 1.417 2.322 1.006
60
1.743 0.891 1.981 0.917 193 1.003 2.554 1.144 2.379 1.040 2.974 1.301 4.077 1.444 2.470 1.061
48 0.274 0.426 0.286 0.567 0.306 0.488 0.352 0.496 0.301 0.621 0.382 0.498 0.312 0.640 0.361
Traffic
0.401 0.275 0.413 0.283 0.611 0.337 0.485 0.336 0.511 0.312 0.645 0.376 0.451 0.297 0.668 0.367
144 0.397 0.276 0.405 0.278 6.6 0.322 0.452 0.317 0.498 0.309 0.683 0.402 0.430 0.288 0.681 0.379
192 0.403 0.284 0.404 0,277 0.604 0.321 0.438 0.309 0.494 0.312 0.699 0.405 0.425 0,288 0.692 0.385
48 0.116 0.168 0.140 0.1 138 0.191 0.156 0.198 0.157 0.217 0.201 0.288 0.160 0.206 0.185 0.240
Weather
96 0.154 0.205 0.160 0.206 O. 180 0.231 08186 0.229 0.187 0.250 0.295 0.308 0.189 0.233 0.230 0.279
144 0.172 0.225 0.174 0.221 0. 190 0.244 199 0.244 0.197 0.257 0.394 0.401 0.200 0.245 0.268 0.308
192 0.193 0.241 0.195 0.243 0.212 0.265 0.217 0.261 0.214 0.270 0.462 0.437 0.219 0.263 0.325 0.347
1" Count
34
11
3
0
0
Table 2: Univariate forecasting results for the M4 dataset. We report the weighted average forecasting error from all four subsets and full results are provided in Appendix D.I
Models
Koopa
N-HiTS
N-BEATS
Pat
TimesNet
DJ iinear
MICN
KNF
Fi .M
Autoformer
Weighted Average
sMAPE
11.863
11.960
11.910
13.022
11.930
12.418
13.023
12.126
12.489
14.057
MASE
1.595
1.606
1.613
1.814
1.597
1.656
1.836
1.641
1.690
1.954
OWA
0.858
0.861
0.862
0.954
0.867
0.891
0.960
0,874
0.902
1.029
7

---

Mamory Roolprist
Mamory Foolprint
KNF
077
Autoformer 4.583CR, 167.mm
Autoformer
KNF
TimesNel
TimeaNal
B
2,5500K, 96 dem
26566, 266 Aen
2 5.5
2.5403, 140.9e
DLinear BErGa,
DLinear
FILM
54
114308, 47.
28768
FILM
Koopa
3 AARGA, 53 9P
2:3990K, 68 dm
PatchTST
977G6, Jess
963
03
350
600
700
1200
1500
Training Time (ma/
Training Time (ma/iter]
Figure 4: Model efficiency comparison. The performance comes from Table with forecast window length H 144. Training time and memory footprint recorded with the same batch size and official code configuration. Full results of all six datasets are provided Appendix D.3
Model efficiency We comprehensively evaluate the model efficiency from three aspects: forecasting performance, training speed, and memory footprint. In Figure we compare the efficiency under two representative datasets with different variate numbers (7 in ETTh2 and 862 in Traffic..
Compared with the state-of-th the forecasting model PatchTST, Koopa training time respectively in the ETTh2 and Traffic datasets with only 26.8% % and 2.9% % memory footprint. Concretely, the averaged training time and memory ratio of Koopa compared to PStchTST are 22.7% % and 24.0% % all six datasets (see Appendix [D.3 for the detail). Besides, as an efficient MLP-based forecaster, Koopa is also capable of learning nonlinear dynamics from time-variant and lime-invariatt components, and thus achieves better performance.
62.3% % and 96.5%
Time-maraan Component
Time Series
Tine-variant Competinn
0.15
Degree- of
0.13
DKC
17Fth2 Ebchange
T:afflc Weythr
AG
Figure 5: Left: Comparison of Degree of Variation (the standard deviation of linear weighting fitted on different periods), we plot respective values of disengaged components on all six datasets. Right: case of localized Koopman operators calculated on the Exchange dataset at the interval one year.
5.2 Model Analysis
Dynamics disentanglement To validate the disentangling effect of our proposed Fourier Filter, we divide the whole time series into 20 subsets of different periods and conduct respective linear regression on the components disentangled by Fourier Filter. The standard deviation of the linear weighting reflects the variation of point-to-point temporal dependencies, which works as the manifes- tation of time-variant property. We plot the value as Degree of Variation (Figure Left). It can be observed that larger deviations occur the time-variant component, which indicates the proposed module successfully disentangles two types of dynamics from the perspective of frequency domain.
(Figure())ect),
We present a case study on real- world time series (exchange rate) on the right of Figure We sample the lookback window at the interval of one year and visualize the Koopman operators calculated Time-variant KP. can be clearly observed that localized operators can exhibit changing temporal patterns in different periods, indicating the necessity of utilizing varying operators to describe time-variant dynamics. And interpretable insights are also presented as series uptrends correspond to heatmaps with large value and downtrends reflected with small value.
Case study

---

Ablation study We conduct ablations on Koopa. As shown in Table Time-varian and Time- invariant KPs perform as complementary modules to explore the dynamics underlying time series, and discarding any one of them will lead to the inferior performance. Besides, we evaluate alternative decomposition filters to disentangle time series dynamies. We find the proposed Fourier Filter conducts effective disentanglement, where the amplitude statistics of frequency spectrums from different periods are utilized to exhibit time-agnostic information. Therefore, Koopa tackling the right dynamics with complementary modules can achieves the best performance.
Table 3: Model ablation. Only Kin uses one-block
KP; Only Kaar stacks Time-varian KPs only; Truncated Filler replaces Fourier Filter with High-Low Pass Filter; Branch Switch changes the order of KPs on disentangled components. The averaged results are listed here.
Dataset
ECL
ETTh2
Exchange
ILI
Traffic
Weather
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Only K
0.148
0.250
0.312
0.358
0.120
0.241
2.146
0.963
0.740
0,446
0.170
0.213
Only K
1.547
0.782
0.371
0.405
0.205
0.316
2.370
1.006
0.947
0.544
0.180
0.232
Truncated Filter
0.155
0.255
0.311
0.362
0.129
0.246
1.988
0.907
0.536
0.334
0.172
0.220
Branch Switch
0.696
0.393
0.344
0.385
0,231
0.325
2.130
0.964
0.451
0.304
0.173
0.221
Koopa
0.146
0.246
0.303
0.356
0.111
0.230
1.734
0.862
0.419
0.293
0.162
0.211
Avoiding rigorous reconstruction Unlike previous Koopman Autoencoders, the proposed Koop- man Predictor does not reconstruct the whole dynamics at once, but aims to portray the partial dynamics evolution. Thus we remove the reconstruction branch, which is only utilized during training in previous KAEs. In our deep residual structure, the predictive objective function works as a good optimization indicator. We validate the design in Table where the performance of sorely forecasting objective optimized model is better than with an additional reconstruction loss. Because the end-to- end forecasting objective helps to reduce the optimization gap between training and inference, making it a valuable contribution of applying Koopman operators on end-to-end time series forecasting.
Table 4: Performance comparison of the dynamics learning blocks implemented by our proposed Koopman Predictor (Koopa) and the canonical Koopman Autoencoder 29((KE))
Datasct
ETTh2
Exchange
ECL
Traffic
Weather
ILI
Model
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Koopa
0.303
0.356
0.110
0.230
0.143
0.243
0,404
0.277
0.161
0.210
1.734
0.862
KAE
0.312
0.361
0.129
0.248
0.169
0.269
0.463
0.329
0.170
0.217
2.189
0.974
Promotion
2.88%
14.73%
15.38%
12.74%
5.29%
20.79%
Learning stable operators We turn to analyze our architectural design from the spectral perspective. The eigenvalues of the operator determine the amplitude of dynamics evolution. As most of non- stationary time series experience the distribution shift and can be regarded an unstable evolution, the learned Koopman operator with the modulus far from the unit circle will cause non-divergent and even explosive trending in the long term, leading to training failures.
To tackle this problem generally faced by Koopman- based forecasters, we propose to utilize the disentanglement and deep residual structure. We measure the stability of the operator as the average distance of cigenvalues from the unit circle. As shown in Figure [ the operator can become more stable by the above two techniques. The disentanglement helps to describe complex dynamics based on the decomposition and appropriate inductive bias can be applied. The architecture where each block is employed to fill the residual of the previously fitted dynamics reduces the difficulty of directly reconstructing complicated dynamics. Each block portrays the basic process driven by a stable operator within its power, which can be aggregated for complicated non-stationary process.
5.3 Scaling Up Forecast Horizon
Most deep forecasting models work as a settled function once trained (e.g. scenarios where the prediction horizon is mismatched or long-term. it poses two challenges for the trained model: (1) reuse parameters learned from observed series; (2) utilize incoming ground truth for model adaptation. The practical scenarios, which we name scaling up forecast horizon, may
For
9

---

(0)
[b]
Imaginary
AVG[ II) 0.1082
Imaginary
AvoCl 11) 0.0887
Imaginary
AVG 0.0813
Real
Real
Real
Figure 6: Visualization of the operator stability on the highly non-stationary Exchange dataset. We plot the first block time-invariant operator eigenvalues of the following design: (a) Single-block model with only time-invariant operator. (b) Single-block model with time-invariant and time-varian operators. (c) Two-block model with time-invariant and time-variant operators.
lead to failure most deep models but can be naturally tackled by Koopa. In detail, we first train Koopa with forecast length Hu and attempt to apply it on larger forecast length He.
KP has learned the globally shared dynamics and Time-variant KP can calculate localized operator K within the lookback window, we freeze the parameters of trained Koopa but only use the incoming ground truth to adapt Kvar- The na've implementation uses incremental Koopman embedding with dimension D and conducts Equation to obtain an updated operator, which has complexity of O (HHeee We further propose an iterative algorithm with improved O(H + D²) complexity. The detailed method implementations and complexity analysis can be found in Appendix
Method Koopa scales up forecast horizon as follows: Since
Results As shown in Table 5 the proposed operator adaption mechanism further boosts the performance on the scaling up scenario, which can be attributed to more accurately fitted time-vate variant dynamics with incoming ground truth snapshots. Besides, the promotion becomes more significant when applied to non-stationary datasets (manifested as large ADF Test Statistic [10)..
Table 5: Scaling up forecast horizon: (Hhr, He) (24, 48) for ILI and (Hr Hie) = (48, 144) for others. Koopa conducts vanilla rolling forecast and Koopa OA further introduces operator adaptation.
Dataset
Exchange
ETTh2
ILI
ECL
Traffic
Weather
Test Statistic
(-1.889)
(-4.135)
(-5.406)
(-8.483)
(-15.046)
(-26.661)
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Koopa
0.214
0.348
0.437
0.429
2.836
1.065
0.199
0.298
0.709
0.437
0.237
0.276
Koopa OA
0.172
0.319
0.372
0.404
2.427
0.907
0.182
0.271
0.699
0.426
0.225
0.264
Promotion (MSF)
19.6%
14.9%
14.1% 1..%%
8.5%
1.4%
5.1%
Conclusion
This paper tackles time series as dynamical systems. With disentangled time-variant and time- invariant components from non-stationary series, the Koopa model reveals the complicated dynamics hierarchically and leverages MLP modules to learn Koopman embedding and operator. Experimen- tally, our model shows competitive performance with remarkable efficiency and the potential to scale up the forecast length by operator adaptation. In the future, will explore Koopa with the dynamic modes underlying non-stationary data using the toolbox Koopman spectral analysis.
Acknowledgments
This work was supported by the National Key Research and Development Plan (2021 YFC30000))))))) National Natural Science Foundation China (62022050 and 02021002), Beijing Nova Program 1221100006820011 and BNRist Innovation Fund (BNR202 RC01002).
10

---

References
Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, Jean- Can-Christophe Gagnon-Audet, Yoshua Bengio. Ioannis Mitliagkas, and Irina Rish. Invariance principle meets information bottleneck for out-of-distribution generalization. NeurlPS, 2021.
[2] O. Anderson and M. Kendall. Time-series. 2nd edn. J. R. Stat. Soc. (Series D), 1976.
[3] Omri Azencot, N Benjamin Erichson, Vanessa Lin, and Michael Mahoney. Forecasting sequential data using consistent koopman autoencoders, In International Conference on Machine Learning, pages 475-4855 PMLR, 2020.
[4] Shaojie Bai, Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01227. 2018.
[5] Nimrod Berman, Naiman, and Omri Azencot. Multifactor sequential disentanglement structured koopman autocncoders. arXiv preprint arXiv:2303 2023.
[6] Steven L Brunton, Marko Budišić, Eurika Kaiser, and Nathan Kutz. Modern koopman theory for dynamical systems, arXiv preprint arXiv:2102. 2021.
[7] Cristian Challu, Kin G Olivares. Boris N Oreshkin, Federico Garza, Max Mergenthaler, and Artur Dubrawski. N-hits: Neural hierarchical interpolation for time series forecasting. arXiv preprint arXiv:2201. 12886, 2022.
[8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
[9] Hamidreza Eivazi, Luca Guastoni, Philipp Schlatter, Hossein Azizpour, and Ricardo Vinuesa. Recur- rent neural networks and koopman- based frameworks for temporal predictions low-order model of turbulence. International Journal of Heat and Fluid Flow, 90:108816, 2021.
[10] Graham Elliott, Thomas Rothenberg, and James Stock. Efficient tests for an autoregressive unit root, 1992.
[11] lan Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.62111 2013.
[12] S. Hochreiter and J. Schmidhuber: Long short-term memory. Neural Comput., 1997.
[13] Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural networks, 4(2):251- 257, 1991.
[14] Rob J Hyndman and George Athanasopoulos. Forecasting: principles and practice. OTexts, 2018.
[15] Xiaoyong Jin, Youngsuk Park, Danielle Maddix, Hao Wang, and Yuyang Wang. Domain adaptation for time series forecasting via attention sharing. International Conference on Machine Learning, pages 10280-10297. PMLR, 2022.
[16] Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. Reversible instance normalization for accurate time-series forecasting against distribution shift. In ICLR, 2022.
[17] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: arXiv:1412.6900 1412.6980. 2014.
[18] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska Barwinska, al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521 -326, 2017.
[19] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer:
The efficient transformer. In ICLR, 2020.
[20] Bernard Koopman. Hamiltonian systems and transformation in hilbert space. Proceedings of the National Academy of Sciences, 17(5):315-315 1931.
[21] Vitaly Kuznetsov and Mehryar Mohri. Discrepancy based theory and algorithms for forecasting non- stationary time series. Annals of Mathematics and Artificial Intelligence, 88(4):367339) 2020.
11

---

[22] Guokun Lai, Wei-Cheng Chang, Yiming Yang. and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In SIGIR, 2018.
[23] Yucheng Lan and Igor Mezié. Linearization in the large of nonlinear systems and koopman operator spectrum. Physica Nonlimear Phenomena, 242(1):42 53, 2013.
[24] Henning Lange, Steven L Brunton, and Nathan Kutz. From fourier koopman: Spectral methods for long-term time series prediction. The Journal of Machine Learning Research, 1881118881,,,,, 2021.
[25] Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In ICCV, 2017.
[26] Zongyi Li, Nikola Kovachki, Kamyar Arizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.0885,, 2020.
[27] Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Lowpeemplyxity pyramidal attention for long- mange time series modeling and forecasting. In ICLR, 2021.
[28] Yong Liu, Haixu Wu, Jiammin Wang, and Mingsheng Long. Non-stationary transformers: Rethinking the stationarity in time series forecasting. arXiv preprint Arxi 14415, 2022.
[29] Bethany Lusch, Nathan Kutz, and Steven L Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature communications, )(1):4950, 2018.
[30] Jeremy Morton, Antony Jameson, Mykel Kochenderfer, and Freddic Witherden. Deep dynamical modeling and control of unsteady fluid flows. Advances in Neural Information Processing Systems, 31, 2018.
[31] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.44730 2022.
[32] Boris Oreshkin, Dmiiri Carpov, Nicolas Chapados, and Yoshua Bengio. N-BEATS: Neural basis expansion analysis for interpretable time series forecasting. ICLR, 2019.
[33] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. TKDE, 2009.
[34] Adam Paszke, S. Gross, Francisco Massa, A. Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Z. Lin, N. Gimelshein, L. Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurlPS, 2019.
[35] Quang Pham, Chenghao Liu, Doyen Sahoo, and Steven CH Hoi. Learning fast and slow for online time series forecasting. arXiv preprint arXiv:2202.11672 2022.
[36] Joshua L Proctor, Steven Brumton, and J Nathan Kutz. Dynamic mode decomposition with control. SIAM Journal on Applied Dynamical Systems, 15(1):142-161, 2016.
[37] David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3):188-1199) 2020.
[38] Peter Schmid. Dynamic mode dccomposition of numerical and experimental data. Journal of fluid mechonics, 656:5-28. -6655-287770 2010.
[39] Spyros Makridakis. M4 dataset, 2018.
[40] Naoya Takeishi, Yoshinobu Kawahara, and Takehisa Yairi. Learning koopman invariant subspaces for dynamic mode decomposition. Advances neural information processing systems, 30, 2017.
[41] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint Xrviv: K 000.08002, 2016.
[42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrIPS, 2017.
[43] Huiqiang Wang. Jian Peng. Feihu Huang, Jince Wang. Junhui Chen, and Yifei Xiao. Mien: Multi-scal local and global context modeling for long-term series forecasting, In The Eleventh International Conference on Learning Representations, 2023.
12

---

[44] Rui Wang, Yihe Dong. Sercan O Arik, and Rose Yu. Koopman neural forecaster for time series with temporal distribution shifts. arXiv preprint arXiv:2210006675 2022.
[45] Matthew O Williams, Joannis G Kevrekidis, and Clarence W Rowley. A data-driven approximation of the koopman operator: Extending dynamic mode decomposition. Journal of Nonlinear Science, 25:1307-1346, 2015.
[46] Herman Wold. study in the analysis of stationary time series. PhD thesis, Almqvist & Wiksell, 1938.
[47] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. arXiv preprint arXiv:2110.02186, 2022.
[48] Haixu Wu, Jichui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with Auto- Cuto-Correlltion for long-term term series forecasting. In NeurIPS, 2021.
[49] Wei Xiong, Xiaomeng Huang, Ziyang Zhang, Ruixuan Deng. Pei Sun, and Yang Tian. Koopman neural operator as mesh-free solver of non-lincar partial differential equations. arXiv preprint arXiv:2301.1002., 2023.
[50] Enoch Yeung, Soumya Kundu, and Nathan Hodas. Learning deep neural network representations for koopman operators of nonlinear dynamical systems. In 2019 American Control Conference (ACC), pages 4832-4839. IEEE, 2019.
[51] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? arXiv preprint arXiv:2205.1 13504, 2022.
[52] Tranping Zhang, Yizhuo Zhang, Wei Cao, Jiang Bian, Xiaohan Yi, Shun Zheng, and Jian Li. Less is more: Fast multivariate time series forecasting with light aumpling-oriented mlp structures. arXiv preprint arXiv:2207.000 186, 2022.
[53] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI, 2021.
[54] Tian Zhou, Ziqing Ma, Qingsong Wen, Liang Sun, Tao Yao, Wotao Yin, Rong Jin, al. Film: Frequency improved legendre memory model for long-term time series forecasting. Advances in Neural Information Processing Systems, 15:12677-, 2022.
A Scaling Up Forecast Horizon
In this section, we introduce the capability of Koopa to scale up forecast horizon. In detail, we train Koopa model with forecast length Hu and attempt to apply it larger length Hu. The basic approach conducts rolling forecast by taking the model prediction as the input of the next iteration until the desired forecast horizon all filled. Instead, we further assume that after the model gives a prediction, the model can utilize the incoming ground truth for model adaptation and continue rolling forecast for the next iteration. It is notable that we do not retrain parameters during model adaptation, since it will lead to overfitting the incoming ground truth and Catastrophic Forgetting
Koopa naturally cope with the scenario by learning Koopman embedding and operator Kin in Time-invariant KPs while calculating localized operator K to describe the dynamics in the temporal neighborhood. Therefore, we freeze the parameters of Koopa but only use the incoming ground truth for operator adaptation of K in Time-variant KPs.
A.1 Implementation of Operator Adaptation
At beginning of rolling forecast, the Encoder in Time-variant KP outputs D-dimensional Koopman embedding R. for each observed series segment as [21, .2...., .., 2F), where F = 5 is the segment number with S as the segment length. The operator Kew in Time-variant KP is calculated as follows:
Zrack [21, ................................ ,F 2F-1), Ziae = [M2, Ka,..., ær), KI = Zian Ziaks
(15)
where 7 7tses E DIF Kur RDDD With the calculated operator, we obtain the next predicted Koopman embedding by one-step forwarding:
2p41 Kmrrp.
(16)
After decoding the embedding 4 to the series prediction, we can utilize the true value of incoming Koopman embedding zr+1 obtained by Koopa with frozen parameters. Instead of using Ker obtain the next embedding
13

---

2r42. we use incremental embedding collections Zuuctt Ziro E DxP to obtain more accurate operator
Kun+ E R D to describe the local dynamics:
Zuack
bZacck 2F], Zture (Zbck 2F+1), Kea+ Ziore+
Doock+
(17)
The procedure repeats L times (L oc Hu) until forecast horizon filled, we formulate it Algorithm And experimental results (Koopa OA) in Section 5.3 have demonstrated the promotion of forecasting performance due to more precisely fitted dynamics.
A.2 Computational Acceleration
naive implementation shown in Sesoridumm[pppeteeeee conducts Equation on the incremental embedding collection to obtain new operators, which has a complexity of O(LD") We propose an equivalent algorithm with improved complexity of O(L + D²) as shown in Algorithm
Theorem. Algorithm gives the same Korr Algorithm in each iteration with O(D²) complexity.
Proof. We start with the first iteration analysis. By the definition of Moore-Panrose inverse, we have Aoue 7ouk = Tr where T is an identity matrix with the dimension of
When the model receives the incoming emhedding 2F41, incremental embedding m = 2P, n = 2F-1 will be appended to Zrack and Zive respectively. Instead of calculating new Kn from incremental collections, we utilize calculated K to find the iteration rule on Kun. Concretely, suppose
Luuc (Lang
Z
R
(18)
where A (R(F-1)XD, b IR. are variables to be identified. By the definition of Moore Pennose inverse, we have Ttuc+ Zunc = Ir. By unfolding it, we have the following equations:
Atrack
bJ Zuack Ō' b" m L, 4 m Am Õ.
(19)
We suppose A 5b where 8 E F-1, such that when b" Znuc Õ, then AZpack 0. Then we have "n m 8t m = ZNa m 8 = Ö, thus A mbb Given equations that 6J "Zmck = Ö and bT m 1.
we have the analytical solution of b:
b ||r|| where
Treac 7 m.
(20)
Therefore, we find the equation hetween the incremental version Kaat and calculated K
7t (JD 1-D-mo mb mbb"]] Kvar+ Zioret Zinekt
|]
Trcck
Kva + (n Kram))"
(21)
T
where m, n are the incremental embedding of Zeaks Zier and can be calculated by Equation 20 the iteration rule on X Zrac Zima to obtain b, which is formulated as follows:
We also derive
X+ Zoace Dbactt = X + (m - Xm)6" = X + rb
(22)
By adopting Equation 21 22 and permuting the matrix multiplication order, we reduce the complexity of each iteration to O (P). Theretore, Algorithm 2 has a overall complexity of O(LL ++ D)D?). Since L x Hu.
Algorithm 2 have O(H, and O((H, D)D?) complexity respectively.
B Implementation Details
Koopa is trained with L2 loss and optimized by ADAM [17 with an initial learning rate of 0.001 and hatch size set 32. The training pracess early stopped within 10 epochs. We repeat each experiment three times with different random seeds obtain average test MSE/MAF and detailed results with standard deviations listed Table Experiments are implemented P PoTch
3 and conducted on NVIDIA TITAN RTX 24GB GPUs.
All the baselines that we reproduced are implemented based on benchmark TimesNet 47 Repository, which is fairly built on the configurations provided by each model's original paper or official code. Since several baselines adopt Series Stationarization from Non-stationary Transformers 28 while others do not, we equip all models with the method for fair comparison.
C Hyperparameter Sensitivity
Considering the efficiency of hyperparameters search, we fix the segment length $ = 7/2 and the number of Koopa blocks B = in all our experiments. We verify the robustness of Koopa of other hyperparameters
14

---

Algorithm 1 Koopa Operator Adaptation.
Require: Observed embedding = [.i...., 2p] and successively incoming ground truth embedding [BF+1...., with each embedding E R...
1: bback 2 1]; Zore -2,-..:: 2r)
,F-,,
Znack Zore (x(-1(1
2: Kvar Zetoe Timee
P Kar DxD > b8Ft1 E ERD 2F41 comes successively D m RD
3: 8F+1 = KvarT
4: for in (1,...,
5: m =2 2841
6: Zoack +- IZback: m), Zeoro - [tore: m
D Zuacky Zoor E DD
7: K Zioee Bback
> K E E DxD
8: årt++1 K
2rtitt RDD
9: End for 2 2++++1]
10: Return
>> Return predicted embedding
Algorithm 2 Accelerated Koopa Operator Adaptation.
Require: Observed embedding Z = [Zi,..., 2F] and successively incoming ground truth embedding
[EF+I,..., with each embedding E P.
Zback 121,..., 2F 1). Zoore ........ 2F
> Zuack; Ziore E Dx
2: Kvar = Ziore Tback X = Buack Zback
b Kvar X E R DxD
3: 8F+1 = Kvarn
> 2241 E ERD
4: for in (1,...,11) T.}:
2F+1 comes successively D Dm, R DRERD D
5: m =IF+1-1, n 2F+1
6:
m - -mm
7:
b = =r/lr1
8:
Kvar t Kvar + (n K m/)
K E RRD
9:
X + X + rb
& E DXD
10:
fft+1 Kvr?
2rtttt E R
11:
End for
12: Return ( 2F44+1]
> Return predicted embedding
Table 6: Detailed performance of Koopa. We report the MSE/MAE and standard deviation of different forecast horizons {H1 H2, H3. H4} (24, 36, 48, 60} for ILI and (48, 96, 144, 192} for others.
Dataset
ECL.
ETTh2
Exchange
Horizon
MSE
MAE
MSE
MAE
MSE
MAE
H1 0.130=20.03
0.23440.000
0.226=0.003
0.300=0.003
0.042+0.002
0.143++0003
H₂
0.136=0.004
2.005
0.297=0.004
0.349=0.004
0.083+0.004
0.207+0.004
Ha
0..093
0.247-1..00
0.3331..004
0.381
0. 0.130.10.005
0.26111.003
0.156-10.004
0.254410003
0.356
0.393
0.18490.009
0.30910.005
Dataset
II.I
Traffic
Weather
Horizon
MSE
MAE
MSE
MAE
MSE
MАΕ
1.62140.008
0,800-10.000
0.415=.0003
0.274=0.005
0.126:000005
0.168:10.000
Ha
1.803-+0.040
0.855+0.000
0.401+00005
0.275=0.004
0.154
0.205++0.003
Ha
1.768-100015
0.903-10.008
0.397+00.04
0.276=0.003
0.172++0.005
0.225++0000
Ha
1.743-10.040
0.891-10.009
0.403
0.284=000009
0.193:+0.003
0.24140.004
15

---

03
0.9
MSE
0.3
5TThL ETTN2
5TTm ETTm2
ETT1
ETTm1 ETTmz
ETTh
Dimention of Koopman Embadding
Hidden Loyer Numbar
Hidden Dimersion
Figure 7: Left: Hyperparameter sensitivity with respect to the dimension of Koopman embedding. hidden layer number, and hidden dimension of Encoder and Decoder in Koopa.
follows: dimension of Koopman embedding D, the hidden layer number the hidden dimension used in Encoder and Decoder. As is shown in Figure 7 we find the proposed model is insensitive to the choices of above hyperparameters. which can be beneficial for practitioners to reduce tuning burden.
Intuitively, a larger dimension of Koopman embedding D can bring about a lower approximation error. We further dive into it and find that stacking blocks can enhance the model capacity, and thus the performance is insensitive D when the model is deep enough. To address the concern, under varying block number in Figure g can be seen that a larger B generally leads to lower error even D is small. And the performance can be sensitive to D when the model is not deep enough (B = 1,2).
further check the sensitivity of D
Besides, we find the proposed model can be insensitive to S on several datasets while sensitive on ECL and Traffic datasets (The difference is about 10%). There many variables in these two datasets, but our current design shares the S for all variables. Since different variables with distinct evolution periods implicitly require different optimal the performance of the dataset with more variables more likely to be influenced by Therefore, we set T/2 with relatively small performance fluctuation to deal with most situations.
0.15
oBcck
BBokcc
3 Blocks
Traffic
ETTh2
MSE
MSE
Weather
2
0.1
1/8
T/4
Koopman Embedding
Segment Langth
Figure 8: Left: Hyperparameter sensitivity the dimension Koopman embedding under different settings of the block number. Right: Hyperparameter sensitivity of the segment length.
D Supplementary Experimental Results
D.1 Full Forecasting Results
Due to the limited pages, we list additional multivariate benchmarks on ETT datasets [3 in Table 7 which includes the hourly recorded ETTh2 and 15- 15-minutely recorded ETTm 1/ETT/22 and the full univariate results of M4 (39) in Table g which contains the yearly, quarterly and monthly collected univariate marketing data. Notably, Koopa still achieves competitive performance compared with state-of-the of-the-artt deep forecasting models and specialized univariate models.
D.2 Full Ablation Results
We elaborately conduct model ablations verify the necessity of each proposed module: Time- tmme-invrriant KP, Hime-variant KP, Fourier Filter and evaluate alternative choices to disentangle dynamics. As shown Table Koopa conducts effective disentanglement and tackles the right dynamics with complementary KPs. and thus achieves the best forecasting performance.
16

---

Table 7: Forecasting results with different forecast window lengths H E [48,,96 96, 144, 192) on ETT dataset. We set the lookback window length T
2H.
Models
KooPA
PatchTST 31 TimesNet
DLinear 53]
MICN 35
KNF
FILM
Autoformer
48
Metric MSE MAE
MSE MAE MSE MAE MSE MAE MSE MAE
MSE
MAE
MSE
MAE MSE
MAE
48 0.283 0.333 0.286 0.336 0,308 0.354 0.322 0.355 0.294 0.353 1.026 0.792 0.324 0.353 0.592 0,419
ETim!
96 0.294 0.345 0.299 0.346 0.329 0.370 0.309 0.346 0.306 0.364 0.957 0.782 0.311 0.346 0.493 0.469
144 0.322 0.366 0.325 0.363 0.358 0.387 0.327 0.359 0.342 0.390 0.921 0.760 0.328 0.358 0.735
0.569
192 0.337 0.378 0.343 0.375 0.462 0.441 0.337 0.365 0.386 0.415 0.896 0.731 0.339 0.366 0.592 0.506
48 18 0.226 0.135 0.231 0.142 0.234 0.144 0.240 0.131 0.238 0.621 0.623 0.146 0.243 0.191 0.280
iTlm2
96 0.171 0.254 0.171 0.255 08187 0.269 0.172 0.256 1.7 0.295 1.535 1.012 0.174 0.257 0.241 0.311
144 0.20 0.280 0.205 0.282 0.216 0.291 0.200 0.276 0.210 0.297 1.337 0.876 0.204 0.279 0.300 0,352
192 0.226 0.298 0.221 0.294 0.243 0.313 0.219 0.290 0.248 0.328 1.355 0.908 0.224 0.293 0.324 0.370
48 0.336 0.377 0.337 0.375 0.365 0.399 0.343 0.371 0.375 0,406 0.876 O. 709 0.407 0.427 0.442 0.438
96 0.371 0.405 0.372 0.393 0.411 0.430 0.379 0.393 0.406 0.429 0.975 0.744 0.429 0.431 0.634 0.523
144 0.405 0.418 0.394 0,412 0.442 0.447 0.393 0.403 0.437 0.448 0.801 0.662 0.451 0.448 0.522 0.491
192 0.416 0.429 0.416 0.439 0.469 0.470 0.407 0.416 0.518 0.496 0,941 0.744 0.460 0.459 0.525 0.501
Table 8: Full univariate forecasting results for M4 dataset. We follow the same data processing and forecasting length settings used in TimesNet [47) benchmark. Additional forecasting models N-HiTS 7 and N-BEATS [2] are also included.
Models
KooPA
N-HITS
N-BEATS
PatchTST
TimesNet
DLinear
MICN
KNF
FiLM
Autoformer
SMAPE
13.352
13.371
13.466
13.517
13.394
13.866
14.532
13,986
14.012
14.786
Year
MASE
2.997
3.025
3.059
3.031
3.004
3.006
3.359
3.029
3.071
3.349
OWA
0.786
0.790
0.797
0.795
0.787
0.802
0.867
0.804
0.815
0.874
Quarter
sMAPE
10.159
10.454
10.074
10.847
10.101
10.689
11,395
10,343
10.758
12.125
MASE
1.189
1.219
1.163
1.315
1.183
1.294
1.379
1.202
1.306
1.483
OWA
0.895
0.919
0.881
0.972
0.890
0.957
1.020
0.965
0.905
1.091
Month
SMAPE
12.730
12.794
12,801
14.584
12,866
13.372
13.829
12.894
13.377
15,530
MASE
0.953
0.960
0.955
1.169
0.964
1.014
1.082
1.023
1.021
1.277
OWA
0.901
0.895
0,893
1,055
0.894
0.940
0,988
0.985
0.944
1.139
Others
sMAPE
4.861
4.696
5.008
6.184
4.982
4.894
6.151
4.753
5.259
5.841
MASE
3.124
3.130
3.443
4.818
3.323
3,358
4.263
3.138
3.608
4,308
OWA
1.004
0.988
1.070
1.140
1.048
1.044
1.319
1.019
1.122
1.294
Weighted
Average
sMAPE
11.863
11.960
11.910
13.022
11,930
12.418
13.023
12.126
12.489
14.057
MASE
1.595
1.606
1.613
1.814
1.597
1.656
1.836
1,641
1.690
1.954
OWA
0.858
0.861
0.862
0.954
0.867
0.891
0.960
0.874
0.902
1.029
D.3 Model Efficiency
We comprchensively compare the forecasting performance, training speed, and memory footprint of our model with well-ackmowledged deep forecasting models. The results recorded with the official model configuration and the same hatch size. We visualize the model efficiency under all six multivariate datasets in Figure 9 In detail, compared with the previous sate-of-the-art model PSt [8) Koopa consumes only 15.2% training time and 3.6% memory footprint respectively in ECL, 37.8% 8% training time and 26.8% memory in ETTh2, 23.5% training time and 37.3% memory in Exchange, 50.9% training time and 47.8% memory in ILI, 3.5% training time and 2.9% memory in Traffic, and 5.4% training time and 25.4% memory in Weather, leading the averaged 77.3% and 76.0% saving of training time and memory footprint in all six datascts. The remarkable efficiency can be attributed to Koopa with MLPs as the building blocks, and we find the budget saving becoming more significant on datasets with more series variables (ECL., Traffic).
Besides, as efficient linear model, the performance of Koopa still surpasses other MLP-based models. Especially, Compared with DLinear [5T), our model reduces 38.0% MSE MSE (0.452-+0.397) in Weather. And the average MSE reduction of Koopa compared with the previous state-of art basebased model reaches 12.2%. Therefore, our proposed Koopa is efficiently built with MLP networks and shows great model capacity to exploit nonlinear dynamics and complicated temporal dependencies in real-world time series.
-1.66) in ILI and 13.6%
17

---

Table Model ablation with detailed forecasting performance. We report forecasting results with
different prediction lengths (24, 36, 48, 60) for ILI and H E {48, 96, 144, 192) for others. For columns: Only Kim uses one-block Time-invariant KP: Only Karr stacks Time-variant KPs only; Truncated Filter replaces Fourier Filter with High-Low Frequency Pass Filter; Branch Switch changes
the order of KPs to deal with disentangled components.
Models
KooPA
Only Kin
Only Kar
Truncated Filter
Branch Switch
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
48
0.130
0.234
0.150
0.243
1.041
0.777
0.149
0.245
0.137
0.234
ECL
96
0.136
0.236
0.137
0.242
4.643
1.669
0.172
0.280
2.240
0.724
144
0.149
0.247
0.150
0.252
0.238
0.327
0.149
0.246
0.226
0.331
192
0.156
0.254
0.158
0.260
0.267
0.355
0.152
0.248
0.181
0.284
48
0.226
0.300
0.235
0.304
0.271
0.334
0.340
0.310
0.245
0.317
ETTh2
0.297
0.349
0.311
0.353
0,382
0.405
0,301
0.352
0.343
0.384
0.333
0.381
0.337
0.379
0.427
0.444
0.338
0.386
0.403
0,418
192
0.356
0.393
0.363
0.397
0,402
0,437
0,363
0.400
0.384
0,420
Exchange
48
0.042
0.143
0.046
0.150
0.065
0,184
0.048
0.150
0.055
0.165
96
0.083
0.207
0.083
0.210
0.147
0.274
0.087
0.210
0.151
0.277
144
0.130
0.261
0.149
0.281
0.222
0.351
0.150
0.278
0.254
0.369
192
0.184
0.309
0.200
0.322
0.385
0.456
0.229
0.345
0.463
0.490
24
1.211
0.800
2.165
0.882
1.972
0.919
2.140
0.874
2.092
0.894
ILI
1.803
0.855
1,815
0.882
2.675
1.091
1.692
0.844
2.116
0,950
48
1.768
0.903
2.107
0.981
2,446
1.045
1.762
0.895
2.394
1.084
60
1.743
0.891
2,496
1.108
2.387
0.970
2.357
1.018
1.917
0.926
48
0.415
0.274
0.445
0.295
0.915
0.536
0.668
0.363
0.468
0.300
Traffic
96
0.401
0.275
0.403
0.277
0.833
0.465
0.441
0.323
0.429
0.298
144
0.397
0.276
0.400
0.278
0.816
0.452
0.436
0.321
0.438
0.307
192
0.403
0.284
1.371
0.788
1.224
0.723
0.597
0.331
0.469
0.312
Weather
48
0.126
0.168
0.142
0.181
0.140
0.190
0.125
0.166
0.130
0.173
96
0.154
0.205
0.164
0.209
0.169
0,224
0.154
0.202
0.163
0,210
144
0.172
0.225
0.178
0.226
0.194
0.247
0.176
0.226
0.187
0.238
192
0.193
0.241
0.195
0.245
0.217
0.268
0.195
0.244
0.212
0.261
ETTh2
Traffic
0.5
Mamory Roolpris!
0.7
Foolprint
KNF
1,ASPA,, 143
4,5980R, 167.me
Autoformar
Autoformer
KNF
MSE
2.535Ga B6
3,25508, 205
Timeshitt
MSE
9.44G 140 Pen
12335Ga,
TimesNet
b.eoros, 70mm
Dlineor
PatchTST S 64GGR, 50,9m
19930E, 263
FILM
544
14800, 473
Koopa
FILM
Koopa
> 3390A, 58 Bro
PotchTST
97708, 20 3ei
350
600
950
1200
1500
Training Time (ma/)tur
Training Time (ms/terr
Figure 9: Model efficiency comparison with forecast length = 144 for ETTh2 and Traffic.
18

---

Exchange
Memory Footprint
Mamory Foolgrinl
KNF
FILM
7581CS,
OGa DGA
2008
Autoformer
MSE
Timeshel
MSEE
0.38
MICN
4.8000K
Autoformer
23,22008, 292 2mm
KNF
TimesNal
Dlineor
02
1143GE, 1O0 Srms
Koopa
FILM 2,40P 266 Ste
0.15
1,2108, 29.2ma
PatchTST 39.1620,
Koopa
PatchTST
DLineor
97766,
320 Training (ma/ter)
500
Training Time (ma/iter]
Figure 10: Model efficiency comparison with forecast length H 144 for Exchange and ECI.
Weather
ILI
Memory
Memory Foolprint
0.4
KNF
3.2
Autoformer
1.241 GK, 329.4m
120768, 455.6ms)
MISE
DLinsor
Times Net
MSE
0.800, $4
EASGE 4.7.
Timashie
PatchTST
MICN
1109Ga 137P
MICN
Koope
Autoformer 7.46666, 268
0.9601 7dmm
Koopa
PatchTST 107GA,
6660 292
FILM
99308, 75.2mz
100
20p
300
100
200
500
400
600
Training Time (ms/iter)
Training Time (mx/ /fe))
Figure 11: Model efficiency comparison with forecast length = 144 for Weather and 48 for ILI.
D.4
Training Stability
Since the eigenvalues of the operator determine the amplitude of dynamics evolution, where modulus close one causes non-divergent and explosive evolution in the long-term, we highlight that a stable Koopman operator only describes weakly stationary series well. While most of the doopman-basee forecasters can suffer from the operator convergence problem induced by complicated non-stationary series variations, employ several techniques stabilize the training process.
Operator initialization We adopt the operator initialization strategy, where the operator starts from the multiplication of cigenfunctions with standard Gaussian distribution and all-one eigenvalues.
Hierarchical disentanglement In our model, each block learns weak stationary process hierarchically and feeds the residual of fitted dynamics for the next block to correct. Thus Koopman Predictor aims not to fully reconstruct the whole dynamics at once, but to partially describe dynamics, rigorous reconstruction is not forced in each block, reducing the difficulty of portraying the non-stationary series as dynamics.
Explosion checking We introduce an explosion checking mechanism that replaces the operator encountering nan number with the identity matrix when the exponential multiplication of multiple time steps is detected.
Based on the proposed strategies, we provide the model training curves in Figure [2 check the convergence of our proposed model and other forecasters. The training curves of the proposed model in blue demonstrate consistent and smooth convergence, indicating its effectiveness in converging toward optimal solution.
E Broader Impact
E.1
Impact on Real-world Applications
Our work copes with roaldworld time series forecasting, which is faced with intrinsic non-stationarity that poses fundamental challenges for deep forecasting models. Since previous studies hardly research the theoretical basis that can naturally address the time-varian property in non-stationary data, propose novel Koopman
19

---

ETTh2
Weather
Kngp
5,4
Troisin Los
Keeps
0:00
430
035
PCemo
Epach
Epoch
Epoch
Figure 12: Training curves of different models on the ETT and Weather datasets.
forecaster that fundamentally considers implicit time-variant and iime-invrriant dynamics based Koopman theory. Our model achieves the state-of-the-an performance on six real- world forecasting tasks, covering energy, economics, discase, traffic, and weather, and demonstrates remarkable model efficiency in training time and memory footprint. Therefore, the proposed model makes it promising tackle real-world forecasting applica- tions, which helps society prevent risks in advance and make better decisions with limited computational budgets. Our paper mainly focuses on scientific research and has no obvious negative social impact.
E.2 Impact Future Research
In this paper, find modern Koopman theory natural learn the dynamics underlying non-stationary time series. The proposed model explores complex stationary patterns with temporal localization inspired by Koopman approaches and implements respective deep network modules to disentangle and portray time-variant and time-invariant dynamics with the enlightenment of Wold's Theorem. The remarkable efficiency and insights from the theory can be instructive for future rescarch.
Limitation
Our proposed model does not respectively considers dynamics in different variates, which leaves improvement for better multivariate forecasting with the consideration of various evolution patterns and series relationships. And Koopman spectral theory is still under leveraging in our work, which discover Koopman modes interpret the linear behavior underlying non-stationary data high-dimensional representation, Besides, Koopman theory for control considering factors outside the system can be promising for series forecasting with covariates. which leaves our future work.
20


Reviews:
[{"review_id": "fmQQ8UVTaH", "reviewer": "Reviewer_36zV", "summary": "The paper studied time series forecasting problem for weather forecasting, energy consumption, and financial assessment. To propose a model that generalizes on varying distribution, the authors proposed Koopa which is composed of modular Koopman predictors to describe and advance forward series dynamics. The motivation is Koopman-based methods are appropriate to learn non-stationary time series dynamics. \nThe implementation is straightforward in the supplement material. The experimental results show the effective performance of the proposed  methods compared with state-of-the-art methods.", "questions": "1. Is there any correlation between Koopa blocks if the curves share similarity as shown in Figure 5 ?\n2. How is the convergence of the proposed method and compared methods?\n3. What are the parameters, such as the number of epochs for training, batch size for experiments?", "limitations": "There is no theoretical analysis but the main contribution is to propose a novel architecture. \n\n", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper proposed a novel method to solve the time series forecasting problem. The introduction of Koopa Block is interesting. The experimental results show that the model outperforms existing methods with more efficient memory usage.  ", "weaknesses": "It is interesting to see more analysis about Koopa blocks. For example on Figure 5, authors showed the visualized blocks are different which corresponds to different curves. But it is clear that the curves of K1, K2, and K2 share the similarity that they goes down first, then goes up. It is interesting to see the correlation analysis if there is some. \n\nThe experiments show the proposed method achieved better performance with efficient training time and memory usage. I am not sue if there is convergence analysis and what are the parameters, such as the number of epochs for training, batch size.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks to Reviewer 36zV for providing thorough insightful comments. \n\n**Q1:** The convergence of the proposed method and compared methods.\n\nAs shown in $\\underline{\\text{Figure 1 of the global response}}$, we provide the model training curves to check the convergence of our proposed model and other methods. The training curves of the proposed model in blue demonstrate a consistent and smooth convergence, indicating its effectiveness in converging towards an optimal solution.\n\n**Q2:** Analysis of learned operators in Koopa blocks.\n\nAs shown in $\\underline{\\text{Figure 5 of the main text}}$, we have qualitatively revealed the correlation that non-contemporaneous subseries that differ in **global tendency** will be distinguished by the heatmap weight of operators. As the reviewer mentioned insightfully, they also share **similar series variations, but with varied durations**. We find it not easy to be directly visualized since it can be manifested by varying decay rates of respective evolution of eigenfunctions. It leaves instructive for us to improve model interpretability with further consideration of Koopman eigenfunctions analysis.\n\nBesides, for a quantitative correlation analysis, we sample subseries monthly from Exchange dataset and calculate the correlation between **global tendency** and **operators**:\n\n- We use linear regression to fit each subseries and use the slope as a manifestation of global tendency.\n- We consider several properties of learned operators: the sum of elements (Sum), the sum of eigenvalues (Trace), the maximum absolute value of eigenvalues (2-Norm), and the Frobenius norm (F-Norm).\n\nThe Pearson correlation $\\tau$ between slopes and operator properties is listed as follows:\n\n|        | Sum       | Trace | 2-Norm | F-Norm |\n| ------ | :---: | :---: | :---: | :---: |\n|   $\\tau$   | **0.845** | 0.605 | 0.523  | 0.624  |\n\nWe observe a strong correlation between the global tendency and the sum of elements. This suggests that there is indeed a correlation between Koopa blocks when their curves exhibit similarity.\n\n**Q3:** About the detailed training parameters.\n\nAs per your request, we provide training parameter as follows, which is shared on all six benchmarks.\n\n| Batch Size | Training Epochs | Learning Rate | Early Stoping Patience | LR Decay Strategy                           |\n| :---: | :---:| :---:| :---: | :---: |\n| 32         | 10              | 0.001         | 3                      | ExponentialLR with decay rate: $\\gamma=0.5$ |\n\n"}, {"review_id": "vuyZeEYzuO", "reviewer": "Reviewer_M3Xz", "summary": "This paper addresses non-stationary time series forecasting, in which case the temporal distribution of time series is changing over time. Koopman theory is introduced in this paper for its fundamental capacity on modeling dynamical systems and a Koopman forecaster called Koopa is proposed. Concretely, in this paper, time series is disentangled into time-variant and time-invariant components. For time-invariant component, Koopa introduces a learnable matrix as Koopman operator, and for time-variant component, Koopa leverages eDMD to find the best fitted matrix that advances forward the system. The proposed framework is technically sound but somewhat lacks novelty. ", "questions": "See weaknesses.", "limitations": "Not applicable.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "1. Non-stationary time series forecasting is challenging and it is reasonable to apply Koopman theory to tackle this task.\n2. Disentangling time series into variant and invariant components is intuitive.\n3. The methods of constructing Koopman operators for both time-variant and -invariant components are straightforward.", "weaknesses": "1. The proposed model lacks novelty. Both disentangling time series and applying Koopman theory are commonly used techniques in time series analysis. \n2. The technical contribution of this paper is limited. The extraction of time variant and invariant components is too simple and lacks analysis. The application of Koopman theory is simply the construction of Koopman operators.\n3. Hierarchically disentangling time series lacks intuition and motivation. What's the point of further disentangle variant component into invariant and variant components?\n4. The experiment does not involve highly non-stationary datasets.\n5. Koopman operators are highly correlated to measurement functions. In the branch of time variant component, although the model constructs diverse Koopman operators, the measurement, i.e., the encoder, is static. For more technical contributions, a deeper integration of Koopman's theory is needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks to Reviewer M3Xz for providing a detailed review. \n\n**Q1:** Novelty of disentangling module.\n\nWe'd like to emphasize our proposed method differs from previous works as follows:\n- It considers time-variant and invariant disentanglement, which is essential for Koopman forecasters, and few series decompositions specialize in it.\n- The module is not a traditional filter. It filters components by **spectrum distribution of the dataset**, instead of pre-defined frequency thresholds.\n\nConcretely, it collects statistics by FFT. The subset in the top percentage sorted by amplitude indicates **dominant time-invariant properties (e.g. periodicity) shared among the dataset**.\n\nWe have conducted analysis in $\\underline{\\text{Figure 5 of the main text}}$, which presents the disentangling effects. And ablation studies in $\\underline{\\text{Table 3 of the main text}}$ check its necessity and compares the performance of other filter choices.\n\n**Q2:** Motivation of the disentanglement.\n\nKoopa is essentially based on disentanglement, which is **motivated by Wold's Theorem**. We sum up motivations as follows:\n- Wold's Theorem implies weakly stationary series can be decomposed into deterministic and stochastic parts. The former (e.g. sine wave) can be easily modeled by an operator, while the latter is the output of a linear filter with stationary processes as input.\n- We utilize data-dependant encoder to learn underlying stationary process and use time-variant operators to portray the varying behavior of the linear filter. The disentanglement is natural on series dynamics and has not been explored in previous works.\n\nAs an intuitive example: $y_t=\\alpha sin(\\omega_0 t)+\\beta sin(\\omega_t t) + \\epsilon_t$, where $\\alpha, \\beta, \\omega_0$ do not change with time, $\\epsilon_t$ is white noise and $\\omega_t$ varies with window. $\\alpha sin(\\omega_0 t)$ behaves time-invariantly because of stable periodicity, while $\\beta sin(\\omega_t t)$ has varying periodicity. The components learned by Koopa are in $\\underline{\\text{Figure 3 of the global response}}$, we find that\n* Time-invariant KP predicts invariant components with stable period.\n* Time-variant KP can predict variant components with window-wise varying period.\n\n**Q3:** Experiments of highly non-stationary datasets.\n\nThe reviewer mentioned it \"does not involve highly non-stationary datasets.\" Based on ADF statistics, there is 99% confidence to accept that a unit root is presented, which indicated these datasets are highly non-stationary: Exchange, ETTh2, and ILI.\n\nWe also include Cryptos evaluated in non-stationary forecaster KNF. Koopa still surpasses existing forecasting models.\n\n|MSE|Koopa|PatchTST| TimesNet| DLinear| KNF|\n|-| -| -|- | -| - |\n|Cryptos|**1.105**|1.111|1.110 |1.116|1.108|\n\n**Q4:** Motivation of the hierarchical disentanglement.\n\nThe review questioned why \"disentangle variant component into invariant and variant components\". Instead, we **disentangle the residual of fitted time-variant component.**, which is motivated by the applicability of Koopman theory and scalability of deep model:\n- The operator eigenvalues determine long-term evolution, where modulus not close to one causes non-divergent evolution. We highlight **a stable Koopman operator only reconstruct well on weakly stationary series**. So we desire each layer to learn weak stationary process hierarchically and feed the residual of fitted dynamics for the next layer to correct.\n- Koopman Predictor aims not to fully reconstruct the whole dynamics at once, but to partially describe dynamics, so we do not force rigorous reconstruction in each layer.\n- Deep decomposition has been demonstrated as effective architecture (e.g.NBEATS, Autoformer) and makes the module scalable for complicated patterns.\n\nWe show the effect of stacking in $\\underline{\\text{Table 1 of the global response}}$. Here are the observations:\n- Sinlge-layer model always leads to the worst forecasting, which is significant on non-stationary datasets (Exchange and ILI).\n- Stacking multi-layer Koopa blocks leads to increased performance on most datasets.\n\n**Q5:** Reclarify the contributions of applying Koopman theory on deep models.\n\nThe reviewer stated \"Koopman theory are commonly used techniques in time series analysis.\" We agree with the argument but want to highlight **incorporating theory into the architecture of deep models** is non-trival. We sum up previous works on Koopman forecasters as follows:\n- Utilize AutoEncoder to learn measurement functions.\n- Introduce extra constraints to refine operator stability on non-stationary series.\n- Use local and global operators to deal with temporal distribution shift.\n\nInstead, our work tackles the above as follows:\n- Design Koopman Predictor with reconstruction loss raveled out and incorporate it into deep residual structure.\n- Not employ any consistency constraint but exquisitely design hierarchal stackable blocks to refine stability.\n- Though previous work also utilizes respective operators, they do not explicitly design and apply time-variant and invariant disentanglement.\n\nOur work further copes with realistic problems that previous works have not explored:\n- End-to-end optimization: We observe in $\\underline{\\text{Table 2 of the global response}}$ that forecasting objective works as good indicator and end-to-end optimizing always achieves better forecasting.\n- Operator adaptation: We derive operator updating rule with reduced complexity for the first time and Koopa can utilize new snapshots to scale up forecast horizon. \n\n**Q6**: About the encoder design.\n\nThanks for your suggestion for encoder design with deeper theory integration. We currently use MLP for the sake of efficiency. It still leaves an open problem and we have tried to make it not \"static\", such as learning the weighting on pre-defined measurements, but it does not achieve further performance improvement. We will appreciate it a lot if the reviewer could inspire us for other potential designs."}, {"review_id": "VrkIH9WtcS", "reviewer": "Reviewer_a2Yj", "summary": "The authors address the issue that most real-world time-series are non-stationary. To tackle this problem they introduce Koopa, a globally learned (time-invariant) and localized (time-variant) linear Koopman operators, to exploit respective dynamics underlying different components. In practice, Koopa is composed of stackable Koopa Blocks, where each block learns operators hierarchically by taking the residual of previous block fitted dynamics as input. The authors evaluate the proposed model on variety of time-series data sets and compare their method with transformer and convolution based deep architectures. Koopa outperforms or performs on par with the existing methods, yet has a significantly lower computational costs. Moreover, the proposed method can be use in settings where the forecast horizon must be increased beyond the training set-up. ", "questions": "For time-variant KP how do you choose S, is the Time-variant KP sensitive to S? ", "limitations": "The authors have not addressed limitation of their work. I would suggest the authors to include such section in the final version of their work. ", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Originality: \nThe related work section in well written. It covers both relevant model types in the field, for (a) time series forecasting, TCN, RNN and transformers are mentioned, as well as the authors position the current work in its context; for (b) Koopman operators, the authors provide an overview of the related work and how their work differs. \nThe work proposes a novel method leveraging Fourier analysis to disentangle time-series components, eDMD for estimating time-variant Koopman Predictor, and time-invariant Koopman Prediction as a learnable parameter. It differentiates itself from KAEs due to its block like structure, where the residual of the previous block is fed as input into the next block. \n\nQuality: \nThe submission is techinncally sound. The authors build upon Koopman operator theory and Wold's Theorem. In addition, experimentally they validate whether their proposed sub-components of Koopa, Fourier Filter, truly disentangles time-variant from time-invariant series. As well as provide an ablation on Koopa structure. \n\nClarity: \nThe work is well written providing sufficient relevant background knowledge on Koopman operator theory also for reader who is unfamiliar with this theory.  \n\nSignificance: \nThe obtained results are significant. The proposed method outperforms or performs on par with existing time-series forecasting methods. More importantly, the method is very efficient with respect to training time and memory costs compared to transformer or convolution based methods. Lastly, the authors showcase that their method is applicable for scaling up forecast horizon, where standard deep learning architectures fail. In all, based on the submission the author's suggest a very promising modelling technique, however, I am not an expert of Koopman operator theory (nor the works in this sub-domain). ", "weaknesses": "Clarity: \nWhen first mentioned, I would recommend using the complete name of the model type rather than only the acronym, for example, temporal convolutional network rather than just TCN, it makes it easier for the reader to read. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks to Reviewer a2Yj for providing thorough detailed comments.\n\n**Q1:** Hyperparameter sensitivity of the segment length $S$.\n\nAs per your suggestion, we have checked the robustness of the proposed method with respect to the hyperparameter $S$, which varies in $\\{H/8,H/4,H/2,H\\}$, where the forecast length is $H$ and the lookback length $T=2H$.\n\n| segment length $S$ | Exchange  | Traffic   | ETTh2     | ILI       | Weather   | ECL       |\n| :----------------- | :-------- | :-------- | :-------- | --------- | --------- | --------- |\n| $H/8$              | 0.046     | **0.458** | 0.244     | **2.326** | 0.129     | 0.157     |\n| $H/4$              | **0.044** | 0.481     | 0.244     | 2.351     | 0.127     | 0.149     |\n| $H/2$              | 0.045     | 0.494     | **0.241** | 2.354     | 0.127     | 0.141     |\n| $H$                | 0.047     | 0.494     | **0.241** | 2.331     | **0.126** | **0.132** |\n\nWe find the proposed model insensitive to $S$ on several datasets but sensitive on ECL and Traffic datasets. It may be because there are many variables in these two datasets, but we currently share $S$ on all variables. Since the prediction of some variables may be greatly affected by $S$, the forecasting performance of the dataset with more variables can be more sensitive to $S$.\n\n\n\n**Q2:** About the writing issues.\n\nThanks for your valuable suggestions. We will replace the acronym of models with their complete names. All the changes will be included in the final version of our work.\n\n**Q3:** About the limitations of the proposed method.\n\nThanks a lot for your concern, we evaluate the limitations of our proposed method in $\\underline{\\text{Section 6 of supplementary materials}}$, which can be summarized as follows:\n\n- The design space of the encoder and decoder. We will further consider the embedding of various evolution patterns for better multivariate forecasting and have deeper integration of Koopman measurement functions.\n- The model interpretability. We will dig into Koopman modes decomposition to reveal the linear behavior underlying non-stationary time series.\n- Incorporation with DMDc. We will consider factors outside observations to tackle time series forecasting with covariates."}, {"review_id": "x7gnNgrVuv", "reviewer": "Reviewer_EGYL", "summary": "This work propose a new architecture inspired by Koopman theory for time-series forecasting task. To address the non-stationarity problem, this work follow the idea of disentangle local and global time-series representation and forecasting model respectively. Particularly, this work propose to use Fourier filter on long sequence to extract more robust and stable patterns for global model and Koopa Block to hierarchically disentangle local and global representation in a residual structure. The proposed method achieves competitive performance when comparing with recent baselines.", "questions": "how to set the hyperparameters  D?", "limitations": "N.A.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.  The structure is well organized. Figures 2 and 3 provide a clear visual representation of how the proposed method works.\n\n2. The experiments are comprehensive. The proposed method can beat the recent method PatchTST on the commonly used long sequence forecasting benchmark.", "weaknesses": "1. Although the paper's title revolves around the Koopman theory, the proposed method is generic and not closely related to it. \nFrom my understanding, most of technique contributions lie in how to extract local and global representation and design forecasting model for non-stationary time-series. The work solely follow the the conclusion of Koopman theory and model the latent dynamic with linear transition model. \n\n2. Technique novelty is limited. The general idea of using local and global presentation/model to tackle non-stationary time-series tasks has been used. This work builds upon these foundations and introduces several modifications by integrating some other methods used in recent deep time series forecasting, like residual block, Fourier filter.\n\n3.  Section 1 needs improvements. It is clear for me how Koopman operator address the non-linear dynamics, but section 1 does not clearly show how existing work of Koopman theory tackle non-stationary problems. Could the authors elaborate more on line 40-42? In line 52-54, could the authors explain the motivation of replacing reconstruction loss? In line 30-31, the authors introduce \"Non-stationary  ...... in different periods\". However, the proposed method do not show the capability to detect period boundary.\n\n4. The experiments are comprehensive, but it would be better to design experiments revolve around the topic and the proposed method. Ablation studies can successfully validate the effectiveness of the design of local and global model, but the experiment associated time and memory cost analysis might lose focus. The authors can also explore the effect of number of block to verify the capability of hierarchical disentanglement.  \n\n5. It would be better to discuss the related work of non-stationary time series forecasting, like regime switching state-space models, time-varying state-space models, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks to Reviewer EGYL for the detailed and insightful review.\n\n**Q1:** Technique novelty of proposed method.\n\nAs clarified in $\\underline{\\text{Q5 of the reviewer M3Xz}}$, we sum up \"how existing work of Koopman theory tackles non-stationary problems\" and highlight our difference with previous Koopman forecasters. \n\nAnd we'd like to clarify our work is not straightforward as \"integrating some other methods used in recent deep time series forecasting, like residual block, Fourier Filter\":\n- To our best knowledge, no previous Koopman forecasters consider stackable modular design and hierarchical forecast. To address the issue, we propose specialized Koopman Predictors to learn respective dynamics.\n- We make scalable incorporation with deep residual structure. It is totally different from residual connection proposed in ResNet. Instead, we feed **the residual of fitted input** of one layer to the next layer in the hope to learn hierarchical dynamics and **aggregate forecast** from all layers as the model output.\n- We propose a way to **extract the spectrum distribution of dataset**, which is based on FFT but is just the first step of our filter. Fourier Filter is not like canonical filters, which are implemented by truncating pre-defined frequency. Besides, the proposed filter is specially designed for Koopman forecasters, and the extraction of time-variant component is hardly explored in previous work.\n\n**Q2:** The motivation to stack Koopa Blocks and replace reconstruction. \n\nThanks a lot for your valuable suggestion to verify the effect of block stacking, please refer to the results and our analysis in $\\underline{\\text{Q4 of the reviewer M3Xz}}$.\n\nBesides, with the introduction of hierarchical forecasting, the ground truth of reconstruction should be the residual of fitted dynamics, which is unavailable during training. Therefore, we adopt the deep residual structure, where solely forecasting objective works as a good optimization indicator, and we further validate it in $\\underline{\\text{Table 2 of the global response}}$.\n\n**Q3:** How does the proposed model detect the period boundary?\n\nFirstly, Fourier Filter helps the model detect window-shared periodicity. Based on the disentangled time-invariance component with almost shared periodicities, the global operator learns lookback-forecast **transition**, which relies less on the specific period boundary but more on the evolution pattern (such as how the phase changes and the decay rate of eigenfunctions).\n\n**Q4:** Reclarify the position and contribution of our work.\n\nThe reviewer mentioned that our work \"solely follows the conclusion of Koopman theory\". We agree with this argument but also would like to clarify that the position of our work is to **propose an applicable architecture based on Koopman theory for real-world forecasting**. Concretely, our contributions mainly revolve around architecture and can be listed as follows:\n- Specialized designed disentanglement for Koopman forecasters.\n- Koopman Predictor with reconstruction loss raveled out and achieve end-to-end objective optimization.\n- Hierarchically disentangling and deep residual structure for model scale up.\n- Derive operator updating rule with reduced complexity based on the linear properties of Koopman operator.\n\n**Q5:** Discuss related works of non-stationary time series forecasting.\n\nPrevious works adopt series stationarization to attenuate non-stationarity for better predictability, such as Adaptive Norm, DAIN, and ReVIN.\n\nSeveral recent works have also explored model architecture for non-stationary forecasting, especially with theory support that addresses time-variant properties. As the reviewer insightfully mentioned, the State Space Models (SSM) share lots of similarities with Koopman forecasters. Representatives such as Kalman Filters widely applied on control systems, and deep SSMs are good at modeling long sequences. Therefore, we'd like to discuss their similarity and differences as follows:\n\n(1) Similarities:\n- They portray the time series as the state transition of the system.\n- They are able to portray time-variant transitions.\n- They can introduce external driving factors to address non-stationarity caused by regime/concept drifting. \n\n(2) Differences:\n- Approach to describing time-variance: Regime Switching State Space Models assume pre-defined regime number and conditions to transit (Markov Process modeling). Time-variant SSM requires exquisitely designed transitions such as the HiPPO matrix. While our Koopman forecaster analyzes time-variant dynamics by disentanglement and leveraging DMD.\n- Model implementation: SSM is always implemented by RNN, with memory to model long-term dependencies. Canonical Koopman forecasters are always incorporated with AutoEncoder. Besides, the transition learner of Koopman forecasters (e.g. parameterized matrix) can be more simple and interpretable.\n- Objective optimization: SSM relies on Variational Bayesian Inference and always uses KL-divergency (CrossEntropy) for optimization. Koopman forecasters are traditionally optimized by reconstruction and dynamics advancing loss (MSE). And our model is solely optimized by end-to-end forecasting loss.\n\nTo further address your concern, we also include the widely used SSM: LSSL and Regime Switching LSSL as additional baselines. Here are the results. Koopa still achieves the best performance in all benchmarks.\n\n|Average(MSE)|ECL|ETTh2|Exchange|ILI|Traffic|Weather|\n|-|-|-|-|-|-|-|\n|LSSL|0.284|1.426|1.036|4.580|0.699|0.216|\n|RegimeSwitching|0.164|0.361|0.176|2.841|0.495|0.189|\n|Koopa|**0.143**|**0.303**|**0.110**|**1.734**|**0.404**|**0.161**|\n\n**Q6:** Hyperparameter analysis of the dimension of Koopman embedding $D$.\n\n$D$ is selected from $\\{64, 128, 256, 512\\}$. We have also provided hyperparameter sensitivity analysis in $\\underline{\\text{Section 3 of supplementary materials}}$, where we find the performance insensitive to the choices of $D$."}, {"review_id": "qomcnLxvv3", "reviewer": "Reviewer_pXJ3", "summary": "This paper tackles the task of non-stationary time series forecasting with Koopman operator. Specifically, the authors propose Koopa Blocks, which uses Fourier transform to disentangle the time series into time-variant and time-invariant components, and leverages Koopman \noperator to model their dynamics respectively. The method is evaluated on six real-world benchmark datasets.", "questions": "1. The authors implement the DMD with torch.linalg.lstsq.  Were there any concerns regarding gradient explosion during the autograd process through this layer? Did the authors employ any techniques to stabilize the training process?", "limitations": "Yes.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. This study represents a pioneering endeavor in integrating Fourier transform and Koopman theory, with the aim of disentangling time-variant and time-invariant components within the Fourier spectral domain. The idea is both innovative and thought-provoking, setting a precedent for future investigations into the research of deep Koopman methods.\n3. The additional \"Scaling Up Forecast Horizon\" experiments demonstrate that Koopa is capable of accurately forecasting in situations involving length mismatches or long-term scenarios. In the realm of the Koopman method, these long-term prediction results are exceptional.\n4. The proposed model is memory and time efficient compared to previous methods.", "weaknesses": "1. The performance improvement of univariate forecasting is incremental. Also, the improvement of multivariate forecasting is also not so significant under most cases.\n2. The essence of the deep Koopman method lies in the spectrum of the Koopman operator, because the eigenvalues determine the model's behavior during long-term evolution. However, the author did not provide any analysis or visualized results regarding the eigenvalues. This lead to the following issues:\n- Given the considerable forecasting length (up to 192 in this case), it is remarkable how the model manages to achieve accurate predictions without encountering issues such as explosion or decay during the forward pass. Is it true that all the eigenvalues of the system have a modulus of one, ensuring stable and non-divergent behavior throughout the prediction process? \n- Intuitively, it would be expected that the time-variant Koopman operator possesses more eigenvalues with larger phase angles compared to the time-invariant operator. I wonder whether this is true in this case. What does the spectrum for these two operator looks like?\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nMany thanks to Reviewer pXJ3 for providing a detailed review and insightful questions. \n\n**Q1:** About the performance of the proposed method.\n\nTo our best knowledge, the benchmark of time series forecasting has been excavated for a long time. And PatchTST as the SOTA forecasting model has surpassed concurrent work by a large margin. However, the Transformer-based model still suffers from the efficiency problem, which makes it less applicable for practitioners. Meanwhile, the recent revival of linear models presents a simple but effective approach, but the performance is still under promotion on non-stationary time series. \n\nTherefore, we delve for **applicable and efficient time series forecasting** based on Koopman theory and achieve **77.3%** training time and **76.0%** memory cut-off on average, while exceeding current SOTA in **34 out of 48** benchmarks. We hope it would help practitioners achieve performance on par with a heavily trained Transformer-based model while benefiting from the efficiency of linear models. \n\n**Q2**: Techniques for stabilizing the training process.\n\nAs shown in our code implementation, we employ several techniques to stabilize the training process：\n\n- **Operator initialization:** We adopt the operator initialization as previous Koopman-based forecasters, where the eigenfunctions start from standard Gaussian distribution with all-one eigenvalues.\n- **Explosion checking**: When we find explosion occasionally happens in the proposed Koopman Predictor, we introduce an explosion checking mechanism that replacs the operator encountering nan with the identity matrix.\n- **Hierarchically disentanglement**: While reproducing the results of previous Koopman forecasters, we often find the training fails with explosive outputs. As the reviewer mentioned insightfully, we analyze the phenomenon and find the hierarchical disentanglement could alleviate the problem (Please refer to $\\underline{\\text{Q2 and Q4 of the reviewer M3Xz}}$ for the analysis).\n\nBesides, motivated by your suggestion of analyzing of the eigenvalues, we conduct spectral experiments to ablate the disentanglement and hierarchical forecasting mechanism. We use $average(||z|-1|)$ to measure the the stability of operator describing the pattern evolution. On Exchange dataset, we plot the eigenvalues $z$ and the measurement of the time-invariant operator in the first layer in the following cases:\n\n- a. Single-layer model with only time-invariant operator.\n\n- b. Single-layer model with time-invariant and time-variant operators.\n\n- c. Two-layer model with time-invariant and time-variant operators.\n\nThe visualization results are shown in $\\underline{\\text{Figure 2 of the global response}}$, with the introduction of disentanglement and hierarchically stacking, we observed the eigenvalues become more close to the unit circle and present a more stable operator.\n\n**Q3**: Spectral analysis of time-invariant and time-variant operators.\n\nThanks for this valuable suggestion. We visualize the eigenvalues of time-invariant and time-variant operators on Exchange dataset in $\\underline{\\text{Figure 4 of the global response}}$, And we have the following two findings:\n\n- We find it not obvious that more eigenvalues with larger phase angles exist in the time-variant Koopman operator than in the time-invariant operator. For example, in time series $y_t=\\alpha sin(\\omega_0 t)+\\beta sin(\\omega_t t) + \\epsilon_t$, where $\\alpha, \\beta, \\omega_0$ do not change with time, $\\epsilon_t$ is white noise and $\\omega_t$ varies window-wisely, the first term $\\alpha sin(\\omega_0 t)$ behaves time-invariantly because of stable periodicity, while $\\beta sin(\\omega_t t)$ with varying periodicity is close to our desired time-variant component. However, no partial order assumption is made between $\\omega_0$ and $\\omega_t$. Thus, the phase angles of eigenvalues may not have explicit relations.\n- Time-variant Koopman operators have many multiple eigenvalues around zero, which indicates simpler evolution patterns than Time-invariant operators. It's possible because Time-variant KP learns the dynamics within one lookback window, while the other learns the dynamics underlying the whole dataset."}]

Author rebuttal:
##  Global Response to All Reviewers

We sincerely thank all the reviewers for their insightful reviews and valuable comments, which are instructive for us to improve our paper further.

To cope with non-stationary time series forecasting, this paper proposes Koopman-based forecasters (Koopa) to portray the underlying time-variant properties. Inspired by Koopman theory, we design renovated Koopman Predictors and disentanglement to hierarchically reveal complicated dynamics. We delve into architectural design for applicable forecasters on real-world non-stationary series, with end-to-end forecasting objective optimization, fairly increased model efficiency, and operator adaptation to scale up forecast horizon. **Comprehensive experiments and module analysis are included. Koopa achieves competitive performance while saving 77.3% training time and 76.0% memory on six real-world benchmarks.**

The reviewers generally held positive opinions of our paper, in that the proposed method is "**novel**", "**technically sound**", "**a pioneering endeavor**", and "**the idea is innovative and thought-provoking, setting a precedent for future investigations into the research of deep Koopman methods**", this paper is "**well-written**" and "**provides a clear visual representation** " and the experiments are "**comprehensive**", "**significant**".

The reviewers also raised insightful and constructive concerns. We made every effort to address all the concerns by providing sufficient evidence and requested results. Here is the summary of the major revisions:

- **Analysis of operators (Reviewer 36zV, pXJ3)**: We provide the correlation analysis between time series and learned operators. To address the issue of operator stability, we visualize the respective eigenvalues in Koopman Predictors. The results present the effectiveness of our architectural design for better convergence of training operators.
- **Motivation of proposed modules (Reviewer M3Xz)**: We illustrate our motivation in both theoretical and experimental aspects. We clarify disentanglement and block stacking based on Wold's Therom, Koopman theory and the scalability of deep model. By conducting ablation and evaluating the other possible designs, we verify that our model still achieves the best performance and good generality.
- **Technical novelty (Reviewer EGYL, M3Xz)**: We highlight our difference with previous works on decomposition and Koopman forecasting. We specifically design and apply the disentanglement for Koopman-based forecasters and renovate canonical KAEs with raveled-out reconstruction branch by incorporating it into deep residual structure.
- **Analysis of hyperparameters (Reviewer a2Yj, EGYLj)**: We newly add hyperparameters sensitivity analysis on the segment length and the dimension of Koopman embedding. And we also clarify the selection strategy of model hyperparameters.
- **Discussion of related works (Reviewer EGYL)**: We discuss the similarities and differences between our method and related works on non-stationary time series forecasting. We especially compare Koopa with State Spaces Models in the aspects of implementation, optimization, and the way to describe time-variance. We also conduct experiments on the full benchmarks to check their performance.

The valuable suggestions from reviewers are very helpful for us to revise the paper to a better shape. We'd be very happy to answer any further questions. 

Besides, shared Tables of the global response are listed as follows, and Figures are provided in PDF.

> Table.1  Analysis of block number.
| Block Number (MSE) | ILI       | ETTm2     | Exchange  | Weather   | ETTh2     | ECL       |
| ------------------ | :-------- | :-------- | :-------- | :-------- | --------- | --------- |
| 1                  | 2.184     | 0.136     | 0.520     | 0.129     | 0.242     | 0.153     |
| 2                  | 1.980     | 0.133     | 0.506     | 0.126     | 0.241     | **0.141** |
| 3                  | 1.974     | 0.132     | 0.479     | 0.125     | **0.236** | 0.142     |
| 4                  | **1.850** | **0.131** | **0.473** | **0.124** | 0.239     | 0.158     |

> Table.2 Analysis of introducing reconstruction 
| ETTh2 (MSE\|MAE)      | Predict 48             | Predict 96             | Predict 144            | Predict 192            |
| --------------------- | --------------- | --------------- | ---------------------- | ------------|
| Koopa                 | **0.226** \| **0.300** | **0.297** \| **0.349** | **0.333** \| **0.381** | 0.356 \| 0.393         |
| + Reconstruction | 0.244 \| 0.311         | 0.309 \| 0.356         | 0.339 \| 0.385         | **0.354** \| **0.392** |
||||||||
| **Exchange**   |             |             |             |            |
| Koopa                 | **0.042**\| **0.143** | **0.083** \| **0.207** | **0.130** \| **0.261** | **0.184** \| **0.309** |
| + Reconstruction | 0.047 \| 0.152        | 0.102 \| 0.225         | 0.131 \| 0.263         | 0.235 \| 0.352         |
||||||||
| **ECL**   |             |             |             |            |
| Koopa                 | **0.130** \| **0.234** | **0.136** \| **0.236** | **0.149** \| **0.247** | **0.156** \| **0.254** |
| + Reconstruction | 0.183 \| 0.282         | 0.159 \| 0.260         | 0.165 \| 0.265         | 0.167 \| 0.269         |
||||||||
| **Traffic**     |              |             |   |          |
| Koopa                 | **0.415** \| **0.274** | **0.401** \| **0.275** | **0.397** \| **0.276** | **0.403** \|**0.284** |
| + Reconstruction | 0.518 \| 0.353         | 0.446 \| 0.317         | 0.445 \| 0.325 | 0.444 \| 0.322        |
||||||||
| **Weather**     |              |             |           |           |
| Koopa                 | **0.126** \| **0.168** | **0.154** \| **0.205** | **0.172** \| **0.225** | **0.193** \| **0.241** |
| + Reconstruction | 0.143 \| 0.186         | 0.163 \| 0.208         | 0.178 \| 0.226         | 0.196 \| 0.246         |

Meta-review: The authors propose a deep architecture for time series forecasting, inspired by the theory of Koopman operators. After a thorough discussion with many clarifications from the authors, the reviewers converged on a positive assessment of the work. While the connection to Koopman theory is somewhat tenuous, the architectural proposal is interesting and the results are promising.

AC note: In addition to addressing the reviewers' concerns and adding the promised improvements to the final manuscript, I encourage the authors to correct grammatical mistakes throughout the paper. For example, the first line of the abstract has a singular/plural error.
Decision: Accept (poster)