Koopman Neural Operator Forecaster for Time-series with Temporal Distributional Shifts
Authors: Rui Wang
Yihe Dong
Sercan O Arik
Rose Yu
Venue: ICLR 2023 poster (2023)
Keywords: Time series forecasting
Temporal distributional shifts
Koopman Theory

Abstract: Temporal distributional shifts, with underlying dynamics changing over time, frequently occur in real-world time series and pose a fundamental challenge for deep neural networks (DNNs). In this paper, we propose a novel deep sequence model based on the Koopman theory for time series forecasting: Koopman Neural Forecaster (KNF) that leverages DNNs to learn the linear Koopman space and the coefficients of chosen measurement functions. KNF imposes appropriate inductive biases for improved robustness against distributional shifts, employing both a global operator to learn shared characteristics and a local operator to capture changing dynamics, as well as a specially-designed feedback loop to continuously update the learnt operators over time for rapidly varying behaviors. We demonstrate that KNF achieves superior performance compared to the alternatives, on multiple time series datasets that are shown to suffer from distribution shifts.

Published as a conference paper at ICLR 2023
KOOPMAN NEURAL FORECASTER FOR TIME SERIES WITH TEMPORAL DISTRIBUTION SHIFTS
Rui Wang" UC San Diego
Yihe Dong Google Cloud Al
Sercan Ö. Arik Google Cloud AI
Rose Yu UC San Diego
ABSTRACT
Temporal distributional shifts, with underlying dynamics changing over time, frequently occur in real-world time series, and pose a fundamental challenge for deep neural networks (DNNs). In this paper, we propose a novel deep sequence model based on the Koopman theory for time series forecasting: Koopman Neural Forecaster (KNE) that leverages DNNs to learn the linear Koopman space and the coefficients of chosen measurement functions. KNF imposes appropriate inductive biases for improved robustness against distributional shifts, employing both global operator to learn shared characteristics, and local operator to capture changing dynamics, as well as specially-designed feedback loop to continuously update the learnt operators over time for rapidly varying behaviors. We demonstrate that KNF achieves the superior performance compared to the alternatives, multiple time series datasets that are shown to suffer from distribution shifts. We open-source our code at https: /tttttpp://gtttbbbs wtt.:thuw..c.oooooeoe.eo 1 research/ e wew.reeerereerrtteeeeeeeeeeeeeee
/KNF
INTRODUCTION
Temporal distribution shifts frequently occur in real-world time-series applications, from forecasting stock prices to detecting and monitoring sensory measures, to predicting fashion trend based sales. Such distribution shifts over time may due to the data being generated in highly-dynamic and non-stationary environment, abrupt changes that are difficult to predict, or constantly evolving trends in the underlying data distribution (Gama et al. 2014).
Temporal distribution shifts pose a fundamental challenge for time-series forecasting (Kuznetsov & Mohri 2020). There are two scenarios of distribution shifts. When the distribution shifts only occur between the training and test domains, meta learning and transfer learning approaches (Jin et al., 2021 Oreshkin et al., 202T) have been developed. The other scenario is much more challenging: distribution shifts occurring continuously over time. This scenario is closely related to "concept drift" (Lu et al. 2018) and non-stationary processes (Dahlhaus 1997) but has received less attention from the deep learning community. In this work, we focus the second scenario.
To tackle temporal distribution shifts, various statistical estimation methods have been studied, including spectral density analysis (Dahlhaus 1997), sample reweighting (Bennett & Clarkson 2022 McCarthy & Jensen 2016) and Bayesian state-space models (West & Harrison| 2006). However, these methods are limited to low capacity regressgee models and typically designed for short-horizon forecasting. For large-scale complex time series data, deep learning models (Oreshkin al. 2021: Woo al., 2022 Tonekaboni al. 2022 Zhou al. 2022) now increasingly outperform traditional statistical methods. Yet, most deep learning approaches are designed for stationary time- series data (with i.i.d. assumption), such as electricity usage, sales and air quality, that have clear seasonal and trend patterns. For distribution shifts, DNNs have been shown to be problematic in forecasting on data with varying distributions (Kouw Loog 2018
Wang al. 202T).
DNNs are black-bor box models and often require large number of samples to learn. For time series with continuous distribution shifts, the number of samples from given distribution is small, thus DNNs would struggle to adapt to the changing distribution. Furthermore, the non-linear dependencies in DNN are difficult to interpret or manipulate. Directly modifying the parameters based on the change in dynamics may lead to undesirable effects (Vlachas et al. 2020). Therefore, if we can
Work done during the internship at Google Cloud AL

---

Published as a conference paper at ICLR 2023
reduce non-linearity and simplify dynamics modeling, then we would be able to model time series in
much more interpretable and robust manner. Koopman theory (Koopman 1931) provides convenient tools to simplify the dynamics modeling. It states that any nonlinear dynamics can be modeled by linear Koopman operator acting on the space of measurement functions (Brunton et al. 2021), thus the dynamics can be manipulated by simply modifying the Koopman matrix.
In this paper, we propose a novel approach for accurate forecasting for time series with distribution shifts based on Koopman theory: Koopman Neural Forecaster (KNF). Our model has three main features: 1) we combine predefined measurement functions with learnable coefficients to introduce appropriate inductive biases into the model. 2) our model employs both global and local Koopman operators to approximate the forward dynamics: the global operator learns the shared characteristics; the local operator captures the local changing dynamies. 3) we also integrate a feedback loop to cope with distribution shifts and maintain the model's long-term forecasting accuracy. The feedback loop continuously updates the learnt operators over time based on the current prediction error.
Leveraging Koopman theory brings multiple benefits to time series forecasting with distribution shifts: 1) using predefined measurement functions (e.g., exponential, polynomial) provide sufficient expressivity for the time series without requiring a large number of samples. 2) since the Koopman operator is linear, it is much easier to analyze and manipulate. For instance, we can perform spectral analysis and examine its cigenfunctions, reaching a better understanding of the frequeney of oscil- lation. 3) Our feedback loop makes the Koopman operator adaptive to non-stationary environment. This is fundamentally different from previous works that learns single and fixed Koopman operator (Han et al. 2020; Takeishi et al. 2017 Azencot et al. 2020).
In summary, our major contributions include:
Proposing a novel deep forecasting model based on Koopman theory for time-series data with temporal distributional shifts.
The proposed approach allows the Koopman matrix both capture the global behaviors and evolve over time to adapt to local changing distributions.
Demonstrating stateoof-theaa performance on highly non-stationary time series datasets, including M4, cryptocurrency return forecasting and sports player trajectory prediction.
Generating interpretable insights for the model behavior via eigenvalues and eigenfunctions of the Koopman operators.
2 RELATED WORK
DNNs for time-series forecasting. DNNs are shown to increasingly outperform traditional statis- tical methods (such as exponential smoothing (ETS) (Gardner Jr 1985) and ARIMA (Ariyo et al.) 2014)) for time series forecasting. For example, Tonekaboni et al. (2022); Wang et al. (2019) pro- posed to DNNs to learn the local and global representations of time series separately, showing high accuracy on sales and weather data. Woo et al. (2022) leverages inductive biases in different architectures and also specially-des -esiined contrastive loss to learn disentangled seasonal and trend representations, Sen et al. (2019) utilized a global TCN to avoid normalization before training when there wide variations scale. But it focuses mainly on better modeling the relationships between time series rather than advances in modeling over time as ours. Transformer-basee approaches are particularly effective in time series forecasting, particularly on datasets including electricity and traffic (Zhou et al., 2022, Wu et al. 2021] Zhou et al. 2021 which are relatively stationary and have clear seasonality and trend dynamics.
Robustness against temporal distribution shifts. Non-stationarity poses a great challenge for time series forecasting. To cope with varying distributions, one approach is to stationarize the input data. Kim et al. (2022) proposes a reversible instance normalization technique applied on data to alleviate the temporal distribution shift problem. Similarly, |assalli et al. (2019) utilizes DNN to adaptively stationarize input time series. But these approaches did not improve the generalizability of DNNs. Liu et al. (2022) proposes
technique to stationarize time series, but only for transformer- based models. (Arik et al., (022)) proposes test-time adaptation with a self-supervised objective to better adapt against distribution shifts. Another line of work is to combine DNNs and statistical approaches. for better accuracy on non-stationary time series data (Makridakis
al. 2020 Malinin et al. 2021). [Smyl] 2020) combines ETS with RNN, where the seasonality and
2

---

Published as a conference paper at ICLR 2023
Transformer Encoder
[G(V, -otth
G(V))
Lookback Window
Forecasting Window
Enc
Ki
Enc V
Enc V.
Ryf tope?
suf
K +
K k
K K!
KY + K KT
K K
Dec
Dec
Dec
Dec
Prediction
Observation
K2
Prediction Error on the Lookbuck Window
Feedback
Figure 1: Architecture of the proposed KNF model. It encodes every multiple steps of observations into a measurement vector ((VV) = [g] (v. (.(,")) G(V,) is computed based on the set of predfined measurement functions G and their input values V, learnt by the encoder. The model utilizes global operator K and an adaptive local operator IC; learnt by a Transformer encoder to model the evolution of measurements. On the forecasting window, additional adjustment operator KG is learnt by a MLP module based on the prediction error on the lookback window.
smoothing coefficients, are fitted concurrently with the RNN weights using the same gradient descent optimization. Oreshkin et al. (2019) proposes a deep residual architecture for univariant time series data, in which each block used MLP to learn coefficients of basis functions, such as polynomials and sine functions. Indeed, we show that our model outperforms both Smyl (2020) and Oreshkin et al. (2019) the M4 financial time series dataset (Makridakis al. 2020).
Koopman theory. Koopman theory (Koopman 1931 Strogatz 2018) shows that nonlinear dynamical system can be represented as an nnfinite-dimensiinall linear Koopman operator acting on a space of measurement functions. The spectral decomposition of the linear Koopman operator provide insights on the behaviors of nonlinear systems. Traditionally, dynamic mode decomposition (DMD) (Brunton et al. 2016) is commonly used for approximating the Koopman Operator. However, it is highly nontrivial to find appropriate measurement functions well as the Koopman operator. Many works at the intersection of machine learning and Koopman theory employ DNNs learn
measurement functions from data (Han et al., 2020 Li et al. [2200] Morton et al.) 2019 Li & Jiang
2021: Takeishi et al., 2017 Fan et 2022). Yeung et al. (2019) and Lusch et al. (0018) propose to use fully connected DNNs to learn the transformation between the observations and measurement functions that span a Koopman invariant subspace and the Koopman operator is approximated by linear layer. Azennct et al. (2020) also designed autoencoder architecture based on Koopman theory to forecast fluid dynamics. Different from these approaches, we allow the Koopman matrix to evolve time for each time series adapt to the changing distribution.
METHODOLOGY
3.1 TIME-SERIES FORECASTING WITH KOOPMAN THEORY
Time series data {x:)²= can be considered observations of dynamical system states consider following discrete form: Vt+1 F(x,), where æ E X C IR is the system state, and F is the underlying governing equation. We focus multi-step forecasting task of predicting the future states given a sequence past observations. Formally, we seek a function map of such that:
(It-q (tt-qtl.........................
Iz) - (Z.t)...., Itth), I.th)),)
(1)
where & is the lookback window length and h is the forecasting window length.
3

---

Published as a conference paper at ICLR 2023
Koopman theory (Koopman an infiite-dimmnsional linear Koopman operator acting on the space of all possible measurement functions. More specifically, there exists a linear infinite- diriiitedddesiinnaal operator K. g(x) + (
[931] shows that any nonlinear dynamic system can be modeled by
that acts on a space of real-valued measurement functions ((*') == {g : X' + IR.. The Koopman operator maps between function spaces and advances the observations of the state to the next step:
Kg(æ;)) = g (((())) = (x(++))
We propose Koopman Neural Forecasting (KNF), a deep sequence model based on Koopman theory to forecast highly non-stationary time series, shown in Fig. D It instantiates encoder-decoder architecture for each time series segment. The encoder takes in observations from multiple time steps the underlying dynamics may contain higher-order time derivatives. Our model has three main features: 1) we use predefined measurement functions with learned coefficients to map time series to the functional space. 2) the model employs both global Koopman operator to learn the shared characteristics and a local Koopman operator to capture the local changing dynamics. 3) we also integrate feedback loop to update the learnt operators over time based on forecasting error, maintaining model's ' long-term forecasting performance.
3.2 LEVERAGING PREDEFINED MEASUREMENT FUNCTIONS
We define a set of measurement functions G == I9..... 9n] that spans the Koopman space, where each 9: : + R. For example, 91 (z) sin(x). These functions are canonical nonlinear functions and are often used to model complex dynamical systems, such as Duffing oscillator and fluid dynamics (Brunton et al. 202T: Kutz et al., 2016). They also provide sample-efficient approach represent highly nonlinear behavior that may be difficult to learn for DNNs.
We use an encoder to generate the coefficients of the measurement functions V X,), such as the frequencies of sine functions. Let n be the number of measurement functions for each feature, d be the number of features in a time series and be the number of steps encoded by the encoder V dx + I xxdxk every time. The lookback window length q is multiple of l and we denote (t) as X, E Raxk for simplicity.
As shown in the Eq below, we first obtain latent matrix V, = (o 2y 2s (1) Raxd Every vector es E R is a different linear transformation of the observations, where the weights are learnt by the encoder V
(2)
Vi[i, = Σ ((X.)) isisn, 1sj 151sk.
(3)
Our measurement functions are defined in the latent space rather than the observational space. We apply a set of predefined measurement functions g to the latent matrix V₁:
(VV [gi (o) (.) "
(,,,), ",..,,,(( (o(")) E Rexd
(4)
In our implementation. we flatten G(V;) into vector and then finite Koopman operator should be nd x nd matrix. Finally, we use a decoder d Raad + gexd to reconstruct the observations from the measurements:
= (C(VV)
(5)
Here, the encoder and the decoder & can be any DNN architecture, for which we use multi-layer perceptron (MLP). The set of measurement functions G contains polynomials, exponential functions, trigonometric functions as well interaction functions. These predefined measurement functions are useful in imposing inductive biases into the model and help capture the non-linear behaviors of time series. The encoder model needs to approximate only the parameters of these functions without the need of directly learning non-stationary characteristics. Ablation studies (in Sec. 4.8) demonstrate that using predefined measurement functions significantly outperforms the model with leamned measurement functions in the previous works.
3.3 GLOBAL AND LOCAL KOOPMAN OPERATORS
Dynamic mode decomposition (DMD) TTu et al. 2013) is traditionally used to find the Koopman operator that best propagates the measurements. But for time series with temporal distribution shift,
4

---

Published as a conference paper at ICLR 2023
need to compute spectral decomposition and learn Koopman matrix for every sample (i.e. slice of a trajectory), which is computationally expensive. So we utilize DNNs to learn Koopman operators.
In classic Koopman theory, the measurement vector ((V)) is affiite-dimensiie to learn. We assume that encoder is learning a finite approximation and G Vi) forms a finite Koopman- invariant subspace. Thus, the Koopman operator IC that need to find is the finite matrix that best advances the measurements forward in time.
which is impossible
While the Koopman matrix should vary across samples and time in case, it should also capture the global behaviors. Thus, we propose to use both a global operator XC# and a local operator K.) to model the propagation of dynamics in the Koopman space, defined below:
(V) = (KC2 + (V,) = G(V++1), t t O.
(6)
The global operator 1 is an nd x nd trainable matrix that is shared across all time series. We use the global operator to learn the shared behavior such as trend. The local Koopman operator K2, on the other hand, is based on the measurement functions on the lookback window for each sample, shown in Fig. [ The local operator should capture the local dynamics specific to each sample. Since we generate the forecasts in an autoregressive way, the local operator depends on time / and varies across autoregressive steps, adapting to the distribution changes along prediction. We use a Transformer architecture with single-head as the encoder, to capture the relationships between measurements at different steps. We use the attention weight matrix in the last layer as the local Koopman operator.
3.4 FEEDBACK LOOP
Suppose an abrupt distributional shift occurs in the middle of the look-back window, the model would still try to fit two distributions before and after the shift but a single proporgration matrix is never good enough to model multiple distributions. This will results in the inaccurate operator used for the forecasting window. To address it, we add an additional feedback closed-loop, in which we employ an MLP module to learn the adjustment operator KC; based on the prediction errors in the lookback window. It is directly added to other operators when making predictions on the forecasting window, as shown Fig. [ More specifically, we apply global and local operators recursively to the measurements at the first step in the lookback window to obtain predictions:
X X-g/kk++ = ((CC + (VV) 0 0<isg/k
Then, the feedback module I' uses the difference between the predictions on the lookback window and the observed data generate additional adjustment operator Kū; which a diagonal matrix:
K = - Xt-g/ka)) = I(ā:-q. Btt-qit)
If the predictions deviate significantly from the ground truth within the lookback window, the operator Ki would learn the temporal change in the underlying dynamics and correspondingly adjust the other two operators. Thus, for forecasting, the sum of all three operators is used:
S = (<CC + K2, + (VV), i> 0.
In word, the feedback module is designed to detect the distributional shifts in the lookback window and adapt the global+local operator to the latest distribution in the lookback window.
3.5 Loss FUNCTIONS
KNF is trained in end-to-end fashion, using superposition of three loss terms L = Liec + Lbuck L Denote as a distance metric for which we use the L.2 loss. The first term is the reconstruction loss, ensure the decoder & can reconstruct the time series from the measurements:
Liee = ((x,, ((G(V
t 120 0.
(10)
The second term is the prediction loss on the lookback window to ensure the sum of global and local operators is the best-fit propagation matrix on the lookback window.
Track = 1 (X ((CK'
Xx-//k)), O<isq/h
(11)
The third term is for prediction accuracy in the forecasting window to guide the feedback loop to learn the correct adjustment placed on the Koopman operator.
Lior = I(X ((C" + xc| + (V))((V)) i> 0.
(12)
5

---

Published as a conference paper at ICLR 2023
4 EXPERIMENTS
4.1 DATASETS
We benchmark method on three time series datasets: M4 (Makridakis et al. 2020), Crypto (Arik) et al. 2022) and Basketball Player Trajectories (Li et al. 2021). These time series are particularly chosen they are difficult forecast due high nonstationarity and abrupt temporal changes, as analyzed in Sec. 4.2.
M4. It contains 10000 highly nonstationary univariate time series with different frequencies from hourly to yearly and different categories from financials to demographics. The forecasting horizon varies across different frequencies. We directly compare KNF with the M4 competition winner (Smyl 2020). the second place (Montero-Manpy et al. 2020) and the ensemble N-Beats-I4e (Oreshkin 2019 that has achieved the competitive prediction performance on M4.
et al.
Crypto This multivariate dataset contains 8 features on historical trades, such as open and close prices, for 14 cryptocurrencies. The original challenge to predict 3-step ahead 15-minute relative future returns. Since we focus on long-term forecasting, we train all models to make 15-step predictions of 15-minute relative future returns. We use the original training set from the competition and do 80%- 109%-0%%-100 10%-10% training-validationns validation-test split.
Player Trajectory P This dataset contains basketball player movement trajectories from NBA games in 2016. We randomly sample 300 player trajectories for training and validation and 50 trajectories for testing. All models are trained to yield 30-step ahead predictions.
4.2 DISTRIBUTION SHIFTS AND FORECASTABILITY
As way of motivating for the improvement scenarios of our method, we show that the three datasets we focus are much more difficult to predict than a commonly-use datasets like Electricity (Harries
& Wales 1999) with the following three metrics: (1) Forecastability (Goerg) 2013): it is one minus the entropy Fourier decomposition of the time series; (2) Lyapunov exponents (LEs) (Dingwelll
2006 Schölzzl 2019): measure of how sensitive a dynamical system is to initial conditions, Trend: the slope of the linear regression fitted on the time series scaled by its own magnitude: and (4) ACF test (Witt et al., 1998) to test if there is clear seasonality. We report the mean forecastability, mean trend and the percentage of the slices that have seasonality in Table [ averaged over slices with length 20. We can that seasonality is dominant for Electricity dataset, which also has significantly higher forecastability compared to the datasets we experiment with. Moreover, we show that KNFachieves the state-of-the-ar prediction performance on those datasets that have high Lyapunov exponents, low forecastability, no clear trends and seasonality, including Crypto, Player Trajectories, M4-monthly, M4-weekly and M4-daily.
Seasonality:
Electricity Crypto
Player Traj.
M4- monthly
M4-
M4-
M4-
M4-
M4- quarterly
weekly
daily
hourly
yearly
Forecastability
0.77
0.35
0.49
0.44
0.43
0.44
0.46
0.58
0.47
LEs
0.005
0.026
0.052
0.011
0.013
0.020
0.003
0.004
0.003
Trend
0.00
0.02
0.01
0.48
0.13
0.05
0.02
4.32
1.06
Seasonality 100%
0.00%
0.00%
66.34%
0.00%
0.00%
99.76%
0.00%
84.51%
Table The forecastability, Lyapumov exponents, trend and seasonality of the datasets that we used for our experiments, compared with commonly-used Electricity benchmark. The boldface datasets are what we use for experiments in this paper.
4.3 BASELINES
For M4, compare with the winner solution (Smyl 2020), the second best (Montero-Manso et al. 2020) and N-beats (Oreshkin et al. 2019). On the Crypto and Player Trajectory datasets, we compare KNF with four different types of models.

---

Published as a conference paper at ICLR 2023
Vector ARIMA (VARIMA) (Stock & Watson 2001 It is an extension of the classic ARIMA model for multivariate time series.
Multi-layer Perceptron (MLP) that maps the historic observations the future and roll out autoregressively to yield long-term predictions. It is commonly DL model for time series
forecasting (Arik al. 2022 Wang et al. 2021 Faloutsos et 2018
Random Forest (RF) applied autoregressively same as MLP. It is a traditional ML model that
have achieved competitive performance on many time series benchmarks (Godahewa et al. 202T
Masini et al. 2021 Kane et al. 2014 Hyndman & Athanasopoulos 2018).
Long Expressive Memory (LEM) (Rusch et al. long-term sequential dependencies.
2022 An SOTA recurrent model for learning
FedFormer (F eFedFo o (Zhou et al. 2022 is a state-of-art transformer-basee model, which has outperformed other transformme-basedd models, such Informer (Zhou al., 2021) and Autoformer (Wu et al. 2021), on many datasets, including electricity, traffic, weather, etc.
Variational Beam Search (VBS) (i al. 2021 is Bayesian online learning model proposed to detect and adapt to temporal distributional shifts. Since VBS is not designed for time series forecasting, we modify by feeding its own prediction of next step back to the input (instead of the ground truth) to yield multi-step predictions.
4.4 TRAINING STRATEGIES
We find that the following two training strategies can improve the prediction accuracy of models by varying degrees. The first one is the reversible instance normalization ReVIN (Kim et al., [022)). which normalize the input sequence and de-normalize the predictions at every autoregressive step for every instance. We use ReVIN for MIP and KNF since can improve their prediction accuracy. An ablation study of RIV on KNF can be found in Table The second one is, temporal bundling TB (Brandstetter et al. 2022) that asks autoregressive models to generate multiple-step predictions instead of just one on every call, to reduce number of model calls and therefore error propagation speed. We observe this strategy can improve prediction accuracy of both MLP and RF.
4.5
SETUP
sMAPE
Monthly(18)
Weekly(13) Daily(14)
For all datasets, we use sliding win- dow approach to generate training sam- ples. On Crypto and Player Trajectory datasets, we perform grid search of hy- perparameters, including learning rate, input length, hidden dimension, number of predictions made in each autoregres- sive step, etc. for all models. The hyper- parameter tuning ranges can be found in the Appendix [A.3 Table 6 The default
Montero et.al
12,639
7.625
3.097
Smyl
12.126
7,817
3.170
Nbeats-14S KNF'
12.024
11.930
7.254
2.990
Table sMAPE for KNF and baselines on M4 datasets. The numbers in parentheses are the number of predic- tion steps. KNE achieves the state-of-the-art prediction performance at Weekly, Daily and Monthly frequencies.
set of measurement functions used in KNF includes polynomials up the order of four, one exponen- tial function and trigonometric functions with the same number of input steps for each feature, as well as pairwise product interaction functions between features if the time series data is multivariate.
We report the mean and standard deviation averaged across five runs We follow the literature and use RMSE for evaluation on Player Trajectories and the weighted RMSE on Crypto, where the weight corresponds to the importance of each cryptocurrency same as in the competition. Since VBS is deterministic once its inverse temperature parameter is fixed, we do not report its standard deviation. On M4, we evaluate models with the sMAPE metric (Makridakis et al., 2020 Jused in the original competition. Ensembling is used by most models in the M4-competition and N- N-Beats (Oreshkin et al., 2019), so we ensemble five KNF with best hyperparameters but trained with random seeds. Since the evaluation in M4 competition is only based on a single submission, we also report the sMAPE of a single ensembled prediction without standard deviation.
MAPEE the mean absolute error scaled by the magnitude of the predictions and target.
7

---

Published as a conference paper at ICLR 2023
15.
taget BF-TB Endiorer Rocpran
lid
20.
25.0
20.2
TedFormar
Gospman
Predbse Stop
35.0
100 12% 1se me
Figure 2: Left: Example forecasts on Crypto dataset. Right: Example forecasts on NBA player
trajectory dataset. We observe that KNF can be superior capturing complex non-stationary patterns.
4.6 RESULTS ON M4
Table reports the prediction SMAPE of KNF and three baselines, including the M4 winner (Smyl 2020). the second place (Montero- Manso et al. 2020) and the ensembled N-beats-I+G (Oreshkin et al. 2019), on six datasets with different frequency in M4. Oreshkin et al. (2019) does not report breakdown sMAPE of N-beats on weekly, daily and hourly, so we did not include them in the table. The number behind each frequency in the table is the number of prediction steps required in the competition. We can see that KNF achieve the state-of-the-an accuracy on Weekly, Daily and Monthly data that have longer forecasting horizons, and no clear trends and seasonality. This highlights the value proposition of KNF for accurate long-term forecasting accuracy especially for time-series with nonstationary characteristics and low forecastability. KNF also achieve competitive performance on Yearly, Quarterly and Hourly and the full results on M4 are in Appendix A.4
4.7 RESULTS CRYPTO AND PLAYER TRAJECTORIES
Table 3 shows the Prediction RMSE on Crypto and Player Trajectories datasets. KNF consistently achieves the best performance on both, across different forecasting horizons. Fig. 2 exemplifies the predictions of KNFand best rest-performing baselines. We observe that KNF can capture both overall trends and small fluctuations in a superior way, yielding much higher accuracy. Fig. 2 right visualizes the predictions on trajectory with changing direction. KNF correctly predicts the change of moving direction while FedF ormer fails. Regarding baseline performances, RevIN and TB greatly improves MLP, and MLP+RevIN TB the best performing one on Crypto, and FedF rmer performs better on Player Trajectory, that are less chaotic and irregular. Though VBS was shown to perform well on the online change point detection task, does not appear be as successful for time series forecasting based on its performance on these two datasets. We also include additional experimental results on the Traffic and Exchange Datasets (Zeng al. 2023) in Appendix A.5
4.8 ABLATION STUDY ON M4
We performed an ablation study of KNF on the M4-Weekly data to understand the contribution of each component our model. Table shows the sMAPE of ensembled predictions from five funs of each variant. We denote KNF-base as the basic backbone of our model that only has an encoder-decoder architecture and local Koopman operator. KNF-base-G uses purely data-driven measurements,
Crypto (Weighted RMSE 10 3)
Basketball Player Trajectory (RMSE)
Model
(I~5)
(6~10)
(1I~15)
Total
(1~10)
(11~20)
(21~30)
Total
VARIMA
6.09
8.83
10.74.10.00
0.00
0.22
0.90.10.0
1,98.0.00
1.26.1.00
MLP
6.68=1.53 7.95=0.33
8.64
7.85
0.73++020
1.641:031
2.77+042
1.91+0.32
MLP +RP+RIII++T TB
5.03:10.08
7.16=0.13
8.41
7.01
0.37
1.16:0003
2.25:00.4
1.48:025
RF+TB
6.62=1.30 7.99=0.24
8.51
7.84
0.86=+0.01
100000001
3.48+0.02
2.40++001
FedFormer
5.61-0.0 7.50
8.89
7.46110.4
0.43
0.9210003
1.971004
1.29
L.M
5.27-a.m2 7.23=
8.23:10.05
7.0240004
0.33
1.08400.4
2.18:0.02
1.424n.2
VHS
15.233.000
14.46:20.00
26.49::0.0
19.52-0.0
0.90-10:0
2.84:0000
9.24::0.0
5.60
XNE
5.24=0.00
7.03:10.01
7.63:10.01
6.91:10.11
0.26:1001
0.84
1.81
1.1620001
Table 3: Prediction RMSE on Cryptos and Player Trajectories datasets.
8

---

Published as a conference paper at ICLR 2023
M4-Weekly
sMAPE
Target Cooomann feedback, removed
Tirget
X,00
Koopmann, feecback remeved
Koosmon (WR Stedbockk
Koopmaon, un feebbockk
KNE-base-G
14.175
KNF-base-I
9.122
X00
+RevIN 8.435
5000
+RevIN+K's 7.500
400
+RevIN+<<++eedbbad 7.254
590
Table 4: Ablation study of KNF on M4-weekly data. It shows the importance of every component in the model architecture.
Figure 3: Predictions on M4-weekly data from original KNF and with its version without the feedback loop. We observe that the feedback loop can help providing robustness against temporal distributional shifts and maintain the long-term prediction accuracy.
which is similar the Koopman autoencoders proposed in (Yeung et al. 2019 Azencot et al. 2020). This can be considered a baseline for our model. KNF KNF-base- uses predefined measurement functions, which significantly outperforms KNF-base-G. This demonstrates the benefits of leveraging hard-coded functions. Moreover, adding any of ReVIN, the global Koopman operator and the feedback loop also brings great improvement based on the results shown in Table
To further demonstrate the effectiveness of the feedback loop, we visualize the predictions on M4- weekly data from KNF and KNF with the feedback module removed after training. We can observe that the predictions from the model with the feedhack loop removed start to deviate from the ground truth after few steps. That means the feedhack loop can cope with the temporal distributional shifts and thus improve the long-horien hong-horizon forecasting accuracy. We perform additional ablation studies measurement functions on M4-yearly data shown Appendix A.l demonstrating the necessity of each type of measurement functions.
0.5
4.9
INTERPRETABILITE RESULTS
0.0
Trrge
0.5
KNF offer unique interpretability capabilities via spectral analysis of the Koopman operators. We per- form eigen-decompositton on K KC} for KNF trained
0.0
0.2
-1
Bipon Function
on M4-weekly data to investigate what individual eigen- functions have learnt. Fig. 4 shows the predictions in the lookback window with different eigenfunctions. From the top the bottom, the plot shows the target, the reconstruction from KNF with only the first eigen- function in the lookback window and the reconstruction only using the second eigenfunction. We observe that the some eigenfunctions captures the trend while some other functions focus on the seasonality.
0.00
Figure 4: Reconstructions from KNF the lookback window with only one eigen- functions of K AC; M4 time series.
To show that our model can also correctly learn the time series governed by known equations. We run experiments with our model on a simple oscillator system, given in Appendix A.2 We try our models with different sets of polynomial functions and train/test them on different initial conditions. Fig. 5 shows that our model can make accurate predictions on the test set with different dictionaries.
5 CONCLUSION
In this paper, we propose a novel model based on the Koopman theory, Koopman Neural Forecaster (KNF), designed for accurate long-term forecasting for non-stationary time series with temporal distribution shifts. KNF leverages predefined measurement functions to capture the nonlinear and nonstationary characteristics that may pose a great difficulty for neural nets learn. employs both a global operator to learn shared characteristics, and local operator to capture changing dynamics. We also use a judiciously-desieee for rapidly varying behaviors. We demonstrate that KNF achieves the fate-of-theear performance on wide range of time series datasets that are particularly known to suffer from distribution shifts. M4, Cryptos and NBA player trajectory datasets and
feedback loop to continuously update the learnt operators over time
KNF achieves SOTA prediction performance provides interpretable results. Future work includes forecasting higher- dimensional dynamics and investigating distributional shifts between training and test domains.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
We'd like to thank Tomas Pfister, Nate Yoder, and Jinsung Yoon for insightful discussions on this work. This work was supported in part by U.S. Department Of Energy, Office of Science, U. Army Research Office under Grant W91 INF-20-1-0334 and NSF Grants #2134274, #2107256 and #2134178.
REFERENCES
Sercan O Arik, Nathanael C Yoder, and Tomas Pfister. Self-adaptive forecasting for improved deep learning non-stationary time-series. arXiv preprint arXiv:2202.02403. 2022.
Adebiyi Ariyo, Adewumi O Adewumi, and Charles Ayo. Stock price prediction using the arima model. In 2014 UKSim-AMSS 16th international conference computer modelling and simulation, pp. 106-112. IEEE, 2014.
Omri Azencot, N Benjamin Erichson, Vanessa Lin, and Michael Mahoney. Forecasting sequential data using consistent koopman autoencoders. International Conference on Machine Learning, pp. 475-485. PMLR, 2020.
Stefanos Bennett and Jase Clarkson. Time series prediction under distribution shift using differentiable forgetting. arXiv preprint arXiv:2207.11488 2022.
Johannes Brandstetter, Daniel E. Worrall, and Max Welling. Message passing neural PDE solvers. In International Conference on Learning Representations, 2022. URL /tt/itts///peeeeeiieee
net /uat//oon Btt/E/ouunnivviii111111111
Bingni W Brunton, Lise A Johnson, Jeffrey G Ojemann, and J Nathan Kutz. Extracting spatial- temporal coherent patterns in large-scale neural recordings using dynamic mode decomposition. Journal of neuroscience methods, 258:1-15, 2016.
Steven L Brunton, Marko Budišić, Eurika Kaiser, and J Nathan Kutz. Modern koopman theory for dynamical systems. arXiv preprint arXiv:2102. 2021.
Rainer Dahlhaus. Fitting time series models to nonstationary processes. The annals of Statistics, 25 (1):1-37, 1997.
Jonathan B Dingwell. Lyapunov exponents. Wiley encyclopedia of biomedical engineering, 2006.
Christos Faloutsos, Jan Gasthaus, Tim Januschowski, and Yuyang Wang. Forecasting big time series: old and new. Proceedings of the VLDB Endowment, 11 11(12):102--2105, 2018.
Fletcher Fan, Bowen Yi, David Rye, Guodong Shi, and lan R Manchester. Learning stable koopman embeddings. 2022 American Control Conference (ACC), pp. 2742-2747. IEEE, 2022.
João Gama, Indré Žliobaité, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia. A survey on concept drift adaptation. ACM computing surveys (CSUR), 46(4):1-37, 2014.
Everette Gardner Jr. Exponential smoothing: The state of the art. Journal of forecasinng. 4(1):1-28, 1985.
Rakshitha Godahewa, Christoph Bergmeir, Geoffrey I. Webb, Rob J. Hyndman, and Pablo Montero- Manso. Monash time series forecasting archive. Neural Information Processing Systems Track Datasets and Benchmarks, 2021.
Georg Goerg. Forecastable component analysis. In International conference on machine learning, pp. 64-72. PMLR, 2013.
Yiqiang Han, Wenjian Hao, and Umesh Vaidya. Deep learning of koopman representation for control. In 2020 59th IEEE Conference on Decision and Control (CDC), pp. 1890-1895. IEEE, 2020.
Michael Harries and New South Wales. Splice-2 comparative evaluation: Electricity pricing. 1999.
10

---

Published as a conference paper at ICLR 2023
Rob J Hyndman and George Athanasopoulos. Forecasting: principles and practice. OTexts, 2018.
Xiaoyong Jin, Youngsuk Park, Danielle C Maddix, Yuyang Wang. and Xifeng Yan. Domain adaptation for time series forecasting via attention sharing. International Conference on Machine Learning, 2021.
Michael J Kane, Natalie Price, Matthew Scotch, and Peter Rabinowitz. Comparison of arima and random forest time series models for prediction of avian influenza h5nl outbreaks. BMC bioinformatics, 15(1):1-9, 2014.
Taesung K Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. Reversible instance normalization for accurate time-series forecasting against distribution shift. In Interna- tional Conference on Learning Representations, 2022. URL
neet
Bernard O Koopman. Hamiltonian systems and transformation in hilbert space. Proceedings of the National Academy of Sciences, 715)1315-318, 1931.
Wouter M Kouw and Marco Loog. An introduction to domain adaptation and transfer learning. arXiv preprint arXiv:88/2 /806, 2018.
J Nathan Kutz, Joshua L Proctor, and Steven L Brunton. Koopman theory for partial differential equations. arXiv preprint arXiv:1607.07076. 2016.
Vitaly Kuznetsov and Mehryar Mohri. Discrepancy-basse theory and algorithms for forecasting non-stationary time series. Annals of Mathematics and Arlificial Intelligence, 88(4):367-399, 2020.
Aodong Li, Alex Boyd, Padhraic Smyth, and Stephan Mandt. Detecting and adapting to irregular distribution shifts in bayesian online learning. Advances in Neural Information Processing Systems, 34:6816 16-6888, 2021.
Mengnan Li and Lijian Jiang. Deep learning nonlinear multiscale dynamic problems using koopman operator. Journal of Computational Physics, 446:1 10660, 2021.
Yunzhu Li, Hao Hc, Jiajun Wu, Dina Katabi, and Antonio Torralba. Learning compositional koopman operators for model-based control. 2020.
Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Rethinking the stationarity in time series forecasting, arXiv preprint arXiv:2205.14415, 2022.
Jie Lu, Anjin Liu, Fan Dong, Feng Gu, Joao Gama, and Guangquan Zhang. Learning under concept drift: review. IEEE Transactions on Knowledge and Data Engineering, 31(12):3366-233, 2018.
Bethany Lusch, Nathan Kutz, and Steven L Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature communications, 9()):1110, 2018.
Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The m4 competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting, 36(1):54-744 2020.
Andrey Malinin, Neil Band, Alexander Ganshin, German Chesnokov, Yarin Gal, Mark J. F. Gales, Alexey Noskov, Andrey Ploskonosov, Liudmila Prokhorenkova, Ivan Provilkov, Vatsal Raina, Vyas Raina, Denis Roginskiy, Mariya Shmatova, Panos Tigar, and Boris Yangel. Shifts: A dataset of real distributional shift across multiple large-scale tasks. arXiv preprint arXiv:2107.07455 2021.
Ricardo P Masini, Marcelo Medeiros, and Eduardo Mendes. Machine learning advances for time series forecasting. Journal of Economic Surveys, 2021.
Daniel McCarthy and Shane T Jensen. Power-weighted densities for time series data. The Annals of Applied Statistics, 10(1):305-334 2016.
Pablo Montero-Mansee George Athanasopoulos, Rob Hyndman, and Thiyanga S Talagala. Fforma: Feature-based forecast model averaging. International Journal of Forecasting, 66:86-92, 2020.
11

---

Published as a conference paper at ICLR 2023
Jeremy Morton, Freddie D Witherden, and Mykel J Kochenderfer. Deep variational koopman models: inferring koopman observations for uncertainty-awant dynamics modeling and control. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pp. 3173-3179, 2019.
Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint 1905. 10437, 2019.
Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. Meta-learning framework with applications zero-shot time-series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 9242-9250, 2021.
Nikolaos Passalis, Anastasios Tefas, Juho Kanniainen, Moncef Gabbouj, and Alexandros Iosifidis. Deep adaptive input normalization for time series forecasting. IEEE transactions on neural networks and learning systems, 3160):376033765 2019.
T. Konstantin Rusch, Siddhartha Mishra, N. Benjamin Erichson, and Michael W. Mahoney, Long ex- pressive memory for sequence modeling. In International Conference on Learning Representations, 2022. URL htRpstt::: http//ppeeeveeeeeeeeeeeeeeeeeeee net/ forummid=vver (er
Christopher Schölzel. Nonlinear measures for dynamical systems, June 2019. URL https:
//doi. wtt.:oo.o....111///... 20/81/eenod..o 3814723
Rajat Sen, Hsiang-Fu Yu, and Inderjit Dhillon. Think globally, act locally: deep neural network
approach to nigh-ddmensional time series forecasting. Advances in neural information processing systems, 32, 2019.
Slawek Smyl. A hybrid method of exponential smoothing and recurrent neural networks for time series forecasting. International Journal of Forecasting. 36(1):75-88) 85, 2020.
James H Stock and Mark W Watson. Vector autoregressions. Journal of Economic perspectives, 15 (4):101-115, 2001.
Steven H. Strogatz. Nonlinear dynamics and chaos: with applications to physics, biology, chemistry, and engineering. CRC press, 2018.
Naoya Takeishi, Yoshinobu Kawahara, and Takehisa Yairi. Learning koopman invariant subspaces for dynamic mode decomposition. Advances in Neural Information Processing Systems, 30, 2017.
Sana Tonekaboni, Chun-Liang Li, Sercan O Arik, Anna Goldenberg, and Tomas Pfister. Decoupling local and global representations of time series. International Conference on Artificial Intelligence and Statistics, pp. 8700-8714 PMLR, 2022.
JH Tu, CW Rowley, DM Luchtenburg, SL Brunton, and JN Kutz. On dynamic mode decomposition: theory and applications. arXiv preprint 1rXiv:1312.004 2013.
Pantelis Vlachas, Jaideep Pathak, Brian R Hunt, Themistoklis Sapsis, Michelle Girvan, Edward Ott, and Petros Koumoutsakos. Backpropagation algorithms and reservoir computing in recurrent neural networks for the forecasting of complex spatiotemporal dynamics. Neural Networks, 126:
191-217, 2020.
Rui Wang, Danielle Maddix, Christos Faloutsos, Yuyang Wang, and Rose Yu. Bridging physics-based and data-driven modeling for learning dynamical systems. In Learning for Dynamics and Control, pp. 385-398 PMLR, 2021.
Yuyang Wang, Alex Smola, Danielle Maddix, Jan Gasthaus, Dean Foster, and Tim Januschowski. Deep factors for forecasting. In International conference on machine learning, pp. 6607-6617. PMLR, 2019.
Mike West and Jeff Harrison. Bayesian forecasting and dynamic models. Springer Science & Business Media, 2006.
12

---

Published as a conference paper at ICLR 2023
Annette Witt, Jürgen Kurths, and A Pikovsky. Testing stationarity in time series. physical Review E, 58(2):1800, 1998.
Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. CoST: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. International Conference on Learning Representations, 2022.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 34:22419-22433 2021.
Enoch Yeung, Soumya Kundu, and Nathan Hodas. Learning deep neural network representations for koopman operators of nonlinear dynamical systems. In 2019 American Control Conference (ACC), pp. 4832 4839. IEEE, 2019.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? 2023.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 11106-11115, 2021.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Fre-
quency enhanced decomposed transformer for long-term series forecasting. arXiv preprint 120.:2201.12440, 2022.
13

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 ABLATION STUDIES ON PREDEFINED MEASUREMENT FUNCTIONS
Table shows the results from the ablation study of measurement function M4- YeaYeyrl data. G represents generic, purely data-driven. P is polynomial function, E is the exponential function, is the sine function. The integers in the first row are the number of each type of function. Basically, we gradually add more and different types of predefined measurement functions to purely data-drive model and observe improvements brought by the polynomial, exponential function, sine functions.
KNF G
P1
P2
P3
P2+E1
P2+E2
P2+E3
P2+E14SI
P2+E1+S2
P2+E1+S3
P2+E1+S4
SMAPE
14.66 14.65
14.56
14.79
14.48
14.63
14.66
14.35
14.16
14.01
14.15
Table 5: Ablation study measurement function on M4- M4-Yearly data. G represents generic, purely data-driven. P is polynomial function, E is the exponential function, S is the sine function. The integers the first row are the number of each type of function.
A. SIMPLE NONLINEAR SYSTEM WITH FINITE KOOPMAN SPACE
We consider the following simple nonlinear system with discrete spectrum.
2
\ - a
This system has minimal Koopman invariant subspace spanned by {x1, 32,, xf); can also be spanned by three eigenfunctions
d
F4
0
0
x2
=
0
-
dt
0
2p.
(on,
Ø2
Øal
[2
32
x-2 z
+
0
+
d
244
Φ
0
@
Any multiplication of (oh, 02, Φ3]
still an eigenfunction.
We want to show our model with only trainable global operator can correctly identify the Koopman invariant subspace from the synthetic data generated based on this system. We generate 36 time series with μ -0.1 and X = 1 and the initial values are uniformly sampled from [-1, 1 We tried four different basis dictionaries, including D1 = [Ii 22, xf). D2 = {x1, Ø2, 21. x }. D3 {I1, D2, x 22 21, 2,), D₁ {I1, 22, of 22, 21 22 Ti 22) and we trained and tested the models on different initial conditions. The left figure in Fig. [5) shows eigenvalues learnt by KNF and some true eigenvalues. Since the learned Koopman matrix may not be unique given the flexibility in the coefficients of measurement functions, and the compositions also eigenfunctions, there many possible eigenvalues. But can still see that most of the learned cigenvalues match the true eigenvalues. The right three figures in Fig. 5 shows that KNF can make accurate predictions with different dictionaries.
eigenfunctions are
A.3 ADDITIONAL EXPERIMENTAL DETAILS
Table/osshow the hyperparameter tuning ranges, including the learning rate, the hidden dimension and number of layers of deep neural network modules in both baselines and our model, number of predictions made at each autoregressive step, the length of input observations, the forecasting window size during training, and whether to use ReVIN. For different modules a model, we tune hyperparameters, such the number of layers/hidden dimensions separately. For instance, in FedF o omer. x, the encoder and decoder may have different numbers of layers. As for the nonlinearity, use ReL.U for all layers.
14

---

Published as a conference paper at ICLR 2023
-300
-2.3
-30
-3.6
-1.
Teath
3.00
2.0
332
11.01
Figure 5: Left: Learnt eigenfunctions by KNF with different dictionaries. Right: Predictions from KNF with on the simple nonlinear system.
Learning rate
Hidden dim
#Layers
#forecasting window size during training
#Predictions made each autoregressive step
Input length
Whether use ReVIN
1e-1~1e-5
64~1024
3~7
1~11
1~-10
5~-50
True/False
Table 6: Hyperparamter Tuning Ranges.
A.4 ADDITIONAL RESULTS
sMAPE
Monthly(18)
Weekly(13)
Daily(14)
Hourly(48)
Yearly(6)
Quarterly(8)
Montero et et.al
12,639
7.625
3,097
11.506
13.528
9.733
Smy1
12.126
7.817
3.170
9.328
13.176
9.679
Nbeata- KNF
12.024
12.924
9.212
11.930
7.254
2.990
11.294
13.800
10.008
Table 7: sMAPE for KNF and baselines six M4 datasets for different sampling frequencies. The numbers in parentheses are the number of prediction steps. KNF achieves the state-of-the-art prediction performance Weekly, Daily and Monthly frequencies.
A.5 EXPERIMENTS THE TRAFFIC AND EXCHANGE BENCHMARK DATASETS
We also experiment with KNF on the the Traffic and Exchange Benchmark Datasets. Our results in Table 8 demonstrate that KNF achieves MAE scores comparable to the state-of-the-art model (Zeng al.) 2023). Additionally, KNF outperforms the SOTA model in terms of MSE on the traffic dataset, indicating that KNF generates relatively stable predictions across the forecasting horizon.
Models Metrics
NLinear DLinear
KNF
MAE
MSE
MAE
MSE
96
0.279
0.410
0.254
0.158
192
0,284
0.423
0.276
0.176
Traffic
336
0.290
0.435
0.301
0.197
720
0.307
0.464
0.344
0.247
96
0.203
0.081
0.234
0.095
0.293
0.157
0.331
0.175
Exchange
336
0.414
0.305
0.456
0.327
720
0.601
0.643
0.589
0.517
Table 8: Prediction MAE and MSE the Traffic and Exchange Benchmark Datasets. KNF achieve MAE scores comparable to the SOTA scores and significantly better MSEs on the traffic dataset.
15


Reviews:
[{"review_id": "mD0VoitCQTi", "reviewer": "Reviewer_CZDy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is about a new method called Koopman Neural Forecaster (KNF) for forecasting time series that have distributional shifts. It combines three operators: a global one, a local one, and one that can update over time via feedback.", "summary_of_the_review": "I think that this paper is working on an important problem with broad applications. The approach seems interesting, and the results are impressive. I'll be interested in trying this method out once the paper is published. I have some comments and questions, but they are not major. \n", "strength_and_weaknesses": "To my knowledge, this a novel way to use Koopman theory for time series predictions. I think that the comparisons against baselines are impressive, especially when comparing to the recent M4 competition. Table 1, demonstrating that the datasets are difficult to predict, is helpful, and I thought the ablation studies were well-done. \n\nI have some comments in the next box, but none of them are major or difficult to fix. For sure, the claims along the lines of  \"To the best of our knowledge, this is the first time that Koopman theory is applied to real-world chaotic time series without known governing laws\" need to be fixed. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "iPC5XfDhJgr", "reviewer": "Reviewer_ab4B", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new deep learning model for long-term time series forecasting with distribution shifts based on Koopman operator theory. In comparison to existing work, the authors utilize measurement functions (e.g. sine functions) in latent space, and they formulate forecasting using global and temporally local Koopman operators, as well as an additional adjustment operator based on a feedback loop. The authors evaluate their approach on three challenging datasets in comparison to several baseline methods, showing several promising results.\n", "summary_of_the_review": "In summary, this paper suggests a new and interesting Koopman-based approach for long-term of time series data with distribution shifts. The benchmark results position this method in line with state-of-the-art forecasting (statistical and learning) approaches, which will definitely inspire others to build on the proposed architecture to further improve forecasting capabilities. The exposition shortcomings could be handled during the revision period, and thus should not prevent publication.", "strength_and_weaknesses": "Probably the strongest point of the paper is the results obtained on several challenging datasets in comparison to several strong baselines. To the best of my knowledge, the particular datasets considered in this work were not previously considered in Koopman-based approaches. Moreover, surpassing strong baselines such as N-BEATS and Smyl on M4 benchmark is particularly encouraging and highlights the capabilities of the proposed approach.\n\nThere are also a few weaknesses. The related work section and the overall discussion regarding previous work on Koopman methods can be improved significantly. For instance, the comment in the conclusion (and abstract): ``... this is the first time that Koopman theory is applied to real-world time series without known governing laws.'' is inappropriate and should be re-phrased. Two examples (that even appeared in ML venues) and deal with sequential data for which governing laws are unknown is \"DeSKO: Stability-Assured Robust Control with a Deep Stochastic Koopman Operator\" by Han et al. that showed multiple examples of real-world control tasks. The second paper \"Learning Compositional Koopman Operators for Model-Based Control\" by Li et al. also considered a few real-world control problems. Actually, there is another relevant paper that comes to mind in this context which deals with Koopman-based prediction of ECG signals (\"12-lead ECG Reconstruction via Koopman Operators\" by Golany et al.). Also, even in papers such as Azencot et al. where the authors claim the tasks were \"synthetic time series with known governing equations\", examples involving forecasting of sea surface temperature were investigated. While these phenomena generally follow the Navier--Stokes equations, real-world data such as sea surface temperature can be modeled with NS only on its dominant factors, whereas fast and slow dynamical processes are not modeled well. Finally, please change the DMD citation of Brunton et al. 2016 to the correct one (\"Dynamic mode decomposition of numerical and experimental data\" by Schmid). You may also want to include several Koopman-based papers discussing Koopman with control, given that the paper deals with a basic version of control via a feedback loop.\n\nAnother weakness is related to the methodology section. Specifically, the formulation around Eq. (4) could be improved significantly. For instance, there is a $v_i \\in \\mathbb{R}^k$ which seems like a leftover from a previous formulation. Also, it is not clear to me what you mean by $V_t = \\Psi(X_t) X_t$ (left of Eq. (4)). Is it matrix multiplication? is it batch multiplication? The dimensions as they appear in the document do not make sense. It would be good if you can add a discussion describing the differences between training and inference with respect to how the global, local and adjustment operators are being computed and updated. Why do you use an adjustment as a multiplication operator and not add the residual in Eq. (9) to the forecast in E. (8)? The equations (11)-(13) seem to not really support the ansatz in Eq. (7). Specifically, while you compute local and adjustment operators per time step, Eq. (12)-(13) use powers of the operators at the *same time*. Shouldn't it be a product of different operators? Please discuss this point. How do you choose the set of measurement functions? Is the method sensitive to the choice of functions?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Im6pbDCHzVl", "reviewer": "Reviewer_BYPh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduce a new architecture for forecasting non-stationary time series, the ‘Koopman Neural Forecaster (KNF)’. KNF combines local and global Koopman matrix operators assembled from a set of basis functions, and also includes a feedback operator based on local prediction errors. Parameters of the Koopman operators are learned through a Transformer encoder-decoder network. The KNO is compared to several SOTA baseline models on challenging, non-stationary real-world time series from the M4 dataset as well as Crypto and basket ball player trajectories.", "summary_of_the_review": "Potentially nice contribution with superior prediction for non-stationary time series (an important problem), and a model that can in part be interpreted. In my mind somewhat weak anchoring in theory, although not essential here I think. Some more baselines should be considered and presentation of results should be improved.", "strength_and_weaknesses": "Strengths:\n- superior prediction of non-stationary time series\n- interpretability of model\n\nWeaknesses:\n- only loose anchoring in theory (see below)\n- some parts unclear (see below)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DVPcvaj9Ur", "reviewer": "Reviewer_63Af", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a deep sequence model, KNF, via the Koopman theory for time series forecasting. KNF employs a global operator to learn shared characteristics, and a local operator to capture changing dynamics, which is claimed to impose appropriate inductive biases for improved robustness against distributional shifts. The experiments conducted on several datasets confirm the effectiveness of the proposed model.\n\nThe presentation of this paper is clear, leading to an easy understanding. I have two doubts about the claimed contributions\n1. The KNF is proposed for handling time series with temporal distribution shifts. Hence, it is natural to verify the performance of KNF using real-world datasets with distribution shifts. As introduced in Subsection 4.2, the conducted Electricity dataset seems to exhibit only seasonality. Does distribution shift refer to periodicity? Thus, I have a doubt that how to identify the time series with temporal distribution shifts, or the killer applications of KNF in practice. Otherwise, it is better to provide certain theory to support this claim.\n\n2. It is also a need to point out the key component of the proposed KNF model that enables translation invariance. Besides, it is better to provide a comparison or discussion with previous \"global-local\" models, such as [Sen, R., Yu, H. F., & Dhillon, I. S. (2019). Think globally, act locally: A deep neural network approach to high-dimensional time series forecasting. Advances in neural information processing systems, 32.]\n\nMinor issues:\n1. What do the bold notes in Table 1 mean?\n2. When using KNF to forecast time series, does the user have to guarantee that the size of the forecast window is the same as the size of the lookback window? Is there any trick to setting this window size? What is the impact if the window size is much smaller than the period?", "summary_of_the_review": "Overall, I tend to accept this paper if the authors fixed my doubts in the next phase.\n\n---- after rebuttal ----\n\nI have read the authors' rebuttal, and I believe the responses have fixed my doubts. Therefore, I tend to accept this paper.\n", "strength_and_weaknesses": "mentioned above.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster