Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper is about a new method called Koopman Neural Forecaster (KNF) for forecasting time series that have distributional shifts. It combines three operators: a global one, a local one, and one that can update over time via feedback.
strength_and_weaknesses: To my knowledge, this a novel way to use Koopman theory for time series predictions. I think that the comparisons against baselines are impressive, especially when comparing to the recent M4 competition. Table 1, demonstrating that the datasets are difficult to predict, is helpful, and I thought the ablation studies were well-done. 

I have some comments in the next box, but none of them are major or difficult to fix. For sure, the claims along the lines of  "To the best of our knowledge, this is the first time that Koopman theory is applied to real-world chaotic time series without known governing laws" need to be fixed. 

clarity,_quality,_novelty_and_reproducibility: Clarity:

I'm confused about the encoder. Is the output really supposed to be 3D? Is v_i really k-dimensional?  

"But computing spectral decomposition for every sample is computationally expensive, so we utilize DNNs to learn the Koopman operator... While the Koopman matrix should vary across samples and time in our case..." I found these uses of "sample" confusing, but eventually figured out that you're using "sample" to mean one trajectory. Maybe that could be stated explicitly. (You could apply DMD to multiple trajectories by stacking them up if you expect one matrix A to be valid across all of the trajectories. However, this is certainly an approximation and, especially in the context of studying temporal distribution shift, it makes sense that just applying DMD once wouldn't work.)


Reproducibility: 

The networks seem vaguely described. For example, it's mentioned in the appendix that the hyperparameter tuning range for number of layers is 3-7. Do the encoder, decoder, transformer encoder, and the feedback module lamda all have the same number of layers? We also don't know what the nonlinearity is, etc. However, the authors say they will release the code, so that would help. 


Novelty: 

The abstract says, "To the best of our knowledge, this is the first time that Koopman theory is applied to real-world chaotic time series without known governing laws," and the conclusion contains a similar sentence. However, this is not the first. For example, see [A].  They use three real-world datasets, including measles outbreak data, and specifically mention that it's been shown that measles outbreaks are chaotic. On the other hand, it is not shown that the datasets in this paper are chaotic. 

The main distinction given from previous Koopman papers is "Most of these works use rather simple DNN architectures, and are applied to synthetic time series with known governing equations. Different from these, we focus on the real-world time series with no governing laws, such from finance." Since none of these papers are used as baselines, and previous Koopman papers have certainly been applied to real datasets, I would suggest emphasizing the approach of letting the Koopman matrix evolve over time in a three-piece way (global, local, and feedback loop). As far as I know, this is novel. As a **bonus**, it would be interesting to see to what extent the Koopman matrix evolves. For example, how much influence does the global piece have, or does it get dominated by the others? If there is a clear distribution shift halfway through the data, is that easy to see in the pieces of the Koopman operator?  


Quality/Correctness:

"Fig. 6 shows that our model can always make perfect predictions on the test set..." It cannot be true that a deep learning model can "always make perfect predictions" of this system. There has to at least be rounding error. :) What is the error? 

"We define a set of measurement functions G := [g1, · · · , gn] that spans the Koopman space..." The Koopman matrix is nd x nd. Is there a way to know that the few measurement functions you picked will span the space? 

In A.1.1:
- "It can also be spanned by three eigenfunctions..." There are two listed. Is there a comma missing? 
- "our model can always learn the correct eigenfunctions." What do you mean by "correct"? Did you check that the eigenfunctions are the expected ones, or just that the prediction is accurate? I could imagine that the encoder transforms the data and finds some other valid combination that results in low error. 

"Table 2 reports the prediction sMAPE... on six datasets with different frequency in M4." There are only 3 datasets in this table. I see from later in the paragraph that the rest were moved to the appendix. 

"We run experiments with our model on a simple oscillator system..." This isn't an oscillator system. The eigenvalues are real and the system decays down to an attractor. 


[A]  Brunton, S.L., Brunton, B.W., Proctor, J.L. et al. Chaos as an intermittently forced linear system. Nat Commun 8, 19 (2017). https://doi.org/10.1038/s41467-017-00030-8.

summary_of_the_review: I think that this paper is working on an important problem with broad applications. The approach seems interesting, and the results are impressive. I'll be interested in trying this method out once the paper is published. I have some comments and questions, but they are not major.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a deep sequence model, KNF, via the Koopman theory for time series forecasting. KNF employs a global operator to learn shared characteristics, and a local operator to capture changing dynamics, which is claimed to impose appropriate inductive biases for improved robustness against distributional shifts. The experiments conducted on several datasets confirm the effectiveness of the proposed model.

The presentation of this paper is clear, leading to an easy understanding. I have two doubts about the claimed contributions
1. The KNF is proposed for handling time series with temporal distribution shifts. Hence, it is natural to verify the performance of KNF using real-world datasets with distribution shifts. As introduced in Subsection 4.2, the conducted Electricity dataset seems to exhibit only seasonality. Does distribution shift refer to periodicity? Thus, I have a doubt that how to identify the time series with temporal distribution shifts, or the killer applications of KNF in practice. Otherwise, it is better to provide certain theory to support this claim.

2. It is also a need to point out the key component of the proposed KNF model that enables translation invariance. Besides, it is better to provide a comparison or discussion with previous "global-local" models, such as [Sen, R., Yu, H. F., & Dhillon, I. S. (2019). Think globally, act locally: A deep neural network approach to high-dimensional time series forecasting. Advances in neural information processing systems, 32.]

Minor issues:
1. What do the bold notes in Table 1 mean?
2. When using KNF to forecast time series, does the user have to guarantee that the size of the forecast window is the same as the size of the lookback window? Is there any trick to setting this window size? What is the impact if the window size is much smaller than the period?
strength_and_weaknesses: mentioned above.
clarity,_quality,_novelty_and_reproducibility: The presentation of this paper is clear, leading to an easy understanding. There are only two doubts mentioned above.
summary_of_the_review: Overall, I tend to accept this paper if the authors fixed my doubts in the next phase.

---- after rebuttal ----

I have read the authors' rebuttal, and I believe the responses have fixed my doubts. Therefore, I tend to accept this paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new deep learning model for long-term time series forecasting with distribution shifts based on Koopman operator theory. In comparison to existing work, the authors utilize measurement functions (e.g. sine functions) in latent space, and they formulate forecasting using global and temporally local Koopman operators, as well as an additional adjustment operator based on a feedback loop. The authors evaluate their approach on three challenging datasets in comparison to several baseline methods, showing several promising results.

strength_and_weaknesses: Probably the strongest point of the paper is the results obtained on several challenging datasets in comparison to several strong baselines. To the best of my knowledge, the particular datasets considered in this work were not previously considered in Koopman-based approaches. Moreover, surpassing strong baselines such as N-BEATS and Smyl on M4 benchmark is particularly encouraging and highlights the capabilities of the proposed approach.

There are also a few weaknesses. The related work section and the overall discussion regarding previous work on Koopman methods can be improved significantly. For instance, the comment in the conclusion (and abstract): ``... this is the first time that Koopman theory is applied to real-world time series without known governing laws.'' is inappropriate and should be re-phrased. Two examples (that even appeared in ML venues) and deal with sequential data for which governing laws are unknown is "DeSKO: Stability-Assured Robust Control with a Deep Stochastic Koopman Operator" by Han et al. that showed multiple examples of real-world control tasks. The second paper "Learning Compositional Koopman Operators for Model-Based Control" by Li et al. also considered a few real-world control problems. Actually, there is another relevant paper that comes to mind in this context which deals with Koopman-based prediction of ECG signals ("12-lead ECG Reconstruction via Koopman Operators" by Golany et al.). Also, even in papers such as Azencot et al. where the authors claim the tasks were "synthetic time series with known governing equations", examples involving forecasting of sea surface temperature were investigated. While these phenomena generally follow the Navier--Stokes equations, real-world data such as sea surface temperature can be modeled with NS only on its dominant factors, whereas fast and slow dynamical processes are not modeled well. Finally, please change the DMD citation of Brunton et al. 2016 to the correct one ("Dynamic mode decomposition of numerical and experimental data" by Schmid). You may also want to include several Koopman-based papers discussing Koopman with control, given that the paper deals with a basic version of control via a feedback loop.

Another weakness is related to the methodology section. Specifically, the formulation around Eq. (4) could be improved significantly. For instance, there is a $v_i \in \mathbb{R}^k$ which seems like a leftover from a previous formulation. Also, it is not clear to me what you mean by $V_t = \Psi(X_t) X_t$ (left of Eq. (4)). Is it matrix multiplication? is it batch multiplication? The dimensions as they appear in the document do not make sense. It would be good if you can add a discussion describing the differences between training and inference with respect to how the global, local and adjustment operators are being computed and updated. Why do you use an adjustment as a multiplication operator and not add the residual in Eq. (9) to the forecast in E. (8)? The equations (11)-(13) seem to not really support the ansatz in Eq. (7). Specifically, while you compute local and adjustment operators per time step, Eq. (12)-(13) use powers of the operators at the *same time*. Shouldn't it be a product of different operators? Please discuss this point. How do you choose the set of measurement functions? Is the method sensitive to the choice of functions?
clarity,_quality,_novelty_and_reproducibility: As mentioned above, the method section can be improved. This is also important in terms of reproducibility, which at the current version would be difficult to achieve given that many hyper-parameters are not listed. It would also help if a proper network architecture illustration or table is given, listing all components, network layers, activations, etc. (beyond Fig. 1).

The paper is of high quality in terms of methodology and evaluation setup. To the best of my knowledge the proposed method differs from existing Koopman-based approaches in the measurement functions applied in the latent space (although similar ideas exist in the extended DMD paper and derived work and the decomposition of the Koopman operator to three different objects. In my opinion, these changes are novel and warrant a publication at ICLR. The paper could be stronger if the decomposition could be associated to Koopman theory, however, I do not have a good idea on how to approach this.
summary_of_the_review: In summary, this paper suggests a new and interesting Koopman-based approach for long-term of time series data with distribution shifts. The benchmark results position this method in line with state-of-the-art forecasting (statistical and learning) approaches, which will definitely inspire others to build on the proposed architecture to further improve forecasting capabilities. The exposition shortcomings could be handled during the revision period, and thus should not prevent publication.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors introduce a new architecture for forecasting non-stationary time series, the ‘Koopman Neural Forecaster (KNF)’. KNF combines local and global Koopman matrix operators assembled from a set of basis functions, and also includes a feedback operator based on local prediction errors. Parameters of the Koopman operators are learned through a Transformer encoder-decoder network. The KNO is compared to several SOTA baseline models on challenging, non-stationary real-world time series from the M4 dataset as well as Crypto and basket ball player trajectories.
strength_and_weaknesses: Strengths:
- superior prediction of non-stationary time series
- interpretability of model

Weaknesses:
- only loose anchoring in theory (see below)
- some parts unclear (see below)

clarity,_quality,_novelty_and_reproducibility: In general I found this a potentially valuable contribution, depending on whether my understanding of some of the details is correct. I also liked the interpretability aspect of the KNF.

1) The presentation of the model, incl. Fig. 1, I didn’t find very clear, and so I’m not sure I understood everything correctly. For instance, in eq. 1 the problem is stated as one of learning a dynamical system, but right below in eq. 2 it’s then formulated as essentially a time series (auto-) regression problem, which is a quite different objective. My understanding is the authors compile a library of basis functions (similar as in Brunton et al. 2016 pnas), the parameters of which and their weights are learned through the Transformer encoder-decoder? If correct, this should be made more explicit and formal (I didn’t really find this information, incl. the specific type of parameterization and the complete library used, in the text). I also didn’t quite get how (and why?) the time series were transformed into a latent code (just an affine transformation?), and how these parameters were learned. So sect. 3.2 I think needs to be made *much* clearer with all mathematical details on the transformations used and the specific parameterizations of learned functions, what exactly the output of the encoder is etc. etc. Fig. 1 should also be improved.
Much more details about hyper-parameter tuning for all models are also necessary I think. Also, I didn’t find any statement on code availability (did I miss it)?

2) In general I found the anchoring in Koopman operator and dynamical systems theory only very loose at best, and the model is not really geared toward extracting governing equations from the data, unlike much of the recent efforts in ML for dynamical systems (e.g. [1-3]). And I don’t think it needs to be, it’s fine, in my mind, if it just takes inspiration from these fields. So I would suggest to simply tune down and clarify any claims in this direction, i.e. introduce this work as what it is, namely an auto-regressive model for non-stationary time series forecasting which is based on just a library of fixed basis functions. If the authors want to make any stronger (theoretical) claims about their model, e.g. in direction governing equations, there is much more they need to show in my mind (see [1-3]), as well as different models to compare to!

3) This is important: The authors include a local feedback loop, which is similar in spirit to a Kalman filter correction in my understanding. Fig. 4 shows this component is essential for good predictions adapting to the non-stationarity. If I got this right, this component is actually adjusted based on the forecasting time window? But isn’t this then basically violating the basic training-test split scenario, as part of the model gets re-adjusted within the forecasting horizon? At the very least, isn’t this giving the author’s model an unfair edge over all the other competitors which do not use such local information?

4) Baseline models: This paper doesn’t develop novel theory but focuses on engineering a good forecasting model based on, essentially, known components and techniques. For instance, sliding window approaches are very common for non-stationary data, the feedback loop is in my understanding just a variation of the nonlinear Kalman filter theme, and the ‘Koopman operator’ part here basically boils down to the classic technique of providing the model with a set of fixed basis functions (and perhaps all these aspects should be pointed out in a revision!). So I really see the main strength of this contribution in the development of an architecture that outperforms other state-of-the-art models on challenging real-world benchmarks. At the same time, for the same reasons, I would have liked to see a few more comparisons:
- I believe it’s really important to also compare to very simple baselines, like ARIMA or just moving averages or kNN for time series. In my mind there is generally a tendency to develop overly complicated models in DL without properly checking whether very simple and naive methods could already do the job.
- I’m not quite sure why different models have been used as competitors for the two sets of time series in Table 2 and Table 3? Why not apply the same set of models to all?
- LEM [4] in my mind is another strong candidate that should have been tested, and perhaps also Informers. Less important, but just for curiosity I would have been interested how models perform actually designed for extracting governing equations (like those in [1-3]), although not build for non-stationary time series prediction.

5) Presentation of data: Table 2 needs to be in the main text in full (i.e., Table 7). As it’s presented now, this is really cherry-picking of the results where the own model outperforms others. Also I find Tables (like Table 2) which do not include any indication of standard error essentially meaningless from a statistical point of view. It is pretty clear in my mind that there will always be some bias toward the own model, let alone because authors simply know their own model best, and so at the very least an indication of significance needs to be provided. I would furthermore suggest to also indicate the % improvement over the best competing method.

Minor things:
- p.6: In my view forecastability should be based on Lyapunov exponents (not on a linear decomposition like Fourier spectrum), since these are really what determines the forecast horizon in a dynamical system.
- p.9: It’s clear almost any model will perform well on a simple oscillator. But most real world data are likely chaotic, and so these would constitute a more interesting test case (e.g. [5]).

[1] https://arxiv.org/abs/2106.06898

[2] https://arxiv.org/abs/2207.02542

[3] https://arxiv.org/pdf/2201.05136

[4] https://arxiv.org/abs/2110.04744

[5] https://arxiv.org/abs/2110.07238

summary_of_the_review: Potentially nice contribution with superior prediction for non-stationary time series (an important problem), and a model that can in part be interpreted. In my mind somewhat weak anchoring in theory, although not essential here I think. Some more baselines should be considered and presentation of results should be improved.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a new model, based on Koopman operator theory for forecasting time series with distributional shifts. The method combines global operators shared by time series, local ones that adapt in time for each series, together with a feedback loop for abrupt changes. The evaluation is performed on challenging datasets and shows promising results.

The reviewer agree that the paper is solid and presents new ideas to use Koopman theory for time series prediction. In particular, the authors introduce a novel way to model measurement functions and apply them in a latent space. The reviewers highlight the quality of the results obtained on challenging benchmarks which allows the method to surpass well established SOTA methods. Several reviewers qualify these experimental results as impressive. The responses and modifications during the rebuttal period were acknowledged as satisfying by the reviewers. 
justification_for_why_not_higher_score: This is a solid paper that cleverly combines existing ideas but does really not introduce new concepts. The stronger point is the experimental results obtained on several challenging datasets. This does not deserve in my opinion a spotlight, but I have nothing against.
justification_for_why_not_lower_score: original contribution and strong results on challenging datasets
Decision: Accept: poster