Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper is about improving Koopman autoencoder models in two ways: (1) a better initialization of the Koopman matrix by directly changing the eigenvalues, and (2) penalizing the eigenvalues of the Koopman matrix during training. These changes are tested on 4 datasets (some synthetic & some real-world), showing improved convergence and long-term prediction error.
strength_and_weaknesses: **Strengths:**

I think it makes sense that if you have some physical knowledge of the system, you could use that to influence the eigenvalues of the Koopman matrix and have better results. I appreciate that this was tested on four datasets, which is a pretty good number. The improvements over the Standard KAE are pretty strong.

**Weaknesses:**

My primary concern is correctness: When the initial Koopman matrix is defined with new eigenvalues, it may be complex. The real part of each matrix element of A is kept. Does that change the eigenvalues, or are they still \lambda^{\tilde}? I tried a small test, and it seems that the eigenvalues change. Please let me know if I'm missing something. I set A = [1, 2; 3, 4]. The eigenvalues are approximately  -0.3723 and 5.3723. I changed the second eigenvalue to 0.5 + .1i and reconstructed the matrix: A_new = V D_new inv(V). The eigenvalues are as expected, but the matrix is complex. I took the real part: real(A_new) and the eigenvalues change to approximately -0.3723, 0.5. 

It would be nice to have more benchmark methods to compare against. In lieu of that, I think some of the language should be softened. I think it's overstatement to say that the results are compared against "several state-of-the-art baselines." Similarly "was introduced by Azencot et al. (2020), yielding stable Koopman systems and state of the art performance" is an overstatement, since that paper only compares to one of many previous papers. (There are many papers with Koopman autoencoders cited in this paper.) Further, the Azencot et al. (2020) comparison to the Lusch et al., 2018 paper is not a complete one, since they left off the key feature of letting eigenvalues vary for a continuous spectrum. 

It's mentioned that "backpropagation through the eigenvalues of a matrix is numerically stable if there are no repeating eigenvalues." It sounds expensive to backpropagate through an eigendecomposition. How fast is this method? Does converging in fewer epochs come at a cost of each epoch being much slower? 

Although this paper shows improved results with the eigenloss and/or eigeninit approaches, there's a lot of a variability in whether it's best to use one or the other or both, and the improvements are much more modest when compared to the consistency baseline. The only column that is better than the consistency baseline more than half of the time in terms of test data the init-only column. Looking at Table 1 & Figure 4, if you were to try all of these approaches for a new dataset and pick the best approach based on the validation loss, that would usually not correspond to the best approach based on the cumulative test error. Is this because of differences in loss vs. error, or are there variable amounts of overfitting? 

I also have some comments in the next section. 
clarity,_quality,_novelty_and_reproducibility: **Clarity**

I think it would be good to elaborate more on when the "spectral radius is <= 1" assumption and the eigenloss are helpful and when they aren't applicable. 

- I understand that it **can** be advantageous to require/encourage that the spectral radius is <= 1 for stability. However, this might be restrictive. For example, in the fluid flow example in the Lusch et al., 2018 paper, trajectories that start inside the attractor exhibit growth until they reach the attractor. I know this sentence is motivation to focus on systems with spectral radius is <= 1: "Namely, Koopman eigenvalues of stable dynamical systems are constrained in the unit circle of the complex plane (Mauroy & Mezic, 2016)." However, I could be wrong, but I think this paper has a more narrow scope than this statement. From the abstract: "The main results establish the (necessary and sufficient) relationship between the existence of specific eigenfunctions of the Koopman operator and the global stability property of hyperbolic fixed points and limit cycles." The paper seems to focus on hyperbolic attractors and not systems with, say, a non-empty continuous spectrum. 
- Further, it seems that the eigenloss encouraging eigenvalues to have magnitude near 1 could be detrimental for systems that have decay toward an attractor or dissipation, as mentioned in the paper. It's mentioned that in the case of dissipation, there could be a weaker weight. However, would you know that you have dissipation and should use a weaker weight? And for significant dissipation, would you be better off with no eigenloss in that case? For example, a common test case for Koopman methods is the "2D fixed point attractor" in the Pan & Duraisamy (2020) paper, which has two eigenvalues, both causing decay. I suspect that the eigenloss would be detrimental for this example. 
- Again, I agree that these assumptions **can** be helpful, so I think this method has value, but I think it's useful to make it more clear that these assumptions aren't always helpful.  

I spent a while trying to understand which combinations of ideas were tested, so I think the language in Section 5 could be more clear. For example, in Table 1, I'm guessing that "None" refers to the Standard KAE, and "init," "loss," and "both" refer to models with the eigninit and/or eigenloss ideas applied but not the "consistency" idea? And then "consistency & best" refers to using the consistency idea plus the best of the previous 3 columns? 

Relatedly, I don't understand the sentence "The particular schemes applied to the consistency architecture were chosen by a two-stage selection process: first employing the best dataset eigenloss or eigeninit and then taking the lead performer out of these." 

How is cumulative test error defined? Averaging test error over the prediction steps? 

I'm confused about Figure 5. The left side and the right side have very different sizes of eigenvalues. Are they both accurate? It doesn't make intuitive sense to me that the same dynamics could be predicted well with such different sets of eigenvalues. 

**Quality/Correctness**

"All models with both eigeninit and eigenloss have significantly lower final validation loss than the control." I think this is an unfair summary of the table since test data is what ultimately matters, not validation data. 

There are a couple of mathematical details that are a bit incorrect. 
- The discussion around "eigenvector diminishes with time... the norm of v_j does not change..." is not quite right. Typically, eigenvectors are normalized. As the exponents on the eigenvalues increase, that does not change the eigenvectors. 
- "The Koopman operator is a linear object with a complex spectrum, i.e., the associated eigenvalues are complex-valued, when eigendecomposition exists." The Koopman eigenvalues are not necessarily complex. For example, the "2D fixed point attractor" in the Pan & Duraisamy (2020) paper has real eigenvalues only. Of course, you could say that real numbers are a special case of complex numbers, but I still think that the sentence is misleading because then it's an empty statement: of course the eigenvalues are complex when you count real numbers because what else could they be? The next sentence is also a bit wrong: the Lusch et al. (2018) paper mixes blocks for complex conjugate eigenvalues & blocks for real eigenvalues. 

In Table 1, first row, the consistency result should be bolded as best, but the init result is bolded. 

**Reproducibility**

- There are several options for eigenvalue distributions on the bottom of page 5 and two options for the eigenloss, but then it's unclear which one(s) work the best and with which values of parameters such as sigma. There is some hint of this in the appendix, but not a complete answer. It seems that the best option is inconsistent.    
- The "Standard KAE" is not clearly defined. I'm guessing it's the Azencot et al. (2020) model with some consistency features removed? Specifically adding the backward & consistency losses from Azencot et al. (2020)?

summary_of_the_review: I think that this paper presents a couple of interesting ideas and shows results that are sometimes better than the Azencot et al. (2020) paper. However, I think that the eigeninit is not doing what they think it is: it seems that the eigenvalues will change when they take the real part of the new matrix. I believe there are multiple incorrect or exaggerated statements. Also in order to improve upon the Azencot et al. (2020) paper, many models were trained with variants of the new ideas, so (a) the improvements may be mostly due to having many models to pick from, and (b) since there aren't consistent winners, there may need to be a lot of tuning to use these ideas.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents an initialization strategy for Koopman operators and a loss regularization in terms of spectral information for Koopman autoencoders. For initialisations, the paper claims that approximated Koopman operators should be initialized such that their spectral norm should be less than 1. For regularization, the loss is penalized such that the eigenvalues are concentrated on the unit circle.
strength_and_weaknesses: ### Strength
- The paper is well-motivated in the introduction and method section.
- In my opinion, regularizing eigenvalues is an interesting idea. This is possible for the Koopman operator because of its linearity. 

### Weaknesses
- The paper shows an empirical approach, lacking theoretical investigations.This does not mean that I do not appreciate the paper contributions.
- Even as an empirical study, the experiments of the paper fall short and are not comprehensive enough. For example, the paper may need to ablate the effect of changing the weight of $\epsilon_\lambda$. It’s not clear what the initialization strategy (among 4), and the regularizer (among 2) are used in the main text (Table 1, 2 and Figures 4).

clarity,_quality,_novelty_and_reproducibility: Question:
- The paper mentions that backprop through eigenvalues is numerically stable if there is no repetition among eigenvalues. However, can we avoid the instability by just using the proposed regularization? If yes, it is not clear to me.
- How are the convergence rate ratios measured? Specifically, Table 3 is not clear what the numbers are and how to compute them. Also, what is "best scheme and control"?

summary_of_the_review: In general, the paper proposes an nteresting way for egeinvalue intialization and regularization for Koopman operators in dynamic systems. However, the experiments should be polished more to support the claims.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper investigates the regularization and initialization of the Koopman Autoencoders (KA). KA aims at learning the lifting functions to transform the input dynamical system to the space where the state of the system evolves linearly. Existing methods for well-conditioning the KA rely on specific parameterizations (block-diagonal, tri-diagonal) or indirectly regularize the dynamics (Lyapunov-based, forward-backward soft constraints). Motivated by the fact that KA with stable dynamics has spectral radius with a magnitude less than one, the authors propose to initialize the Koopman layer transition matrix to have eigenvalues near one, and to regularize the model during training to keep its eigenvalues near one. Results are shown on synthetic and real datasets showing that the proposed initialization and regularization scheme outperforms the alternatives on the test data.
strength_and_weaknesses: **Strengths**
* The problem studied is timely and significant, and is relevant to the ICLR community.
* The introduction and related work sections contain a nice summary of the existing theoretical and empirical results on the initialization and regularization.
* The figures and schematics are clear, helping the reader to follow the arguments. The background on the Koopman Operators (KO) is sufficient and easy to follow.
* Overall, the logical flow of the paper is well-presented. Starting from the intro and background, the problem considered is motivated and connections to the existing literature are mentioned. The presented methods are logical consequences of the existing gaps.
* The presented results specifically on the real datasets help support the arguments.

**Weaknesses**
* I understand that the stable dynamics have a KO with a spectral radius below one, but if the dynamics evolve on a lower-dimensional space some eigenvalues can be closer to zero in magnitude while there are a few eigenvalues with near one magnitude. There seems to be a need to identify the intrinsic dimension of the system in order to find what is the right number of near-one eigenvalues. Did the authors experiment with the effect of model mismatch in this sense? Meaning that if the number of near one eigenvalue considered is not consistent with the intrinsic dimension f the system. And how this affects the overall quality of the results.
* There is no mention of the computational complexity of the system. The way I imagine this to be implemented is either using a parameterization that directly incorporates eigenvalues and the regularizations therein. In this scenario, the errors are backpropagated directly through the parameters (eigenvalues and eigenvectors) but the downside is that in every forward iteration one needs to compute the weight matrix. The second scenario (which is less likely) is if the weight matrix is parameterized in the traditional way, but at every iteration, the eigendecomposition is computed to evaluate the regularization. In both cases, there seems to be some computational burden with jointly initializing or regularizing all the weights. I would like the authors to clarify this (and possibly add a paragraph to the paper) and change the x-axis of the right-hand side plots on Fig. 4 to wall clock time instead of epochs.
* The authors motivate the closeness of the eigenvalue magnitudes to one, but at the bottom of page 5, the distributions considered seem to be a bit problematic. First, doubleGaussianEigen is not a distribution (it should be divided by 2, right?). Second, gaussianEigen and uniformEigen do not seem to encourage the eigenvalues to be of magnitude one. Third, unitPerturb seems to enforce all the eigenvalues to be near one magnitude. Based on my comment regarding intrinsically low-dimensional systems this seems to be a limiting factor for modeling those types of dynamics. Can the authors include a spike and slab type prior and report the results?
* My biggest concern about this paper is with respect to the experiments. Below I elaborate on how the results section can be improved.
    * It is argued that the weight decay regularization is not compatible with the assumptions of stable dynamics. But in the experiments, no comparison against weight decay regularization is presented. Can the authors include the comparisons and ablations (different combinations of initialization (eigeninit, He, Xavier) with regularization (eigenloss, l1, l2)?
    * The results are shown on limited architectures and the effect of the nonlinearity, width, depth, and architectural choices are not investigated systematically.
    * How is the dimension of the KO layer chosen? Can the authors include dimension mismatch studies showing that the method still outperforms alternatives when the wrong dimension is chosen?
    * In Fig. 4, on the right-hand side plots, it seems that all the methods eventually get to similar test errors, if so it's important to compare the time that it takes for different methods to get there. As I suggested above, changing the x-axis to wall clock time will address this. For each method, which epoch is chosen for the results on the left side plot? If the criterion for choosing a  model is the epoch number then the left-side plots and right-side plots seem to deliver the same message. What's more interesting is to change the criterion to convergence. This can address the following question: once all the models achieve similar test performance, which one learns a better dynamics model and generalize better to more time points?
 
**Minor Comments**
* What does the following line mean and how is this solved by the presented methods?
> Importantly, backpropagation through the eigenvalues of a matrix is numerically stable if there are no repeating eigenvalues.
* Can you include standard (with some sort of normalization that is comparable across different datasets) units on the y-axis of the plots of figure 4?
* Is the following paragraph systematically tested? While it sounds very interesting I would appreciate it if the authors do a systematic experiment where the amount of dissipation changes in the data generating system and show that different values of the hyperparameter are optimal with varying dissipation.
> For instance, if the underlying system is measure-preserving, we can weight $\epsilon_{\lambda}$ more strongly with respect to the reconstruction and prediction losses. In contrast, if the dynamics are dissipative, we can use a weaker weight.
* In the last paragraph of page 3, shouldn't $MSE(x_k, \hat{x_k})$ be changed to $MSE(x_k, \hat{x}_{k+1})$?
* Are the methods only developed for autonomous systems? Can this be extended to input-driven systems (which might be the case for many physical systems)? What would change if one was interested in training a KA with intrinsic dynamics, inputs, and outputs? Do we still need to encourage the eigenvalues to be near one?

## Post Rebuttal

I thank the authors for the clarifications and for taking my comments into account for their revision. Specifically, I appreciate that the authors included the suggested prior, did the wall-clock experiments, included the discussion about the intrinsic dimension of the system, and normalized the y-axis in the results figure.

Given the new plots, I am not quite convinced that the method outperforms alternatives. Fig. 3 which contains the experimental summary of the paper shows that all methods achieve similar validation performances. Given the noisiness of the validation loss plots, it's hard to conclude that the proposed method achieves better results. If the test and validation data have similar statistics the level of noise observed across epochs in the validation loss plots should appear in the test data as well. In addition, the training/validation plots and results of the test data in the table are not accompanied by their corresponding variances. There are multiple ways to include this information, but this is done commonly using random seeds (since the eigenvalues are sampled from spikeAndSlab randomly every initialization will change the eigenvalues which can give rise to training statistics). This is specifically problematic given the computational burden of the eigenLoss raising the question that given a similar computational budget (memory and time) which method achieves the best performance when the within-model variance is accounted for? The experiments included in the supplementary (section E) do not show a clear trend of eigenLoss or eigenInit being helpful for the presented datasets. In the Ocean dataset "No penalty" model performs similarly to "Eigenloss and eigenInit" and in the Cyclone dataset, there's so much noise in the validation loss that it's hard to determine between "No penalty" and "Eigenloss and eigenInit" which one is doing significantly better (also the losses seem very small even in the first time point which is strange).

In the rebuttal round, the authors fixed the conceptual error raised by the other reviewer and re-did all the experiments. The paper is fully restructured and the figures are changed, given that the changes are major, I do believe that the submission requires another round of careful reviewing. Moreover, it's unfair to other submissions since the authors used more time for the major changes. Given this, I tend to keep my score as is. I do believe however that the paper is well motivated, and careful experimentation may better represent the merits of the eigenLoss and eigenInit. Therefore I'd encourage the authors to do another round of polishing and submitting elsewhere if not accepted here.
clarity,_quality,_novelty_and_reproducibility: The presentation of the paper is clear, the methods are novel and question is timely. There seems to be enough information for reproducing the results although I did not try to reproduce them myself.
summary_of_the_review: The paper proposes a timely solution to an important problem which is well-conditioning the Koopman layer in the Koopman Operators. Overall the paper has good quality, however, the results and experiments can be improved to make the case stronger. I'm happy to increase my score to acceptance if my detailed comments above are addressed.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies the problem of training Koopman Autoencoders, which is a family of models used to solve physics-informed learning problems. For the Koopman layer in such models, the paper proposes a special type of eigenvalue-tailored initialization (as compared to standard random init schemes such as He or Xavier). Moreover, the model is regularized to ensure that the eigenvalues are kept close to unity. Experiments on both synthetic and real datasets indicate the benefits of this approach.

On the positive side, the reviewers agreed that this is a well-motivated paper that proposes an intuitive approach and validates it on real datasets.
 
On the negative side, several concerns were raised by the reviewers. Several claims about the method were not well-supported (or over-stated); detailed ablation studies were lacking; comparisons to other standard regularization procedures (such as weight decay) were missing; and the numerical stability of eigenvalue computation was a challenge. A big concern (echoed by multiple reviewers) was computational complexity: eigenvalue computation is slow, and it is not quite clear how the approach scales up. 

In the response phase, the authors did address many of these concerns by providing a revised version of the paper. However, the issue of computational complexity (and wall-clock running time comparisons) still persists, particularly on larger problem instances. The issue of real/complex eigenvalues also persists: the authors dealt with this by using a "hack" (in my opinion) and only adjusting eigenvalue magnitudes, but this somewhat weakens the principled motivation behind the proposed method.

The authors are encouraged to consider the above comments while preparing a new revision.
justification_for_why_not_higher_score: As mentioned in the summary above: this is a good paper, but requires fairly extensive revisions, particularly related to the (revised) method's soundness as well as a proper treatment of computational complexity. I believe this set of revisions is beyond what is feasible at this stage to merit an accept decision.
Decision: Reject