Eigenvalue Initialisation and Regularisation for Koopman Autoencoders
Authors: Jack William Miller
Charles O'Neill
Navid C Constantinou
Omri Azencot
Venue: Submitted to ICLR 2023 (2023)
Keywords: koopman
deep learning
dynamical systems
autoencoders
physics-constrained learning
neural networks

Abstract: Regularising the parameter matrices of neural networks is ubiquitous in training deep models. Typical regularisation approaches suggest initialising weights using small random values, and to penalise weights to promote sparsity. However, these widely used techniques may be less effective in certain scenarios. Here, we study the Koopman autoencoder model which includes an encoder, a Koopman operator layer, and a decoder. These models have been designed and dedicated to tackle physics-related problems with interpretable dynamics and an ability to incorporate physics-related constraints. However, the majority of existing work employs standard regularisation practices. In our work, we take a step toward augmenting Koopman autoencoders with initialisation and penalty schemes tailored for physics-related settings. Specifically, we propose the "eigeninit" initialisation scheme that samples initial Koopman operators from specific eigenvalue distributions. In addition, we suggest the "eigenloss" penalty scheme that penalises the eigenvalues of the Koopman operator during training. We demonstrate the utility of these schemes on two synthetic data sets: a driven pendulum and flow past a cylinder; and two real-world problems: ocean surface temperatures and cyclone wind fields. We find on these datasets that eigenloss and eigeninit improves the convergence rate by a factor of 2 to 5, and that they reduce the cumulative long-term prediction error by up to a factor of 2.5. Such a finding points to the utility of incorporating similar schemes as an inductive bias in other physics-related deep learning approaches.

Under review as a conference paper at ICLR 2023
EIGENVALUE INITIALISATION AND REGULARISATION FOR KOOPMAN AUTOENCODERS
Anonymous authors Paper under double-blind review
ABSTRACT
Regularising the parameter matrices of neural networks ubiquitous in training deep models. Typical regularisation approaches suggest initialising weights using small random values, and to penalise weights to promote sparsity. However, these widely used techniques may be less effective in certain scenarios. Here, we study the Koopman autoencoder model which includes an encoder, Koopman operator layer, and decoder. These models have been designed and dedicated to tackle physics-related problems with interpretable dynamics and an ability to incorporate physics-related constraints. However, the majority of existing work employs standard regularisation practices. In our work, we take a step toward augmenting Koopman autoencoders with initialisation and penalty schemes tailored for physics- related settings. Specifically, we propose the "eigeninit" initialisation scheme that samples initial Koopman operators from specific eigenvalue distributions. In addition, we suggest the "eigenloss" penalty scheme that penalises the eigenvalues of the Koopman operator during training. We demonstrate the utility of these schemes on two synthetic data sets: driven pendulum and flow past a cylinder; and two real-world problems: ocean surface temperatures and cyclone wind fields. We find on these datasets that eigenloss and eigeninit improves the convergence rate by up to factor 5, and that they reduce the cumulative long-term prediction error by up to factor of 3. Such a finding points to the utility of incorporating similar schemes as an inductive bias in other physics-related deep learning approaches.
INTRODUCTION
Modern neural networks are often overparameterised, i.e., their number of learnable parameters significantly larger than the number of available training samples -hulen-Zhu et al.) 2019a|b) To guide optimisation through this immense parameter space, and to potentially improve performance by avoiding overfitting, neural networks are trained with regularisation techniques (Goodfellow et al., 2016). The importance of regularisation has been shown in the theory and practice of deep learning.
Prominent examples include the initialisation of parameter matrices (He al. 2015 Hanin & Rolnick 2018), and constraining the parameters' norm via loss penalties (Hinton 1987 Krogh & Hertz] 1991). Initialising weights and penalising them with small random values and weight decay are arguably the most common regularisation techniques employed in training deep models with stochastic gradient descent algorithms. However, specific neural architectures, data domains, and learning problems may require different initialisation and penalty schemes, In this paper, we empirically study the effect of regularisation on physics-aware architectures.
The ground-breakinn success of deep learning in solving complex tasks in vision and other domains has inspired the physics community develop deep models suited deal with real-world problems arising in the field (Willard et al. 20200 Karniadakis et al. (02I). In this context, we focus on dynamical systems analysed and processed using Koopman-bun based approaches (Takeishi et al. 2017] Lusch et al.) 2018). Koopman theory (Koopman) 1931) proves that under certain assumptions, nonlinear and finite-dimensional systems can be transformed to a linear (albeit nnfinte-dimmesiioaal) representation via the Koopman operator. Using finite-dimensional approximations of this Koopman operator is advantageous as they facilitate the analysis and understanding of dynamical systems by utilising linear analysis tools. Despite the theoretical and practical advances that have significantly improved Koopman- based learning methods, the majority of existing models still apply regularisation practices designed for general neural networks. Our investigation aims answer the research

---

Under review as a conference paper at ICLR 2023
question: one exploit properties of Koopman operators improve regularisation and to promote better overall performance?
The Koopman operator is linear object with a complex spectrum. Some groups in the past have saught to place a bias on the operator's s form. For example, Pan & Duraisamy (2020) propose a skew-symmetric tridiagonal form to guarantee stable Koopman models. However, there has not been a systematic investigation that evaluates the effect initialisation and penalty regularisation schemes on the behavior of Koopman-basee neural networks. The overarching objective of this paper is to help bridge this gap. Key to our regularisation schemes is the observation that spectral properties of linear Koopman operators follow a rypical structure, shared by many dynamical systems. Namely, Koopman eigenvalues of stable dynamical systems are constrained in the unit circle of the complex plane MMauroy Mezic 2016).
Motivated by the theoretical observation that, for this important class of stable systems, eigenvalues need be within the unit circle, we propose two novel regularisation techniques: random eigenvalue generation from known distributions to initialise key weights in the network ("eigeninit"), and a loss penalty on the eigenvalues to alter their expansion ("eigenloss"). Importantly, our regularisation schemes can be incorporated with all Koopman-based approaches as a means to regularise the associated Koopman operator. We evaluate our approach on several challenging physics-relater datasets and in comparison to standard regularisation and initialisation approaches as well state-of-theeaar baseline. Our results indicate that our spectral schemes for initialisation and penalty improve the performance of Koopman- poosmad--baeddd based networks, leading to faster and smoother convergence in the objective loss, as well as yielding models that generalise better in long-term prediction tests. Although these schemes require tuning, guide to which we provide, they are easily applicable in any context where a Koopman approximate is used. As such, they present a useful tool for practitioners in the field.
RELATED WORK
Regularisation of neural networks is a fundamental research topic in machine learning. Common regularisation techniques include dropout (Srivastava et al., 2014). batch normalisation (loffe & Szegedy 2013), and data augmentation (Perez Wang 2017). In what follows, we mainly discuss parameter initialisation approaches and weight penalty methods.
Parameter initialisation. Proper initialisation of deep models is known to be crucial to their successful training (Sutskever et al. 2013). Standard initialisation schemes for tanh(-) (Glorot & Bengio 2010) and ReI.U (He et al. 2015) activations sample small random values for the weight matrices of the network. These choices are backed by theoretical results showing that initial small scale weights generalise better (Woodworth et al. 2020), and random initialisation converges to local minimisers on differentiable losses (Lee et al.] | 2016). While guaranteed, convergence may be exponentially slow (Du et al., 2017). Other approaches promote information flow using initial orthogonal weight matrices (Saxe et al.. 20144; Mishkin & Matas 2016 [Pennington et al. (007)) Koopman-based approaches (Pan & Duraisamy (2220) suggest to initialise the Koopman operator using the dynamic mode decomposition (DMD) estimates (Schmid 2010). Still, the general problem of weight initialisation in neural networks remains active research topic in practice (Arpit et al.
2019), and theory (Hanin & Rolnick 2018: Stöger Soltanolkotabi 2021).
Parameter loss penalties. Penalising the norms of weight matrices using L1 and L2 metrics is a common practice in machine learning, collectively termed weight decay (Hinton 1987). More recently, Arjovsky al. (2016) showed that recurrent neural networks can avoid the issue of exploding gradients if their hidden-to- to-h-dot--h matrices are parameterised to be unitary. Similarly, Yoshida & Miyato (2017) introduce regularisation scheme based on penalising the spectral norm of weight matrices. Greydanus et al. (2019) assume the underlying system is geeasure-preservving network learns the Hamiltonian during training. Lusch et al. (2018) use block diagonal Koopman operators to support continuous spectra. To promote stability, Erichson et al. (019)) employ Lyapunov- based constraints, whereas Pan & Duraisamy (2020) guarantees that Koopman eigenvalues remain in the unit circle via tridiagonal Koopman operators. In contrast, a soft penalty the forward and backward dynamics was introduced by Azencot et al. (202)), yielding stable Koopman systems and state of the art performance.
and their
2

---

Under review as a conference paper at ICLR 2023
3 BACKGROUND
3.1 KOOPMAN OPERATOR THEORY
Suppose we have discrete time dynamical system given by:
Ek+1 ((())) k E No
Let f (where is the system's state space) real-valued observable of the system. By real-valued observable we mean function of the state of the dynamical system, where an observed data-variable its range. The collection of all such observables forms a linear vector space. The Koopman operator K is linear transformation on this functional space:
K x p(x))
Essentially, the Koopman operator may viewed as a lifting of the dynamics from the state space to the space of observables (Mezić 2015). Whilst the Koopman operator maps between function spaces and is thus infinite dimensional, it has been shown empirically that finite-dimensional approximations (U) generally quite expressive, where U/ is usually found through dynamic mode decomposition or deep learning (Mauroy et al. 2020).
3.2 KOOPMAN AUTOENCODERS
Pioneered by several groups (Takeishi al.) 2017 Lusch al. 2018). the Koopman autoencoder (KAE) is deep neural network designed to solve physics-related problems. Primarily, KAFs are based on an autoencoder architecture (Hinton Zemel 1993) which has a "bottleneck" structure including encoder v which learns a low-dimensional representation of the input signal, and decoder w which recovers the original signal from the low-dimensional code. Koopman autoencoders extend these family of autoencoder models by introducing Koopman operator module in-between & and w whose purpose is to produce a finite-dimen dimensionl approximation of the Koopman operator. The Koopman module U is essentially linear layer (with no bias) aiming advancing latent codes forward time. Put together, the KAE architecture can be described by the following equations.
yk (Ixx) 3h (yy)) uk+1 Uyk $8411 w(K+1))
where is the input signal, म사 is its reconstruction, and 4x+1 is the reconstruction of the latent code obtained with the Koopman matrix U. The model trained with reconstruction loss MSB(I Tx.), and a prediction loss MSE(x 2x+1),, where MSE is the mean squared eтror. Several existing methods adopt this general framework which we illustrate in Fig. and they further generalize it to promote stability (Pan & Duraisamy] [020 Azencot et al. 20200. to incorporate control (Han et al.
2021), and to other settings (Morton et al. 2018 Li al. 2020 Yeung al., 2019 2019
Otto & Rowley
Iwata & Kawahara 2020).
Input 26
Output ft
yk Uk
Figure 1: Diagram Koopman autoencoder architecture.
3.3 SPECTRAL ANALYSIS OF KOOPMAN APERATORS
We briefly mentioned above that one of the main advantages to working with Koopman- bass approaches, and KAE in particular, is the linearity of the Koopman matrix U. Specifically, can
3

---

Under review as a conference paper at ICLR 2023
harness tools from linear analysis and spectral approaches to study the underlying dynamics (Strogatz 2018). For instance, the cigendecomposition of U to eigenvectors and eigenvalues is instrumental to qualitative characterisation of the behavior of the system. We call (0z.Ag Ag) an eigenvector value pair of the Koopman matrix if they satisfy: Uny = Ay where X, E C. The effect the eigenvalues on the hehavior of the system can be realized by considering the long-term evolution of inputs. Denote by V the eigenvectors in columns, and A the diagonal matrix with the eigenvalues along its main diagonal, then can advance the latent code yı by
Uk+! = U yk VAV yk.
(4)
Thus, advancing Mı forward in time from time k to time k + amounts to simply multiplying yn with powers of U. Eq. 4 also shows that every eigenvector 2j is scaled by its associated X/. and therefore, the modulus of \Ayl determines its long-term behavior. We identify three qualitatively different evolution profiles, determined by the modulus of eigenvalues: if [Ayl < 1 then the associated action of its eigenvector diminishes with time (i.e. - oo). if A > 1, then the action increases arbitrarily when I + oc, and if |yyl 1 then the dynamics remain stable. In this way, the eigenvalues of the Koopman matrix encode the "memory" of an evolution rule.
Our regularisation schemes described in Sec. based prior soft bias the number of eigenvalues in each dynamical mode- vanishing, exploding and stable.
Xavier Initialisation W-- 17 FETRUT
Gaussian Initialisation w- M1,
250
Avg
300
200
frequency
Frequency
150
200
150
100
50
50
0.50
1.00 1.25
2.00
Avarage Magnitude of Eigenvatues
1e-1
Average Magnitude of Elgenvalues
Figure 2: Eigenvalue distribution of various initialisation schemes for x operator with six layers.
METHODS
Our discussion in Sec. 3 suggests that the optimal eigenvalue spectra of the approximate Koopman matrix may change based on the nature of the system. In our experiments we focus on the class of systems for which the eigenvalues are approximately contained in the unit circle. However, one could easily adapt our schemes to promote different distributions.
4.1 EIGENVALUE INIIIALISATION OF LEARNABLE PARAMETERS
Initialising deep neural network of any architecture is critical to its training and model performance. In Fig. 2 we empirically evaluate the eigenvalue distribution of parameter matrices as attained by two of the most common initialisation schemes: Xavier (Glorot & Bengio 2010) and random Gaussian initialisation. We find that the average magnitude of eigenvalues are approximately 0.1 for Xavier, whilst they are 0.4 for a Gaussian initialisation. Overall, these initialisations produce Koopman matrices whose eigenvalues span distribution with low variance. However, these spectra may negatively affect training cases where the optimal Koopman matrix consists of eigenvalues with modulus Ayy x 1.
Using initial small random values for the parameter weights of deep neural networks is motivated by theoretical results which show that such initialisations converge to local minima differentiable loss functions when solved with stochastic gradient descent (Lee et al. 2016). However, convergence may be extremely slow (Du et al. 2017). Can improve convergence speed and model generalisation by developing initialisation schemes tailored to Koopman autoencoder architectures? Typical Koopman- based approaches utilise of the two initialisation schemes above (see e.g., Azencot al. (2020));
4

---

Under review as a conference paper at ICLR 2023
however, these choices are agnostic to the structure of the Koopman operator layer. Other techniques employ a DMD-based initialisation (see for instance, Pan & Duraisamyl (2220), which may be an unrealistic choice for highly- nonlinerr and sigh-dimenstonal dynamical systems (Zhang et al. 2017). Inspired by the discussion in Sec. 3 we would like to exploit the unique role of the approximate Koopman operator U as an evolution matrix and its special structure.
Therefore, we choose to focus on eigenvalue- based initialisation rather than element- wise initialisation whose effect on the eigenvalues is indirect, and leads to eigenvalues with lower magnitudes. Namely, we propose initialisation schemes that directly control the eigenvalues of the Koopman matrix U and their distribution. To this end, we propose the following eigenvalue initialisation method (termed "eigeninit") "): we start with an element-wise sampling from a Gaussian distribution to produce Uo. Then, we perform an eigendecomposition of Ug- = VAV -1, where V is matrix with the eigenvectors "j organized in its columns, and A is diagonal matrix with the eigenvalues Aj along its main diagonal. Each eigenvalue X is, in general complex, and can be re-written r, cid,,, We modify the modulus ry = |را| E IR of the eigenvalues without altering their phases 0,. Each modulus ri is sampled from pre-defined distribution D; D; + F D. Care is taken for complex- complex-conjugate conjuuatte eigenvalue pairs: the modified modulus of both eigenvalues from a complex- conpux--conjggaee eigenvalue pair is kept the same, ry e + i3 edi, Then, the initial Koopman matrix U is defined via == VÁV where Á is a diagonal matrix with λ₃ Fy e along its main diagonal. Following this procedure and given that the initial matrix Un is real, then U is also real (up to machine precision). All other weights in the network are initialised according to the He initialisation as we utilise ReLU activation layers (Hc et al. 2015).
Taking into consideration the above discussion, natural question arises: what distributions D should we sample from? If one is not looking to model divergent dynamics, the focus should be on parameter
matrices U initialised such that their spectral radius satisfies p(U) S 1. On the other hand, existing
works on recurrent neural networks (Arjovsky et al., 2016) advocate the use of orthogonal weights
for which p(U)= = 1. In what follows, we prefer the former option. i.e., U matrices with p(U) <
since orthogonal and unitary matrices are less expressive (Kerg et al. 2019). In particular, p(U) < 1
allows U to immediately capture any dissipative dynamics in system. In practice, == 0 (4, 1) + (1 - 0) 8(1)
sample the
eigenvalue moduli from a spike and slab distribution: D DsspikeAndSlaal
where 4(00 1) is a uniform distribution from 0 to and 5 is the delta function. This choice is motivated by the notion of an intrinsic dimension of the system, which will naturally give rise to a certain number of eigenvalues on the unit-circle (corresponding inside the unit-circle (corresponding to dissipative or finite-time horizon dynamics of the system). Whilst [0, 1] (which governs the proportion of modulus 1 eigenvalues sampled) may be chosen through a hyperparameter search, another feasible approach is to use a dynamic mode decomposition to estimate the number of non-unitary eigenvalues, and use this to inform A.
conserved quantities) and others
4.2 EIGENVALUE REGULARISATION OF LEARNABLE PARAMETERS
While proper initialisation of weights is crucial for neural network to start optimisation from successful trajectory that generalises well (Stöger & Soltanolkotabij 202T), we note that without an additional mechanism that penalises non-preferable solutions, optimisation may become stuck inferior local minima. In our setting, this means that the resulting Koopman matrix may settle on a spectrum which is not commensurate with the system being studied.
Similarly to our initialisation scheme which directly affects the eigenvalues of U (Sec. 4.1 we propose a penalty method (termed "eigenloss") which biases the spectrum of the approximate Koopman operator: we augment the reconstruction and prediction loss components of the Koopman autoencoder (Sec. B with a new and novel loss term ex whose role is penalise eigenvalues of U which are too distant from a certain value or distribution. While we could have used the same ansatz as in Sec. 4.1 promoting the spectral radius of U to satisfy p(U) S 1, we find that on our datasets, encouraging the eigenvalues to be close or on the unit circle yields better models in terms of convergence and generalisability features. Thus, we consider the penalty scheme (MSE) := Σ, | - 112 where Xy is an eigenvalue of U. That is, we encourage the modulus of the eigenvalues to be approximately one. Noteably, backpropagation through the eigenvalues of a matrix is always numerically stable, even if there are repeating eigenvalues.
We also experimented with uniform distribution hetween 0.9 and
5

---

Under review as a conference paper at ICLR 2023
The choice of eigenvalue penalty imposes different soft biases the Koopman approximation U. Based on the -owaiinnnnwwweeeee of the relevant dynamical system, one can tune the above penalties better. For instance, if the underlying system is preserring,..,
weight ελ more strongly with respect to the reconstruction and prediction losses. In contrast, if the dynamics are dissipative, can a weaker weight. As with initialisation, one might use the derived to guide this choice. If 0 is near zero, it is likely better to drive cigenvalues closer to the unit circle.
5 EXPERIMENTS
The empirical results for eigeninit and eigenloss applied the standard Koopman autoencoder and consistent Koopman autoencoder are demonstrated in various experiments, Both were implemented in Pytorch (Paszke al. 2019).
5.1 DATASETS
Datasets were chosen as combination of synthetic and real-world data, varying in complexity and including varying amounts of dissipation. Our initial dataset was a driven frictionless pendulum governed by: d + w fo sin(wt), where wh the characteristic frequency, and fo and the amplitude and frequency of the forcing
We also applied our techniques to cyclone wind fields, flow over cylinder and sea surface tempera- tures. The cyclone prediction dataset was extracted from the ERA5 Re-Analysis dataset provided by ECMWF using the International Best-Tracks Archive for Climate Stewardship (IBTrACS). Prediction sequences were sub-sampled from the u component of wind at the pressure level of 650 hPa in 20 x 20 sliding window around the cyclone. The sea surface temperature benchmark was taken from (Reynolds et al., 2007). It is a subset of the NOAA OI SST V2 High Resolution Dataset over the gulf of Mexico with spatial resolution of 100 x 100 spanning time horizon of 1305 days. Finally, the fluids dataset modeled flow over cylinder and was produced by Kutz et al. (2016). We took the u-component of the flow vector field.
5.2 TEST AND VALIDATION PREDICTION ERROR WITH DATASETS
One can see the application of different schemes (including their combination) and the influence of these on the prediction test prediction across given prediction horizons and validation loss in Fig. 3 In addition, the state of the art consistency architecture presented by Azzencot et al. (2020) is shown as an additional baseline and an architecture results see Tab. [ Of particular note is that all of our schemes (except for their combination with the pendulum dataset) outperform the standard KAE and do by up to a factor of 3 in cumulative test error. Additionally, our initialisation schemes increased convergence rates (see Appendix C for summary). Regarding the consistency benchmark, our schemes outperform or match its performance in all datasets except for the pendulum. In addition, augmentation of the consistent KAE with cigenloss or eigeninit improves its performance on the real-world data and rate convergence on all datasets.
which our schemes are applied. For a summary of
5.3 INVESTIGATING EIGENVALUE EVOLUTION
Although we have demonstrated the utility of our initialisation and regularisation schemes, have yet to investigate the evolution of eigenvalues within the networks themselves. We can plot a heatmap of eigenvalue magnitudes of the Koopman operator during training (Appendix B Fig. 0 By doing so, we see that the control network has increasing eigenvalues even though there is no bias towards this and that using the unit circle MSE penalty promotes this tendency. Alternatively, with initialisation close to the unit circle, the eigenvalues tend towards the different eigenvalue classes mentioned in Sec. 33
We take the standard Knopman autoenender to he architecture used by Azencot et al. 2020 without the consistency modifications.
We use wp 3.13, fo and 1, and initial conditions E [[ n]] and /z/dt [-1,1].

---

Under review as a conference paper at ICLR 2023
Pendulum Dataset
Normalised Cumulative test for prediction horizon
Normalised Valication loss for training
1.0
0.8
Mean squares
Spenta
0.6
1.2
0.2
1.0-
10
1'5
20
25
10
20
30
50
60
70
Fluids Dataset
0.8
Moein squareo erdr
0.6
10.0
0.0
0.0
6
10
12
14
10
20
40
50
Ocean Dataset
Mean squared enror
0.0
5
20
25
2.5
7.5
10.0
12.5
15.0
Cyclone Dataset
Mean squared ereer
1.0
0.8
12-
0.4
0.2
1.0
0.0
0
10
15
25
50
75
100
Time steps
Epochs
Figure 3: Averaged prediction horizon test loss and training validation loss for selected library members eigeninit and eigenloss. Models are trained on the longest prediction horizon shown in the test loss graphs. Hyper parameters are given in appendix A.
5.4 COMPUTATIONAL COMPLEXITY
One reasonable concern with our regularisation technique is computational complexity. The eigen- decomposition of the Koopman operator must be upplied at each iteration in order to calculate the loss with respect to the cigenvalues, which scales as 2.3366) (Demmel et al. 2007). Thus, it is possible that our techniques do not actually converge faster, at least terms of actual CPU wall time (as opposed to epoch number). However, this concern is rendered largely irrelevant by the size of Koopman approximations used. Even when modelling cyclone paths, which are considered relatively complex dynamical system, we only required a Koopman matrix of size 16 16. This is an advantage of compressing the problem to a smaller latent space with the autoencoder architecture. Hence, is likely that the computational complexity of our methods is not concern. Indeed, as shown Fig. (Appendix E), we see that our methods still converge faster in terms absolute wall time compared to the control.
7

---

Under review as a conference paper at ICLR 2023
Table Minimum validation loss and final cumulative test prediction error for benchmark data sets. See bold for minimum.
Validation Loss
Dataset
None
Xavier
Cons.
Init
Loss
Both
Cons. & Both
Pendulum
10.8
11.68
10.9
11.01
11.05
11.09
11.01
Fluid
0.19
0.24
0.18
0.24
0.29
0.19
0.26
Ocean (x 10)
0.26
0.21
0.27
0.14
0.27
0.16
0.19
Cyclone
16.18
14.60
13.77
14.25
15,44
14.34
14.33
Cumulative
Test Error
Pendulum
176.69
178.45
161.68
165.71
174.76
181.78
191.47
Fluid
1.25
2.30
1.00
0.58
0.82
0.96
1.04
Ocean
3.12
1.69
2.82
1.02
2.85
1.24
1.66
Cyclone 554.5
626.4
631.29
494.2
501.3
458.5
484.7
5.5 ABLATION WITH HYPERPARAMETERS
We also tested the sensitivity of our methods the choice of hyperparameters on the fluids and ocean datasets. In particular, completed ablations with differing values of éx for the penalty term, 0 for the spike and slab distribution, the size of the Koopman approximate and the width of the encoder and decoder. The results of this can be seen in Fig. 4 The ocean dataset always improved compared to the standard Koopman autoencoder for all hyperparameter choices. However, performance on fluids was sensitive to operator size and 0.
Ablation changing
Ablation changing
Gauszon intoisation
10f
1x-10-
ocean
Bx10--
ratio of
10a
30
1.7
10' Diffement.
107
Different
Ablation with changing
Ablation with changing ng width
R320 with Couszion
10f
6
6x10
6x10
4xx
0.4 Different
10'
Different
Figure 4: Testing the sensitivity of schemes to changes in network hyperparameters. Each data point the averaged final cumulative test error. For ablation with changing width and Koopman operator size, the scheme compared is eigenloss for the fluid dataset and initialisation for the ocean dataset.
5.6 DYNAMIC MODE DECOMPOSITION INFORMS INITIALISATION AND REGULARISATION
The use of spike and slab distribution naturally raises the question of how to choose the parameter 9 E [0. 1]. which governs the proportion of modulus eigenvalues sampled. Whilst we could do so through simple hyperparameter search, another interesting approach could be to dynamic mode decomposition to give an estimate the number of non-zero eigenvalues and hence the magnitude of 0. Dynamic mode decomposition (DMD) achieves the dimensionality reduction of
8

---

Under review as a conference paper at ICLR 2023
dynamical system by finding the eigenvectors associated with fixed frequency and growth (Schmid 2010). We hypothesised that using the DMD to find an approximation of the evolution matrix A, with =k++1 Azg, may give computationally cheap way to estimate the number of unitary cigenvalues. We could then determine the proportion of eigenvalues to sample from the unit circle, and the proportion to sample from zero-centred Gaussian, using this estimation. To test this, we computed the n n. DMD evolution matrix Á of the ocean dataset, matching the size of the n X n Koopman approximation in our autoencoder. We then estimated 0 by the proportion of Á's eigenvalues with magnitude within le-3 of 1. We see from Fig. 5 that our estimated 0 gave the lowest average final loss.
0.0250
00.25
00200
Avcoage test loss
0.0.0
0.077
0.02%
0.0500
0.022
00:35
0.0
0.5
0.6
Figure 5: Examining the eigenvalues of the DMD of the ocean dataset gave the best initialisation hyperparameter 0 (estimated f shown in red).
DISCUSSION
In this research we have asked whether an initial and ongoing bias on the eigenvalues of Koopman autoencoders would improve their accuracy and convergence. We tested this via empirical analysis on the field's existing benchmarks and through the introduction of a new one (cyclone wind fields). It seems clear that certain well developed biases do in fact have utility for training these networks. In particular, our schemes outperform state of the art baseline models which do not use them. In addition, they augment one such bascline introduced by Azencot et al. (2020), and improve its performance on real-world datasets.
These techniques important because they marry together the purposes of dynamical systems modelling: prediction and explanation. Whilst deep autoencoders have frequently been used for predicting complex dynamical systems, they often criticised because they lack explainability. Koopman autoencoders are a front-running scheme to address this. However, inserting a Koopman approximation between the encoder and decoder introduces bottleneck in the network and reduces its capacity for learning.
Because the linear dynamics are so useful in understanding the system, we want be able keep this interpretability whilst providing necessary forecasting capacity. The results above demonstrate that this is possible by exploiting the spectral properties of the Koopman approximation U. Specifically, we have shown that Koopman autoencoders using our eigeninit and eigenloss techniques (i) converge faster; (ii) reach a lower validation loss level on real-world data; (iii) achieve better results across multiple prediction horizons than state of the art baseline networks; and (iv) can productively augment other inductive biases on the Koopman approximate. Our ablations have shown the effect of various hyperparameters on model performance, depending the nature of the dataset. Further, dynamic mode decomposition (DMD) has been shown to provide an informative way to choose the initialisation distribution. Further applications of the eigenvalues from DMD may provide insight into the conservative or dissipative aspects of the relevant system, and thus inform the appropriate in regularisation.
eigenloss
These techniques will be of practical help for those using Koopman methods in dynamical systems. It is likely that the schemes presented here be extended in many ways, particularly by devising better initialisation and regularisation candidates and tuning them to the nature of the problem. Whilst application to other autonomous dynamical systems is evident, may also be possible to extend the Koopman initialisation and regularisation techniques for input-driven systems, significant area of future potential research.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
In order to ensure reproducible results, we have provided an anonymised source code in Supplemental materials with synthetic pendulum dataset, along with the scripts used for data processing and pipelining in this case. The Koopman autoencoder with all combinations of our techniques is included as model, as well as other previous state-of-the-art baselines for comparison. Further, the training script used is included. Using these models and data, training runs were performed several times and averaged. Other datasets are available upon request from the referenced author. The full source code will be released upon conclusion of Open Review.
10

---

Under review as a conference paper at ICLR 2023
REFERENCES
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. Advances neural information processing systems, 32, 2019a.
Zeyuan Alue Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. In International Conference on Machine Learning, pp. 242-252, PMLR, 2019b.
Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In International conference on machine learning, pp. 1200-1128, PMLR, 2016.
Devansh Arpit, Victor Campos, and Yoshua Bengio. How to initialize your network? robust initialization for weightnorm & resnets. Advances in Neural Information Processing Systems, 32, 2019.
Omri Azencot, N Benjamin Erichson, Vanessa Lin, and Michael Mahoney. Forecasting sequential data using consistent Koopman autoencoders. In International Conference Machine Learning. pp. PMLR, 2020.
James Demmel, Ioana Dumitriu, and Olga Holtz. Fast linear algebra is stable. Numerische Mathematik, 108(1):59-99) 91, 2007.
Simon S Du, Chi Jin, Jason D) Lee, Michael I Jordan, Aarti Singh, and Barnabas Poczos. Gradient descent can take exponential time to escape saddle points. Advances neural information processing systems, 30, 2017.
Benjamin Erichson, Michael Muehlebach, and Michael W Mahoney. Physics-informed autoen- coders for lyapunov-stable fluid flow prediction. arXiv preprint arXiv: 1905. 10866, 2019.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256 JMLR Workshop and Conference Proceedings, 2010.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Minghao Han, Jacob Euler-Rolle, and Robert K Katzschmann. DeSKO: Stability robust control with a deep stochastic Koopman operator. In International Conference on Learning Representations, 2021.
Boris Hanin and David Rolnick. How to start training: The effect of initialization and architecture. Advances in Neural Information Processing Systems, 31, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEFE international conference on computer vision, pp. 1036-1034, 2015.
Geoffrey E Hinton. Learning translation invariant recognition in a massively parallel networks. In International Conference on Parallel Architectures and Languages Europe, pp. 1-13. Springer, 1987.
Geoffrey E Hinton and Richard Zemel. Autoencoders, minimum description length and Helmholtz free energy. Advances in neural information processing systems, 6, 1993.
Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Tomoharu Iwata and Yoshinobu Kawahara. Neural dynamic mode decomposition for end-to-end modeling of nonlinear dynamics. arXiv preprint arXiv:2012.06/// 2020.

---

Under review as a conference paper at ICLR 2023
George Em Karniadakis, Ioannis G Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics-inffe informed machine learning. Nature Reviews Physics, 3(6):422 440, 2021.
Giancarlo Kerg, Kyle Goyette, Maximilian Puelma Touzel, Gauthier Gidel, Eugene Vorontsov, Yoshua Bengio, and Guillaume Lajoie. Non-normal recurrent neural network (nnrnn): learning long time dependencies while improving expressivity with transient dynamics. Advances in neural information processing systems, 32. 2019.
Bernard O Koopman. Hamiltonian systems and transformation in Hilbert space. Proceedings of the National Academy of Sciences, 17(5):315 318, 1931.
Anders Krogh and John Hertz. A simple weight decay can improve generalization. Advances in neural information processing systems, 4, 1991.
J. Nathan Kutz, Steven L.. Brunton, Bingni W. Brunton, and Joshua I.. Proctor. Dynamic Mode Decomposition: Data-driven Modeling of Complex Systems. SIAM-Society for Industrial and Applied Mathematics, Philadelphia, PA, USA, 2016. ISBN 161 1974496.
Jason D Lee, Max Simchowitz, Michael Jordan, and Benjamin Recht. Gradient descent only converges minimizers. In Conference on learning theory. pp. 1246-1257. PMLR, 2016.
Yunzhu Li, Hao He, Jiajun Wu, Dina Katabi, and Antonio Torralba. Learning compositional Koopman operators for model-based control. In International Conference on Learning Representations, 2020.
Bethany Lusch, Nathan Kutz, and Steven L Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature communications, 91):1-10,, 2018.
Alexandre Mauroy and Igor Mezić. Global stability analysis using the eigenfunctions of the Koopman operator. IEEE Transactions on Automatic Control, 61(11))3356333) 2016.
Alexandre Mauroy, Yoshihiko Susuki, and Igor Mezić. Introduction to the Koopman operator in dynamical systems and control theory. The Koopman Operator Systems and Control, pp. 3-33. Springer, 2020.
Igor Mezić. On applications of the spectral theory of the Koopman operator in dynamical systems and control theory. In 2015 54th IEEE Conference on Decision and Control (CDC), pp. 7034-7041. IEEE, 2015.
Dmytro Mishkin and Jiri Matas. All you need is good init. In International Conference on Learning Representations, 2016.
Jeremy Morton, Antony Jameson, Mykel J Kochenderfer, and Freddie Witherden. Deep dynamical modeling and control of unsteady fluid flows. Advances in Neural Information Processing Systems, 31, 2018.
Samuel E Otto and Clarence W Rowley. Linearly recurrent autoencoder networks for learning dynamics. SIAM Journal on Applied Dynamical Systems, 18(1):558-593 2019.
Shaowu Pan and Karthik Duraisamy. Physics-informet probabilistic learning linear embeddings of nonlinear dynamics with guaranteed stability. SIAM Journal on Applied Dynamical Systems, 19 (1):480-509, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer. James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. Advances in neural information processing systems, 30, 2017.
Luis Perez and Jason Wang. The effectiveness of data augmentation in image classification using deep learning. arXiv preprint arXiv 2017.
12

---

Under review as a conference paper at ICLR 2023
Richard W Reynolds, Thomas M Smith, Chunying Liu, Dudley B Chelton, Kenneth S Casey, and Michael G Schlax. Daily righ- blended analyses for sea surface temperature. Journal of climate, (DN22):5433-599) 2007.
Andrew M. Saxe, James L. McClelland, and Surya Ganguli. Exact solutions the nonlinear dynamics of learning in deep linear neural networks. In International Conference on Learning Representations, 2014.
Peter Schmid. Dynamic mode decomposition of numerical and experimental data. Journal of fluid mechanics, 656:5-28, 2010.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 19(1):1929-1 1958, 2014.
Dominik Stöger and Mahdi Soltanolkotabi. Small random initialization is akin spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstruction. Advances in Neural Information Processing Systems, 34:23831 2021.
Steven H Strogatz. Nonlinear dynamics and chaos: with applications to physics, biology, chemistry, and engineering. CRC press, 2018.
llya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pp. 1139-1144 147. PMLR, 2013.
Naoya Takeishi, Yoshinobu Kawahara, and Takehisa Yairi. Learning koopman invariant subspaces for dynamic mode decomposition. Advances in Neural Information Processing Systems, 30, 2017.
Jared Willard, Xiaowei Jia, Shaoming Xu, Michael Steinbach, and Vipin Kumar. Integrating physics- based modeling with machine learning: survey, arXiv preprint arXiv:2003.04919 1-1), 2020.
Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Conference on Learning Theory, pp. 3635-3673. PMLR, 2020.
Enoch Yeung, Soumya Kundu, and Nathan Hodas. Learning deep neural network representations for Koopman operators of nonlinear dynamical systems. In 20/9 American Control Conference (ACC), pp. 4832-4839. IEEE, 2019.
Yuichi Yoshida and Takeru Miyato. Spectral norm regularization for improving the generalizability of deep learning. arXiv preprint arXiv: 1705. /0941, 2017.
Hao Zhang, Scott Dawson, Clarence W Rowley, Eric A Deem, and Louis N Cattafesta. Evaluating the accuracy of the dynamic mode decomposition. arXiv preprint arXiv11710.00745,00 2017.
13

---

Under review as a conference paper at ICLR 2023
A HYPERPARAMETERS FOR MAIN RESULTS
Table 2: Hyperparameters for results in Fig. and Tab.
Noteably the ocean and pendulum datasets had uniform distributions between 0.9 and rather than between 0 and 1 as it worked slightly better.
Dataser
Operator Size
Eigeninit
Eigenloss
Pendulum
0
10a
Fluid
16
1
20
Ocean
16
0.7
2
Cyclone 16
0.7
20
14

---

Under review as a conference paper at ICLR 2023
B
EIGENVALUE DISTRIBUTION HEATMAPS
Gaussian Element
Eigeninit (Spike and Slab)
Index of Eigenvalue
0
1.00
Index of Eigenvaluu
3
0.4
3
6
0.75
9
0.2
15 12
15 12
0.50
10 20 30 40
50
60
70
80
90
0
10 20 30 40 50 60
80
Epochs
Epochs
Eigenloss (Unit circle MSE)
Both
Index of Eigenvalue
0
0.4
Index of Eigenvalue
0
-1.0
3
3
6
0.3
9
9
0.5
15 12
0.2
15 12
10 20 30 40 50 60 80 80
90
0
10 20 30
40
50 60
70 80
Epochs
Epochs
Figure 6: Change in eigenvalue magnitudes over time.
15

---

Under review as a conference paper at ICLR 2023
C CONVERGENCE STATISTICS
The convergence of solutions also improves with eigeninit and eigenloss. One can see this in Tab. where the approximate epoch of convergence is noted for different data and scheme. Convergence was defined based on the ratio of the maximum validation difference between two points and the current validation difference between two points (excluding the first epochs).
Table Epoch of model convergence for benchmark data sets. See bold for minimum.
Datasel
Regular KAE
Eigeninit
Eigenloss
Both
Pendulum
15
13
Fluid
42
9
33
11
Ocean
13
5
18
Cyclone
10
5
11
6
16

---

Under review as a conference paper at ICLR 2023
D WEIGHT DECAY INEFFECTIVENESS
Ocean
0.11
0.07
0.36
0.05
Veldafen Doo
0.28
0.00
10
15
25
30
Epach
Cyclone
0.01
16-
12-
30
Epoch
Figure 7: Minimal effect of weight decay in regularising the spiky loss, particularly for eyclone.
17

---

Under review as a conference paper at ICLR 2023
E ABSOLUTE WALL-TIME PLOTS
Ocean
0.10
penalty
Eigenloss only
Eigenloss and eigeninit
0.08
Forward validation
0.06-
0.04
0.02-
10
20
30
40
50
Wall time (s)
Cyclone
24-
No penalty
Eigenloss only
22-
Eigenloss and eigeninit
Forward validation loss
20-
18-
16-
14
12-
10
8-
50
100
150
200
250
300
350
Wall time (s)
Figure 8: Comparing convergence time the ocean dataset; this can be seen measure of the computational complexity.
18


Reviews:
[{"review_id": "vXiDeWwDp6b", "reviewer": "Reviewer_ibgi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is about improving Koopman autoencoder models in two ways: (1) a better initialization of the Koopman matrix by directly changing the eigenvalues, and (2) penalizing the eigenvalues of the Koopman matrix during training. These changes are tested on 4 datasets (some synthetic & some real-world), showing improved convergence and long-term prediction error.", "summary_of_the_review": "I think that this paper presents a couple of interesting ideas and shows results that are sometimes better than the Azencot et al. (2020) paper. However, I think that the eigeninit is not doing what they think it is: it seems that the eigenvalues will change when they take the real part of the new matrix. I believe there are multiple incorrect or exaggerated statements. Also in order to improve upon the Azencot et al. (2020) paper, many models were trained with variants of the new ideas, so (a) the improvements may be mostly due to having many models to pick from, and (b) since there aren't consistent winners, there may need to be a lot of tuning to use these ideas.  ", "strength_and_weaknesses": "**Strengths:**\n\nI think it makes sense that if you have some physical knowledge of the system, you could use that to influence the eigenvalues of the Koopman matrix and have better results. I appreciate that this was tested on four datasets, which is a pretty good number. The improvements over the Standard KAE are pretty strong.\n\n**Weaknesses:**\n\nMy primary concern is correctness: When the initial Koopman matrix is defined with new eigenvalues, it may be complex. The real part of each matrix element of A is kept. Does that change the eigenvalues, or are they still \\lambda^{\\tilde}? I tried a small test, and it seems that the eigenvalues change. Please let me know if I'm missing something. I set A = [1, 2; 3, 4]. The eigenvalues are approximately  -0.3723 and 5.3723. I changed the second eigenvalue to 0.5 + .1i and reconstructed the matrix: A_new = V D_new inv(V). The eigenvalues are as expected, but the matrix is complex. I took the real part: real(A_new) and the eigenvalues change to approximately -0.3723, 0.5. \n\nIt would be nice to have more benchmark methods to compare against. In lieu of that, I think some of the language should be softened. I think it's overstatement to say that the results are compared against \"several state-of-the-art baselines.\" Similarly \"was introduced by Azencot et al. (2020), yielding stable Koopman systems and state of the art performance\" is an overstatement, since that paper only compares to one of many previous papers. (There are many papers with Koopman autoencoders cited in this paper.) Further, the Azencot et al. (2020) comparison to the Lusch et al., 2018 paper is not a complete one, since they left off the key feature of letting eigenvalues vary for a continuous spectrum. \n\nIt's mentioned that \"backpropagation through the eigenvalues of a matrix is numerically stable if there are no repeating eigenvalues.\" It sounds expensive to backpropagate through an eigendecomposition. How fast is this method? Does converging in fewer epochs come at a cost of each epoch being much slower? \n\nAlthough this paper shows improved results with the eigenloss and/or eigeninit approaches, there's a lot of a variability in whether it's best to use one or the other or both, and the improvements are much more modest when compared to the consistency baseline. The only column that is better than the consistency baseline more than half of the time in terms of test data the init-only column. Looking at Table 1 & Figure 4, if you were to try all of these approaches for a new dataset and pick the best approach based on the validation loss, that would usually not correspond to the best approach based on the cumulative test error. Is this because of differences in loss vs. error, or are there variable amounts of overfitting? \n\nI also have some comments in the next section. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "mUtfxfvCPFh", "reviewer": "Reviewer_eMgi", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents an initialization strategy for Koopman operators and a loss regularization in terms of spectral information for Koopman autoencoders. For initialisations, the paper claims that approximated Koopman operators should be initialized such that their spectral norm should be less than 1. For regularization, the loss is penalized such that the eigenvalues are concentrated on the unit circle.", "summary_of_the_review": "In general, the paper proposes an nteresting way for egeinvalue intialization and regularization for Koopman operators in dynamic systems. However, the experiments should be polished more to support the claims.", "strength_and_weaknesses": "### Strength\n- The paper is well-motivated in the introduction and method section.\n- In my opinion, regularizing eigenvalues is an interesting idea. This is possible for the Koopman operator because of its linearity. \n\n### Weaknesses\n- The paper shows an empirical approach, lacking theoretical investigations.This does not mean that I do not appreciate the paper contributions.\n- Even as an empirical study, the experiments of the paper fall short and are not comprehensive enough. For example, the paper may need to ablate the effect of changing the weight of $\\epsilon_\\lambda$. It’s not clear what the initialization strategy (among 4), and the regularizer (among 2) are used in the main text (Table 1, 2 and Figures 4).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5CbNDPUkj8C", "reviewer": "Reviewer_AUHg", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates the regularization and initialization of the Koopman Autoencoders (KA). KA aims at learning the lifting functions to transform the input dynamical system to the space where the state of the system evolves linearly. Existing methods for well-conditioning the KA rely on specific parameterizations (block-diagonal, tri-diagonal) or indirectly regularize the dynamics (Lyapunov-based, forward-backward soft constraints). Motivated by the fact that KA with stable dynamics has spectral radius with a magnitude less than one, the authors propose to initialize the Koopman layer transition matrix to have eigenvalues near one, and to regularize the model during training to keep its eigenvalues near one. Results are shown on synthetic and real datasets showing that the proposed initialization and regularization scheme outperforms the alternatives on the test data.", "summary_of_the_review": "The paper proposes a timely solution to an important problem which is well-conditioning the Koopman layer in the Koopman Operators. Overall the paper has good quality, however, the results and experiments can be improved to make the case stronger. I'm happy to increase my score to acceptance if my detailed comments above are addressed.", "strength_and_weaknesses": "**Strengths**\n* The problem studied is timely and significant, and is relevant to the ICLR community.\n* The introduction and related work sections contain a nice summary of the existing theoretical and empirical results on the initialization and regularization.\n* The figures and schematics are clear, helping the reader to follow the arguments. The background on the Koopman Operators (KO) is sufficient and easy to follow.\n* Overall, the logical flow of the paper is well-presented. Starting from the intro and background, the problem considered is motivated and connections to the existing literature are mentioned. The presented methods are logical consequences of the existing gaps.\n* The presented results specifically on the real datasets help support the arguments.\n\n**Weaknesses**\n* I understand that the stable dynamics have a KO with a spectral radius below one, but if the dynamics evolve on a lower-dimensional space some eigenvalues can be closer to zero in magnitude while there are a few eigenvalues with near one magnitude. There seems to be a need to identify the intrinsic dimension of the system in order to find what is the right number of near-one eigenvalues. Did the authors experiment with the effect of model mismatch in this sense? Meaning that if the number of near one eigenvalue considered is not consistent with the intrinsic dimension f the system. And how this affects the overall quality of the results.\n* There is no mention of the computational complexity of the system. The way I imagine this to be implemented is either using a parameterization that directly incorporates eigenvalues and the regularizations therein. In this scenario, the errors are backpropagated directly through the parameters (eigenvalues and eigenvectors) but the downside is that in every forward iteration one needs to compute the weight matrix. The second scenario (which is less likely) is if the weight matrix is parameterized in the traditional way, but at every iteration, the eigendecomposition is computed to evaluate the regularization. In both cases, there seems to be some computational burden with jointly initializing or regularizing all the weights. I would like the authors to clarify this (and possibly add a paragraph to the paper) and change the x-axis of the right-hand side plots on Fig. 4 to wall clock time instead of epochs.\n* The authors motivate the closeness of the eigenvalue magnitudes to one, but at the bottom of page 5, the distributions considered seem to be a bit problematic. First, doubleGaussianEigen is not a distribution (it should be divided by 2, right?). Second, gaussianEigen and uniformEigen do not seem to encourage the eigenvalues to be of magnitude one. Third, unitPerturb seems to enforce all the eigenvalues to be near one magnitude. Based on my comment regarding intrinsically low-dimensional systems this seems to be a limiting factor for modeling those types of dynamics. Can the authors include a spike and slab type prior and report the results?\n* My biggest concern about this paper is with respect to the experiments. Below I elaborate on how the results section can be improved.\n    * It is argued that the weight decay regularization is not compatible with the assumptions of stable dynamics. But in the experiments, no comparison against weight decay regularization is presented. Can the authors include the comparisons and ablations (different combinations of initialization (eigeninit, He, Xavier) with regularization (eigenloss, l1, l2)?\n    * The results are shown on limited architectures and the effect of the nonlinearity, width, depth, and architectural choices are not investigated systematically.\n    * How is the dimension of the KO layer chosen? Can the authors include dimension mismatch studies showing that the method still outperforms alternatives when the wrong dimension is chosen?\n    * In Fig. 4, on the right-hand side plots, it seems that all the methods eventually get to similar test errors, if so it's important to compare the time that it takes for different methods to get there. As I suggested above, changing the x-axis to wall clock time will address this. For each method, which epoch is chosen for the results on the left side plot? If the criterion for choosing a  model is the epoch number then the left-side plots and right-side plots seem to deliver the same message. What's more interesting is to change the criterion to convergence. This can address the following question: once all the models achieve similar test performance, which one learns a better dynamics model and generalize better to more time points?\n \n**Minor Comments**\n* What does the following line mean and how is this solved by the presented methods?\n> Importantly, backpropagation through the eigenvalues of a matrix is numerically stable if there are no repeating eigenvalues.\n* Can you include standard (with some sort of normalization that is comparable across different datasets) units on the y-axis of the plots of figure 4?\n* Is the following paragraph systematically tested? While it sounds very interesting I would appreciate it if the authors do a systematic experiment where the amount of dissipation changes in the data generating system and show that different values of the hyperparameter are optimal with varying dissipation.\n> For instance, if the underlying system is measure-preserving, we can weight $\\epsilon_{\\lambda}$ more strongly with respect to the reconstruction and prediction losses. In contrast, if the dynamics are dissipative, we can use a weaker weight.\n* In the last paragraph of page 3, shouldn't $MSE(x_k, \\hat{x_k})$ be changed to $MSE(x_k, \\hat{x}_{k+1})$?\n* Are the methods only developed for autonomous systems? Can this be extended to input-driven systems (which might be the case for many physical systems)? What would change if one was interested in training a KA with intrinsic dynamics, inputs, and outputs? Do we still need to encourage the eigenvalues to be near one?\n\n## Post Rebuttal\n\nI thank the authors for the clarifications and for taking my comments into account for their revision. Specifically, I appreciate that the authors included the suggested prior, did the wall-clock experiments, included the discussion about the intrinsic dimension of the system, and normalized the y-axis in the results figure.\n\nGiven the new plots, I am not quite convinced that the method outperforms alternatives. Fig. 3 which contains the experimental summary of the paper shows that all methods achieve similar validation performances. Given the noisiness of the validation loss plots, it's hard to conclude that the proposed method achieves better results. If the test and validation data have similar statistics the level of noise observed across epochs in the validation loss plots should appear in the test data as well. In addition, the training/validation plots and results of the test data in the table are not accompanied by their corresponding variances. There are multiple ways to include this information, but this is done commonly using random seeds (since the eigenvalues are sampled from spikeAndSlab randomly every initialization will change the eigenvalues which can give rise to training statistics). This is specifically problematic given the computational burden of the eigenLoss raising the question that given a similar computational budget (memory and time) which method achieves the best performance when the within-model variance is accounted for? The experiments included in the supplementary (section E) do not show a clear trend of eigenLoss or eigenInit being helpful for the presented datasets. In the Ocean dataset \"No penalty\" model performs similarly to \"Eigenloss and eigenInit\" and in the Cyclone dataset, there's so much noise in the validation loss that it's hard to determine between \"No penalty\" and \"Eigenloss and eigenInit\" which one is doing significantly better (also the losses seem very small even in the first time point which is strange).\n\nIn the rebuttal round, the authors fixed the conceptual error raised by the other reviewer and re-did all the experiments. The paper is fully restructured and the figures are changed, given that the changes are major, I do believe that the submission requires another round of careful reviewing. Moreover, it's unfair to other submissions since the authors used more time for the major changes. Given this, I tend to keep my score as is. I do believe however that the paper is well motivated, and careful experimentation may better represent the merits of the eigenLoss and eigenInit. Therefore I'd encourage the authors to do another round of polishing and submitting elsewhere if not accepted here.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject