Learning Chaos In A Linear Way
Authors: Xiaoyuan Cheng
Yi He
Yiming Yang
Xiao Xue
Sibo Cheng
Daniel Giles
Xiaohang Tang
Yukun Hu
Venue: ICLR 2025 Poster (2025)
Keywords: Dynamical systems
operator learning
chaos
physics-informed learning
Primary area: learning on time series and dynamical systems

Abstract: Learning long-term behaviors in chaotic dynamical systems, such as turbulent flows and climate modelling, is challenging due to their inherent instability and unpredictability. These systems exhibit positive Lyapunov exponents, which significantly hinder accurate long-term forecasting. As a result, understanding long-term statistical behavior is far more valuable than focusing on short-term accuracy. While autoregressive deep sequence models have been applied to capture long-term behavior, they often lead to exponentially increasing errors in learned dynamics. To address this, we shift the focus from simple prediction errors to preserving an invariant measure in dissipative chaotic systems. These systems have attractors, where trajectories settle, and the invariant measure is the probability distribution on attractors that remains unchanged under dynamics. Existing methods generate long trajectories of dissipative chaotic systems by aligning invariant measures, but it is not always possible to obtain invariant measures for arbitrary datasets. We propose the Poincaré Flow Neural Network (PFNN), a novel operator learning framework designed to capture behaviors of chaotic systems without any explicit knowledge of the invariant measure. PFNN employs an auto-encoder to map the chaotic system to a finite-dimensional feature space, effectively linearizing the chaotic evolution.  It then learns the linear evolution operators to match the physical dynamics by addressing two critical properties in dissipative chaotic systems: (1) contraction, the system’s convergence toward its attractors, and (2) measure invariance, trajectories on the attractors following a probability distribution invariant to the dynamics. 
Our experiments on a variety of chaotic systems, including Lorenz systems, Kuramoto-Sivashinsky equation and Navier–Stokes equation, demonstrate that PFNN has more accurate predictions and physical statistics compared to competitive baselines including the Fourier Neural Operator and the Markov Neural Operator.

Published as a conference paper at ICLR 2025
LEARNING CHAOS IN A LINEAR WAY
Xiaoyuan Chengl Yi He Yiming Yang!. Xiao Xue
Sibo Cheng'
Daniel Giles+ Xiaohang Tang² Yukun Hu
'Dynamic Systems, University College London, United Kingdom SStatistical Science, University College London, United Kingdom
SCEREA, ENPC and EDF R&D, Institut Polytechnique de Paris, France
"Computer Science, University College London, United Kingdom
i Equal contribution Corresponding author (yukun. www.hluuu.....o ukk
ABSTRACT
Learning long-term behaviors in chaotic dynamical systems, such turbulent flows and climate modelling, is challenging due to their inherent instability and unpredictability. These systems exhibit positive Lyapunov exponents, which signif- icantly hinder accurate long-term forecasting. As a result, understanding long-term statistical behavior is far more valuable than focusing on short-term accuracy. While autoregressive deep sequence models have been applied to capture long-term behavior, they often lead exponentially increasing errors in learned dynamics. To address this, we shift the focus from simple prediction errors to preserving an invariant measure dissipative chaotic systems. These systems have attractors, where trajectories settle, and the invariant measure is the probability distribution on attractors that remains unchanged under dynamics. Existing methods generate long trajectories of dissipative chaotic systems by aligning invariant measures, but it is not always possible to obtain invariant measures for arbitrary datasets. We propose the Poincaré Flow Neural Network (PFNN), novel operator learning framework designed to capture behaviors of chaotic systems without any explicit knowledge the invariant measure. PFNN employs auto-encoder map the chaotic system to a finitedimmensional feature space, effectively linearizing the chaotic evolution. It then learns the linear evolution operators match the physical dynamics by ad- dressing two critical properties in dissipative chaotic systems: (1) contraction, the system's 's convergence toward its attractors, and (2) measure invariance, trajectories on the attractors following a probability distribution invariant to the dynamics. Our experiments on a variety of chaotic systems, including Lorenz systems, Kuramoto- Sivashinsky equation and Navier-Stokes equation, demonstrate that PFNN has more accurate predictions and physical statistics compared to competitive baselines including the Fourier Neural Operator and the Markov Neural Operator.
INTRODUCTION
View Understand a Chaotic System. Imagine a box filled with 10a1 identical gas molecules,
each with specific positions and momenta. Physicists (Szász 1996 Zund 2002) posed the question:
Suppose the system starts in a certain state; will it eventually return to a state arbitrarily close to the initial one?
The question seems intractable for an enormous number (o 1024) of coupled Hamiltonian equations governing the system evolution. However, Poincaré discovered that the system's 's long-term behavior could be understood without explicit solutions (Poincaré 1928). He showed that in a closed system with finite measure, almost all initial states would eventually return close to their starting conditions (detailed proof refers to C.I). This is shown to be the case because the probability measure of the phase space is invariant under the dynamics. This led to the development the mean ergodic theorem (Birkhoffl 1927] Koopman & Neumann, 1932 Neumann. 1932a Birkhoff & Koopman 1932). which asserts that in ergodic systems, time averages converge to space averages. Standing on the shoulders of these scientific giants, we recognize that while solving stepwise precise solutions for

---

Published as a conference paper at ICLR 2025
chaotic dynamics may be intractable, adopting measure- theorreie view can significantly simplify the problem.
Understanding the behaviour of chaotic systems remains crucial for applications in weather forecast- ing, climate modelling, and fluid dynamics (Tang et al. 2020). Over the past decade, forecasting the long-term behavior of chaotic systems has posed substantial challenge to the machine learning com- munity (Yu et al. 2017; Mikhaeil et al., 2022] Wan et al.) 2023] Yang et al. (025). These dynamical systems are characterized by instabilities and sensitivity initial conditions. A small perturbation to any given state can cause trajectories to diverge exponentially, a phenomenon attributed to positive Lyapunov exponents. Consequently, accurately predicting the long-term trajectory of such systems is non-trivial To tackle these challenges, two primary streams have emerged in learning dynamical systems: 1) deep sequence models, which leverage sequential neural networks to learn the temporal patterns, and 2) operator learning, learning the interrl-differential operators directly without knowing the differential equations.
Deep Sequence Models. Previous studies have focused on predicting short-term dynamics, mainly
using deep sequence models by minimizing the mean squared error (MSE) of the next-step prediction.
Commonly used models include recurrent neural networks (RNNs) (Lipton et al. 2013 Vlachas
cet al. 2020). long short-term memory networks (LSTMs) (Mikhaeil et al.
2022). reservoir computing
(Pathak et al. 2018: Tanaka et al.) 2019) and Transformers (Woo et al. 2024). These approaches
have been applied to classic examples like the Lorenz 63 system (Lorenz, 1963) and turbulent flows
(Pathak al., 2018). However, due to the instability of trajectories, these methods often suffer from exponential error accumulation (Ribeiro et al.) 2020), which hampers their ability to model chaotic systems effectively. To stabilize predictions, various strategies have been proposed, such as constraining the recurrence matrix to be orthogonal (Helfrich et al. 2018 Henall et al.) 2016)) skew-symmetric (Chang al. 2019), or ensuring globally stable fixed point solutions (Kag et al.. 2020). However, these methods struggle with simulating chaotic systems since chaotic systems are usually hyperbolic and aperiodic (Hasselblatt & Katok| 2002). To address this issue, specialized models tailored to chaotic systems are necessary to accurately capture their complex patterns and behaviors. Recognizing that dissipative chaotic systems with attractors exhibit ergodic behaviours, new sequential models have been developed that go beyond simply minimizing MSEs. For instance, Jiang et al. (2024) improve the learning of chaotic systems by incorporating transport distances to the invariant measure, which is assumed to be known. This ensures that the model's predictions aligned with the true distribution of the system. The methods proposed in these two papers (Jiang al. 2024 Schhff' et al. 2024) generate samples directly from the neighborhood of the attractors and estimate the invariant probability distribution from these generated samples. Differently, |Schiff et al. (2024) removed the need for knowledge of the invariant measure of underlying systems, enabling direct measurement of the invariant distribution from trajectories. By incorporating the estimated invariant distribution as a regularized transport term, they train deep sequence models by enforcing long trajectories to match the estimated invariant distribution. However, significant challenge for these methods lies in accurately estimating the invariant measure on attractors from arbitrary datasets, where this invariant distribution is usually unknown.
Operator Learning. Another prominent approach to studying the long-term behavior of dynamical systems is through operator theory. Two classic methods in this domain are the transfer operator (Demers& & Zhang 201T) and the Koopman operator (Bevanda et al. 202I). The transfer operator captures the evolution of probability density functions in chaotic dynamical systems, making it powerful tool for analyzing statistical mechanics (Lagro et al.) 2017), chaos (Jiménez 2023). and fractals (Ikeda et al., 2022). The Koopman operator, often considered the adjoint of the transfer operator, focuses on the evolution of feature functions. Both of these methods primarily aim to capture the long-term behaviors of chaotic dynamical systems with an invariant measure (Adams & Quas| 2023 Das et al. 2021 Valva & Giannakis] 2023). Although the dynamics on invariant sets govern long-term behavior (Mori & Kuramoto Omsteei 1989 Li et al. 2017). focusing solely on ergodic behaviors can sigmificanlly distort short-term predictions. Traditionaly, the Koopman operator is approximated using kernel methods (Ikeda et al. 2022, Kostic et al. 2022) or dynamic mode decomposition (DMD) (Williams et al., (2155] Takeishi et al. 2017). Recently, Koopman learning has emerged as a prominent approach for modeling the dynamics of differential equations by leveraging autoencoder architectures to approximate a finite-rank linear operator within a learned feature space (Lusch et al.) 2018 Nathan Kutz et al., 20188) Brunton & Kutz, 2023). This approach has facilitated the development of various methods for capturing general dynamical behaviors, such
2

---

Published as a conference paper at ICLR 2025
time series (Wang al. 2022 Liu al., 2024) graph dynamics (Mukherjee et al. 2022) However, no specific Koopman learning framework has been designed effectively capture the behaviors of dissipative chaos. Integrating deep learning techniques with operator theory has led to the development various neural operators aimed at solving differential equations. For instance, Deep Operator Networks (DeepONet) (Lu al. 2019)
universal approximators for initial value problems, while Fourier Neural Operators (FNO) et al. 2020; Kovachki et al., (2221) learn deep Fourier features minimize loss functions in Sobolev space, effectively solving differential equations with high-order terms. Both DeepONet and FNO aim to capture accurate solution operators for initial value problems in fefnite-dimensionee function space. However, chaotic systems often require analysis beyond traditional initial value problems with unstable solutions, necessitating specialized architecture to address. address this, the Markov Neural Operator (MNO) (Li et al. 2022) improves long-term predictions for dissipative chaotic systems by introducing concept called an "absorbing ball." This "absorbing ball" acts like boundary that ensures the chaotic system does not go completely off course over time, guiding it back to stable range. However, choosing the right size for this absorbing ball is tricky. If the ball is too large, it might make inaccurate predictions, as the system could stray too far from the actual behavior.
To address the aforementioned challenges, we propose the Poincaré Flow Neural Network (PFNN) established Koopman theory, novel model designed to capture both dynamics and long-term behaviors of dissipative chaotic systems without requiring prior knowledge or assumptions of the invariant measure from the raw data. Unlike the previous Koopman deep learning methods for general dynamics (Li et al. 2017, Lusch et al. 2018 Nathan Kutz et al. 2018 Brunton & Kutz 2023). our method tailored learn dissipative chaos by embedding intrinsic physical properties. Specifically, dissipative chaos can be separated into two phases based on the physical properties: a) Contraction Phase: where the system converges toward attractors due to energy dissipation, which governs transient short-term dynamics; b) Heasure-Invarian Phase: where the system fully explores bounded invariant set according to the invariant distribution over time, enabling the derivation of statistical properties from time averages (ergodic behaviours). PFNN learns a linear contraction operator and a unitary operator to accurately model in an corresponding physical behaviors of the contraction phase and ceaasre-invariint phase (as shown Figure T). respectively. Our approach avoids relying on a known invariant measure for guiding long- term predictions. Instead, employ simple one-step forward learning, regularized with physical constraints (contraction and unitary) on the linear operators.
feature space the
Trajectories
Invariant Set
Probulilyy desaity
Prebubiity denily
deautt
col
6.30
M22
Contuniaol Mednore ploce
invorisos
Contaction Mossar
Ievaiiset piltul
Conmrntion Mesust
plase Inversint
plose
piose
photo
(a) Trajectory samples
(b) The evolution of probability density function of three dimensions
Figure 1: Contraction and measure-invariant phase in chaotic dynamics (Lorenz 63 system): (a) trajectories from random initial states all spiral toward attractors, ultimately moving within the butterfly-shaper invariant set; (b) the probability density of the trajectories samples in each state dimension keeps changing over the earlier evolution, which indicates contraction phase in the beginning; whilst the probability density becomes invariant over the later evolution, which echos the states finally move consistently within the invariant set.
2 BACKGROUND AND PROBLEM FORMULATION
Notation. (X, B, μ.) denotes the measure space, with set X, Borel o B and measure μ.. The forward map on the state space is denoted as T. The Lebesgue space with p-norm is represented as CX(X, p)), which is abbreviated as L in this paper. Specifically, the Lebesgue space L is equipped with an inner product structure as (..)). The functions o, are denoted as feature functions in the L space. 7f denotes the transpose of real matrix, and D denotes the conjugate transpose.
3

---

Published as a conference paper at ICLR 2025
O()) denotes an asymptotically tight upper bound while o(-) indicate that the upper bound is not asymptotically tight. The ceiling function is denoted as [>] = min E Z x S n).
Problem Setup. In what follows, focus forecasting dissipative chaotic dynamical systems that can be described as
2++1 (=*), 2 E I"".
(1)
where denotes the state of the system at time &o E IN and . is bounded space. The forward map T: : M M is nonlinear, pushing states from time k to time
The above model assumes the states depend only on the current state 2₁ regardless of information from long historical states.
To forecast future states, one might use neural network to approximate the forward map T. However, the resulting model ignores prior knowledge related to the problem, and it can be challenging to analyze. In contrast, our model based on learning the evolution of latent features way. In this framework, the evolution of the system becomes linear in the function space L2, Such an evolution map is the so-called neural operator. Our model adopts the Koopman operator (Koopman 1931), the forward map I" induces a linear operator G as
_ in a linear
G =o
where is the feature map as .M + R. with o F C being function space F on M. Essentially, the operator G describes the linear evolution of factorized features. Consequently, future states of the features can be obtained by iteratively applying G. More specifically, the feature ((zx) is mapped to p(*++1) under the operator g. The evolution of the entire feature space can be expressed
linear combination of features:
G(aιo + a202 + +...) = (a1œ1 + 0202 + .....) o
where og E IR is the weight corresponding to the feature functions di.
Definition 2.1 (Measure Preserving Transformation and Ergodicity (Walters) 2000)) Let (X, B, p. T) be measure preserving transformation (MPT), meaning that for every E. E B, the measure satisfies p(T E)= =(E). MPT is said to be ergodic if. for any invariant set E, either ((E) = or p(X X E) = O. In this case, is referred to as an ergodic measure.
Ergodicity T : X + X ensures that u(T-²E) = ((E) + ((E) = {0, 1}, meaning the system almost surely follows the behavior described by the invariant measure. Simply speaking, the ergodicity implies that spatial statistics are the same as the temporal statistics.
Definition 2.2 (Global Attractor (Hasselblatt & Katok, 2002)) A compuct, invariant set is called global attractor if. for any bounded set B M and existing a time k* e N such that k >> k* such that T (B) is contained within the neighbourhood of A.
In this paper, we aim to forecast dissipative chaotic dynamical systems by considering invariant measures on attractors. Given an arbitrary initial state, such systems typically converge to their attractors (an interpretation of this can be found in Appendices B. and B.2). Due to the dissipative nature of the dynamics, the phase space contracts, meaning that all trajectories will eventually approach specific regions within the phase space. The contraction reflects energy dissipation from physical standpoint. Once the system enters the neighbourhood of an attractor, becomes ergodic, and the system's 's trajectories distribute across the attractor following an invariant measure that does not vary over time (see II.). Many dynamical systems have been proven to exhibit this property, including the Lorenz 63 (Tucker 1999), Lorenz 96 (Maiocchi et al. (024)) and Auramoto-Sivashing (Temam 2012). In other words, the trajectory of a chaotic system will span the attractor in a way that accurately reflects the statistical properties of the entire invariant set, as shown in Figure []
METHOD
In the following sections, we introduce the Poincaré Flow Neural Network (PFNN) designed for forecasting dissipative chaotic dynamical systems. PFNN consists of two steps: first, mapping the system into miite-dimensional feature space using an auto-encoder (AE); second, embedding physical properties in the finite-rank operator to capture behaviors contraction and measure- invariant phases.
4

---

Published as a conference paper at ICLR 2025
3.1 FINITE-RANE OPERATOR APPROXIMATION
A popular approach for operator learning G C + L2 is to construct a Unite-dimensional approxi- mation. Specifically, convergent approximation method for prediction problems can be developed by utilizing the fact that G is a bounded (and therefore continuous) línear operator, without explicitly considering its spectral properties. Given an arbitrary orthonormal basis {Φ1, $2....3 of L we define the operator's s orthogonal projection
I L + span f. - The finite-rank opera- tor is then given by Gr = 11, I,I,, where the operator Gr, is characterized by the matrix elements G1g.2 (0, GØz) with 1 S j S L. Due to the continuity of G, the sequential predictions of the operator Gr. converge pointwise those of G on L²
The finite-dimnniinal feature space is spanned by learned adaptive features of the neural networks used in this paper. The learned encoder and decoder are denoted as 9ø, and g8.- with parameters A and 02 respectively. Specifically, the encoder 98, functions as the projection operator onto the learned latent feature space, effectively embedding the state space into the function space. The decoder 902 then maps the feature space back to the original space. In this setup, the operator G and 01. 0g are learned jointly by minimizing the following loss function:
arg min EEZD [1[95]] (2x) 96, (2*+1)ll + 1lol,, 96 (zx) - z4 =x|12]; G,0,00
where G is the approximated finite-rank operator, and y is data distribution. The first term in Equation represents the prediction loss, while the second term accounts for the reconstruction loss with a coefficient ? denoted as I reee. To ensure consistent solution in the feature space, the auto- encoder must satisfy the bijective property, meaning that encoder and decoder satisfy the constrained relationship Ta 96, o 98, where Ia denotes the identity matrix. For the class of finite-rank operators, we prove Theorem 3.TT regarding the PFNN. This result shows that the finite-rank operator can approximate the ciinite-dimensiona operator with a sufficiently small error.
Theorem 3.1 (Convergence of Finite-Rank Operator) Let M C Ihm be compact set, the solu- tion operator G L + L² associated to the dynamics is locally Lipschitz. Then, for a sufficient large dimension L E N and a sufficiently small number > O, there exists approximated finite-rank operator g L L converging pointwise such that sup=EM [19 (2) Go(2))l2 S.
The theorem reflects that the finite-rank operator, derived from the infiniteddimensinal operator, operates in a function space spanned by learned latent feature functions. By selecting sufficiently large L as the dimension of the latent feature space, stepwise prediction can be achieved (see the procedure in Appendix(C) CC. While the existence of the finite-rank operator can be proven, relying solely on this approximation for time-stepping can lead to exponential error accumulation. To address this, long-term predictions can be improved by incorporating physical properties. In the following sections, introduce loss functions designed to enhance long-term prediction accuracy. To distinguish the different operators, those applied during the contraction phase and the measure-invariant phase are denoted Go and Gm. respectively.
3.2 STAGE I: CONTRACTION PHASE
In accordance with the definition of attractors in Section 2.2] a dissipative system induces a contraction operator (Lumer& & Phillips, 1961) due to the presence of attractors. In this stage, there exists volume contraction. It means any bounded set containing the invariant set A will contract towards constrain the spectral properties of the operator G... ensuring || || < 1. Specifically, we focus on GT Ge instead of G. because G Ge is symmetric (Franklin 2012) with real, non-negative eigenvalues, and enforce the condition:
A. To characterize this contraction,
(G.d, G.d) S > 0<A51.
(5)
Eigenvalues of G. with a modulus bounded by 1 reflect the contraction (refer to Lumer-Philips theo- rem in Appendix D. To enforce these contraction properties in practice, we introduce specialized
loss function as
arg min Eg [118-98 (zx) 98, (zx+1)) 1)l + 7I con + 7I rec]
(6)
0,,0
5

---

Published as a conference paper at ICLR 2025
where the contraction term is defined as coon == -e Ia)) = max (ol0 Ia). with coefficient ?1 E (0, 1). (-) is denoted as eigenvalues of operator, and ReLU g. - 1a)) constraints the negative semidefinite of G Se La- The I con encourages the contraction of the state space by making the eigenvalues of Ć Ge larger than one, based on Equation 5 The data distribution y is lying in the bounded set as M \ A, which encourages the state space to contract towards the invariant set A from outside.
However, the invariant set of the global attractor is typically not directly observable in the raw dataset. To address this, one can utilize the dissipation property to effectively truncate the dataset, preparing it for training various operators. Dissipative chaotic systems take time to approach their invariant distribution (Znidarič 2015), which corresponds to the relaxation time in physics (or mixing time in ergodic theory). The relaxation time is bounded by the log-Sobolev time (Bauerschmidt & Dagallier 2024; Mori & Shirai 2020), which provides a stronger constraint on the convergence rate to the invariant distribution. More specifically, the relaxation time estimates the time & required for a system to approach its invariant distribution μt, over the invariant set A, starting from an arbitrary initial state. The relaxation time can be computed using the log-Soboley inequality, typically in the form of an exponential rate decay O (cer.7 - (og( (Feng & lyer, 2019), where CLSI represents
the Sog--oolev constant and is the tolerance. In this case, the following holds: d(uuk U.)) after k ix| cLaT log())) steps, where d(:-)) measures the total variation distance between the two distributions. Although determining the log-Soboley constant can be challenging, it can be inferred
that the probability of lying outside A is sufficiently small, such that the timestep k is proportional to CLST log(
[ )1. Note that á is the feature function of state 2. Thus, the empirical calculation
of relaxation time c2.91 log( [=lle>] and the norm of state ||=||2 can be understood as energy-like quantity in physics. As illustrated in Figure the initial distribution contracts toward the invariant
distribution. Thus, it is feasible to use the trajectory data before k for the training of the contraction phase.
3.3 STAGE II: MEASURE-INVARLAN PHASE
Once the system reaches the global attractor, it explores the bounded set A based on the invariant measure, with spatial and long-term temporal statistics aligned. Von Neumann (Neumann) 1932b) established that MPT induces unitary operator the corresponding Hilbert space L Consequently, a unitary operator G can be learned to describe the system's 's evolution with feature space.
Proposition 3.2 (Unitary Property) When the map T is MPT. the operator Gm is unitary on L That is, for all o, v E C²,
(gm Gm = (0, 1b),
where (ò, v) = Sx (0(z), (z)))dp is the inner product on L²
Theorem 3.3 (Koopman-von Neumann (KvN) Ergodic Theorem (Neumann 1932b)) Let
(X, (X T) be an MPT. If & L3, then lim Iv EN- g d lim EEE To where Φ is invariant. If is ergodic, then Φ == ʃ ddl..
The proof can be referred to Appendix
and C.4 The KvN ergodic theorem shows that for an ergodic MPT the associated operator G is unitary, preserving norms L This means the time averages of _ functions converge to their space averages, ensuring the long-term predictability of chaotic systems. The theorem links chaos with operator theory, providing a framework to study the statistical behavior through Gm. The spectrum of the operator Gm. lies the complex unit circle a (9m) C fr e C |rl== = 1}. with positive measure on both discrete and continuous spectrum for chaotic systems (Koopman & Neumann| 1932). The presence of a continuous spectrum indicates chaotic behavior, meaning the learned operator should have its spectrum densely distributed along the unit circle. The validity of the learning results can be assessed by analyzing the spectrum of the operator, as shown in Figure 2
Py T (2014) supports the auto-differentiation of eigenvalue function of symmetric matrix or Hermitian matrix, which improves computational stability.
6

---

Published as a conference paper at ICLR 2025
Reppas
3010
Imaginary Part
Imaginury Part
10
150
PINN
Eaergy
Energy
80
10'
15 Wavenumber
Real Part
Wavenumher
(a) Lorenz eigenvalues (b) Lorenz energy spectrum
(c) KS eigenvalues
(d) KS energy spectrum
Figure Operator eigenvalues and energy spectrum of the Lorenz 96 and KS equations: The eigenvalues in subfigures (a) and (c) display the difference of eigenvalue distribution of learned operator layer from the aspect of the measure-invariant constraint. Red points represent the PFNN's eigenvalues in the maasure-invarian phase, densely distributed near the unit circle in the complex plane, indicating invariant measure under evolution. In contrast, blue points represent eigenvalues without the measure- mecsure-invariant invariant constraint, showing values both inside and outside the unit circle, leading to gradient vanishing and blow-up for long-term predictions. Subfigures (b) and (d) show the corresponding energy spectrum for the Lorenz 96 and KS equations, illustrating the system's 's energy distribution across modes. The energy spectra of PFNN's prediction fit the true energy spectra most by the introduction of the measure-invariant constraint.
To determine the unitary operator in the ergodic state (see Proposition 3.2 we leverage the intrinsic property that 9m and its conjugate transpose 9T, satisfy 9m - la. This property ensures that Gm unitary, preserving norms and inner products, and establishes well-defined backward dynamic. Specifically, the backward operator 9m is equal to G This conjugate transpose relationship guarantees that 9%, Gm 9 la. indicating that 9m is invertible and the inversion is consistent with unitarity (see theoretical aspects in Appendix D.3). To learn Gm effectively, we use a bi- directional approach that captures both forward and backward dynamics. Gm predicts the forward process, while Ir governs the reverse. Aligning the learning of Gm and G ensures the operator remains unitary and accurately represents the system's dynamics in both directions.
According to the estimated relaxation time, the trajectories timestep > J log( [=ll used to train the measure-invariant phase. The measure-invariant loss function becomes
are <
arg min Eznyul [| (zx) - 98" (=x+1)ll 11 (Z++ 9 (zx) 2x))l 91,02
forward loss
backward loss
(8)
+72 ( lall F + im LallF) reec
consistent constraint
where 7/2 E (0, 1) is the regularized coefficient, || || is denoted as the Frobenius norm, and the distribution of p follows the distributions with timestep larger than integer k. The loss function the first line is represented as forward and backward prediction, respectively. The last term in Equation is consistency constraint on the backward and forward process of G. The combination of three terms in Equation
8 iimplie the unitary evolution in ergodic theorem in Sections 3.2 and 3.3
Equations and will be used for training the corresponding operators to predict the distinct
chaotic behaviors. We provide the pseudocode in Appendix E
NUMERICAL EXPERIMENTS AND ABLATION STUDY
In this section, we empirically evaluate the performance PFNN chaotic systems by comparing
against four baselines: (a) LSTM (Vlachas et al.) 2018), type of RNN architecture designed to learn long-term dependencies in sequential data; (b) Koopman operator (Pan et al. 2023). linear operator
that represents the evolution of functions in a dynamical system; (c) Fourier neural operator (FNO)
(Kovachki et al. 2023), which uses fast Fourier transform (FFT) to efficiently parameterize integral operators and learn the system evolution; and (d) Markov neural operator (MNO) (Li et al. 2022). which enhances FNO by imposing a hard-coded constraint to better capture dissipative chaos. These baselines cover both deep sequence models and operator learning methods.
7

---

Published as a conference paper at ICLR 2025
3-dimensional simplified model of atmospheric convection, known for its chaotic behavior and sensitivity to initial conditions. (2) Lorenz 96 (Lorenz) 1996): A surrogate model for atmospheric circulation, characterized by the coupled differential equations with periodic boundary conditions and varying state dimensions. To test PFNN across different dimensionalities,
Learning tasks. (I) Lorenz 63 (Lorenz
generated datasets with state dimensions of 9, 40, and 80, respectively. (3) Kuramoto- Kuramoto-Sivashinsky (KS) equation (Papageorgiou & Smyrlis [1991): A fourth-order nonlinear partial differential equation that models diffuusive instabilities and chaotic behavior in systems, such as fluid dynamics, and reaction-diffusee processes. In this case, we uniformly sub-sampled 128 states from a dense spatial discretization during integration as system states. (4) Kolmogorov flow governed by Navier-Stokes (NS) equation Temam )2012): A two-dimensional shear flow commonly used in fluid dynamics to study turbulence, characterized by sinusoidal velocity fields one direction and external forcing in the perpendicular direction. In our experiments, we utilize 64 x 64 resolution field for studying.
Training details. Given the generated trajectories, each trajectory was divided into two phases: contraction and measure-invariant with the split occurring at an estimated relaxation time. The data from each stage then segmented into single-step pairs of observations and true states, which were subsequently shuffled to create the training data. All trajectories are sampled from random initial states, with prior statistical information available guide the training process.
Evaluation metrics. The models are evaluated using specific metrics: (1) Normalized Root Mean Square Error (NRMSE) for short-term performance, measuring prediction accuracy within 10-step roll-out; (2) Kullback-Leilerr divergence (KLD), and (3) Maximum Mean Discrepancy (MMD) for long-term performances, both of which compare the predicted and true distributions estimated from roll-out samples of the models and the dataset; (4) Turbulent Kinetic Energy (TKE) is distribution of turbulent energy districture, which reflects the error of TKE between the learned model and ground truth. Notably, KL divergence can be problematic in high-dimensione spaces, where probability distributions are overly spread out. To address this, we performed kernel principal component analysis (KPCA) to identify oow-dimensional subspace (m 3), and used kernel density estimation (KDF) to estimate the predicted and true distributions. Further details on the learning tasks, training data, evaluation metrics, and baseline models can be found in Appendix G
Table 1: Performance comparison of models (FNO, LSTM, Koopman, MNO, and PFNN) across various dynamical systems (the Lorenz 63, Lorenz 96, Kuramoto-Sivashinskt and Kolmogorov Flow) with corresponding evaluation metrics: NRMSE, KL divergence, MMD, and TKE. The best performance highlighted in bold. NRMSE evaluated 50 times using different initial states, with the results reported as standard deviation. Here, represents the state dimension.
Metrics
LSTM
Koopman
FNO
MNO
PFNN
NRMSE
1,83=0,85
0.67
0.53
0.44-0.20
0.49
Lorenz 63 (m 3)
KLD
+
0.66
0.61
0.37
0.29
MMD
1.05
0.46
0.51
0.48
0.39
NRMSE
1.57-10.44
0.61100.3
0.42-10.36
0.28-0.09
1.21-10.14
Lorenz 96 (m
KLD
3.49
2.61
2.12
2.01
1.87
MMD
0.21
0.13
0.11
0.10
0.10
NRMSE
4.77-22.15
0.74:10.12
0.09
0.04
0.11-10.06
Lorenz 96 (m 10)
KLD
+000
0.16
0.12
0.11
0.11
MMD
0.854
0.0239
0.015
0.011
0.011
NRMSE
0.28
L64:10.49
0.51
0.37=0.16
0.32-10.09
Lorenz 96 (m 80)
KLD
1.32
0.24
0.15
0.18
0.10
MMD
0.056
0.0088
0.0094
0.0084
0.0083
NRMSH
13.8744.69
1.71
0.53
0.31
0.25
(m 128)
KLD
+000
52.51
142,73
24.53
9.37
MMD
0.996
0.204e
0.915
0.031
0.011
NRMSE 10.65 + 4.09
10.06 +: 4.23
4.22:14.23
3.81=4.48
3.42.13.79
NS (m = 64 x 64)
TKE
2.2965
1.8092
2.0973
1.5828
1.3638
Thh L.S yields constant predictions, resulting in a degenerate (Dirac delta) distribution that value, which has different support from the true distribution, leading an infinite KI. divergence.
8

---

Published as a conference paper at ICLR 2025
4.1 RESULTS AND COMPARISON
Table presents the performance of all baseline methods and PFNN, evaluated using four metrics across the selected learning tasks. PFNN consistently outperforms the baselines, particularly ex- celling in KLD and MMD, which highlights its capability to capture the long-term dynamics of complex, chaotic systems. For short-term prediction, both PFNN and MNO show better accuracy, as reflected by the NRMSE metric. This is likely due to their ability to account for the initial phase of energy dissipation, which characterizes transient behaviors. PFNN (contraction operator) and MNO incorporate physical properties during this phase, resulting in predictions that more accurately reflect the system's 's dissipative nature. However, MNO enforces a fixed dissipation rate and pre-defined absorbing ball, which can reduce its effectiveness in short- soort-temm forecasting. Models like LSTM and Koopman, are hard to capture transient behaviors for states lying outside attractors, exhibit large deviations from the ground truth. Notably, while FNO uses FFT to extract features and provides relatively accurate short-term predictions, its reliance on integer Fourier modes tends to produce periodic behavior. This periodicity conflicts with the non-periodic nature dissipative chaos.
In terms of long-term teng-termmnn statistical properties, PFNN's ' unitarity directly leads to invariant distribution, which results in better performance in KLD and MMD. contrast, the periodic tendencies of integer Fourier modes in FNO and MNO disrupt ergodicity, preventing a full exploration of the invariant sets. This limitation is reflected in the higher KLD and MMD values for FNO and MNO. A key observation that further validates this distinction is evident in the eigenvalue distribution of the learned unitary operator 9m for the Lorenz 96 and KS systems. Specifically, the eigenvalues along the unit circle in Figure This distribution aligns with the presence of a continuous spectrum in chaotic systems, predicted by the KvN theorem, indicating positive measure the continuous spectrum. Additionally, Figure demonstrates the effectiveness of our consistent loss function in maintaining the unitarity of the operator and accurately learning key invariant distributions, which ensures stable long-term predictions in chaotic systems. In contrast, when the model is not constrained by regularized terms in Equation X. it results in unstable/stable modes. Empirically, this instability causes rapid forecast divergence//onvergenne and failure to learn the invariant distribution, as shown in the cigenvalue distributions (see blue points in Figure 2(a) and ) Furthermore, the unitarity of more accurate energy spectrum, as the operator reflects underlying conservation laws, resulting
densely distributed
demonstrated in Figure 26b 2(d) and Figure 3 Due to the periodic behaviors in MNO and FNO, the distribution of TKE inconsistent with ground truth. More experimental details showcasing the short- and long-tem prediction and physical statistics are provdied in Appendix
TKE
Koopman
PFNN
60
60
60
60
100
40
40
40
40
40
7.5
20
20
20
20
20
20
0
0
Figure 3: Visualization of the prediction error for Kolmogorov flow trajectories in terms of TKE, demonstrating the model's accuracy in capturing the kinetic energy distribution. We compare the model-predicted trajectories with the ground truth and plot the absolute error in TKE.
4.2 ABLATION STUDY
In this section, we revisited the KS system and conducted ablation studies to evaluate the effects of key hyperparameters in PFNN, including the relaxation time k. the unitary regularized coefficient 72 (with 71 fixeds), and the finite feature dimension L.. Table includes the metrics evaluated for each configuration, providing insight into how these hyperparameters affect short- and long-term performance.
We retrained each model, systematically varying T2 and found that 722 [0.1, 0.5) consistently performs well and achieves a good balance between short-term precision
Regularized coefficient.
SSnce the does not affect long-term performance and is trained with smaller dataset, performed grid search over (0. 1] to determine its value.

---

Published as a conference paper at ICLR 2025
and long-term consistency. Outside this range, lower "/2 reduces long-ten terg-termmm consistency, while higher values overly constrain the model, degrading short-term accuracy.
Feature dimensions. We explored the impact of latent feature dimension L in the PFNN by varying L while keeping other hyperparameters fixed. From T we found that L, = 256, 512 consistently perform well, while smaller values reduce the model's expressiveness to capture complex patterns, while larger values require more data for training and risk overfitting. This ablation study result reflects classic trade-off in deep learning between model complexity and data volume.
Log-Sobolev constant. Systems exhibit varying dissipation rates, making it important to test different relaxation times. In most dissipative chaotic systems (Garbaczewski & Olkiewicz 2002), the approach to the invariant distribution follows an exponential contraction rate. In this experiment, we varied the relaxation time k and constructed two corresponding datasets for training the contraction and unitary operators in PFNN. Due to the logarithmic relationship, the has weak effect on relaxation time k. Therefore. we adjust the relaxation time by fixing c = 0.01 and varying the constant term CLSS" For the KS equation, dissipation predominantly occurs in the higher Fourier modes, typically for wave numbers exceeding (Papageorgiou & Smyrlis 1991), leading slow dissipation characterized by a log-Sobolev constant bounded as cLaI 2 1. In such a situtation, we 11,3,55 5, 7, 7, the model consistently performs well once the exceeds test the constant CLa! = in KS system. Stable performance was observed when 1 2 5, a certain threshold highlighting that an appropriate relaxation time is crucial: if too short, it introduces bias, while too long, many samples lie on the attractor, leading to data contamination from transient behaviors. Furthermore, continuously increasing the parameter beyond this range does not have an obvious effect on the long-term statistics. If the hyperparameter C1.11
is set too high, the data volume available for the measure- measure-invariant phase in Equation & becomes insufficient for training. In other experiments, we set the constant term
5 obtaining a similar result.
CLST
Table 2: Ablation experiments showing the effects of varying PFNN hyperparameters. NRMSE (100 steps) measures the normalized root mean square егтоr over 100 steps. KLD is estimated using sample size of 110,000 for long-term statistical analysis. In the left column, we fix ? 0.3, set
the feature dimension L = 512, and assign the constant term CL3! 5. In the middle column, we
fix 71- 0.3. 1/2 0.5, and CLSI 5. In the right column, we fix ?1 0.3, 7/2 0.5, and set the
feature dimension 512.
Parnmeters
Regularized coefficient, 1/2
Feature dimension, L
Log- Sobolev constant,
Metrics
0.3
128
1024
NRMSE 100 steps
26.59
26.67
23.11
15.79
26,51
21.59
19.13
15.79
43.44
585k
46.77
15.79
15.07
14.23
KL.
136.68
85.33
37.21
6,55
48.09
50.12
41.33
6.55
1637.46
186k
9.17k
6.55
491
5.78
CONCLUSIONS AND LIMITATIONS
Conclusions. Learning chaotic behavior poses a significant challenge due the inherent instability and unpredictability of these systems. To address this complexity, we adopt a physically informed approach and introduced the Poincaré Flow Neural Network (PFNN), a novel operator learning frame- work designed to capture both the contraction and measure-invariant phases of dissipative chaotic systems. Our method outperforms traditional deep sequence models and neural operators, providing superior short- and long-term predictions while ensuring more consistent statistical outcomes across a variety of dissipative chaotic dynamical system experiments.
Limitations. We evaluate the limitations from both theoretical and practical perspectives. Theoreti- cally, the assumption of ergodicity in the second stage does not apply to all chaotic systems, which can lead to a breakdown in operator unitarity for non-ergodic cases. A promising direction for future www.mmaaauueeeeeereeieee
work is to extend the method to handle bi-directional training results in slow convergence due to the high computational complexity of the Frobenius norm. To alleviate this issue, future research could leverage the Hermitian properties of unitary operators employ Hutchinson's 's trace estimation to optimize trace calculations and reduce computational load. Lastly, a promising direction is to extend the discretized -rid input in PFNN to a inreetizationaannsste input.
chaotic systems. Practically, the
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
The authors thank the reviewers and the program committee of ICLR 2025 for their insightful feedback and constructive suggestions that have helped improve this work. We also acknowledge the support from the University College London (Dean's 's Prize, Chadwick Scholarship), the Engineering and Physical Sciences Research Council projects (EP/W007762/11 EP/T517793/1, EP/W524335/1), and the Royal Academy of Engineering
7 REPRODUCIBILITY STATEMENT
The full algorithm of the proposed method can be refered to Appendix E Code details of numerical experiments are available at bltttps://gtthut com/ Hy23333/33333
REFERENCES
Terrence Adams and Anthony Quas. Ergodicity and mixing properties. In Ergodic Theory. pp. 35-60. Springer, 2023.
Bipin Kumar Agarwal. Statistical mechunics. New Age International, 2007.
Ayya Alieva, Dmitrii Kochkov, Jamie Alexander Smith, Michael Brenner, Qing Wang, and Stephan Hoyer. Machine learning accelerated computational fluid dynamics. Proceedings of the National Academy of Sciences USA, 2021.
Roland Bauerschmidt and Benoit Dagallier. Log-sobolev inequality for near critical ising models. Communications on Pure and Applied Mathematics, 77(4):2568-2577 2024.
Petar Bevanda, Stefan Sosnowski, and Sandra Hirche. Koopman operator dynamical models: Learn- ing, analysis and control. Annual Reviews in Control, 12:197-2 2021.
George Birkhoff and Bernard Koopman, Recent contributions to the ergodic theory. Proceedings of the National Academy of Sciences, 18(3):279-282, 1932.
George David Birkhoff.. Dynamical systems, volume 9. American Mathematical Soc., 1927.
Steven L Brunton and J Nathan Kutz. Machine learning for partial differential equations. arXiv preprint 2..3::3303./7774 2023.
Bo Chang, Minmin Chen, Eldad Haber, and Ed H Chi. Antisymmetriernn: A dynamical system view on recurrent neural networks. arXiv preprint arXiv:1902.09689 2019.
Xiaoyuan Cheng, Yiming Yang, Wei Jiang, Xiaohang Tang, and Yukun Hu. Keec: Embed control on an equivariant geometry. arXiv preprint arXiv:2312.01544, 2023.
Isaac P Cornfeld, Sergei Vasilevich Fomin, and Yakov Grigor' evic Sinai. Ergodic theory, volume 245. Springer Science Business Media, 2012.
Predrag Cvitanović, Ruslan L Davidchack, and Evangelos Siminos. On the state space geometry of the turamoto-sivashinsk flow in periodic domain. SIAM Journal Applied Dynamical Systems, 9(1):1-33,2011
Suddhasattwa Das. Dimitrios Giannakis, and Joanna Slawinska. Reproducing kereel hilbert space compactification of unitary evolution groups. Applied and Computational Harmonic Analysis, 54: 75-136, 2021.
Mark Demers and Hong-Kun Zhang. Spectral analysis of the transfer operator for the lorentz gas. Journal of Modern Dynamics, 5(4), 2011.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICI.R, 2021.
11

---

Published as a conference paper at ICLR 2025
Klaus-Jochen Engel, Rainer Nagel, and Simon Brendle. One-parameter semigroups for linear evolution equations, volume 194. Springer, 2000.
David P Feldman. Chaos and dynamical systems, volume 7. Princeton University Press, 2019.
Yuanyuan Feng and Gautam Iyer. Dissipation enhancement by mixing. Nonlinearity, 22(5):1810 2019.
Joel N Franklin. Matrix theory. Courier Corporation, 2012.
Piotr Garbaczewski and Robert Olkiewiez. Dynamics of dissipation, volume 597. Springer Science & Business Media, 2002.
Jerryman Gyamfi. Fundamentals quantum mechanics liouville space. European Journal of Physics, 41(6):063002, 2020.
Boris Hasselblatt and Anatole Katok. Handbook of dynamical systems. Elsevier, 2002.
Kyle Helfrich, Devin Willmott, and Qiang Ye. Orthogonal recurrent neural networks with scaled cayley transform. In International Conference on Machine Learning, pp. 1969-1978. PMLR, 2018.
Mikael Henaff, Arthur Szlam, and Yann LeCun. Recurrent orthogonal networks and mangmeeory tasks. In International Conference on Machine Learming, pp. 2034-2042. PMLR, 2016.
Masahiro Ikeda, Isao Ishikawa, and Corbinian Schlosser. Koopman and perron-frobenius operators on reproducing kernel banach spaces. Chaos: Interdixciplinary Journal of Nonlinear Science, 32(12), 2022.
Ruoxi Jiang, Peter Lu, Elena Orlova, and Rebecca Willett. Training neural operators to preserve invariant measures of chaotic attractors. Advances in Neural Information Processing Systems, 36. 2024.
Javier Jiménez. A perron-frobenius analysis of wall-bounded turbulence. Journal of Fluid Mechanics, A68:A10, 2023.
Anil Kag, Ziming Zhang, and Venkatesh Saligrama. Runs incrementally evolving on an equilibrium manifold: A panacea for vanishing and exploding gradients? In International Conference on Learning Representations, 2020.
Mihail Vladimirovi_ Karasev, Viktor Pavlovi_ Maslov, et al. Nonlinear Poisson brackets: geometry and quantization, volume 119, American Mathematical Soc., 2012.
Bernard O Koopman. Hamiltonian systems and transformation in hilbert space. Proceedings of the National Academy of Sciences, 1755):315-318, 1931.
Bernard O Koopman and J v Neumann. Dynamical systems of continuous spectra. Proceedings of the National Academy of Sciences, 18(3):255-263, 1932.
Vladimir Kostic, Pietro Novelli, Andreas Maurer, Carlo Ciliberto, Lorenzo Rosasco, and Massimiliano Pontil. Learning dynamical systems via koopman operator regression in reproducing kernel hilbert spaces. Advances Neural Information Processing Systems, 35:4017 2022.
Nikola Kovachki, Samuel Lanthaler, and Siddhartha Mishra. On universal approximation and error bounds for fourier neural operators. Journal of Machine Learning Research, 22(290):1-76, 2021.
Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces with applications to pdes. Journal of Machine Learning Research, 24(89):1-97, 2023.
Matthew Lagro, Wei-Shih Yang, and Sheng Xiong. perron-frobeniut frobenius type theorem for quantum operations. Journal of Statistical Physics, 169:38-62, 2017.
12

---

Published as a conference paper at ICLR 2025
Qianxiao Li, Felix Dietrich, Erik M Bollt, and Ioannis G Kevrekidis. Extended dynamic mode decomposition with dictionary learning: A data-driven adaptive spectral decomposition of the koopman operator. Chaos: An Interdisciplinary Journal of Nonlinear Science, 27(10), 2017.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895, 2020.
Zongyi Li, Miguel Liu-Schiaffini, Nikola Kovachki, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Learning dissipative dynamics in chaotic systems. arXiv preprint arXiv:2106.06890 2022.
Zachary C Lipton, John Berkowitz, and Charles Elkan. A critical review of recurrent neural networks for sequence learning. arXiv preprint arXiv: 100.. 00019, 2015.
Yong Liu, Chenyu Li, Jianmin Wang, and Mingsheng Long. Koopa: Learning non-stationary time series dynamics with koopman predictors. Advances in Neural Information Processing Systems, 36, 2024.
Edward N Lorenz. Deterministic nonperiodic flow. Journal of atmmsppheric sciences, 20(2):130-1411 1963.
Edward N Lorenz. Predictability: A problem partly solved. Proc. Seminar on predictability. volume 1. Reading, 1996.
Lu Lu, Pengzhan Jin, and George Em Karniadakis. Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv preprint www.:v:1910.3330. 2019.
Günter Lumer and Ralph Phillips. Dissipative operators in a banach space. 1961.
Bethany Lusch, Nathan Kutz, and Steven L Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature communications, (1):4990, 2018.
Chiara Cecilia Maiocchi, Valerio Lucarini, Andrey Gritsun, and Yuzuru Sato. Heterogeneity of the attractor of the lorenz '96 model: Lyapunov analysis, unstable periodic orbits, and shadowing properties. Physica D: Nonlinear Phenomena, 457:133970, 2024.
Jonas Mikhaeil, Zahra Monfared, and Daniel Durstewitz. On the difficulty of learning chaotic dynamics with rnns. Advances in Neural Information Processing Systems, 15:11297-11322, 2022.
Hazime Mori and Yoshiki Kuramoto. Dissipative structures and chuos. Springer Science & Business Media, 2013.
Takashi Mori and Tatsuhiko Shirai. Resolving discrepancy between liouvillian gap and relaxation time in boundary-dissipatee quantum many-body systems. Physical review letters, 125(23):230604, 2020.
Sayak Mukherjee, Sai Pushpak Nandanoori, Sheng Guan, Khushbu Agarwal, Subhrajit Sinha, Soumya Kundu, Seemita Pal, Yinghui Wu, Draguna Vrabie, and Sutanay Choudhury. Learning distributed geometric koopman operator for sparse networked dynamical systems. In Learning on Graphs Conference, pp. 45-1, PMLR, 2022.
J Nathan Kutz, Joshua L Proctor, and Steven L Brunton. Applied koopman theory for partial differential equations and data-driven modeling of spatio-temporal systems. Complexity, 2018(1): 6010634, 2018.
J v Neumann. Physical applications of the ergodic hypothesis. Proceedings of the National Academy of Sciences, 18(3):263-266 1932a.
J v Neumann. Proof of the quasi-ergodic hypothesis. Proceedings of the National Academy of Sciences, 18(1)700-82, 1932b.
Donald S Ornstein. Ergodic theory, randomness, and" chaos". Science, 243(4888) 1989.
13

---

Published as a conference paper at ICLR 2025
Shaowu Pan, Eurika Kaiser, Brian M de Silva, J Nathan Kutz, and Steven L Brunton. Pykoop- man: a python package for data-driven approximation of the koopman operator. arXiv preprint TrXiv:2306.12966 2023.
Demetrios Papageorgiou and Yiorgos S Smyrlis. The route to chaos for the kuramoto-sivashinkk. equation. Theoretical and Compulational Fluid Dynamics, 3(1):15-42, 1991.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetoto, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style. High- Bigh-Performance Deep Learning Library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché Buc, E. Fox. and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
Jaideep Pathak, Brian Hunt, Michelle Girvan, Zhixin Lu, and Edward Ott. Model-free prediction of large spatiotemporally chaotic systems from data: A reservoir computing approach. Physical review letters, 120(2):02400,, 2018.
Henri Poincaré. (Euvres de Henri Poincaré: publiées sous les auspices de l'Académie des sciences.... volume 1. Gauthier- Villars et cic, 1928.
PyTorch. bacch.linalg.eigh
2014. [Accessed 18-08-2024).
Antônio Ribeiro, Koen Tiels, Luis A Aguirre, and Thomas Schön. Beyond exploding and vanishing gradients: analysing training using attractors and smoothness. In International conference artificial intelligence and statistics, pp. 2370-2380. PMLR, 2020.
Yair Schiff, Zhong Yi Wan, Jeffrey B Parker, Stephan Hoyer, Volodymyr Kuleshov, Fei Sha, and Leonardo Núñez. Dyslim: Dynamics stable learning by invariant measure for chaotic systems. arXiv preprint arXiv:2402.04466 2024.
Konrad Schmüdgen. Unbounded self-adjoint operators on Hilbert space, volume 265. Springer Science & Business Media, 2012.
Domokos Szász. Boltzmann's 's ergodic hypothesis, a conjecture for centuries? Studia Scientiarum Mathematicarum Hungarica, 31(1):299-322, 1996.
Naoya Takeishi, Yoshinobu Kawahara, and Takehisa Yairi. Learning koopman invariant subspaces for dynamic mode decomposition. Advances in neural information processing systems, 30, 2017.
Gouhei Tanaka, Toshiyuki Yamane, Jean Benoit Héroux, Ryosho Nakane, Naoki Kanazawa, Seiji Takeda, Hidetoshi Numata, Daiju Nakano, and Akira Hirose. Recent advances in physical reservoir computing: review. Neural Networks, 115:100-123, 2019.
Yang Tang, Jürgen Kurths, Wei Lin, Edward Ott, and Ljupco Kocarev. Introduction to focus issue: When machine learning meets complex systems: Networks, chaos, and nonlinear dynamics. Chaos: An Interdisciplinary Journal of Nonlinear Science, 30(6), 2020.
Terence Tao. The mean ergodic theorem. https::/ wwwpppree..wooorss........ httpapprrrss..ss../eeeeoe /
01/30 01/30/2548 lecture- cessed 25-08-2024).
mean www.aaeeeooccctttoeeeeeeeo -tt.://moooeeeeee 2500 2008a. [Ac-
Terence Tao. Cohomology in dynamical system. https: http::///terryaaa.vvvvrreeeee 2024].
www.wwesw.ss...... com/
for dynamiccall. cty.amiccl-syysttemsss 2008b. [Accessed 13-07-
Roger Temam. Infinite dimeesoadi dynamical systems in mechanics and physics, volume 68. Springer Science Business Media, 2012.
Warwick Tucker. The lorenz attractor exists. Comptes Rendus l'Académie des Sciences-Serier I- I-Mathematies, 197-11222. 1999.
14

---

Published as a conference paper at ICLR 2025
Claire Valva and Dimitrios Giannakis. Consistent spectral approximation of koopman operators using resolvent compactification. arXiv preprint wxXiv2309.000722, 2023.
Pantelis R Vlachas, Wonmin Byeon, Zhong Y Wan, Themistoklis P Sapsis, and Petros Koumout- sakos. Data-driven forecasting of high-dimensional chaotic systems with long short-term memory networks. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 474(2213):0110884, 2018.
Pantelis Vlachas, Jaideep Pathak, Brian R Hunt, Themistoklis Sapsis, Michelle Girvan, Edward Ott, and Petros Koumoutsakos. Backpropagation algorithms and reservoir computing in recurrent neural networks for the forecasting of complex spatiotemporal dynamics. Neural Networks, 126: 191-217, 2020.
Felix Voigtlaender. The universal approximation theorem for complex-valued neural networks. Applied and computational harmonic analysis, 64:33-61, 2023.
Peter Walters. An introduction to ergodic theory, volume 79. Springer Science & Business Media, 2000.
Zhong Yi Wan, Leonardo Zepeda-Núñez, Anudhyan Boral, and Fei Sha. Evolve smoothly, fit consistently: Learning smooth latent dynamics for advection-dominated systems. arXiv preprint 103391, 2023.
Rui Wang, Yihe Dong, Sercan Ô Arik, and Rose Yu. Koopman neural forecaster for time series with temporal distribution shifts. arXiv preprint arXiv:2210.03675. 2022.
Matthew O Williams, Joannis G Kevrekidis, and Clarence W Rowley. A data driven approximation of the koopman operator: Extending dynamic mode decomposition. Journal of Nonlinear Science, 25:1307-1346, 2015.
Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers. arXiv preprint 2:X552240..02552. 2024.
Yiming Yang, Xiaoyuan Cheng, Daniel Giles, Sibo Cheng, Yi He, Xiao Xue, Boli Chen, and Yukun Hu. Tensor-var: Variational data assimilation in tensor product feature space. arXiv preprint arXiv:2501.13312. 2025.
Rose Yu, Stephan Zheng, and Yan Liu. Learning chaotic dynamics using tensor recurrent neural networks. In Proceedings of the ICML, volume 17, 2017.
Marko Žnidarič. Relaxation times of dissipative many-body quantum systems. Physical Review E. 92(4):042143, 2015.
Joseph D Zund. George david birkhoff and john neumann: A question of priority and the ergodic theorems. 1931-1932. Historia Mathematica, 29(2): 2002.
15

---

Published as a conference paper at ICLR 2025
CONTENTS
Introduction
Background and Problem Formulation
3
Method
3.1
Finite-Raid Operator Approximation
3.2
Stage I: Contraction Phase
3.3
Stage II: Pecasure-Ivvarian Phase
Numerical Experiments and Ablation Study
Results And Comparison
4.2
Ablation Study
5
Conclusions and Limitations
10
Acknowledgement
Reproducibility Statement
11
Table of Notations
18
Dissipation and Conservation
19
Proofs Main Theorems
19
Proof of Poincaré's s Claim
19
Proof of 3.
3 Proof of 3.2
22
Proof 3.3
22
D Analysis on Operator
24
Theoretical Aspects Koopman Operator
24
D.2 Contraction Operator Hilbert Space
25
D.3 Unitary Operator
25
Algorithm
27
More Experimental Results
28
Lorenz 63
28
Lorenz 96
28
F22.1
Experiment dimensional dataset
29
F2.
Experiment on 40 dimensional dataset
32
k2.3
Experiment 80 dimensional dataset
32
16

---

Published as a conference paper at ICLR 2025
R3
37
Kolmogorov Flow
40
Experiment Settings
42
Neural Network Architecture
42
G.2
Dataset.
42
G.3
Training details and baselines,
43
17

---

Published as a conference paper at ICLR 2025
A
TABLE OF NOTATIONS
Notations
Meaning
CLST
Nog-Sobolev constant
9en
encoder
gde
decoder
k m
time index dimension of state space state
A B
invariant set or attractor
Borel o algebra bounded set
B
F
function space
G Ge
forward operator on L space contraction operator
9m
unitary operator
la
identity matrix
L D X
finite rank of approximated operator
M
bounded state space
O,
describes limiting behavior of a function
nonlinear forward map
0
parameters of neural networks
Ø, v
feature functions in L space
1
Lebesgue measure
invariant distribution
(.)
cigenvalues of operators total variation distance of probability measures
d(-,))
CP CP(X. 11)
function spaces defined using a natural generalization of the p-norm
(X, 14)
measure space
18

---

Published as a conference paper at ICLR 2025
The Appendix is organized into two main sections: the first three chapters delve into theoretical analysis, while the subsequent chapters address empirical results and algorithm architectures.
We also encourage readers to spend time the seminal monographs ergodic theory (Birkhoff 1927, Koopman Neumann 1932 Neumann 1932a Birkhoff & Koopman 1932), as they offer profound insights that enrich the theoretical foundations presented in the Appendix.
B DISSIPATION AND CONSERVATION
The reason would like discuss Poincaré's claim is offer perspective understanding dissipative chaos from a macro-structural viewpoint. In statistical mechanics, contraction corresponds dissipative systems, while measure preservation corresponds to conservative systems. Below, we introduce two key concepts.
Definition B.1 (Wandering Set (Hasselblatt & Katok 2002)) Let X - X be a flow map in
topological space X. A point x E X is said to be wandering point there a neighbourhood U of I and positive integer N such that for all k > N, the flow map is pairwise disjoint as
(U))U
(9)
Remark B.2 Wandering sets are transient. When a dynamical system has wandering set of non- zero measures, then the system regarded as a dissipative system. This is the opposite of the conservative system in Poincare's 's claim, An intuitive way to understand the wandering set is: if portion of phase space "wanders away" during system evolution, and never visits again, then the system is dissipative. A simple example is Lorenz 63, it will contract the global attractor without visiting the outside again. When the global attractor exists, the invariant set can be expressed as w-limit set as w(x, T)) = =
(T)) The invariant set is thus non-wandering.
Theorem B.3 (Liouville's Theorem in Hamiltonian (Agarwal 2007)) Consider Hamiltonian dynamical system with canonical coordinates qi (q. 9 F.) and momenta p. = (p) p p?) of ifith th molecule for i I....,N. N. Then the phase space distribution uly, q) determines the proba- bility u(p, q)dd qd that the system will be found in the infinitesimal phase space volume d qdl p.
The Liouville equation governs the evolution of plp, 1) in time L as
dp
3u Öt
>
Op 0q:
d: +
p.) 0. Op,
0p
(10)
dt
Or can say the distribution function is constant along any trajectory in phase space.
Remark B.4 In Hamiltonian mechanics, the phase space is smooth manifold that comes naturally equipped with a smooth measure (locally, this measure is the 6N -dieensional Lebesgue measure). The theorem says this smooth measure is invariant under the Hamiltonian flow due to the preservation properties of the symplectic form. By the language in ergodic theory, we can assert that the T : X X is continuous conservative flow such that p{EE) (T E) for any E. C X and lo E N.
C PROOFS OF MAIN THEOREMS
C.1 PROOF OF POINCARÉ'S S CLAIM
Theorem C.1 (Poincaré's Claim) Imagine a box filled with gas made of N identical molecules. Classical mechanics says that we know the positions qi ( (x.))) and momenta pt (p) p..P?) (..) of i-th molecule for 1...,,N, the positions and momenta of each particles at time determines by the Hamiltonian's 's equations as
p (t) 0H
0g4
(11)
0H1
dl
(1)
0p.
19

---

Published as a conference paper at ICLR 2025
where (IN......N.PI........
is the Hamiltonian, measuring the total energy of the system.
The state of entire system is denoted as (q. p) = (g1....., CN.P....... PI.....PN). Let X denote the collection of all possible states. If the Hamiltonian is bounded above, then the transformation of state can be defined by a map as
T: (q. (g(t),
(12)
where T. push the state move towards after steps. When the system is regular, there exist 6N equations shown in Equation]l coupled together. The problem can be regarded as an initial value problem, and T satisfies the conservative law for the Hamiltonian system, which means ar)) X + (x()) E X for all L.
When the system with enormous molecules, Poincaré claimed that the system starts at a certain state (g(0), r(O),, it will eventually return to a state close to (g(0), p(D) after a long enough time.
Proof. Here is Poincaré's solution. Definite T = Tn and T Tx-- Fix 0 and consider the set of W of all states x = (q. p) such that d(z, T (x)) > e for all k: > 1. where d(..-) is the Euclidean distance. Divide W into finitely many disjoint pieces W; of diameter less than
For each fixed i, the sets (W.) 2 1) are pairwise disjoint, otherwise we can derive that T- (W:)TTT- (W,) * = WinT- (W;) + 0. In such situation, this leads contradiction
* x T (WW)) implies that (x) Wi, whence d(x, T*x) S diam(W.) <
x E W. C W implies that d(x:, T%x) by the initial settings of W.
So T (W:) must be pairwise disjoint.
Since WT>*W;Ju21 are pairwise disjoint, x(X) EnS A (aaTauTTWWWW W.). However, by the Liou-
ville's theorem in statistical mechanics in B.3 Agarwal 2007 all (W,) have the same mea- sure, and x oc, so a natural result must be ((W)) for all W The summation of all u(UiEy W.) = 0 means the measure zero of wandering set (Feldman, 2019). In summary, we can infer p(X \ USeN W3) = 1, and the system will explore sufficiently for all possible set X VUenN W,. Consequently, we have the property as d(T* (x),x) < for k 1
Proposition C.2 Suppose (X, B. B.H,Ty T) is an MPT complete measure space, then the following are equivalent (Cornfeld et al. 2012):
μ is ergodic;
if E E and p(T- EAE) = then =(E)= = or p(X EE==0, =
: X is measurable function and o = almost everywhere, then there is
constant s.t. f almost everywhere.
Proposition C.3 Mixing property implies ergodicity.
Proof. Suppose is invariant, then the mixing property and measure preserving property imply u(E) lom 1(E n -T**E) E) + 2(E)? whence μ ((E)² p(E). It follows that u(E) 0 or (FF) = = (()). This result reflects the second item in C.2 and thus ergodicity.
Proposition C.4 A MPT (X,B,p,7 p,T) is strongly mixing iff for every o, vb E L2, lim f dn Th + Tdu f vddu, or equivalent to say lim Covlo, 1 Th) + 0.
Proof. The following fact can be easily derived as
Fact 1. Since o T = μ4, | o T |0012,, for all & = L
Fact 2. [Cou(o, + < 4| @ - vb11
*The Liouville equation governs the evolution probability space invariant under the flow map as (TTFF)) F) = ((E) for E c X. "A is the symmetric difference such that AAB (A (B
20

---

Published as a conference paper at ICLR 2025
Fact 3. The function Covlo, 2)) is bilinear.
The proof of proposition needs to indicate it as a necessary and sufficient condition.
(Necessary.) The condition that for Φ, v/ L the mixing result reflects that Φ sb can be regarded as the indicator function as Φ = 1 and d = r. By the property, we have lim Cool Tho - according to the mixing property. Consequently, the necessity is proved.
(Sufficient.) Let Φ, v be the linear combination of indicator functions on the invariant set. Generally 6, v E L², giving small perturbation e > 0, the finite linear combinations of indicators become de. Wee satisfying | dell2: 11 Vell2 < Then, by the analysis tricks as
lim [Cou(é, x k
< lim [Cov(o - des v/e o T*
lim [Cov(de: dhe o ) k
lim [Cov(e., (1h vb) l*)
k0000
k-00
<Ae||/ila + o(1) + 4(1l012 + c)e,
(13)
where the result tells us limg- Coo(o, v T S 4e/lo/12 + + 4(1øol2 + e)e according to the Fact 1, 2 and 3. When is sufficiently small, the limit becomes zero, the proof is finished.
C.2 PROOF OF 3.1
Since {Φ1. .2.... } is an orthonormal basis of L for L is large enough, we have a projection map Jen & approximates within e-
| - 21 c.
(14)
Since the forward operator G is locally Lipschitz bounded by constant c, the following the inequality holds
||9rr Golla sup cl) (z))l < ce. =EM
sup
(15)
zEM
Now, dr., function lying the space of L since is linear combination of {di The encoder map 90, learns the adaptive feature from the dataset. When the dataset is sufficiently large, there exists a map as
90, ΦL,
(16)
where Jen spans a finite-dimensional linear space, the relation holds due to the universal property of neural networks (Voigtlaender, 2023). In such case the learned operator 9 can be written as
& = =11
(17)
where the finite representation of the operator is consistent with Gr. = II,, Consequently, we have the inequality as
sup 99 g0. (z) G ((2)]l2
=
sup
9 o 9े, (z) - و x(>)
+ sup و| ((2) o ((2)lla
(18)
sEM
EEM
strougly continuous property
condition of strong operator topology
S(c+1) 1)e.
The second line is from the triangle inequality. The third line holds due the strongly continuous operator and strong operator topology in Hilbert space (see D.I). then uniform limit of some sequence of finite-rank operators exists. Therefore, we can assert that existing finite rank L E N to approximate the operator G with arbitrary small error.
'1. Suppose {æ.} sequence of elements in L² that converges to in the norm of the space. i... lim, 12,, - æ|| - 0. The strong continuity of implies that the operator G is continuous with respect to the norm topology when applied to sequences of vectors. Therefore, must hold that limn-roo ||@æ, Gall - 0. 'Let E(X) ) be the space of all bounded, linear operators on Hilbert space X. A net {G,) C 2(X) converges pointwise to G E 2(X) on (X, I. []) (i.e., lim Ge, Garll 0, Vr X) if and only if (9.) converges in the strong operator topology on 2(X).
21

---

Published as a conference paper at ICLR 2025
C.3 PROOF OF
(Unitary property.) The unitary of G follows from the measure-preserving property of T. For Φ 1p E L² we have
GV)
(9 o ((
(19)
(o(T=)(v(To))dde)))
Using the change of variable y = Tv and du = du(X), we have
/ o(y)v()))p()) (6.0).
(20)
A more interesting way to prove the unitarity of G comes from the perspective of quantum mechanics (Gyamfi) 2020). In meesure-connervvng systems, the Liouville operator is new-Hermittian (as seen in the discussion of the Liouville operator and Poisson brackets in (Karasev et al. (012)). which implies that the operator unitary.
C.4
PROOF OF
Fact from 3.2 the MPT property of T implies the isometric transformation as || - ||ó o Til for all E
This convergence of MPT can be proved by the coboundary property such that the function & be set as Ø = v v o I (both o, 1 E C²) (such function &i can be regarded as transfer function from the perspective of cohomology in Tao (008bb), this is easy to see
lim
2 o 1
+00
N
lim
Σ
(1 o T -
o T
N
N
(21)
k= 0
lim
1/ 16 o N-1 11
N
lim
11:/1122 +
No00
Consequently,
N-1 Σ 0 k=
lim Noc
(22)
N
Thus it derived if holds for the element of subspace (v o v E L²). Then by the analysis
tricks, choose a arbitrary function X E {v & T sufficiently large N we have
vb E C2} satisfying ||x øl12 < e. For the
lim N N0000
T
lim
1
Σ
(0 6 x)
+
1
Σ
x r 11
+00
N
(23)
N-1
lim No400
Σ
(Φ x)
+6<20
N
k=0
The proof of the first part is completed.
'Coboundary of a dynamical system. This concept is developed from cohomology theory (an interesting interpretation can be found in (Tao. 2008b)). Two functions f.g X - C are said to he cohomology via transfer function h, f g h T. function which cohomologous is called coboundary.
22

---

Published as a conference paper at ICLR 2025
In particular, & is invariant. I is ergodic, é must be constant and é f odd almost surely. Also, since lim # =E- g + é, we have
[ody lim Σ (1,6oT) (1, li
oo T
(1. +/ f odp
(24)
k=0
The proof is different from the original proof in Neumann 1932b), Von Neumann started the proof by factorizing the spectrum using Fourier analysis and Borel functional calculus and derived the only non-trivial invariant subspaces are those consisting of constant functions.
23

---

Published as a conference paper at ICLR 2025
D ANALYSIS ON OPERATOR
Contraction and unitary operators Hilbert spaces are key tools for predicting and understanding chaotic systems in this paper. To make work self-contained, analysis of these two important types of operators. Generally, the analysis on the one- peeapeeeeeerr group is on Banach space. Since paper is specifically defined on L space, restrict definitions and analysis to Hilbert space.
will provide clear and formal
Definition D.1 -emigrou (Engel et al. 2000)) A strongly continuous one-parameter semi- group on a Hilbert space L² a family G = (9(() IR+ of bounded linear operators G(t) on L satisfying
(()) = la (the identity operator on L2)
Vt, E R+ g(t + s) = (())
lime-soo ((1) = & for lo E R., & E L
The first two axioms algebraic, and state that G is representation of the semigroup (R4,, +); the last is topological, and states that the map G is continuous in the strong operator topology.
If. in addition, [|](t)]| S 1 for all Rt, then G is called strongly continuous one-parameter semigroup of contractions (or contraction operator).
In paper, target is learn continuous linear operators on Hilbert space, which belong Co semigroou..
Definition D.2 (Infinitesimal Generator (Engel al. 2000) The infinitesimal generator of strongly continuous mar-amemete semigroup is the linear operator P on L defined by
Po= lim (g(t) Ia)ø,
(25)
D(P) = 1o f L2 lim,-o
(G((1) |
Ia)sb exists}.
The strongly continuous semigroup (tt) with generator P is often denoted by the symbol ct,_ This notation is compatible with the notation of matrix exponentials. Generally, directly approximating the operator P is more difficult than the operator G, since operator P is usually unbounded and is defined on dense subspace of C2, see the following proposition.
Proposition D.3 (Schmmüdgen 2012 The infinitesimal generator P is a densely defined closed operator on L which determines the strongly continuous one-parumeter semigroup G uniquely. huve
d = & E D(P) and & E R..
(26)
D.1 THEORETICAL ASPECTS OF KOOPMAN OPERATOR
The Koopman operator (t)) KKopman 1931 Das et al. 2021 Ikeda et al. 2022 Cheng al. 2023) acts on function space by composing with the forward map (also known as the flow map), effectively implementing time shifts the function ø. Various choices exist for the function space, such L and spaces of continuous functions. Specifically, for function & E L² and time t R the Koopman operator G(t) : L + L is defined as:
g(t) T
(27)
According to Definition D.1 (t) forms semigroup.
In general, if T is a C flow for some lo 2 0, then G(t) maps the space C into itself for every 0 S k. The infinitesimal generator P of the Koopman operator G()) defined by:
Pg = lim (G(t) Φ). De D(P),
(28)
ch denotes the function space with l-th order differentiabilityy
24

---

Published as a conference paper at ICLR 2025
where D(P) C is the domain of P, consisting of functions for which this limit exists. Typically, P: C' n - L when T is smooth. However, merely considering the semigroup property of the Koopman operator is insufficient for modeling the complex behavior of dissipative chaotic systems. To account for the intrinsic physical properties of such systems, we incorporate this physical knowledge into the learning framework. Specifically, we analyze intrinsic physical properties from spectral theory-contracte and unitarity-as detailed in Section D.4 and Appendix D.3
D.2 CONTRACTION OPERATOR ON HILBERT SPACE
Definition D.4 (SSchmüdgen, 2012)) We shall say that the operator B is accretive if Re(( Φ)) 2 for all @ E D(B) that B is m aa--cecretvve if B is closed, acceretive, and R(B Ao dense in L for some Ag C, Re(Nn)
B is called dissipative if - is accretive and midisiipative if BB is m m-maae -accretivee..
Theorem D.5 (Lumer- Philllps theorem in Hilbert space) A linear operator P Hilbert space 2 is the generator of a strongly continuous on-paarameter contraction semigroup if and only if P is dissipative, equivalently. -P is m-accretive.
Proof. If P the generator of a contraction semigroup G, then in particular P is dissipative, and hence (P ()) for Φ E L by D.4 Thus, the latter fact can be easily derived directly. Indeed, using that (t)) contraction in D.I we obtain
(e(((G(t) La)o, の)) (G(()())))))oooo Ø)) 10ll z ||((+))]llÍ) 11o11 <
(29)
Dividing by t + 0 and passing to the limit & - 0+, we get (Pé, 6) S O.
Combining the D.3 and D.5 states the eigenvalues of infinitesimal generator P = lim;-s0+ G of contraction operator & has non-positive real parts. However, the operator
usually unbounded and is defined on dense subspace of 2 (Engel et al., 2000). Directly learning the infinitesimal generator P can be more difficult than working with operator G. To simplify the problem, we can consider cigenvalues of within the unit disk in the complex plane during the contraction stage Equation
Spectral Analysis on Gd G. We define the operator norm as ||||| == suppec" ||Grb|2. Given that G is contraction, we know
||||| S =
l 9ll S | <
(30)
Since Gd G a positive operator (because GT self-adjoint and non-negative), the norm || gll is equal the spectral radius of G'GG, G, and hence
a g) c [o, 9 [0, 1].
(31)
This show that G is also a contraction operator, since
|| < 1.
D.3 UNITARY OPERATOR
Definition D.6 (Unitary Operator (Schmüdgen, 2012)
strongly continuous one- me-parameterr uni-
tary group briefly unitary group, is family G ( L2 such that
t RR of unitaries Gtt) on Hilbert space
((t) G for all t
limm--n 9(L + h) = (()) for L2 and 1 R.
Axiom (i) D.6 means that 9 group homomorphism of the additive group IR into the group of unitary operators L In particular, this implies that ( and
g = - =
R.
(32)
RRe()) is denoted as the real part, and R is the range of operator B.
25

---

Published as a conference paper at ICLR 2025
strong continuity of G. It clearly suffices to require (ii) for 0 and for Axiom (ii) in D.6
from a dense subset of L². Since the operators (t) are unitaries, is even enough to assume that (9t)) o) (ø, Φ) for & from dense subset of L2
The generator of the one-parameter unitary group (t) satisfying the following relationship
((t) exp(itP) Vt E E.
(33)
The operator P governs the infinitesimal behavior of the group and is defined by the strong limit:
g -
PΦ
lim L
(34)
it
for all é in domain (PP of P, which consists of those functionals for which limit exists.
Since ((t) = exp(itP) where skew-adjoint P P, the spectrum of (tt) must lie on the unit circle.
Spectral Properties. For all g(t). E R, its spectrum a(G(t) lies on the complex unit circle T, which is the set of complex numbers with absolute value
(G(()) c (rEC|1]==1)
Physical Interpretation of Two Operators. During the contraction phase, changes the probabil- ity distribution inherently reflect the asymmetry of entropy with respect to time. This phenomenon is explained by the second law of thermodynamics: the contraction operator, associated with energy dissipation. However, with invariant measure, the system's evolution governed by unitary operator if it satisfies the Liouville equation (in the classical setting) or the Liouville-von Neumann equation (in the quantum setting) (see B.3 which makes the density matrix or the invariant measure sime-independent (Tao 2008a). respectively. Consequently, unitary operator and its conjugate transpose can describe the forward and backward chaotic dynamics the invariant set.
26

---

Published as a conference paper at ICLR 2025
E ALGORITHM
Algorithm Poincaré Flow Neural Network
PFNN:
Training
Require: Training data D = {[(s), ((11))) li=0 with number of trajectories N and trajectory
length T, relaxation time k, training epochs N learning rate regularized coefficients Y1: 72 encoder 96, M + _R, decoder gø, R- + M with latent feature dimension L
(contraction phase) operator Ge E RR
(measure invariant phase) forward operator gm RRXL backward operator G RLXL Separate the dataset D into contraction phase dataset D. = (((s), == It= and measure invariant phase dataset mm = {[(s] s1+1) T--INN
1im0-
2: for training epoch
1, ...., Nuain do
3: Compute contraction loss Leantraction by Equation with regularized coefficient on D..
4: Update Se, 01. 02 = aV G..01.00 _.oorrraattionn
5: Compute measure invariant loss Lunitary by Equation 8 with regularized coefficient )2 on D...
6: Update Smr Gin.
a Gm. 0,,02 Cunitary
7: end for de
8: return F8. 98. Gc, Gms 9:n
9:
PFNN: Evaluating
Given initial condition 80 E M
11: for timestep t = 1,.... k do
12: Predict using Ge, So (8t+1) Go.0 (3:)
13: end for
14: Collection Dstatistics = {
15: for timestep / >> 1: do
16: Predict using Sm. 901 (3++1) = 9mgg (3e) and 8 90z 9 (8c4.)
de
17: Daaaistiss U {8++1}
end for
19: Estimate required long-term statistics {Mr ar .... using D stariistiss
return long-term statistics {u, o...
27

---

Published as a conference paper at ICLR 2025
F MORE EXPERIMENTAL RESULTS
F.1 LORENZ 63
The Lorenz 63 model (Lorenz 1963). which consists of three coupled nonlinear ODEs,
dæ o x), dt
dy x(p -
dz
Bz
(35)
dt
dt
used as a model for describing the motion of fluid under certain conditions: an incompressible fluid between two plates perpendicular the direction of the earth's gravitational force. particular, the equations describe the rate of change of three quantities with respect to time: æ is proportional to the rate of convection, y to the horizontal temperature variation, and z to the vertical temperature variation. The constants a, p. and 3 are system parameters proportional to the Prandtl number, Rayleigh number, and coupling strength.
this paper, we take the classic choices o = 10, p = 28, and 3 which leads to chaotic behavior with two strange attractors BBB(p- 1), B 1), p 1) and (B 1), - /Bp 1), p 1). Its state is (z, y. 2) E R bounded up and below from +30.
Table 3: KL Divergence for long-term prediction distributions of Lorenz 63 system
Lorenz 63 Components
KL Divergence
FNO
LSTM
Koopman
MNO
PFNN
X
0.5045
00
0.8621
0.6263
0.2408
y
0.9339
XC
0.6975
0.3132
0.5155
z
0.3913
DC
0.4172
0.1793
0.1005
F.2 LORENZ 96
This system was introduced in (Lorenz, 1996) as a low-order model of atmospheric circulation at constant latitude circle. The system consists of K variables S (Si.... Sx) S R representing the values of atmospheric velocity measured along a circle of K evenly spaced locations on the certain latitude of the carth. The governing equations given by.
dSk di
- Sk+1) - S₁ + F,
(36)
where the parameter represents the forcing term. Here, the first term models advection, the second represents linear damping, and F is external forcing. We choose the of variables of K = {9, 40, 80} and the external forcing F = 8, parameters where the system is chaotic with the Lyapunov exponent is approximately 1.67. Its dynamics exhibit strong energy-conserving non- linearity, and for a large F 2 10, it can exhibit strong chaotic turbulence and symbolizes the inherent unpredictability of the Earth's 's climate.
we generated 1800 trajectories for training and 200 trajectories for testing, where each trajectory contains 2000 timesteps with integration time 0.01 and sample rate of 10. Meanwhile, in the best alignment with real scenarios, the initial conditions for trajectories in the training and testing set were drawn from a normal distribution.
Data generation:
Experiment setup: for PFNN models, when training the PFNN (consist) model, we discarded the initial 1000 timesteps to avoid the dissipative process; whilst for training the PFNN (contract) model, the initial steps were maintained to specialize the model in learning the dissipativity in the strly-stage emulation. For all other models, the initial 100 timesteps prediction part, the PFNN (full) model autoregressively predicted the system states with the PFNN (contract) model for the beginning 900 timesteps, and then switched to PFNN (consist) model to
discarded. In the model
28

---

Published as a conference paper at ICLR 2025
Method
True
PFNN
20
-20
20
-20
40
20
-25
0
25
-25
0
50
X
Z
Figure 4: Visualization of long-term statistics of model predictions for Lorenz 63 system of dimensions: we visualize the spatial correlation among components of the velocity, focusing on evaluating the learned spatial correlation from the PFNN model's long-term predictions compared with the ground truth.
predict the states for the rest 1000 timesteps. For all other models, the trajectory for 1900 timesteps was predicted in the same autoregressive way.
F.2.1
EXPERIMENT ON DIMENSIONAL DATASET
Contents:
Table of KL divergence (Table
Short-term prediction plot set (Figure 5
Long-term prediction plot set (Figure
Table 4: KL Divergence for long-term prediction distributions by models across 3 principal compo- nents Lorenz 96 system of 9-dimensional states
Principle Components (PC)
KL Divergence
FNO
LSTM
Koopman
MNO
PFNN
PC1
2.1491
2.2315
2,7091
1.8112
1.5920
PC2
2.1152
2.6774
2.7133
2.1292
1.9539
PC3
2.1090
5.5756
2,4214
2.0916
2.0637
29

---

Published as a conference paper at ICLR 2025
Krepetin
PN0
MIN
115
-38
Dimencon
LSTM
END
350
PPAN
108
(a) Prediction visualization. The ground truth trajectory is visualized in the middle of the leftmost side. The predicted trajectories by baseline models and PFNN shown in the row. corresponding absolute error trajectories the predictions against the ground truth are shown in second row.
Seeppan
Keepsan
(74)
0%6
20
20
NR.SE..0 (5)
15
3-
20
68
100
120
140
16
10'
Preduction (A
steps
(b) Accuracy prediction 15s.
Accuracy of prediction 100s.
Figure 5: Visualization of prediction error in NRMSH: visualize the comparison of model predictions of Lorenz 96 dynamics of 9 dimensions over 15 seconds (150 timesteps, short-term) and 100 seconds (1000 timesteps, mid-term).
30

---

Published as a conference paper at ICLR 2025
Method
True
PFNN
15
PCT
-15
15
-15
15
-15
-0
0
10
-10
0
10
-10
0 PC3
10
PC1
PC2
(a) Spatial correlation of PFNN in principle components
Figure 6: Visualization of long-term statistics of model predictions for Lorenz 96 of 9 dimensions: visualize the spatial correlation among principle components the velocity, focusing on evaluating the learned spatial correlation from the PFNN model's long-tern predictions compared with the ground truth.
31

---

Published as a conference paper at ICLR 2025
F.2.2
EXPERIMENT ON 40 DIMENSIONAL DATASET
Contents:
Table of KL divergence (Table
Short-term prediction plot set (Figure
Long-term prediction plot set (Figure
Principle Components
KL Divergence
(PC)
FNO
LSTM
Koopman
MNO
PFNN
PCI
0.1427
62659.3087
0.1723
0.0954
0.1097
PC2
0.0987
62822.3591
0.1384
0.1009
0.0756
PC3
0.0767
64132.3089
0.1475
0.1255
0.1315
PC4
0.1309
63519.8891
0.1772
0.1382
0.1297
PC5
0.1305
62346.3862
0.1647
0.1106
0.1056
Table 5: KL Divergence for long-term prediction distributions by models across 5 principal compo- nents of Lorenz 96 system of 40-dimensional states
F.2.3
EXPERIMENT ON 80 DIMENSIONAL. DATASET
Contents:
Table of KL divergence (Table
Short-term prediction plot set (Figure
Long-term prediction plot set (Figure 10
Table 6: KL Divergence for long-term prediction distributions by models across 5 principal compo-
nents Lorenz 96 system of 80-dimensional states
Principle Components (PC)
KI. Divergence
FNO
LSTM
Koopman
MNO
PFNN
PCI
0.0785
1.3057
0.1650
0.2384
0.0733
PC2
0.1045
1.3413
0.2477
0.2268
0.0801
PC3
0.1546
1.4145
0.1613
0.0574
0.0524
PC4
0.1600
1.2329
0.2324
0.0394
0.1220
PC5
0.2451
1.2965
0,3733
0.3391
0.1724
32

---

Published as a conference paper at ICLR 2025
Truah
Dincasoo
28 Dinandon
Dunetaios
(a) Prediction visualization. The ground truth trajectory is visualized in the middle of the leftmost side. The predicted trajectories by baseline models and PFNN shown in the first row. The corresponding absolute error trajectories of the predictions against the ground truth are shown in the second row.
MNND
Keopea:
1188
7N0
LSTMiee
recx
70
20
15
NRASSD (N)
10
5-
0-
10
20
10
10
Prediction Stopl (8.
0,3a)
Prodiction (A 0.10
(b) Accuracy of prediction over 15s.
(c) Accuracy of prediction over 100s.
Figure 7: Visualization of prediction error NRMSE: we visualize the comparison of model predictions of Lorenz 96 dynamics of 40 dimensions over 15 seconds (150 timesteps. short-term) and 100 seconds (1000 timesteps, mid-term).
33

---

Published as a conference paper at ICLR 2025
6x 10
Truth
LSTM (z)
Kongman
4x 10'
FNO
MNO
PENN
Energy
30
x 10
10
15
Wavenumber
(a) Energy spectrum of velocity
Method
PENN
-20
20
PC2
-20
PC
-20
20
PC4
PCS
-20
0
20
-20
0
20
-20
20
-20
20
-25
(b) Spatial correlation of PFNN in principle components
Figure 8: Visualization of long-term statistics of model predictions for Lorenz 96 system of 40 dimensions: we visualize density plots of each state dimension of the system velocity predicted by all six models; and then visualize the spatial correlation among 5 principle components of the velocity, focusing on evaluating the learned spatial correlation from the PFNN model's long-term predictions compared with the ground truth.
34

---

Published as a conference paper at ICLR 2025
LSTM
Koapesan
PNO
MNO
PENN
Truth
190.00
Dimension
1STM
Kospron
INO
MND
PONN
Dinension
10
Demension
(a) Prediction visualization. The ground truth trajectory is visualized in the middle of the leftmost side. The predicted trajectories by baseline models and PFNN are shown first row. The corresponding absolute error trajectorics of the predictions against the ground truth are shown in the second row.
Keepraa
Suppes
(52)
0%1
20
20
60
100
120
140
16
Profictionn
Prediction stepe (8 0.10
(b) Accuracy prediction 15s.
Accuracy of prediction over 100s.
Figure 9: Visualization of prediction error in NRMSE: visualize the comparison of model predictions of Lorenz 96 dynamics of 80 dimensions over 15 seconds (150 timesteps, short-term) and 100 seconds (1000 timesteps, mid-term).
35

---

Published as a conference paper at ICLR 2025
Truth
6x 10'
LSTM
Koogman
ENO
MNO
Energy
4x 10"
PENN
3 10
x 10
10
20
30
40
Wavenumber
(a) Energy spectrum of velocity
Method
PENN
20
PCI
PC2
PC3
20
PC4
-20
20
PCS
-20
-25
0
25
-25
25
-25
0
25
-25
25
-25
0
25
PC3
(b) Spatial correlation of PFNN in principle components
Figure 10: Visualization of long-term statistics of model predictions for Lorenz 96 system of 80 dimensions: we visualize density plots of each state dimension of the system velocity predicted by all six models; and then we visualize the spatial correlation among 5 principle components of the velocity, focusing on evaluating the learned spatial correlation from the PFNN model's 's long-term predictions compared with the ground truth.
36

---

Published as a conference paper at ICLR 2025
F.3 KURAMOTO-SWASHINSKY
The Suramoto-SSvashing (KS) equation (Papageorgiou & Smyrlis, 1991]. well-known for its chaotic behavior, is nonlinear PDE applied to studying pattern formation and instability in fluid dynamics, combustion, and plasma physics. The dynamics in 1d spatial domain u(z, f) is given by
du dt
Ou
8uu
0*u Fz
0,
(37)
dr
Or
where x E [0, L] with a periodic boundary condition. The interaction of high-order terms produces complex spatial patterns and temporal chaos when the domain length L is large enough (Cvitanović et al. 2010).
Dataset processing for training and testing: The dataset of Kuramoto-Sivashinsk. simulations we utilized was obtained from the public source, consisting of 1 1200 simulated trajectories with 512 spatial dimensions, u(x, t), on the periodic boundary. The simulations employed pseudo-spectral method with exponential time differencing (ETD) to evolve the dynamics. Each trajectory was generated over 2000 timesteps with integration time 0.005s and sample rate of 10. We sampled 188-dimesioonl u and used 1000 trajectories for training and 200 trajectories for testing, where each trajectory was truncated to 1990 timesteps to preserve the contraction phase before the system reached the ergodic state. For model training, the full trajectory length (1990 timesteps) was used to train baseline models, while the trajectory before the contraction step is was used to train the PFNN contraction operator Gc. and the trajectory after i was used to train the PFNN measure-invarian operators G, and Gün, The determination of the contraction step via the ablation study can be found in Table 2 For model evaluation, the full trajectory length in the test set was applied to all models.
PFNN model architecture: The PFNN model is designed to incorporate three pairs of encoder- decoder layers: one operator layer represent the contraction dynamics, and operator layers to represent unitary characteristics in the measure-invarianl forward and backward dynamics. The model is implemented with Py PyTorch version 1. and the details of the layers are presented in Table
g
Benchmarks for Kuramoto-Sivashinsky We compare the PFNN model with classic recurrent neural networks: the Long Short- Memory network (LSTM), Koopman Operator network, Fourier Neural Operator (FNO), and Markov Neural Operator (MNO). We use the Adam optimizer to minimize the relative 1.2 loss with learning rate of 1e-4, and step learning rate scheduler that decays by half every epochs, for a total of 100 epochs. Based on the provided code source in Appendix G (1) for LSTM, we chose 2 layers and latent feature dimension of 2 x 128; (2) for the Koopman Operator, we chose layer and a latent feature dimension of x 128; (3) for FNO and MNO, set the width to 40, considering the down-sampled state dimension, and chose 30 Fourier modes to improve their ability to learn sighffrequency dynamics; (4) for PFNN, the latent feature dimension is x 128 for both the contraction model and the measure-invariant model.
Table 7: KL Divergence for long-term prediction distributions by models across principal components of Kuramoto-Sivashinsky
Principle Components (PC)
KL. Divergence
FNO
LSTM
Koopman
MNO
PFNN
PCI
143.6508
40.6045
23.8991
9.2303
PC2
133.1811
00
43.0569
26.6271
7.1571
PC3
137.7399
00
40.6242
25.9322
5.9686
PC4
122.4125
29.3022
19,4713
11.7539
PC5
176.6664
00
108.9509
26.7227
6.9519
Accuracy results: we provide more short-term and mid-term accuracy evaluations in IT We noted Figure Notes: Visualization of model prediction performance from the NRMSE of short-term
37

---

Published as a conference paper at ICLR 2025
prediction to relevant statisties of long-term prediction. We used 200 test samples and plotted the mean and standard deviation all test results to obtain the NRMSE of the prediction.
Trush
Keeppan
MIND
PENN
Dimezeioo
Dicrenioo
(a) Prediction visualization. The ground truth trajectory visualized in middle the leftmost side. The predicted trajectories by baseline models and PFNN are shown in the first row. The corresponding absolute error trajectories of the predictions against the ground truth are shown in the second row.
MNO
ESTM
Kengeaan
PFN
NONO
IXTM
Eoepens
MNNN
50-
SRASS 050
25
20
20
Prodiction stye (6 1110
(b) Accuracy prediction over 15s.
(c) Accuracy of prediction over 100s.
Figure 11: Visualization of prediction enror in NRMSE: we visualize the comparison of model predictions of Kuramoto-Sivashinsky dynamics over 15 seconds (150 timesteps, short-term) and 100 seconds (1000 timesteps, mid-term).
38

---

Published as a conference paper at ICLR 2025
102
Truth
LSTM
Koopman
FNO
MNO
PUNN
Energy
10
10
10
30
40
50
60
Wavenumber
(a) Energy spectrum of velocity
Method
PFNN
10
PCI
10
PC2
10
PC3
-10
10
PC4
-10
PC5
-10
-10
10
-10
-10
-10
-10
PC1
PC3
PC4
(b) Spatial correlation of PFNN in principle components
Figure 12: Visualization of long-term statistics of model predictions for Kuramoto Kuramoto-Sivashinkk. visualize the energy spectrum model predictions over wavenumbers; and then we visualize spatial correlation among principle components of PFNN long-term predictions over all test states comparing with the spectrum of the ground truth.
39

---

Published as a conference paper at ICLR 2025
F.4 KOLMOGOROV FLOW
The Kolmogorov flow system, introduced by Arnold Kolmogorov, is classic model for studying fluid instabilities and turbulence in two-dimensional incompressible flows Temam (2012). It is described by a nonlinear, incompressible Navier-Stokes equation driven by a sinusoidal forcing term.
Uu aL
1 v sin(k)) = O, Re
+ Vu+ Vp-
(38)
where u(æ y, t) is the velocity field, p is the pressure, // the kinematic viscosity, and sin k represents the external forcing in the y-direction. The experiment setting follows the (Alieva et al. 2021).
PFNN model architecture: The PFNN model for the 2-D task begins with Convolution2D layers to do patch embedding of the input state, and employs the attention mechanism within Transformer Blocks Dosovitskiy al. (2021) to perform feature encoding. Then, one operator layer is used to learn the contraction dynamics; and two operator layers are used to learn the unitary characteristies in the measure-ine -easuue-invariant forward and backward dynamics. The model implemented with Py PyTorch version 2.3.1, with detailed layer configurations provided in Table
Short-term prediction accuracy: We compared the model performance in the short-term forecasting accuracy by evaluating the absolute error of model predictions with the true states at timesteps 2, 4, 8, 16 and 32 in measure-invariant phase in Figure [3] The result showed PFNN exhibits the smallest
error across all steps, closely matching the ground truth and maintaining low errors even at Step 32.
Estimating statistics in mecarret:-measure: We further evaluated PFNN on the turbulent kinetic energy (TKE) metric, which is defined as:
TKE 2
122 12 1 8²2
(39)
where u', uf are the fluctuating components of velocity compared to velocity mean over time in the x and y directions respectively. The result in Figure showed PFNN predicted trajectory preserved the internal physics statistics in (1) reproducing the TKE distribution with low absolute error (top row); and (2) capturing the time-average state of system in invariant-measen comparing the lowest absolute error (AE) of the predicted mean state against the true mean state, PFNN outperforms the baselines as well (bottom row).
(middle row): (3)
40

---

Published as a conference paper at ICLR 2025
Ground Truth
L.STM
Koopman
FNO
MNO
PFNN
Step 2
42
440
23
20
22-
25
24
90
24
25
90
25
Step
25
0
25
25
60
63
60
60-
42
Step
Step 16
42-
40-
40-
23-
21-
20-
30
25
50
25
Step 32
23
20
25
10
30
25
50
25
Figure 13: Visualization of the absolute difference of model predictions from ground truth in short- term steps {2, 4, 8, 16, 32).
Kuopman
60
60
6U
60
10.0
30
40-
80
7.5
20
20-
20
20
0
0
25
50
25
25
50
25
Truth
Koopman
6c:
60-
60-
60-
40
40
40
40
20
20
20
20-
20
0
0
Mcan Truth
AE LSTM
AE Koopman
AE FNO
AE MNO
AE PFNN
60
60-
60
60-
MW
90
40
40
50
25
50
25
50
25
50
0
25
Figure 14: Visualization of the forecast trajectory sample of Kolmogorov Flow system in top row: the turbulent kinetic energy, which illustrates the accuracy of model predictions in reproducing the kinetic energy distribution; middle row: the mean state that demonstrated each model's ability in capturing the time-average velocity field of a system in equilibrium, where plot bottom row: the absolute error (AE) outstands the difference the predicted mean state against the true mean state.
41

---

Published as a conference paper at ICLR 2025
G EXPERIMENT SETTINGS
G.1 NEURAL NETWORK ARCHITECTURE
Table 8: Neural network architecture for the one dimensional chaotic systems. d is the state dimension and I, is the latent feature dimension.
Components
Layer
Weight size
Bias size
Activation
Encoder
Fully Connected
m 10m
10m
ReLU
Encoder
Fully Connected
10m x 10m
10m
ReLU
Encoder
Fully Connected
10m x L
L
Forward operator Ge
Fully Connected
L x L
Forward operator 9m
Fully Connected
Lx T.
0
Backward operator 9
Fully Connected
Lx L
0
Decoder
Fully Connected
L x 10m
10m
ReLU
Decoder
Fully Connected
10m x 10m
10m
ReLU
Decoder
Fully Connected
10m
Table 9: Model architecture for Kolmogorov Flow
Components
Layer type
Layer number
Channels, (H, W)
Activation
Patch embedding
Convolution2d
1 + 32, (64, 64)
Encoder
Transformer Block
32 64, (32, 32)
ReLU ReLU
Encoder
Transformer Block
64 + 128, (8, 8)
Encoder
Transformer Block
3
128 + 128, (2
ReLU
Contraction
Fully Connected
1
128 2 x 2,
operator Ge
Measure-invariant
Fully Connected
128 x 2 x 2,
operator 9m
Backward
Fully Connected
128 2,
operator 9:.
Decoder
Transformer Block
3
128 + 128, (8, 8)
ReLU
Decoder
Transformer Block
128 64, (32, 32)
ReLU ReLU
Decoder
Transformer Block
64 + 32, (64, 64)
Decoder
Convolution2d
32 + 1, (64, 64)
G.2 DATASET.
(1) Lorenz 63: A 3-dimensional simplified model of atmospheric convection, known for its chaotic behavior and sensitivity to initial conditions. To learn our models, generated dataset consisting of trajectories of 80,000 timesteps from random conditions. (2) Lorenz 96: surrogate model for atmospheric circulation, characterized by chain of coupled differential equations. We generated three datasets corresponding to 9, 40, and 80-dimensional states, respectively, each consisting of 2,000 trajectories with 1,500 timesteps. (3) Kuramoto-SSvashinsky equation: fourth-order nonlinear partial differential equation that models diffusive instabilities and chaotic behavior in systems, such as fluid dynamics, and coctiiondiifusion processes. We sampled -2-dimensionnl dataset consisting of 1,000 trajectories with 500 timesteps from the dataset The description of the three dynamical systems listed Appendix
Dataset for Kuramoto -ivasiiiiii http:://encca..rrrccccrt///////s
42

---

Published as a conference paper at ICLR 2025
G.3 TRAINING DETAILS AND BASELINES.
At a high level, PFNN and other baselines are implemented in Pytorch (Paszke et al. 2019). Both the training and evaluations conducted multiple A100s and Mac Studio with c4-core Apple M2 Ultra CPU and 64-core Metal737 Performance Shaders (MPS) GPU. The evaluation conducted on the CPU.
LSTM: The implementation based on the provided code of othttpss//githuub.cmm
pvlachas/ pylachas/RMN-RC- RCN
Koopman operator: The implementation is based on the provided code of https://
github.com/dyynii http://///yyyaaaaaaaaaaa/////opp
FNO: The implementation is based the provided code of hnttps://pypi.org/
ppcjjet/ffurrle--nerra-l-ppppppp
MNO: The implementation is based on the provided code of offttps://githhbb.cm
neuual_pperattor/markkooyuuuuu __ppraatorr
43


Reviews:
[{"review_id": "lBCTdXeORS", "reviewer": "Reviewer_CMhA", "summary": "This paper proposes Poincaré Flow Neural Network (PFNN), which maps chaotic dynamical systems to a latent space where their evolution is approximated by one of two linear operators: a dissipative operator for the initial contraction phase and a unitary operator for the measure-invariant phase on the attractor. PFNN enforces these properties of the learned operators using carefully designed loss functions. The switching time between the two phases / operators is a hyperparameter of the method that must be tuned. The proposed method is evaluated on several chaotic systems including Lorenz 63/96, Kuramoto-Sivashinsky, and Navier-Stokes equations.", "questions": "1. How do you understand the relation of the proposed method to Koopman learning?\n\n1. For the Koopman operator baseline, what is the exact network setup? Are there any other Koopman learning architectures you think are relevant to the current problem?\n\n2. Why is the model called Poincare Flow Neural Network? The name seems to be an homage to Poincare rather than a description of what the network actually does.\n\n3. In Table 1, it’s not clear what is meant by TKE. Is this an error in TKE with respect to ground truth?\n\n4. Could you elaborate on this statement? \"Notably, while FNO uses FFT to extract features and provides relatively accurate short-term predictions, its reliance on integer Fourier modes tends to produce periodic behavior.\"\n\n5. Could you elaborate on this statement? \"Models like LSTM and Koopman, which do not account for transient behaviors for states lying outside attractors...\" \n\n6. Is there a reason the longer-term experimental results are not reported with uncertainty estimates?\n\n7. Is there a reason you haven't compared with echo state networks?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The proposed method is rigorously grounded in dynamical systems theory, and the design of the loss function provides a nice example of how a theoretical or mathematical understanding of a problem can be incorporated into the learning process.\n2. The paper is generally clear and very well presented, although the overall score here is let down by how the proposed method is contextualized with respect to the literature, as discussed below.", "weaknesses": "### Major\n1. A major issue with the presentation of the paper is how it deals with Koopman operator learning. In particular, the core of the proposed method consists of a learned embedding followed by learned linear evolution in the latent space. This is Koopman operator learning [1,2,3,4], which is a vast and active area of research. Despite forming the foundation of the present work, Koopman learning only receives a few passing mentions in the text.\n\n   For example, with regard to the proposed learning setup, the authors state \"This concept is fundamentally aligned with neural operators, such as the Koopman operator (Bevanda et al., 2021)...\" However, rather than simply being \"aligned\" with Koopman operator learning, the method proposed here _is_ Koopman operator learning. For someone not already familiar with Koopman learning, it would be easy to read this paper and think that the core learning strategy -- linear evolution in a learned latent space -- is a novel contribution. \n\n   The actual contribution of this paper is in fact to split the time domain into separate contractive and measure invariant phases, with a carefully designed loss function for each. In other words, what we are doing here is a kind of \"physics-informed Koopman learning\" for dissipative chaotic systems. This is fine, albeit not particularly groundbreaking. Either way, it should be made very clear how the proposed method relates to prior work on Koopman learning.\n\n    Taking all of this into account, I think it is essential that the proposed method is properly and rigorously placed within the context of existing literature on Koopman learning, both theoretically and experimentally. This requires a detailed discussion of key papers on the foundations as well as the state-of-the-art of Koopman learning. It is also necessary to compare empirically to the state-of-the-art in Koopman learning (the authors do include a Koopman operator baseline, but they do not include any details of the exact setup.) Finally, please also include a paragraph clearly stating the marginal contribution of this paper.\n\n    Ultimately, it is for this reason that I have given this paper a 1 for presentation, even though it is generally quite well written.\n\n2. The results presented in Table 1 are inconclusive and could be presented better. Firstly, only the NRMSE is provided with a measure of uncertainty (although I think the standard error would be more useful than standard deviation for judging the significance of the results). For almost every system, there is no significant difference between FNO, MNO, and PFNN for short-term predictions, as measured by NRMSE. For longer-term predictions, as measured by KLD/MMD/TKE, PFNN performs comparably with MNO for every system except KS, where it does appear better, but the lack of uncertainty estimates makes it hard to draw conclusions. Ultimately, the results as they are presented suggest that PFNN offers relatively little advantage over MNO.\n\n3. The authors do not compare to echo state networks, which are extremely effective networks for learning chaotic dynamical systems. In my opinion, they are an essential comparison in a paper like this. See, for example, [5,6], as well as [7] and citations therein.\n\n### Minor\n1. The authors are a bit loose with the term “neural operator”, which I understand to mean a family of models for learning on function spaces. These models possess specific properties, such as grid invariance, that are not shared by the proposed method. See, e.g., [8,9].\n\n2. As a shortcoming of MNO, the authors note that “choosing the right size for this absorbing ball is tricky”. However, the proposed method requires choosing the relaxation time $k$ as a hyperparameter, essentially replacing one hard-to-estimate hyperparameter with another. More generally, I have concerns about the dependence of the proposed method on the choice of $k$, since it requires that the dataset at hand can be neatly split in this manner (what if certain initial conditions are already on the attractor?). In addition, I find the ablation study in Table 2 unconvincing, since there is little meaningful difference between $k = 7$ and $k = 9$; I think the range of $k$ needs to be extended to larger values.\n\n### References\n[1] Lusch, B., Kutz, J.N. & Brunton, S.L. Deep learning for universal linear embeddings of nonlinear dynamics. Nat Commun 9, 4950 (2018). https://doi.org/10.1038/s41467-018-07210-0\n\n[2] Qianxiao Li, Felix Dietrich, Erik M. Bollt, Ioannis G. Kevrekidis; Extended dynamic mode decomposition with dictionary learning: A data-driven adaptive spectral decomposition of the Koopman operator. Chaos 1 October 2017; 27 (10): 103111. https://doi.org/10.1063/1.4993854\n\n[3] Otto, S. E., & Rowley, C. W. (2019). Linearly Recurrent Autoencoder Networks for Learning Dynamics. SIAM Journal on Applied Dynamical Systems, 18(1), 558-593. https://doi.org/10.1137/18M1177846\n\n[4] E. Yeung, S. Kundu and N. Hodas, \"Learning Deep Neural Network Representations for Koopman Operators of Nonlinear Dynamical Systems,\" 2019 American Control Conference (ACC), Philadelphia, PA, USA, 2019, pp. 4832-4839, doi: 10.23919/ACC.2019.8815339.\n\n[5] Jaideep Pathak, Zhixin Lu, Brian R. Hunt, Michelle Girvan, Edward Ott; Using machine learning to replicate chaotic attractors and calculate Lyapunov exponents from data. Chaos 1 December 2017; 27 (12): 121102. https://doi.org/10.1063/1.5010300 \n\n[6] Pathak, J., Hunt, B., Girvan, M., Lu, Z., & Ott, E. (2018). Model-Free Prediction of Large Spatiotemporally Chaotic Systems from Data: A Reservoir Computing Approach. Physical Review Letters, 120(2), 024102.\n\n[7] Yan, M., Huang, C., Bienstman, P. et al. Emerging opportunities and challenges for the future of reservoir computing. Nat Commun 15, 2056 (2024). https://doi.org/10.1038/s41467-024-45187-1\n\n[8] Kovachki, N., Li, Z., Liu, B., Azizzadenesheli, K., Bhattacharya, K., Stuart, A., & Anandkumar, A. (2023). Neural Operator: Learning Maps Between Function Spaces With Applications to PDEs. Journal of Machine Learning Research, 24(89), 1-97.\n\n[9] Azizzadenesheli, K., Kovachki, N., Li, Z. et al. Neural operators for accelerating scientific simulations and design. Nat Rev Phys 6, 320–328 (2024). https://doi.org/10.1038/s42254-024-00712-5", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zEsakKZ6sB", "reviewer": "Reviewer_Fbbj", "summary": "In this paper, the authors proposed a Poincare Flow Neural Network (PFNN) to model dissipative chaotic dynamical systems that improves matching of invariant statistics in forecasting over long horizons. The proposed method uses an autoencoder to reformulate prediction on the state space into prediction on a finite-dimensional learned feature space, on which a linear operator might be able to represent the forward evolution of the system in the feature space theoretically. By estimating the transient time the system needs to converge to the attractor with \"relaxation time\", the trajectory data sets are separated into a contraction phase and a measure-invariant phase. Using soft constraints formulated as regularization loss of the learned linear operator, the physical properties of the system, namely (1) dissipativity, in other words, volume contraction during the contraction phase (2) measure invariance due to a unitary operator construction, are incorporated into the training loss. Overall, the method leverages a Koopman-like structure to encourage the model to match intrinsic physical behaviors of a dissipative chaotic system by adding soft constraints on the linear operator. The improvement in preserving invariant statistics in the true system of the proposed method is validated in a few numerical experiments compared to recent benchmarks in the literature.", "questions": "1. When the finite-dimensional features spaces are constructed (L different learned features), wouldn't the \"operators\" G_c, G_m simply collapse to matrices? If so, please make it clear as matrix transpose notion has already been used in the paper. \n2. What is the main difference between the proposed architecture from the Koopman learning literature? e.g. Lusch, Bethany, J. Nathan Kutz, and Steven L. Brunton. \"Deep learning for universal linear embeddings of nonlinear dynamics.\" Nature communications 9.1 (2018): 4950.\n3. In Theorem 3.1, it seems that the result does not depend on the choice of the autoencoder. Is this true for every set of feature functions learned by the encoder?\n4. Just out of curiosity, does such a linear operator always exist? If so, is there any intrinsic difficulty in approximating such infinite-dimensional operator using finite dimensional feature space for certain systems?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Originality: The proposed method provides a new perspective on incorporating dissipative chaos physical behaviors into a neural network model through simple to compute yet powerful linear operator properties.\nQuality: The proposed method has a strong mathematical foundation, which is explained well in the paper. Additionally, the reviewer finds the numerical experiments to be extensive and the ablation study helpful.\nClarity: Overall the paper is written clearly, with minor editorial comments on presentation listed in other sections of the review.\nSignificance: The paper addresses an important problem in learning chaos, which has extensive practical applications including weather forecast, turbulent flow modeling etc.", "weaknesses": "Overall, the reviewer finds the method innovative and well articulated. However, there are a few technical details and writing concerns that need to be addressed to make sure the paper can be accepted in a form that clarifies its own contributions.\n\nTechnical issue:\n1. It is very crucial in this approach to obtain a good separation of trajectory data into the contraction and measure-invariance phase, since these two phases use different operators. However, it seems like all the approximation techniques introduced in Section 3.2 can only reduce the estimation to the same order of magnitude, which might be problematic if the transient period is relatively short.\nMore importantly, the steps for separation chosen in the paper $\\frac{1}{c_{LSI} log(||\\phi||_2/\\epsilon)$ depend explicitly on $\\phi$, which is the learned feature function. It is confusing how the steps can be chosen using this formula before the network is trained, however, this is a necessary step to preprocess the trajectory dataset.\n\n2. The auto-encoder does not seem to have a structure to help learn the feature functions that are orthonormal or help construct a good finite-rank operator approximation. The reviewer would appreciate clarification of the link between Theorem 3.1, or more generally the design of autoencoder structure, and the Koopman theory.\n\nWriting issues:\n1. To the reviewer's best knowledge, the proposed structure is very similar to the auto-encoder structure inspired by Koopman operator theory proposed by Steven Brunton (which includes a series of papers). Although the authors have cited one of these works, the reviewer suggests that the author to clarify that the auto-encoder structure and Koopman-inspired network design is perhaps not a novel design.\n2. It seems like quite a few theorems in the paper e.g., Proposition 3.2 and Theorem 3.3 are standard results either in functional analysis or Koopman literature. The reviewer fails to see the points of rederiving such results, instead, one might be able to cite these well-known results and directly use them.\n3. It seems like the theory in Theorem 3.1 and functional analysis in relation to constructing the finite-rank operator which approximates the designed linear operator suggests the feature functions need to be orthonomal or at least somewhat independent/orthogonal. Since the feature functions are learned using auto-encoders, it would be very helpful to provide some insights on what features are learned. It's a bit concerning to the reviewer that the authors make the point that FNO/MNO relies on periodic Fourier modes contradictory to the chaos nature without the presence of their own learned feature map results.\n4. This paper clearly has its own contributions in leveraging linear operator theory and incorporating certain properties. In order to keep the literature review fair, the reviewer suggests the authors to consider revising introduction especially comparison to neural operators. To make matters clear, the reviewer is not at any point a co-author of any neural operator work but is familiar with this line of work. It seems like more general forms of neural opeartors (1) do not necessarily use Fourier modes (2) might use nonlinear evolution such as kernel structure. It might not be fair to group this entire line of work into the linear evolution box and claims strict improvement over FNO/MNO (Neural operators with Fourier modes), which is a fraction of them.\n5. It seems like the framework assumes a finite-dimensional ODE structure of the dynamical system. Given a discretization grid, this might be true for a broad range of systems, the reviewer suggests the authors to make this limitation clear and restricts the scope to given spatial discretization. There are other papers in the domain that can deal with more general discretization setup such as DeepOnet or Neural operators. Clearly, this paper would not be able to address discretization issues like the other papers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BXQ8omEFDW", "reviewer": "Reviewer_pS8g", "summary": "This paper proposes the Poincare Flow Neural Network (PFNN), an architecture for learning in dissipative chaotic systems. PFNN maps the input to a latent space and proposes to learn linear evolution operators in this latent space. PFNN is applied on several systems, including Lorenz 96, Kuramoto Sivashinsky, and Navier-Stokes, and it demonstrates improvements over prior baselines.", "questions": "1. At inference-time for a system, how do you choose $k$? How robust is the proposed model to variations in $k$? \n2. Does this $k$ depend on the input trajectory? One could imagine a trajectory initialized very close or very far from the attractor. How is $k$ determined in that case?\n3. How are the operator learning architectures (FNO and MNO) parameterized for the finite-dimensional problems (L63, L96)? Is this different from how they are parameterized for KS and NS?\n4. Does constraining the spectrum of the operator $\\mathcal{G}$ give contraction in the physical space as well? I suggest adding more details about this in the paper to make it clearer.\n5. Is there potential for this method to be applied on function spaces, e.g., to forecast a Navier-Stokes fluid flow in a discretization-agnostic manner?\n6. Since the model is dependent on the input discretization, how does the proposed architecture scale with input resolution? For instance, how does convergence and performance compare on Navier-Stokes at 256x256 resolution vs. on 64x64 resolution?\n7. Does $\\hat G$ in equation 4 correspond to the approximation finite-rank operator?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and includes a good exposition of key concepts (e.g., ergodicity, global attractor, etc.) for audiences that may not be as familiar with the mathematical background. The experiments are in-depth, particularly in the appendix. The proposed method also outperforms prior works across a variety of settings and metrics. The detailed proofs in the appendix are also helpful.", "weaknesses": "The authors should explain in more detail the distinction between their framework and prior works. In particular, some details on the distinction between neural operators and the Koopman operator approach are not clear. To my understanding, the proposed approach is based on Koopman operator theory and is learning, in part, both the discretized operator matrix $\\mathcal{G}_{ij}$ and the encoder and decoder feature functions. However, the chaotic systems in question are assumed to be defined over a subset of $\\mathbb{R}^d$. In contrast, the neural operator (e.g., FNO, DeepONet) setting is one of mapping between function spaces. As such, methods like MNO are most interested in learning the solution operator to a time-evolving function (PDE). To my understanding, the proposed approach cannot do this directly in function space as designed, as it is fixed to a specific data resolution. Is this accurate? To me Koopman and neural operators seem to be different paradigms, but in my opinion the authors do not make sufficient distinction between them. However, my interpretation may be mistaken. Please let me know if I am missing a connection in this case.\n\nI also have some questions regarding the experiments and baselines. These can be found in the “Questions” section below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v2T4pv4dOm", "reviewer": "Reviewer_GyXm", "summary": "In this work, the authors address learning chaotic dynamical systems by introducing a model / learning paradigm that incorporates the physical properties of dissipative chaotic systems, namely that the volume of transient states contracts towards an attractive set on which measure is preserved by the dynamics. Across a variety of well-studied systems, the authors demonstrate the improvement from using their proposed PFNN method.", "questions": "### Q1: Is Equation 2 obvious?\nAs someone less familiar with operator theory, is the existence of $\\mathcal{G}$ in Eq 2 obvious? The authors provide strong citations for the other theoretical aspects of their work, but I think citations are missing / would be helpful for the section where $\\mathcal{G}$ is introduced.\n\n\nAdditionally, does the existence of $\\mathcal{G}$ require any assumptions on the features $\\phi$?\n\n### Q2: In Lines 192-193, why do the authors state that the invariant measures “varies over time”?\nIsn’t the invariant measure consistent over time?\n\n### Q3: Why does $\\gamma_2$ need to be between 0 and 1?\nIs this a strict requirement or simply what the authors explored in their experiments?\n\n### Q4: In the ablations, did the authors explore training only on the attractor?\nIn other words, what if we ignore $k$ in the training and assume all initial states are already on the attractor (e.g., by simply time evolving trajectories in the training/test sets and “throwing out the first $k$ steps). How does this affect performance of PFNN and baselines?\n\n### In Figure 5b/c and other similar figures, where do the error bars / shaded regions come from?\nHow did the +/- get computed here?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "The problem and proposed methodology are well motivated. The exposition of why the various design decisions are made is rigorously stated and clear.\n\nThe experimental section is extensive and the improvements of PFNN over baselines is compelling. The authors provide a comprehensive appendix with all the relevant experimental details.\n\nFigure 1 is very illustrative and a great way to convey the high level idea.", "weaknesses": "Overall, my main criticism is that while the methodology is well-motivated and rigorous, the actual implementation is somewhat difficult to follow. The phases and design decisions seem to be somewhat _ad hoc_, which is reflected in the number of hyperparameters (5 or 6 by my count), and I personally find the implementation difficult to follow. For example, I appreciate that the authors provided code, but looking through the code implementation, I struggle to see how the pseudocode in the algorithm maps to the implementation: in neither the Lorenz96 or NS script do I see the two-phase aspect of the training nor how equations 4, 6, or 8 get consistently mapped to code. I want to be clear, I am not taking the code implementation into account in my review/score, I only bring it up to illustrate that I think this method is less straightforward to implement.\n\nBelow I list more specific comments/concerns. The list roughly follows the order of appearance in the manuscript. In my opinion W1, W3, W5, W7/W9/W11, and W8  are the most important among these.\n\n---\n\n### W1: Is PFNN a parameterization or a training algorithm?\nAt a high level, I think the authors should clarify what PFNN refers to: i.e., is it the parameterization of the learned model (encoder + linear feature evolution) or does it refer to the 2 phase training paradigm with the various objectives proposed?\n\n### W2: Lorenz63 is missing from the list of experimental setups in the abstract.\nNot a big deal, but for completeness, I would recommend listing it.\n\n### W3: Characterization of Schiff et al. (2024) is incorrect.\nThe authors place their work in contrast to Jiang et al. (2024) and Schiff et al. (2024), stating that unlike these works, the current proposal does not make any assumptions about the underlying PDE. While I agree that Jiang et al. (2024) requires knowledge of the system to build their set of invariant properties, this is not the case in Schiff et al. (2024). In fact, Schiff et al. (2024) clearly state that their proposed methodology does not make any assumptions about the underlying system, only that it supports an invariant measure, as in this work. In Schiff et al. (2024), the invariant measure is simply estimated as an empirical distribution from the trajectories in the training/test sets. Indeed, the same way the authors in this manuscript compute KL/MMD as evaluation metrics, Schiff et al. (2024) compute MMD as a regularizer.\n\nI believe this characterization of previous work should be corrected. This does not detract from the fact that the current work, with its parameterization and two phase training, is novel. \n\n**Additionally, it would be interesting to see how PFNN compares to baselines, say MNO (the strongest baseline), trained in conjunction with the regularized objective from Schiff et al. (2024). I highly recommend that this comparison be added to the experimental section.**\n\n### W4: Introduction of the method is a bit confusing/misleading.\nIn the first paragraph the authors state that the algorithm / method has 3 steps. But if I understand the methodology correctly, then step 1 (i.e., learning the autoencode) is happening throughout both phase 1 (contractive) and phase 2 (measure preserving) of the training. This should be re-worded / clarified.\n\n### W5: Consistency regularization seems to be dropped after Eq 4 is introduced.\nThe authors introduced the second term in Eq 4 as a way to enforce bijectivity. However there are a few things I find confusing:\n1) After Eq 4, (e.g., in Eq 6 and Eq 8) this term is dropped. Why?\n2) The hyperparameter $\\gamma$ in Eq 4 is not discussed elsewhere in the manuscript and is not included in ablation studies.\n3) The training algorithm in Algorithm 1 indicates that losses are computed by Eq 4 and Eq 6 (Line 1363) / Eq 4 and Eq 8 (Line 1366), but what does this mean in practice? Is the regularization term from Eq 4 included? The first term of Eq 4 is shared in both Eq 6 and Eq 8, is this term double counted? (presumably not)\n\n### W6: Missing citation to “Evolve Smoothly, Fit Consistently: Learning Smooth Latent Dynamics For Advection-Dominated Systems” Wan et al. (2023).\nIn Wan et al. (2023), the learning dynamics are learned in a latent space and there is consistency regularization to ensure that the autoencoder is invertible. This shares several properties with the proposed methodology in the current manuscript. The authors should include Wan et al. (2023) in their related works and discuss how PFNN differs.\n\n### W7: Isn’t the regularizer in Eq 6 quite expensive to compute?\nTo enforce the contractive property of $\\hat{\\mathcal{G}}$, the authors compute eigenvalues. Isn’t this quite expensive: $\\mathcal{O}(L^3)$?\n\n### W8: The choice of hyperparameter $k$ seems hard / requires knowledge of the system\nThe authors claim their method does not require any knowledge of underlying true dynamics, but the cutoff $k$ between contractive and measure preserving phases seems like it would highly depend on the system of interest.\n\nAdditionally, I would recommend adding more information (perhaps in the appendix) about how $c_{LSI}$ is derived/computed.\n\n### W9: What is the complexity of computing objective in Eq 8?\n$\\mathcal{G}^*_m$ is described as the conjugate transpose, but in the provided code implementation it seems like it is computed as the Moore-Penrose pseudoinverse of the parameters of $\\mathcal{G}_m$, unless I am misunderstanding something here. Is this not quite expensive to do and backprop through on each step/batch?\n\n### W10: Algorithm should be provided in the main text\nWhile I recognize that space is limited, I highly encourage the authors to find a way to include the training algorithm in the main body of the text. There are several components to keep track of in the methodology, and having them centralized in the main text in the form of the training Algorithm is really important and would aid clarity, as opposed to having to refer to the appendix. \n\n### W11: Computation complexity of PFNN vs. baselines should be noted and commented on.\nRelated to my comment in W7/W9, it seems PFNN has several compute intensive overhead components. The complexity (both asymptotic and actual wall clock) of PFNN should be remarked on / included in the results sections.\n\n### W12: Additionally, parameter counts should be included in Table 1\nIn addition to wall clock, the parameter counts of the various baselines and PFNN should be indicated to ensure methods are fairly compared.\n\n### W13: Adding +/- for different random seeds on MMD/KLD would be very helpful\nIn my experience, ranking on metrics for the experimental setups explored here can vary significantly across different random seeds. If possible, adding this to Table 1, particularly for the MMD/KLD metrics, would make the results more compelling.\n\n### W14: Why is MMD/KLD omitted for the NS experiment?\nIn particular, in the appendix, TKE is indicated to be a short-term accuracy metric, and so NS experiments seem to not have any reported “long-term” metrics.\n\n### W15: In the ablation table, why is performance worse for $L = 512$?\nI’d expect the trend (better results) to continue as $L$ grows.\n\n### W16: Figure 5a (and other similar ones) is missing legends/labels\nI am assuming the top row indicates trajectories and the bottom row shows error, but this needs to be indicated.\n\n### W17: Minor typos / Formatting Issues & Suggestions:\n- Defining $\\nu$ as the data distribution should be more clear and included in the notation table in the appendix\n- $m$ as the system dimension should be defined in paragraph 2 of Section 4. I would recommend also recalling what $m$ is in the caption of Table 1.\n- Table 1 seems to have some bold formatting typos, e.g., the standard deviation is bolded for some rows (e.g. Line 401).\n- Line 212: $\\mathcal{G}_{ij}$ is missing the $L$ subscript.\n- Line 221: In equation 4, the expectation should be over $\\nu$ not $\\mu$?\n- Line 259/260: The footnote spacing is off. Should be attached to the previous word; looks like there is an added space.\n\n---\n\n**References:**\n\nRuoxi Jiang, Peter Y Lu, Elena Orlova, and Rebecca Willett. Training neural operators to preserve\ninvariant measures of chaotic attractors. Advances in Neural Information Processing Systems, 36,\n2024.\n\nYair Schiff, Zhong Yi Wan, Jeffrey B Parker, Stephan Hoyer, Volodymyr Kuleshov, Fei Sha, and\nLeonardo Zepeda-Nunez. Dyslim: Dynamics stable learning by invariant measure for chaotic systems. arXiv preprint arXiv:2402.04467, 2024.\n\nWan, Zhong Yi, Leonardo Zepeda-Nunez, Anudhyan Boral, and Fei Sha. \"Evolve smoothly, fit consistently: Learning smooth latent dynamics for advection-dominated systems.\" arXiv preprint arXiv:2301.10391 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)