Phase2vec: dynamical systems embedding with a physics-informed convolutional network
Authors: Matt Ricci
Noa Moriel
Zoe Piran
Mor Nitzan
Venue: ICLR 2023 notable top 25% (2023)
Keywords: dynamical systems
convolutional networks
computational physics
physics-informed

Abstract: Dynamical systems are found in innumerable forms across the physical and biological sciences, yet all these systems fall naturally into equivalence classes: conservative or dissipative, stable or unstable, compressible or incompressible. Predicting these classes from data remains an essential open challenge in computational physics on which existing time-series classification methods struggle. Here, we propose, phase2vec, an embedding method that learns high-quality, physically-meaningful representations of low-dimensional dynamical systems without supervision. Our embeddings are produced by a convolutional backbone that extracts geometric features from flow data and minimizes a physically-informed vector field reconstruction loss. The trained architecture can not only predict the equations of unseen data, but also produces embeddings that encode meaningful physical properties of input data (e.g. stability of fixed points, conservation of energy, and the incompressibility of flows) more faithfully than standard blackbox classifiers and state-of-the-art time series classification techniques. We additionally apply our embeddings to the analysis of meteorological data, showing we can detect climatically meaningful features. Collectively, our results demonstrate the viability of embedding approaches for the discovery of dynamical features in physical systems.

Published as a conference paper at ICLR 2023
PHASE2VEC:
DYNAMICAL SYSTEMS EMBEDDING WITH A PHYSICS- INFORMED CONVOLUTIONAL NETWORK
Matthew Ricei
Noa Moriel
School of Computer Science and Engineering The Hebrew University
School of Computer Science and Engineering The Hebrew University
Jerusalem, Israel matthew. ciciciio @ma www.i.iii..... .ac..a.....
Jerusalem, Israel noa www.elrreeeoooooo.... @mai i 1, www..uiii....... aa...a...... i1
Zoe Piran
Mor Nitzan
School Computer Science and Engineering The Hebrew University
School of Computer Science and Engineering, Racah Institute of Physics, Faculty of Medicine
Jerusalem, Israel zoe. ran@mail il huji. ac. i1
The Hebrew University
Jerusalem, Israel mor ni mnitzan@maill hujji ac i1
ABSTRACT
Dynamical systems found in innumerable forms across the physical and bi- ological sciences, yet all these systems fall naturally into equivalence classes: conservative dissipative, stable or unstable, compressible or incompressible. Predicting these classes from data remains an essential open challenge in com- putational physics on which existing time-series classification methods strug- gle. Here, we propose, phase2vec, an embedding method that learns high- quality, physically menningff representations tems without supervision. Our embeddings are produced by a convolutional back- bone that extracts geometric features from flow data and minimizes a physically- informed vector field reconstruction loss. The trained architecture can not only predict the equations of unseen data, but also produces embeddings that encode meaningful physical properties of input data (c.g. stability of fixed points, conser- vation of energy, and the incompressibility of flows) more faithfully than standard blackbox classifiers and state-of-the-art time series classification techniques. We additionally apply our embeddings to the analysis of meteorological data, showing can detect climatically meaningful features. Collectively, results demon- strate the viability of embedding approaches for the discovery of dynamical fea- tures in physical systems.
oon-iimesioonal dynamical sys-
INTRODUCTION
The application of deep neural networks to the prediction Lusch et al. 2018). control (Haluszezyn-
ski Räth 202I), and basic understanding (Raissi et al. 2019) of dynamical systems has spurred important ad vdvances across the sciences, from neuroscience (Sussillo et al. 2016) to physics (Krri- adakis et al., 2021). from earth and climate science (Reichstein et al.) 2019 Fresca al. 2020) to computational biology (Sapoval et al.) 2022). However, while deep learning has already contributed significantly to the analysis of single systems, its role in understanding the underlying principles of dynamical systems in general remains somewhat limited. The ability predict control one system, after all, tells us little about how to do so with another. Crucially, without the ability to gen- eralize across behaviors of numerous dynamical systems, the problem of constructing systems quite difficult, and most current attempts rely on exhaustive search through system parameters (Hart al. 2012 Scholes et al. 2019).

---

Published as a conference paper at ICLR 2023
X F(X:E)
vF(F))
Xrreen ((X)Eroon -Feccon E
COBY
Cour + RaIU,
Eusbodding
PC
T
1,0,000 Ai
Cony
Figure I: phase2 vec pipeline. phase2vec learns high-quality embeddings of dynamical sys- tems from phase space data. An input vector field (represented as a stream plot, i.e. set of trajec- tories) is passed through series of convolutional, rectification, and downsampling layers. Terminal convolutional features are aggregated in a relatively low-dimensional (d = 100) layer before being mapped to a set of estimated coefficients, Execon These coefficients are used to weight dictio- nary of polynomials in variables (depicted & = 2), and the resulting linear combination (depicted transposed above) comprises the estimated governing equation, Frecon (X, Erecon) associated to the reconstructed vector field.
A machine learning framework that could elucidate the latent dynamical structure across numerous systems having different governing equations and parameters would be an important first step in this direction. However, progress this front has been limited, largely because representations of even a single dynamical system are necessarily high-dimensional (many initial conditions over time) and the problem of understanding single systems in detail is already daunting as it is useful. The few existing approaches that analyze multiple systems simultaneously focus on the limited setting of rapid adaptation new parameter regimes for governing equations (Kirchmeyer al. 2022) or rely on hardwired features and laborious search over model manifolds (Quinn et al. 2021]
To address this challenge, we propose phase2vec, dynamical systems embedding model which learns high-quality, low-dimensional, thysically-meaning (Fig. D.. We focus here on two- and three-dimensional dynamics because of their preponderance in nature, ranging from classical non-linear models of population growth like the Lotka Volterra model to increasingly important models of climate dynamics (for applications to both, see Sec. 4.. Similarly to many word embedding approaches (Mikolov et al., 2013), which approximate the meaning of words using statistical regularity instead formal semantics, our dynamical embeddings seek to model the "semantics" of dynamical systems from data instead of via analytical investigation. phase2ved is so-called since it is based on a vector of convolutional features extracted from the vector field representing the dynamics in phase space of input data. To encourage physically- meaningful solutions from our encodings, we use a decoder which reconstructs input dynamics from library of basis functions. The full pipeline is trained with reconstruction loss placing special emphasis on fixed points, giving phase2vec a physically-informed "learning bias" (following the terminology for principles of wwwsscc--inoomeeeeee informed learning suggested in (Karniadakis al. 2021 Box 2).
representations of dynamical systems
Like other embedding methods, we train our system on an auxiliary task, in our case one involving equation reconstruction. We show that this auxiliary task, combined with physics-informe loss, steers the network towards learning dynamically stable, informative embedding!
The main contributions of this paper are:
phase2vec, a novel neural-network based embedding method that can be used to learn low-dimensional representations of dynamical systems in an unsupervised manner.
2. A demonstration that these embeddings can be used decode the governing equations of testing data, recover sparse underlying models, and denoise input data in way that pre-
Cooee available here: http:///hthhbh.comnniiambhhhhhee
2

---

Published as a conference paper at ICLR 2023
serves dynamical characteristics better than per-equation fitting approaches (e.g. LASSO; (Tibshirami 1996)), ) even on noisy data.
3. Quantitative analysis showing that important physical properties (conservation of energy, incompressibility class of dynamical stability) can be decoded from phase2vec embed- dings with much greater accuracy than both blackbox classifiers and contemporary time series classification approaches.
4. An application of phase2vec to the analysis of climate data, demonstrating that the em- beddings capture meaningful structure in real data, such the characteristics of distinct temperature zones.
RELATED WORK
Data-driven discovery of governing equations There is a large body of work concerning the discovery of governing equations of single systems from time series data (reviewed in (Timme & Casadiego 2014)). Early work by (Bongard & Lipson, 2007 Schmidt & Lipson 2009) fit the coef- licients of an explicit basis of polynomials time series data using least squares, an approach which was modernized in the sparse reconstruction method of SINDy (Brunton et al. 2016) and its autoen- coder formulation in (Champion et al., 2019). Other approaches eschew the explicit modeling of dynamical equations in favor of implicitly representing the dynamical transition as the discrete-time action of an autoencoder on state samples. For example, (Lusch et al. 2018) approximated the ac- tion of the Koopman operator in such a set-up, while (Bramburger et al. (021)) used autoencoder to learn topologically conjugate representations of unseen equations. Importantly, all of the above methods focus specific system at hand and do not generalize to additional settings.
Generalization across physical systems While there has been some progress in estimating equa- tions for single systems, inferring the dynamics of multiple systems simultaneously has proven more challenging (Brown & Sethna 2003). The manifold boundary approximation method of (Quinn) jet al. 2021 has been used in multiple cases (Transtrum & Qiu 2014) to estimate the minimal mechanisms underlying behavioral archetypes within single parameterized dynamical family (e.g. a family of Ising models parameterized by coupling strength (Teoh et al. 2020)). This method can be used to identify dynamics that happen to lie on the boundaries of the model manifold, but its use in identifying arbitrary, user-defined dynamical classes is less straightforward. A more recent line of work (Clavera et al. [2099) Lee al. 2020 Yin al. 202T Kirchmeyer et al., [2022] Wang et al.] 2022) has taken an adaptation approach whereby the representation of dynamics observed during training period can be quickly updated to a new setting in which the parameters of the underlying system have changed. Here, too, the focus is the generalization to new parameter regimes of the same underlying equation. To our knowledge, the problem of learning generalizable representations of dynamical models whose functional forms vary widely has not been systematically addressed.
Vector-field-basel approaches All of the above approaches have taken time series their input data. Instead, vector field representations can be generated from known governing equations or from time series measurements, for instance by binning velocities (Sneed & Komaee 2021) by fitting a statistical model (e.g. (Qiu et al. 2022b). Among the approaches which rely on vector field data (Battistelli & Tesi 2020), which provided theoretical guarantees for the performance of two-class vector field classifiers for linear systems. (Ye et al.) 2020) used supervised convolutional network to analyze a vector field arising from aerodynamic simulations, but they neither addressed the prob- lem of generalization across systems or equations nor the problem of classifying general physical properties. Others sought to construct vector fields subject to the constraints on the number, loca- tion, and stability of fixed points which are mathematically imposed by Conley index theory 1978 Zhang et al. 2006 Chen et al. 2008). Our approach not only combines the classifier method of (Battistelli & Tesi 2020) with the physical biases of (Zhang al.) 2006 Chen et al., 2008), but importantly introduces deep-learning- based feature extraction, supporting generalization across complex non-linear dynamics.
3

---

Published as a conference paper at ICLR 2023
3 METHODOLOGY
Let D = { be set of continuously differentiable, parameterized vector fields on R each with parameter vector E,. Each F associated to dynamical system via
X = F(X
(1)
When F is identified with dynamical system in this manner, we refer to I2 as the system's 's phase
space and the value (t) solution to Eq. 1 the state the system at time t. Our goal
to learn 7-parameterizzd embedding map capture the principal factors of variation the underlying physical system given by Eq. [Fig. [ We reason that a good map, 4/, is one that produces embeddings, 2, of testing data from which governing equations can be faithfully decoded. Much the way word embeddings are learned by optimizing an embedding map self-supervised auxiliary task of context prediction (Mikolov) et al. 2013), we train our embedding map on a self-supervised auxiliary task of governing equation prediction.
()(F;;)) for each F; E 'D so that the E I
3.1 ARCHITECTURE
In practice, we evaluate each on a (potentially different) phase space lattice, each of whose dimensions has resolution n, so that D is set of n² arrays whose elements indicate the q- dimensional velocity equation each discrete location. We set 64 but found no qualitative difference in performance when n was varied between 32 and 128 (Fig. A.14). We instantiate the map, 1b as three-layer convolutional architecture (with kernels having either 2 or 3 spatial dimen- sions depending on q) terminating in one fully-connected layer mapping convolutional features to d-dimensional phase2vec.
Following (Brunton et al. 2016), embeddings are passed through two-layer multi-layer perceptron, producing a vector of estimated coefficients, Erecon E RPxq, These coefficients are used to form linear combinations with a set of basis functions (xX) (X), ..... (x()) for column vectors d; The reconstructed equation taken as
X 4 ((X) Frecon (X, Erecon)
(2)
where (X) ( for Eソ=1a, S all possible unit-coefficient monomials up given
degree, c. We set c (cubic degree) so that for g 2, Erecon R0x2 (Fig. 1 Other bases of larger polynomial degree of non-polynomial functions, e.g. including sin(x). are also possible
(Brunton et al. 2016). We chose e = 3 following earlier work (Iben Wagner 202I) which showed this value was sufficient for the types of dynamical systems considered here.
3.2 LOSS AND TRAINING
Following (Champion al. 2019 train system with loss that balances reconstruction fidelity with sparsity of the underlying equation. However, we also normalize our reconstruction loss in manner that emphasizes special "skeletal" points of the dynamics. In particular, take our reconstruction loss to be
1 - Xreeoollz
Li CXX,X Xreem))
|X +
for small corrective constant, > 0. Here, L1 is especially large where X vanishes, lending special emphasis to fixed points and "slow" regions the dynamics, which are often the loci of important dynamical phenomena (Tredicce & Lippi] 2004) (see Supp. A. These regions of low magnitude in an array only have meaning in the physical setting (as opposed to, for example, in image data), placing phave 2vec under the "learning biases" rubric in physics-informed machine learning (Kar- niadakis et al. (2021), Box 2). We use simple sparsity penalty Lx(X) |reeon (x) and train (X;r) by approximating
7" arg min ErED [(X
X Xrocon)
7
4

---

Published as a conference paper at ICLR 2023
for -= L1 11++CC 3C2 with sparsity regularizer 3. Note that the loss measures the discrepancy between the true and reconstructed vector fields and not between the true and reconstructed parameters.
We investigated both q and q 3 dimensional systems. The training set D in either case was a collection of g-dimensional polynomial ODEs generated from the dictionary & with coefficients with probability .75 and otherwise sampled uniformly [-3,3]. We then tested generalization two data sets, one measuring generalization to new parameter regimes for the training set and another measuring generalization to new functional forms not observed during training. For the former, we used a set of equations sampled from the same distribution generating DD but having disjoint set of coefficients, and, for the latter, we used a collection of "classical", real-world systems ranging from the SuzHHugh-Naguna neuron (FitzHugh) [955) to the Lotka Volterra model (Freedman 1980) to the Lorenz system (Lorenz) 1963). The functional form of equations from this latter set were systematically excluded from the training set. For example, if a testing system had the form ai = + bur; 2 222 then this equation was never used to generate a training datum for any a, b, (see Sec. A.2
4 RESULTS
Generalization to held-out systems Embeddings were trained on an auxiliary task of equation reconstruction before being evaluated on a sequence of dynamics classification tasks. order to ensure phase2vec could indeed learn this auxiliary task, we verified that we could reconstruct the equations of held-out dynamical systems compared to the simplest per-equation fitting baseline incorporating sparsity, LASSO (Tibshirani 1996). The same sparsity regularizer of = 1 x 10 was used both models. We measured both the euclidean error from the true parameters and the sked-pointt normalized reconstruction error (Eq. B) the vector fields.
Saddle-Node (I)
Pitchfork (1)
Transcritical
Simple Oscillator (I)
Lotka- Volterra
Par: LASSO
2.15 x 10-!
2.30 x 10-1
2.33 x 10-
6.54 x 10-1
7.00 x 10"
Par: Vec
1.60 x 10
1.61 x 10
1.78 x 10
5.37 x 10
4.51 x 10
Recon: LASSO
4.16
7.13 10
1.12 10
10
1.99 10"
Recon: Vee
1.75
1.78 x 10
1.71 10
1,53 10
1.26 10P
Homoclinie (1)
Van Der (1)
Selkov (2)
MinHHugh-Nagunnn (4)
Polynomial (20)
Par: LASSO
3.45 10"
1,68 x 10"
6.33 x 10"
1.16 x 10"
3.11 x 10"
Par: Phase2Vee
2.93 10"
1.15 x 10-3
5.32 x 10"
6.07 x 10"
1,78 x 10"
Recon: LASSO
2.32 x 10-3
2.39 x 10'
2.97 x 10"
2.93 x 10-1
5.26 x 10-1
Recon: Phase2Nec
1.93 x 10-1
3.26 x 10-1 8.52 x 10-
4.62 x 10-1
1.47 x 10-
Table 1: Fuclidean parameter estimation error and reconstruction loss of phase2vec or LASSO on two-dimensional systems.
For two-dimensional systems (Ta- ble []. our generalizable approach was found to outperform the per-
Saddle-Node 3d (1)
Lorenz System (3)
Par: LASSO
7.16 x 10
2.14 x 10"
equation LASSO baseline on aver- age across the testing set of "clas- sical" equations (see Sec. A.2 Pa-
Par: Phase2Vee
5.65 x 10-2
1.42 x 10
Recon: LASSO
1.83 x 10-1
8.10 x 10
Recon: Phase2Vec
2.28 10-1
6.20 10
rameter error: phase2vec; 2.70 + 0.29 v.s. LASSO, 3.96 + 0.44.
Table 2: Euclidean parameter estimation error and re- construction loss of phase2vec or LASSO for three- dimensional systems.
Reconstruction error: phase2vec, 0.37 + 0.06 v.s. LASSO, 0.6 + 0.12; standard deviation bounds), al- though performance per-class varied widely. These results were quantitatively maintained even when fixed-point normalization was turned off (Reconstruction error: phase2vec, 0.28 + 0.09 v.s. LASSO, 0.89 + 0.15). Further, in order to assess the ability of phase2vec to reconstruct systems which are not expressible in the output basis, we also included one system, the "simple os- cillator", whose equation has no closed polynomial form (see Sec. A.2 Nevertheless, phase2ves was also found to fit this data comparably well to the LASSO baseline (Table I "Simple Oscillator (1)"). Again, none of the functional forms of the testing systems were observed during training, meaning that even outtt--iistiibttion dynamics encoded and decoded correctly.
5

---

Published as a conference paper at ICLR 2023
To evaluate phase2vec embeddings in the three-dimensional case, evaluated our architecture on equations having qualitatively similar behavior to the two-dimensional case (a pireedimensiioal equations giving rise to dynamics
extension of the saddle-node family; see Sec. A.2) well only possible for > 2 (i.e. the chaotic behavior produced by the Lorenz system three dimen- sions). As in the two-dimensional case, we found that three-dimensional reconstruction performance favored phase2vec (Parameter error: phase2vec, 4.78 + 0.27 v.s. LASSO, 7.17 = 0.51. Re- construction error: phase2vec, 0.36 + 0.08 v.s. LASSO, 0.39 0.15; standard deviation bounds; see Table D. An example reconstruction is given in Fig. A.15
Effects of normalization and sparsity We found that fixed-point normalization was crucial for accurate vector field reconstruction of testing data, especially for systems with complex dynamical features, such as cohabitating fixed points and cycles (Fig. Zap. For example, we compared the quality of reconstruction of the normalized and un-normalized models of a dynamical system ex- hibiting a homoclinic bifurcation in which a limit cycle collides with a saddle point as one of its coefficients is tuned from reconstruction error was, naturally, lower for the model trained with un-normalized loss. How- the ability of the normalized model to predict the ground truth system behavior near dynamical key points (such as fixed points) was much greater than the un- un-normalized model (Fig. 2a To
-1.2 to about .8645 (see Sec. A.2). We found that un-normalized
Euclidean error from true
Normalized
2.4
Unnormalized
0.8-
0.6
0.2
0.0
4.0
Time
(a)
(b)
Figure 2: Fixed-point normalization and sparsity. (a) The euclidean distance between trajectories simulated by phase2vec model trained with (red) or without (blue) physically-infommed normal- ization, and the ground truth model, in the case of system nearing homoclinic bifurcation (Sec. A.2). Relative performance near the fixed point is evident from stream plot reconstructions (black inset: ground true; blue inset: unnormalized; red inset: normalized.) (b) Parameter prediction error, P, as function of the proportion of ground truth (noisy) parameters sparsified, S, over different settings of the sparsity regularizer, 3. Low reconstruction error obtained despite sparsification up to 25% (green box).
observe the necessity of the normalization more closely, we generated 20 reconstructions for each of the normalized and un-normalized model corresponding 20 evenly spaced parameter values in the interval [-1.2, .8645]. We then simulated 1000 trajectories from all of these systems with initial conditions sampled from circle of radius .1 surrounding the fixed point the origin. As the trajectories advanced in time, we measured the euclidean deviation of simulated vs true trajecto- ries, averaged across all initial conditions (Fig. 2ap. We found that trajectories starting near the fixed point diverged much more rapidly from their true course when simulated based on the un-normalized model compared to those based on the normalized model. This is especially noticeable when the reconstructions viewed streamline plots (Fig. 2a ground truth, black border inset; blue border inset, un normaazzdl reconstruction, and red border inset, normalized-recomt reconstruction). In this sense, the normalized loss is critical for inferring meaningful dynamical features while ignoring irrelevant details which influence the euclidean loss (details in Sec. A.I.I).
Furthermore, expect meaningful and useful representations of dynamical systems to correspond to the simplest system that generates the underlying input dynamics, i.e. the system with the sparsest parameter values. Therefore, we next investigated the role of sparsity in the auxiliary equation prediction task by measuring parameter reconstruction error on new version of the classical system testing (i.e. the systems of Table I whose underlying parameters, Epert, were perturbed by zero- mean gaussian noise with standard deviation .I. We varied from 10-3 33 to 1 10-1 in

---

Published as a conference paper at ICLR 2023
20 steps and for each measured the average euclidean distance between the inferred parameters and the true (unperturbed) parameters, P, versus the average proportion the perturbed parameters which were sparsified, S:
Erecon 111
p= Rp | Frcoon 2112], S E
(5)
for Epert We found that the perturbed parameters could be substantially sparsified (approxi- mately over the range 3 E [1 10-3 1 x 10-2)) without incurring large reconstruction error (Fig. 2b green box). This provides evidence that the model has learned to distinguish between parameters that contribute to dynamical structures of the systems and irrelevant nuisance parameters.
Dynamically-informed reconstruction of noisy test data The applicability of our model to real data depends on our ability to produce meaningful embeddings from noisy data, as vector fields are typically acquired by binning derivatives from time series datasets which can be sparse corrupted. We do not propose phase2vec as a denoiser per se, rather intending to show that our learned representations robust to noise compared to per-equation baseline. To evaluate this ability, measured reconstruction performance on the "classical" system testing data subjected to four types of noise: (1) independent, zero-mean gaussian noise added to input vector fields, (2) random masking, (3) random sparsification arising from creating vector fields from binned trajectory data and (4) gaussian noise added the true parameter vector (for details, see Sec. A.2). The magnitude of each type of noise was systematically varied as we probed the comparative ability of phase2vec vs LASSO to reconstruct the uncorrupted data.
We found that phase2vec reconstructions degraded gracefully with noise compared to LASSO over wide range of noise parameter values (Fig. 3 For examples of phase2ved reconstructions, see Sec. A.4 Crucially, as these were testing data, reconstruction is only possible if the embedding map, v. has acquired robust and general notion of how the geometry of vectors phase space relates to governing equations.
Liuue Xrecon)
Gaussian
Zero-masking
Trajectory
Parameter
0.425
0.400
3.
0.55
0.75
5.51
0.55
0.50
0.6
0.45
0.40
0.50
0.39
0.25
3.2
20
40
60
10
100
130
140
118
163
170
160
6.30
15
||Xtrue
Spertl
Figure 3: Dynamicalty-informeed denoising. The accuracy of phase2vec (green) in comparison to LASSO (yellow) in reconstructing noisy testing data four cases. From left to right: gaussian noise applied to the input vector field, random zero masking applied to the input vector field, trajectory- generated noise, and gaussian noise applied to the true parameter vector. For low values of noise phase2vec was always the more robust method. For parameter noise, this trend continued for the full range of noise examined.
Decoding the physies of embedded data Having ensured that governing equations for testing data could be decoded from phase2vec embeddings with performance comparable to a per- equation fitting method including under noisy conditions, we next sought validate the quality of these embeddings on a battery of physics classification tasks. We compared phase2vec to three other representations: (1) the parameter vector of the underlying equation, (2) the first PCA eigenvalues of the raw vector fields where d is set to be the dimension of the embedding space, and (3) time series features extracted by TapNet (Zhang et al. 2020), state-of-the-arr time se- ries classifier. For all representations, except for TapNet (which has built-in MLP classifier), trained a logistic regressor with a cross-validated l penalty to predict the true classes of the un- derlying dynamics. We ran three experiments with classes delineated according either global or local dynamical properties. For the global case, we classified: (1) whether the dynamics re- spect conservation of energy not (Conservatvity; classes), and (2) whether the dynamics

---

Published as a conference paper at ICLR 2023
represent incompressible flow not (ncompressiiility; two classes For the local case, classified linear systems according to the five possible types of fixed points they can exhibit (Lin- stability; five classes: stable node, unstable node, stable spiral, unstable spiral saddle point (Strogatz et al., 1994).. TTpNNet classifications came from voting over classifications made from two-dimensional. For training details, Sec. A.3
10 length-64 time-series. All systems
On all classification tasks, we found that phase2vec embed- dings outperformed the compet- ing representations (Fig. + achieving an score over
Patemneters
Vector field PCA
Tapleet
Phase2Vecc
0.8
S The largest comparative advantage for phase2vec was achieved on the five-class lin- ear stability task, where the oth- erwise high-performing TapNet achieved an FI score of un- der .3. The relative advan- tage of phase2vec over Tap- Net specifically the cases of conservation of energy and incompressibility demonstrates one of the important advan- tages of vector-field based ap- proach. Namely, these are global system properties which naturally captured by the tensive coverage of phase space well the aggregation of phase information which are entailed by convolutional features.
Testing F1
0.6
0.4
0.2
0.0
Conservativity
Incompressibility Linear stability
Figure 4: Classification performance of phase2vec vs. alter- native representations. physics classification, phase2vec (green) outperformed competing representations: (1) the raw 20- dimensional parameter vector of true equation (red), (2) first d 100 PCA eigenvalues of the input vector field (blue), and (3) features extracted using the attentional time series model TapNet (purple). Bars depict F1 score and errors standard de- viation across classes. Details in main text and Sec. A.3)
100-
Temparate
0.7
80-
Pressure (Kb)
PCA
20.
Vector field PCA
004
Phase2Vec
0.4
-20-
0.2
Raf
0.0
PCA 1
(b)
Figure 5: Climate embeddings. (a) Emergent clusters in wind data. Fach point is phase2vec embedding of vector field measuring wind velocity fixed spatial grid covering the eastern hemisphere. Different points correspond to different times and pressures. Points are colored by pressure from .01 to 1.0 Kb. Three clusters emerge, corresponding to the appearance circumglobal streamlines (insets) as function of pressure. (b) Temperature gradient embeddings and Köppen label prediction. Temperature gradient embeddings colored according to a coarse Köppen label, indicating the general climatic zone which predominates in the crop. Labels were better predicted using phase2vec than using raw vector fields matched for dimensionality by PCA.
"These important dynamical classes since Helmholtz's Theorem holds that, under mild assumptions, every vector field can decomposed as sum of conservative and incompressible component. We also confirmed that we could decode the identities of the "classical" systems used above test recon- struction data (F1 score, 8144).
8

---

Published as a conference paper at ICLR 2023
Embeddings of climate data To investigate whether our embeddings could identify meaningful structure in real data, we computed representations of two types of climate data. For a first, qual- itative, demonstration, let al. 2005). Measurements were indexed by month and by year, from 1960 to 2022, as well as by pressure (Kh). We used wind vector measurements from a fixed spatial grid comprising roughly the eastern hemisphere which exhibits distinct visual patterns in the form of horizontal streamlines emerging low pressures (Fig. 5a brown inset for .07 Kb and green inset for .01 Kb). We found that phase2ved vec embeddings could easily identify these pressure-depender bands in the form of the red, brown, and green clusters (Fig. 5a).
computed embeddings for global wind vectors taken from (Blumenthal
For a second, quantitative experiment, we computed embeddings for world average temperature data taken from (Fick& & Hijmans, 2017) which were accompanied by so-called Köppen labels which in- dicate to which of five coarse temperature zones a given spatial location belongs (Köppen) [884] Beck et al. 2018). The ability to make quantitative predictions about these data is especially impor- tant since they have been recently shown to produce accurate forecasts of changing climate zones as a result of global warming (Beck et al. 2018).
To demonstrate the ability of phase2vec to classify data taken from real physical system, we generated embeddings of random spatial crops (sized 153" x 153f latitude by longitude) of tempera- ture gradient fields and used them predict each crop's expected Köppen label. As these fields were indexed by month and labels were not, we used month-averaged embeddings as our representation of given location. We found that phase2vec embeddings of these data (first two PCs, Fig. [5b] left) were substantially more predictive of the expected Köppen label than the raw vector field (Fig. right, VFFI testing score: .484 .011 vs phase2ver Fl testing score: .627 + :079), indicating that phase2vec had extracted dynamical structure relevant to climate annotations.
CONCLUSION
phase2vec is a physics-informed convolutional network for high-quality, low-dimensional repre- sentations of dynamical systems. Learned features are trained on an auxiliary task of generalized equation prediction and are demonstrated to be robust to noise, Importantly, phase2vec can gen- eralize held-out systems and learn the underlying semantics of dynamical systems, which can be used to decode general physical characteristics from data with greater fidelity than competing methods.
A clear area for future work is the extension of our framework to high-dimennsonal systems. This could be accomplished which have been used up to several tens of dimensions (Choy et al. 2020). For higher dimensional real-world systems, like neural circuits or chemical signaling pathways, potential approaches could use architectures taking advantage of sparse measurements in phase space, such graph neural networks.
a straightforward way with the use of higher-dimensionall convolutions,
In its current form, phase2vec makes minimal assumptions about input data, namely, that dynam- ics are qualitatively invariant to translations of the input coordinate system, that "slow" dynamics are important, and that dynamics generally change stably with input parameters. The first two of these assumptions are encoded by our use of a convolutional architecture and fixed-point normal- ization, respectively. The stability assumption is encoded by the fact that the embedding map is continuous and validated by showing that embeddings are robust to noise. Future work could in- troduce new assumptions, for instance by enforcing additional types of symmetry (e.g. rotational symmetry). Existing assumptions could also be adapted, for instance, by making embeddings sen- sitive to qualitative changes in dynamics (i.e. bifurcations, see (Kuznetsov) 1998)) which result from otherwise small changes in parameters. The choice of dictionary functions is also an area for future exploration, and we are intrigued by the use of other bases, like Legendre polynomials or beoogicclllymmeanngee
components, like Hill functions (Frank) 2013 Ingalls 2013).
The ability of phase2vec to encode meaningful dynamical features from data makes it a useful and promising model for the recovery of minimal physical systems and the design of new dynamical systems across biological and engineering applications. Chief among these applications are the design and real-time control of dynamical systems across the sciences.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We thank Bianca Dumitrascu and our group members for discussions and feedback. This work was supported by the Zuckerman Postdoctoral Program (M.R.,, the Israeli Council for Higher Educa- tion Ph.D. fellowship (N.M. and Z.P.), the Center for Interdisciplinary Data Science Research the Hebrew University of Jerusalem (N.M.). Clore Scholarship for Ph.D. students (Z.P.), Azrieli Foun- dation Early Career Faculty Fellowship, and the European Union (ERC, DecodeSC, 101040660) (M.N.). Views and opinions expressed are however those of the author(s) only and do not necessar- ily reflect those of the European Union or the European Research Council. Neither the European Union the granting authority can be held responsible for them.
REFERENCES
Giorgio Battistelli and Pietro Tesi. Classification for dynamical systems: Model-based and data- driven approaches. IEEE Transactions on Automatic Control, 66(4):1741-1745, 2020.
Hylke Beck, Niklaus E Zimmermann, Tim McVicar, Noemi Vergopolan, Alexis Berg, and Eric F Wood. Present and future köppen-geiger climate classification maps at 1-km resolution. Scientific data, 5(1):1-12, 2018.
MB Blumenthal, E Grover-Kopec, M Bell, and J del Corral. The iri/Ideo climate data library: Helping people use climate data. In AGU Fall Meeting Abstracts, volume 2005, pp. IN32A-05, 2005.
Josh Bongard and Hod Lipson. Automated reverse engineering of nonlinear dynamical systems. Proceedings of the National Academy of Sciences, 104(24):9933-9948, 2007.
Jason J Bramburger, Steven I. Brunton, and Nathan Kutz. Deep learning of conjugate mappings.
Physica D: Nonlinear Phenomena, 427:133008, 2021.
Kevin S Brown and James P Sethna. Statistical mechanical approaches models with many poorly known parameters. Phys. Rev. E Stat. Nonlin. Soft Matter Phys., 68(2 Pt 1):021904, 2003.
Steven L Brunton, Joshua L Proctor, and Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proc. Natl. Acad. Sci. U. S. A., 113(15): 3932-3937, 2016
Kathleen Champion, Bethany Lusch, Nathan Kutz, and Steven Brunton. Data-driven discovery of coordinates and governing equations. Proc. Natl. Acad. Sci. U. S. A., 116(45):2444-222551 2019.
Guoning Chen, Konstantin Mischaikow, Robert S Laramee, and Eugene Zhang. Efficient morse decompositions of vector fields. IEEE Transactions Visualization and Computer Graphics, 14 (4):848-866 862, 2008
Christopher Choy, Junha Lee, René Ranftl, Jaesik Park, and Vladlen Koltun. High-dimensional convolutional networks for geometric pattern recognition. In Proceedings of the IEEE/CVF con- ference on computer vision and puttern recognition, pp. 11227-11236, 2020.
Ignasi Clavera, Anusha Nagabandi, Simin Liu, Ronald S. Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments through meta- reinforcement learning. In International Conference on Learning Representations, 2019. URL net wttprr/wwwwee//tuuutteeee forumeid-
Charles C Conley. Isolated invariant sets and the Morse index. American Mathematical Soc., 1978.
Stephen E Fick and Robert J Hijmans. Worldelim 2: new 1-km spatial resolution climate surfaces for global land areas. International journal of climatology, 37(1))43302 4315, 2017.
Richard FitzHugh. Mathematical models of threshold phenomena in the nerve membrane. The bulletin of mathematical biophysics, 17(4):257-278, 1955.
Steven A Frank. Input-output relations in biological systems: measurement, information and the hill equation. Biology direct, 801):1-25, 2013.
10

---

Published as a conference paper at ICLR 2023
Herbert Freedman. Deterministic mathematical models in population ecology, volume 57. Marcel Dekker Incorporated, 1980.
Stefania Fresca, Andrea Manzoni, Luca Dedè, and Alfio Quarteroni. Deep learning- based reduced order models in cardiac electrophysiology. PloS one, 15(10):0039446, 2020.
Alexander Haluszczynski and Christoph Räth. Controlling nonlinear dynamical systems into arbi- trary states using machine learning. Scientific reports, 11(1):1-8, 2021.
Yuval Hart, Yaron E Antebi, Avraham E Mayo, Nir Friedman, and Uri Alon. Design principles of cell circuits with paradoxical components. Proceedings of the National Academy of Sciences, 109 (21):8346-835) 2012.
U Iben and C Wagner. Taylor mapping method for solving and learning of dynamic processes. Inverse Problems in Science and Engineering, 29(13):3190-3215 2021.
Brian P Ingalls. Mathematical modeling in systems biology: an introduction. MIT press, 2013.
George Em Karniadakis, Ioannis G Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics-informed machine learning. Nature Reviews Physics, 3(6):422 440, 2021.
Matthieu Kirchmeyer, Yuan Yin, Jeremie Dona, Nicolas Baskiotis, Alain Rakotomamonjy, and Patrick Gallinari. Generalizing to new physical systems via context-informed dynamics model. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference
Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 11283-11301 PMLR, 17-23 Jul 2022. URL https: https::/procees:: http://ppceeeenn.eee/ee mwr. www.:emww......eemmmmmmmmmmmmeee www.prres/.vv... 62 / ki www.rrceeeeeeeem html
Wladimir Köppen. Die wärmezonen der erde, nach der dauer der heissen, gemässigten und kalten zeit und nach der wirkung der wärme auf die organische welt betrachtet. Meteorologische Zeitschrift, 1(21):5-226, 1884.
Y. Kuznetsov. Elements of Applied Bifurcation Theorem. Springer- Verlag, April 1998.
Kimin Lee, Younggyo Seo, Seunghyun Lee, Honglak Lee, and Jinwoo Shin. Context-aware dynam- ics model for generalization in model-based reinforcement learning. In International Conference on Machine Learning. pp. 5757-5766. PMLR, 2020.
Edward Norton Lorenz. Deterministic nonperiodic flow. Journal of the Amoopheric Sciences, 20 (2):330-141, 1963.
Bethany Lusch, Nathan Kutz, and Steven L Brunton. Deep learning for universal linear embeddings of nonlinear dynamics, Nat. Commun., 911):4950, 2018
Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word repre- sentations in vector space. In International Conference on Learning Representations, ICLR 2013, Scotisdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings, 2013, URL httt:://axxi....t org/ abs httgg//bs/13011. /t///311.3366 3771
Xiaojic Qiu, Yan Zhang, Jorge D Martin-Rufino, Chen Weng, Shayan Hosseinzadch, Dian Yang, Angela N Pogson, Marco Y Hein, Kyung Hoi Joseph Min, Li Wang, al. Mapping transcriptomic vector fields of single cells. Cell, 185(4))690-711, 2022
Katherine N Quinn, Michael C Abbott, Mark K Transtrum, Benjamin B Machta, and James P Sethna. Information geometry for multiparameter models: New perspectives on the origin of simplicity. arXiv preprint arXiv:2111.07176, 2021.
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physies-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
Markus Reichstein, Gustau Camps- Camls-Valls, Bjorn Stevens, Martin Jung, Joachim Denzler, Numo Car- valhais, et al. Deep learning and process understanding for data-driven earth system science. Nature, 566(7743) 2019.
11

---

Published as a conference paper at ICLR 2023
Nicolae Sapoval, Amirali Aghazadeh, Michael G Nute, Dinler A Antunes, Advait Balaji, Richard Baramiuk, CJ Barberan, Ruth Dannenfelser, Chen Dun, Mohammadamin Edrisi, et al. Current progress and open challenges for applying deep learning across the biosciences. Nature Commu- nications, 13112 2022.
Michael Schmidt and Hod Lipson. Distilling free-form natural laws from experimental data. science, 324(5923):81 2009.
Natalie S Scholes, David Schnoerr, Mark Isalan, and Michael PH Stumpf. A comprehensive network atlas reveals that turing patterns are common but not robust. Cell systems, 9(3):243-257, 2019.
Terry-Ann Sneed and Arash Komaee. Nonparametric reconstruction of vector fields from noisy observations of their flow curves. In 2021 American Control Conference (ACC), pp. 3969-3974. IEEE, 2021.
Steven Strogatz, Mark Friedman, John Mallinckrodt, and Susan McKay. Nonlinear dynamics and chaos: With applications to physics, biology, chemistry, and engineering. Comput. Phys. Commun., 8(5):332, 1994.
David Sussillo, Rafal Jozefowiez, LF Abbott, and Chethan Pandarinath. Lfads-latent factor analysis via dynamical systems. arXiv preprint arXiv:1608.066 06315, 2016.
Han Kheng Teoh, Katherine N Quinn, Jaron Kent-Dobias, Colin B Clement, Qingyang Xu, and James P Sethna. Visualizing probabilistic models minkowski space with intensive symmetrized kullback-leibler embedding. Physical Review Research, 2(3):033221, 2020.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), (88(1):267-285) 1996.
Marc Timme and Jose Casadiego. Revealing networks from dynamics: an introduction. Journal of Physics A: Mathematical and Theoretical, 47(34):343001, 2014.
Mark Transtrum and Peng Qiu. Model reduction by manifold boundaries. Phys. Rev. Lett. 113 (9):098701, 2014.
J.R. Tredicce and G.L. Lippi. Critical slowing down at bifurcation. American Journal of Physies, 72(799), 2004.
Jindong Wang. Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip Yu. Generalizing to unseen domains: survey on domain generalization. IEEE Transactions Knowledge and Data Engineering, 2022.
Shuran Ye, Zhen Zhang, Xudong Song, Yiwei Wang. Yaosong Chen, and Chenguang Huang. A flow feature detection method for modeling pressure distribution around a cylinder in non-uniform flows by using convolutional neural network. Scientific reports, 10(1):1-10, 2020.
Yuan Yin, Ibrahim Ayed, Emmanuel de Bézenac, Nicolas Baskiotis, and Patrick Gallinari. Leads: Learning dynamical systems that generalize across environments. Advances in Neural Informa- fion Processing Systems, 34:7561-7573, 2021.
Eugene Zhang, Konstantin Mischaikow, and Greg Turk. Vector field design on surfaces. ACM Transactions Graphics (ToG), 1294)::294-1333 1326, 2006.
Xuchao Zhang, Yifeng Gao, Jessica Lin, and Chang-Tien Lu. Tapnet: Multivariate time series classification with attentional prototypical network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 6845-6852, 2020.
12

---

Published as a conference paper at ICLR 2023
A
APPENDIX
All experiments were carried out using pytorch v. 1.12 using an NVIDIA RIX 3060 GPU.
A.1 ARCHITECTURE AND TRAINING
Here, describe the embedding architecture of phase2vec specifically for the two-dimensional case (q = 2). The three-dimensional case is identical, except that the number of spatial dimensions of kernels is larger (q 3) and the embedding dimension, d, was larger. The convolutional part of the phase vec embedding pipeline consisted of three convolutional blocks, each with kernel size 3 x 3, stride 2 x and 128 channels. This resulted in a sequence convolutional blocks of sizes (128 x 31 x 31), (128 x 15 x 15), (128 x 7 x 7). Convolutional blocks were interleaved by ReLU non- non-linearities and batch norm layers. The final convolutional activations were mapped by a single linear layer to a d 100-dimensional embedding space (d = 256 for three- fire-dimensional case).
To compute predicted coefficients, these embeddings were mapped by a 2-layer MLP whose hidden layers had 128 units to 20-dimensional vectors representing the 10 coefficients possible for each of the 2 dimensions the ODE. For the three-dimensional case, embeddings were mapped to 60- dimensional vectors representing 20 coefficients for each of the 3 dimensions. MLP layers were alternated with batch norm layers as well as dropout layers whose rate we set to p during training.
The model was trained with an ADAM optimizer using a learning rate of 1 x 10-4 over 200 itera- tions.
A.1.1 PIYSICALLY-INFORMED LOSS NORMALIZATION
We provide a visualization of the physically-inorran normalized loss in comparison to the unnor- malized loss showcasing importance in capturing meaningful dynamics. Considering the saddle- node bifurcation system Fig. A. we observe that the normalized loss landscape across phase space captures the properties the system, visually represented by the fixed points Fig. A.6b In contrast, these can not be identified in the unnormalized representation Fig. A.6c
(a)
(b)
(c)
Figure A.6: Fixed-point normalization. (a) Saddle-node vector field used for evaluation of the loss landscape (b)-(c). Gaussian noise was added to the true vector field and losses were measured at each spatial location. (b) shows the physically-inforee normalization and (c) the unnormalized loss.
A.2 DATA, TRAINING AND EVALUATION
A.2.1 SYNTHETIC DATA
The training set D was a collection of polynomial ODEs generated from the dictionary + with coefficients that were 0 with probability .75 and were otherwise sampled uniformly |-3.3 3] and consisted of 10k examples. For evaluation in all settings, a test set of 1000 samples was used.
Network hyperparameters selected minimize validation loss two data sets:
13

---

Published as a conference paper at ICLR 2023
1. Polynomial equations: For generalization to new parameter regimes. A set of equations sampled from the same distribution generating D but having a disjoint set of coefficients.
2. Real-world systems: For generalization to new functional forms not observed during train- ing. The real-world systems, their parameter ranges and labels are given in Table A.3
All equations were re-centered be on the square -, 1]2 on the phase plane and discrete vector fields were computed by measuring the continuous system on an evenly-spaced grid of 64 x 64 points. We only investigated equation reconstruction for two 3-d systems as a proof of concept and did not classify them. They therefore do not have labels.
Name
Equation
Parameler ranges
Class labels
Suddle
0: lia
Transnticcal
21 ex
0: a
Simple Oscillator
8- ria - 11.0 -ipooln
0
-..
6 0: 7
Lalks-Voliere
X for all
Homocline
10 2645
21 1
-all
1. tung
mry 20s
WE [00,, 100. [22.1]]
12 2 2
FidHfoghNaguume
8.
14ffa if
21
B [-1
Lonenz (3:))
F1 61-6 - 1.li
1.0
Fal 13
m
NAA
Table A.3: Real-world systems, their equations and class boundaries. Systems having two classes exhibit a bifurcation (i.e. the emergence of new topological structure in the dynamics: fixed point, limit cycle, etc.)
A.2.2 REAL DATA
Global wind vectors Data was downloaded from IRI/LDEO Climate Data Library (https:// iridl. Idee. coloume coluubiaa a edu/). Measurements were indexed by month and by year, from 1960 to 2022, as well as by pressure (Kb). We used wind vector measurements from a fixed spatial grid comprising roughly the entire eastern hemisphere. A set of 2000 samples was used for testing.
World climate data We took climate data from WorldClim V2 (http:// www. wor morldclln im. org). We used monthly temperature averages, taken over temporal span of 1970-2000 at a spa- tial resolution of 0.083 Corresponding Köppen labels were downloaded from GloH2O, Köppen- Geiger (http:/ www. gloh2o. org/ koppen/). We used coarse-grained level of the labels amounting five temperature zones (tropical, arid, temperate, cold and polar). A set of x 12 (per month) samples was used for testing.
A.3 PHYSICS CLASSIFICATION
All data sets had 1000 samples with equal numbers of class exemplars. Embeddings acquired for all data using the model trained on polynomials systems as described above. We considered three problems:
Conservativity A conservative system represents physical system which respects conservation of energy. Every conservative system can be represented as the gradient of a scalar field. To gen- erate conservative systems, we first generated scalar fields by sampling the coefficients of a single polynomial equation with basis functions in & according to the same distribution used to generate training coefficients. We then manually calculated gradients of these scalar fields. The opposing class were 500 samples from the training set which we ensured were not conservative. We did so by calculating the curl of each counter datum and checking was nonzero (on a simple connected phase space system is conservative if and only if it is irrotational, i.c., has zero curl.
Incompressibility A system is incompressible if it represents the flow of fluid which cannot be "compressed" into or out of phase space. In other words, the divergence of the field is zero. To generate divergence- free fields, first identified the phase plane with C so that each point (æ1,
14

---

Published as a conference paper at ICLR 2023
was identified with 21 +22. We again created polynomial (automatically holomorphic) using the same distribution on parameters and having the form
f(z) f + ix2)) a + ib.
(6)
We then defined the vector field
df iw where
ds
da
(7)
dr
and
aa
(8)
0z
The component b automatically satisfies the equivalence of mixed partials,
8²b 02b
(9)
Or, 0r2
dzzdmi
which gives Vi 0) and was taken divergence-ftee, incompressible system.
Linear stability A linear planar system श = Az exhibits stable node, unstable node, stable spiral, unstable spiral or saddle point as a function the trace and determinant of A. We randomly sampled A until 200 exemplars of each stability class were acquired.
For each task we computed the relevant embeddings and then trained logistic regressor on 80% of the data. The remaining 20% was held out for testing, using a stratified train-test split. An &g penalty was cross-validated using leave-one-out vross-validation on validation data split from training set. We searched for optimal regularizer over 11 values spread logarithmically between x 10 and 10a. For multi-class settings, we used one-versus rest scheme.
A.4 VECTOR FIELD RECONSTRUCTIONS
We provide phases phase2vec reconstructions test data. Starting with demonstration of the differ- ent classes, (1) Conservative (Fig, A.7 (2) Incompressible (Fig. A.8) and (3) Linear (Fig. A.9
Sanaa do Crnnn Them
Pecomstuct
Figure A.7: Conservative dynamics, ground truth vs. reconstruction
15

---

Published as a conference paper at ICLR 2023
Teitirg inc Ground Tnth
05
C4
C3
Reconmnuttin
C2
-01
Figure A.8: Incompressible dynamics, ground truth vs. reconstruction
Theteng linear Ground Tromts
16
14
08
Necorsmattees
06
Figure A.9: Linear stability dynamics, ground truth vs. reconstruction
16

---

Published as a conference paper at ICLR 2023
To assess the stability of phase2vec to noise, perturbed testing data with four types of noise and compared the ability of our method to reconstruct the unperturbed data to that of LASSO baseline. These noise types were:
Gaussian: Zero-mean, independent gaussian noise was added to the vector fields. The standard deviation of the noise was set to true. where true was the true standard deviation of the data to be perturbed. This was done to scale the noise to each data class. We varied a from
in 20 steps.
Zero-masking: A proportion of each vector field was randomly zero-masked. The propor- tion was varied between 0 and .3 in 20 steps.
Trajectory: Data was generated by simulating a certain number of trajectories which were run for 100 steps with step size of .01 and then calculating velocities by binning and averaging. The number of trajectories was used to control the amount of noise. We used between 10 and 2000 random initial conditions in 20 steps. Empty bins were filled with zeros.
Parameter: added zero-mean gaussian noise to the true parameter vector of the data, varying the standard deviation between 0 and .3 in 20 steps.
Cean
Noisy
Recon- LASSO
Recon-Fo
Figure A. A0 Added gaussian noise of order of .3 times the ground-truth standard deviation of each vector field. Rows, top to bottom, depict the ground truth dynamics, noisy, LASSO reconstruction and phase2vec reconstruction.
17

---

Published as a conference paper at ICLR 2023
Clean
Noisy
Recon-LASSO
Recon-P2V
Figure A.11: Random zeroing out of 30% of the vectors. Rows, top bottom, depict the ground truth dynamics, noisy, LASSO reconstruction and phase2vec reconstruction.
Last, we consider the reconstruction for different resolutions the data, taking only n = 32, 64, 128 lattice points. Of note, the losses averaged over the simple oscillator test case were identical, but took longer to train each resolution: 50, 100 and 150 iterations respectively.
18

---

Published as a conference paper at ICLR 2023
Clean
Noisy
Recon-LASSo
Recon-P2V
Figure A.12: Reconstruction based on limited trajectory set (100 initial conditions). Rows, top to bottom, depict the ground truth dynamics, noisy, LASSO reconstruction and phase2vec reconstruc- tion.
19

---

Published as a conference paper at ICLR 2023
Clean
Noisy
Recon -ASSO
Recon- P2V
Figure A. 13: Added gaussian noise of order of times the standard deviation of the true parameter vector to the parameters vector. Rows, top bottom, depict the ground truth dynamies, noisy, LASSO reconstruction and phase2vec reconstruction.
20

---

Published as a conference paper at ICLR 2023
n
64
n 128
Original
Reconstruction
Figure A.14: Reconstruction based on different resolutions of the data. Columns correspond to the number of lattice points taken, 32 (left) 64 (center) n = 128 (right). The top row presents the ground truth dynamics and the bottom row the reconstruction
True
LASSO
P2V
10
10
10
0.5
0.0
0.0
0.0
-0.5
00.
005
-1.0
-1.0
-1.0
1.0
1.0
1.0
0.5
0.5
0.5
-0.5
-0.5
-0.5
-0.5
-0.5
-0.5
0.5
-1.0
0.5
-1.0
0.5
11.0
Figure A.15: Example reconstructions.. Three vector fields from a 3-d saddle-node system are de- picted. Arrows represent velocity at given location and their color represents deviation from the true velocity, with red being the highest error and green being lowest error. (Left) Ground truth (i.e. zero error); (Center) LASSO reconstruction having high error; (Right) phase2vec recon- struction with relatively low error.
21


Reviews:
[{"review_id": "hPBfH_3VL9", "reviewer": "Reviewer_STRm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a method similar to word2vec but applied to dynamic systems. The approach considers the phase space as an image, from which we can extract a latent representation summarizing the characteristics of the dynamics.", "summary_of_the_review": "This paper introduces an interesting idea, but too limited on the number of possible applications. It is, however, an important line of research, which deserves to be taken into account.", "strength_and_weaknesses": "The paper introduces an interesting method for the study of dynamical systems. One of the key points of the approach is the ability to generalize to different dynamical systems without requiring re-training, which is not the case for existing methods. The network architecture seems reasonable, as does the design of the criterion.\n\nHowever, I believe the following points are debatable:\n\n- Embeddings are learned using a phase field reconstruction task, through a sparse regression to the coefficients of the dynamic equation. This has several limitations in my opinion:\n    - This requires the supervision of a relatively complete dynamics map, which is not easy to access in many situations. However, the authors show that their method is sufficiently robust to withstand failures and errors in the phase field.\n    - Predictions are limited to the structure of the function dictionary used $\\Phi_\\alpha$. Since this contains only polynomial structures, a large number of dynamical systems become out of reach. Can the authors discuss this limitation in a more general framework?\n\n- The use of convolution greatly limits the number of systems that can be studied, since the dynamics must be of dimension 2. Convolution can possibly be extended to higher dimensions, but this becomes non-trivial. This is for me one of the most negative points of the approach.\n\n- The experiments are for me rather unconvincing, in particular the reconstruction of the dynamics. Of the 10 systems studied, LASSO greatly outperforms Phase2Vec on 4 systems. When Phase2vec dominates, the gap with LASSO is smaller. However, I acknowledge that Phase2Vec do not requires training to infer on unseen dynamics, while it is not the case for LASSO. The denoising and masking experiments are very interesting, but quite unrealistic for me. Would it be more relevant to generate the phase field from a set of trajectories whose initial conditions will be chosen at random? This will make the phase field more representative of what can be obtained in practice.\n\n- I would also be curious to see how the network behaves beyond the training set, i.e. outside the spatial limits of the phase plane used to estimate the embedding.\n\nMINOR\n- I found experiments on real world data a bit hard to read.\n- 3.1: \"at a discrete LOCATION\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_n4KQD_Q9R", "reviewer": "Reviewer_HbBX", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes phase2vec - a self-supervised framework for learning representations of dynamical systems in 2D. The main technical contribution is to use a 2D convolutional network for embedding dynamical system velocities, and formulating a reconstruction loss based on the data creation process to train the network weights. Experiments demonstrate that phase2vec can generalise to previously unseen dynamics equations, noisy systems, and real-world climate data, as well as serve as a downstream classifier for physical properties of dynamical systems.", "summary_of_the_review": "I currently recommend acceptance. Overall, the paper is well written, the proposed phase2vec method is straightforward, and the experimental evaluation demonstrates its usefulness across several scenarios. I would like the authors to better contextualise the novelty of the proposed approach in the rebuttal.\n\n---\n\n**Post-rebuttal:** Thank you to the authors for their rebuttal and revised manuscript. Based on the rebuttal as well as the other reviewers' summary, I would currently like to keep my original score/assessment.\n\n**Post-discussion:** Apologies for the late action. Based on revisiting the discussion and the authors releasing open source code (I have not tried running it, but it looks modular, well thought out, and extensible), I have upgraded my score. I think the revised manuscript is an improvement over the original and addresses reviewers' concerns. I would like to re-iterate my moderate confidence.", "strength_and_weaknesses": "### Strengths\n+ The paper is generally well written and well motivated. The Introduction is readable by non-experts and contextualises the contributions well.\n+ The proposed technique is straightforward and easy to understand. The main technical contribution - combining a 2D ConvNet and self-supervised loss applied to dynamical systems - is novel to the best of my knowledge. Phase2vec is an autoencoder for dynamical systems - which by itself isn't new - but the formulation of using a 2D ConvNet and self supervision signal is. (I would like to see the authors address the question of novelty during the rebuttal, see my questions below.)\n+ The experimental evaluation seems thorough - phase2vec is probed for its ability to generalise outside its training distribution as well as on real world data. (But I would be keen to read other reviewers' opinion on the novelty of the technical contributions as well as the experimental rigour.)\n\n### Weaknesses\n- As the entire methodology and experiments are only geared towards dynamical systems in 2D Euclidean space, I would have liked to see this mentioned in the Abstract/Introduction. The only place where this is mentioned is in the Conclusion. (I realise this can be opinionated.)\n- I found the Methodology and Related Work sections to have minor clarity issues which could be addressed by re-writing. I would like the authors to better contextualise the novelty of the proposed approach.\n\n### Questions and Feedback\n1. I appreciate the Related Work headers. While the section does a good job at surveying the literature, I was unsure how this paper, its methodology, and its experiments fit into each of those headers. Could the authors contextualise how their work is situated compared to what has been done. This may help to explicitly identify the main novelty of the paper w.r.t. existing literature. As a concrete example, there is a paragraph on vector-field-based approaches - but does phase2vec use them - if yes, how? \n2. Along this line of contextualising the novelty w.r.t. related work, could the authors clarify whether the proposed self supervised reconstruction loss is original? Are there similar works which encode observations to a latent dimension and run dynamics there?\n3. Section 3.1. describes the 2 x n x n input array. How is the size n chosen? Does n need to be fixed? Assuming not, could the authors comment on how sensitive phase2vec is to changes in the input size?\n5. Currently, the 2D ConvNet operates on a 2D grid array where each pixel/entry is a 2D velocity vector. Has phase2vec made any assumptions about the symmetries that may act on this vector space? E.g. is it realistic to assume that the velocity vectors should be rotationally equivariant (i.e. if the system is rotated, they would rotate along with it)? How would additional symmetry constraints impact the design of phase2vec and the underlying ConvNet + self-supervised loss function used?\n\n### Minor Comments\n- Along these lines, in Figure 1, it may be useful to list the dimensionality of the intermediate tensors. Just to double check my understanding: the input is 2 x n x n, it goes through a series of convolutions, the final prediction is 2 x 10? \n- I found the paragraph under equation 2 to be unclear. I assume it describes the basis functions? What is the multi-index? Why cubic degree? Would the resulting basis be: $1, x_1, x_2, x_1^2, x_2^2, x_1 x_2, x_1^2 x_2, x_1 x_2^2, x_1^3, x_2^3$?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MJsjfVA0Ny", "reviewer": "Reviewer_PePd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a methodology for predicting the equations of polynomial ODEs and obtaining embeddings of dynamical systems which can be applied to denoising data and classifying time series based on their physical properties. The authors propose a novel architecture based on CNNs to encode geometric features of vector fields, which is trained to minimize reconstruction losses. A FC layer predicts polynomial ODE coefficients, which are used for predicting equations that haven't been seen previously. They present results in synthetic datasets generated from polynomial ODEs, and utilize meteorological data to showcase their embedding space is meaningful. ", "summary_of_the_review": "The main concern I have with this paper is the lack of systematic evaluation in some sections. Although I have some concerns about the presentation, I hope authors add more details and make it more digestible. In my opinion, this paper is marginally over the threshold for acceptance. I would be willing to increase my score with the authors responding to my concerns on evaluation.\n\n----\n**After authors response:** I changed my decision to ACCEPT. See thread for details.", "strength_and_weaknesses": "Strengths:\n- S1. Interesting and important problem. \n- S2. Recapitulation of relevant related work is very good.\n- S3. The methodology is sound and non-trivial contribution.\n\nWeaknesses:\n- W1. The evaluation does not appear to be systematic. In neither the paper nor the appendix, the authors specify which dataset/ODE they are using for the examples of dynamically-informed denoising. It is possible that this is anecdotal evidence and not a legitimate claim for the paper. Additionally, I wonder why authors did not compare their method with other denoising methods and compute error metrics.\n- W2. I found their method's design choices difficult to understand. There were some of them that I had to read several times from the appendix to the main paper. Other times, the information was simply not available. For example, why should we use the cubic expansion? Did you find this out experimentally?\n- W3. The physics-informed term commonly refers to the physics-informed neural networks (PINNs), but authors use it with a significantly different meaning. It might be a good idea to think of another term or at least another title.\n- W4. Readability would be benefited from putting clear examples, especially when addressing the mathematics in Section 3.\n\nQuestions:\n\n- Q1. Can you please response to questions in W1?\n- Q2. Please see questions stated in W2.\n- Q3. Can you elaborate on why LASSO should be the only baseline for results in generalization to held-out systems? It is hard to judge if these are good results when compared to one simple baseline, and it’s not clear which were the reasons to exclude them from the evaluation to the several methods listed in the related work.\n- Q4. From which literature did you obtain the physical properties of the underlying data (stability of fixed points, conservation of energy, and the incompressibility of flows)? Is this an exhaustive list?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "E7CoN9vtWI", "reviewer": "Reviewer_sZrD", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The manuscript discuss how to encode 2D phase space images to shorter vectors. The embedding can be decoded to reveal the governing equations, recover simple, sparse models or de-noise the input preserving the dynamical relations. As the phase portrait is so fundamental in describing the dynamics, it is not a surprise the physical features like energy conversation, etc can be decoded out, as there are very visible in the phase space image.\n\nThe proposed pipeline feeds the phase portrait, through a bottleneck, to polynomials that can be used to reconstruct the phase plot through solving the decoded equation. A consistency error is set between the original and newly generated phase plots and the sparsity loss is set to keep the equations as simple as possible.\n\nThe training data is an ensemble of polynomial ODEs in such a way that the form of equation where the model was tested were removed from the training data - to check the generalisation capability. The results were compared to a traditional per-equation fitting method.\n\nThe authors demonstrate the encodings with an analysis of atmosphere.", "summary_of_the_review": "The paper is valuable contribution to explainability research, where experimental data can be encoded in a physically meaningful manner that supports common intuition.  The network architecture is a variation of an auto-encoder, where part of the generator is replaced with a world model. I would really be interested seeing this connected to a control problem, where the intuition gained in the encoding is used to control the physical processes....  So, this is not only about seen that, done that, but also opens new ways for machine learning based, explainable, control systems, besides the more straightforward venue to dimensions greater than two.", "strength_and_weaknesses": "Phase space is a traditional way to understand what is happening in dynamical systems. This is a combined in a nice way to a traditional deep learning image classification pipeline, using the strengths of both the physical domain and the machine learning domain.  The idea of using equation terms as \"words\" in the encoding is excellent, as the decoding through solving the resulting sparse differential equation has a real chance of being the optimally sparse representation of the system.\n\nHere lies also the weakness of the system - as small changes to a differential equation terms, can lead to big changes in the phase space, so the encoder may not be very stable. Restraining to polynomial functions of low degree may help on that, but at the same time they system becomes less able to encode such complicated behaviour. Would there be some Lipschitz condition based reasoning that would provide smoothness of the phase space transformation with respect to the changes of polynomial weightings or is the de-noising training with digital ensembles able to find stable encodings and move away for highly sensitive combinations? Or are the described normalization procedures able o solve this?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%