Transformers for Modeling Physical Systems
Authors: Nicholas Geneva
Nicholas Zabaras
Venue:  (2021)
Keywords: Scientific Machine Learning
Deep Learning
Physics
Surrogate Modeling
Koopman
Transformers
Attention

Abstract: Transformers are widely used in neural language processing due to their ability to model longer-term dependencies in text. Although these models achieve state-of-the-art performance for many language related tasks, their applicability outside of the neural language processing field has been minimal. In this work, we propose the use of transformer models for the prediction of dynamical systems representative of physical phenomena. The use of Koopman based embeddings provide a unique and powerful method for projecting any dynamical system into a vector representation which can then be predicted by a transformer model. The proposed model is able to accurately predict various dynamical systems and outperform classical methods that are commonly used in the scientific machine learning literature.

Under review as a conference paper at ICLR 2021
TRANSFORMERS FOR MODELING PHYSICAL SYSTEMS
Anonymous authors
Paper under double-blind review
ABSTRACT
Transformers are widely used in neural language processing due their ability model longer-term dependencies in text. Although these models achieve state-of- the-art performance for many language related tasks, their applicability outside of the neural language processing field has been minimal. In this work, we propose the use transformer models for the prediction of dynamical systems represen- tative of physical phenomena. The use of Koopman based embeddings provide a unique and powerful method for projecting any dynamical system into a vector representation which can then be predicted by transformer model. The proposed model is able to accurately predict various dynamical systems and outperform classical methods that are commonly used in the scientific machine learning liter- ature
INTRODUCTION
The transformer model Vaswwan et al., 2017). built on self-attention, has largely become the state- of-the-art approach for a large of neural language processing (NLP) tasks including language modeling, text classification, question answering, etc. Although more recent transformer work is focused on unsupervised pre-training of extremely large models (Devlin et al.) 2018 Radford al.) 2019 Dai et al.) 2019] Liu et al.) 2201)), the original transformer model garnered attention due to its ability to out-perform other state-of-the-art methods by learning longer-term dependencies without recurrent connections. Given that the transformer model was originally developed for NLP, nearly all related work has been rightfully confined within this field with only a few exceptions. Here, we focus on the development of transformers to model dynamical systems that replace otherwise expensive numerical solvers. In other words, we are interested in using transformers to learn the language of physics.
The surrogate modeling of physical systems is a research field that has existed for several decades and is large ongoing effort in scientific machine learning. A surrogate model is defined as an approximate model of a physical phenomenon that is designed to replace an expensive computational solver that would otherwise be needed to resolve the system of interest. The key characteristie of surrogate models is their ability to model a distribution of initial or boundary conditions rather than learning just one solution. This is arguably essential for the justification of training deep learning model versus using standard numerical solver. The most tangible applications of surrogates are for optimization, design and inverse problems where many repeated simulations are typically needed. With the growing interest in deep learning, deep neural networks have been used for surrogate modeling a large range of physical systems in recent literature.
Standard deep neural network architectures such as auto-regressive Mo et al. 2019
Geneva & 2020), re-
Zabaras 2020a), residual/Euler (González- Garria et al. 1998 Sanchez-Gonzan current and LSTM based models (Mo et al. [019 Tang et al. 2020 Maulik et al., (0200) have been largely demonstrated to be effective at modeling various physical dynamics. Such models gener- ally rely exclusively the system's 's evolution. Particularly for dynamical systems, present machine learning models lack generalizable time cognisant capabilities to predict multi-time-scale phenomena present in systems including turbulent fluid flow, multi-scale materials modeling, molecular dynamics, chemical pro- cesses, etc. Thus currently adopted models struggle maintain true physical accuracy for long-time
al.
the past time-step provide complete information on the current state of
'Code available at: [URL. available after review] "Supplementary videos available http:://wne..oooogg..o////////ep

---

Under review as a conference paper at ICLR 2021
predictions. Much work is needed to scale such deep learning models larger physical systems that are of scientific and industrial interest. This work deviates from this pre-existing literature by investigating the use of transformers for the prediction of physical systems, relying entirely self- attention to model dynamics. In the recent work of Shaalvaa & Oseledets (2020), such self-attention models were tested learn single solutions of several now-dimensional ordinary differential equa- tions. In this work, we propose physics inspired embedding methodology to model a distribution of dynamical solutions. We will demonstrate our model on high-dimensional partial differential equation problems that far surpass the complexity of past works. To the authors best knowledge, this is the first work explore transformer NLP architectures for the prediction of physical systems.
METHODS
When discussing dynamical systems, we interested in systems that be described through dynamical ordinary or partial differential equation:
Φe = F (x, p(t, x, n), V &. V Φ. Φ- V) F +R"
(1)
in which Φ E R" is the solution of this differential equation with parameters 77, in the time interval T and spatial domain Ω2 with boundary I C Ω. This general form can embody vast spectrum of physical phenomena including fluid flow and transport processes, mechanics and materials physics, and molecular dynamics. In this work, are interested in learning the set of solutions for dis- tribution of initial conditions do ~ p(Φo), boundary conditions B(Φ) ~ (B) Va E I' or equation parameters n p (n) This accounts for modeling initial value, boundary value and stochastic problems. We emphasize that this is fundamentally different, more difficult and of greater interest for most scientific applications compared learning single solution.
To make this problem applicable to the use transformer models, the continuous solution dis- cretized in both the spatial and temporal domains such that the solution of the differential equation is = ( $1.... Φ Ø E Rexd for which Φ; has been discretized by d points Ω. We assume an initial state Φ and that the time interval is discretized by T time-steps with a time-step size At. Hence, we pose the problem of modeling a dynamical system as a time-series problem. The machine learning methodology has two core components: the transformer for modeling dynamics and the embedding network for projecting physical states into a vector representation. Similar to NLP, the embedding model trained prior to the transformer. This embedding model is then frozen and the entire data-set is converted to the embedded space in which the transformer is then trained as illustrated in Fig. [ During testing, the embedding decoder is used to reconstruct the physical states from the transformer's predictions.
Embedding Training
Embed Data
Transformer Training
Encoder
Decoder
Transformer
Encoder
Figure 1: The two training stages for modeling physical dynamics using transformers. (Left to right) The embedding model is first trained using Koopman based dynamies. The embedding model is then frozen (fixed), all training data is embedded and the transformer is trained in the embedded space.
2.1 TRANSFORMER
The transformer model was originally designed with NLP as the sole application with word vector embeddings of passage of text being the primary input (Vaswani et al.) 2017). However, recent works have explored using attention mechanisms for different machine learning tasks (Veličkovid (et al. 2017, Zhang et al. 2019 Fu et al.] 2019) and a few investigate the use of transformers for applications outside of the NL. field (Chen et al. 2020). This suggests that self-attention and in
2

---

Under review as a conference paper at ICLR 2021
particular transformer models may work well for any problem that can be posed as sequence of vectors. In this work, the primary input to the transformer will be an embedded dynamical system, where the embedded state time-step i is denoted as &, R. Given that
E = {Eo. (........... .rr) we are interested in the prediction of a physical time series, this motivates the usage of a language modeling architecture that is designed for the sequential prediction of words in body of text. We select the transformer decoder architecture used in the GPT models (Radford et al. 2018 2019)
Transformer Decoder
Fully Hully--oomen Connected Neural Network
Layer Normalization
Transformer Decoder
Transformer Decoder
Multi-head Masked Attention
Layer Normalization
Positional Embedding
Physical Embodding
Figure 2: The transformer decoder model used for the prediction of physical dynamics.
Our model follows the GPT-2 architecture based on the implementation in the Hugging Face trans- former repository (Wolf et al. 2019), but is significantly smaller in size than these modern NLP transformers. This model consists of stack of transformer decoder layers that use masked attention, depicted in Fig. [ The inputs to the transformer the embedded representation of the physical system with the sinusoidal positional encoding proposed in the original transformer (Vaswani et al.)
2017). To train the model, consider data of D embedded i.i.d. time-series D {=) for which we can use the standard time-series Markov model (language modeling) log-likelihod::
Lo ΣΣ Σ
log (E)lE)- ky 0).
is the context window and 0 are the model's parameters. Contrary to the standard NL. approach softmax over dictionary of tokens, the likelihood here is taken as
which poses the likelihood a standard Gaussian between the transformer's prediction and the target embedded value resulting in L2 loss. This is due to the fact that the solution most physical systems cannot be condensed to discrete finite set making tokenization into a finite dictionary not possible and thus softmax approach not applicable. Training is the standard auto-regressive method used in GPT (Radford eet al. 2018), opposed to the word masking (Devlin et al.) 2018), constrained to the embedded space. The physical states, di, have the potential to be very high dimensional thus training the transformer in the
embedded space can significantly lower training costs.
2.2 EMBEDDING MODEL.
The second major component of the machine learning methodology is the embedding model respon- sible for projecting the discretized physical state space into a 1D vector representation. In NL., the standard approach to tokenize then embed finite vocabulary of words, syllables characters us- ing methods such as n-gram models, Byte Pair Encoding (Gage) 1994), Word2Vec (Mikolov et al.) 20) Ve (Pennington et al. 2014), etc. These methods allow language to be represented by series of 1D vectors that serve as the input to the transformer. Clearly a finite tokenization and such NLP embeddings are not directly applicable to a physical system, thus we propose our own embedding method designed specifically for dynamical systems. Consider learning the gen- eralized mapping between the system's 's state space and embedded space: F Raad -> Re and G _ + gn Naturally, multiple approaches can be used especially if the dimensionality of the embedded space less than that of the state-space but this not always the case.
The primary approach that we will propose is a Koopman observable embedding which is tech- nique that can be applied universally to all dynamical systems. Considering the discrete time form of the dynamical system in Eq. the evolution of the state variables be abstracted by Φi+1 = (6:) for which is the dynamic map from one time-step to the next. The foundation of Koopman theory states that for any dynamical system of this form, there exists infinite set of state
3

---

Under review as a conference paper at ICLR 2021
observables, 9 (ø:), that evolve linearly in time such that:
Kg (Φ:) A go ((.).
(3)
where I is known as the Koopman operator (Koopman 1931) which time-invariant. Namely, the Koopman observables can evolve in time through continual matrix multiplication of with the Koopman operator:
($++1) Kgg (d:), g ( =
9 ( K2 (xb.)
in which TT denotes matrix products (e.g. = KC K. X.). Modeling the dynamics of a system through the linear Koopman space can be attractive due to its simplification of the dynamics but also the potential physical insights it brings along with it. Spectral analysis of the Koopman operator can reveal fundamental dynamical modes that drive the system's 's evolution in time.
However, these observables, g (œt), sional. Thus Koopman theory can be viewed as a trade off between lifting the state space into observable space with more complex states but with simpler dynamics. For practical application, the Koopman operator and observables are finitely approximated. In recent years, various machine learning based approaches have been proposed learn both the Koopman operator and observables approximated in finite space for modeling, control and dynamical mode analysis (Takeishi et al., Li et al., 2017] Lusch et al.) 2018 Korda & Mezić 2018 Otto & Rowley [2099] Korda al,, Mezic 2020). While deep learning methods have enabled greater success with discovering Koopman observables and operators (Lusch et al.) 2018 Otto & Rowley 2019), applications have still been limited to fairly simple systems and do not work for long time prediction. This is likely due to the approximation of the dimen-dimeniional Koopman observables, limited data and complete de- pendence on the discovered Koopman operator K. to model the dynamics. Suggesting the prediction of a system through a single linear transform clearly has significant limitations and is fundamentally naive approach from machine learning perspective.
typically unknown and theoretically infinite dimen-
2017
2020
Ø
F
g
Φ
K
Figure 3: Example of a Koopman embedding for a two-dimensional system using a convolution encoder-decoder model. The encoder model, J, projects the physical states into the approximate Koopman observable embedding. The decoder model, G recovers the physical states from the em- bedding.
In this work, we propose using approximate Koopman observations as a methodology to develop embeddings for the transformer model such that F(Ø.)) - g((:). As seen in Fig. 3 the embed- ding model follows a standard encoder-decoder model with the middle latent variables being the Koopman observables. Embedding model architectures for each numerical example are provided in Appendix A We introduce a learnable Koopman operator which takes the form a banded matrix that is symmetrical about the diagonal encourage the discovery of dominate dynamical modes rather than feggufrryenncy oscillations (Lusch et al.) [0088) Otto & Rowley] 2019). This learned Koopman operator is disposed of once training of the embedding model is complete. Given the data set of physical state time-series, Dz = ( following loss:
} the Koopman embedding model is trained with the
D
T ΣΣ (6) +A1 MSE ( (+))) ==
D
LD.
+ |||
(5)
Reconstruction
Dynamics
Decuy
This loss function consists of three components: the first is a reconstruction loss which ensures a consistent mapping to and from the embedded representation. The second is the Koopman dynamics
4

---

Under review as a conference paper at ICLR 2021
loss which pushes y, to follow linear dynamics resulting in time-steps of similar dynamics to have similar embeddings. The last term decays the Koopman operator's parameters to help force the model to discover sower-dimensional dynamical modes.
In reference to traditional NLP embeddings, we believe our Koopman observable embedding has a motivation similar to Word2Vec (Mikolov al. 2013a) well as recent embedding methods such as context2vec (Melamud et al. (016). ELMo (Peters et al.) 2018). etc. Namely, these methods based on the principle of context: words that are contextually related each other have a similar embedding. The Koopman embedding has similar objective encouraging physical realizations containing similar time-invarte iiverinvariant dynamical modes also have similar embeddings. Hence, our goal with the embedding model is to not find the true Koopman observables or operator, but rather leverage Koopman enforce physical context.
CHAOTIC DYNAMICS
As foundational numerical example to rigorously compare the proposed model to other classical machine learning techniques, we will first look at surrogate modeling the Lorenz system governed by:
dr = o x), dt
dy
dz
= x (p- - -) -
dt = z 82.
(6)
dt
We the classical parameters of p 28, a 10, = 8/3. For this numerical example, wish to develop a surrogate model for predicting the Lorenz system given a random initial state zo ~ 40-20, 20, 20). yo 2(-20, 20) and 20 (10, 40). In other words, we wish to surrogate model various initial value problems for this system of ODEs. The Lorenz system is used here because of its well known chaotic dynamies which make extremely sensitive to numerical perturbations and thus an excellent benchmark for assessing a machine learning model's 's accuracy.
Figure 4: Four test case predictions using the transformer model for 320 time-steps.
A total of four alternative machine learning models are implemented: fully coll--connected auto- regressive model, a fully-comnected LSTM model, a deep neural network Koopman model (Lusch et al., 2018; Otto & Rowley 2019) and lastly echo-state model (Chattopadhyay Lukoševičius 2012).. All these models have been proposed in past literature for predicting many dynamical systems. Each are provided the same training, validation and testing data sets containing 2048, 64 and 256 time-series solved using numerical solver, respectively. The train- ing data set contains time-series of 256 time-steps while the validation and testing data sets have 1024 time-steps. Each model is allowed to train for 500 epochs if applicable. The proposed trans- former and embedding model are trained for 200 and 300 epochs, respectively, with an embed- ding dimension of 32. The embedding model is simple fully-connected encoder-decoder model, F 3 - 32 G T2 R33 which is trained first. The transformer is then trained with context length of 64 with 4 transformer decoder layers.
al. 2019
We plot four separate test cases in Fig. for which only the initial state is provided and the trans- former model predicts 320 time-steps. Several test predictions for alternative models are provided in Appendix B In general, we can see the transformer model is able to yield extremely accurate pre- dictions even beyond its context length. Additionally, from a numerical solver and predicted from transformer model Fig. 5a Note both have the same structure, which qualitatively indicates that the transformer indeed maintains physical dynamics.
plot the Lorenz solution for 25k time-steps
5

---

Under review as a conference paper at ICLR 2021
Sape
Dox
83)-
30
033
(a)
(b)
(c)
Figure 5: (a) Lorenz solution of 25k time-steps. (b) The Lorenz map produced by each model. (c)
The test relative mean- Sean-squreddeeror (MSE) with respect to time.
Table 1: Test set relative mean-squared-error (MSE) for surrogate modeling the Lorenz system at several time-step intervals.
Relative MSE
Model
Parameters
(0-64)
[64-128]
[128- 192)
Transformer
36k/54ki
0.0003
0.0060
0.0221
LSTM
103k
0.0041
0.0175
0.0369
Autoregressive
92k
0.0057
0.0253
0.0485
Echo State
7.5k/6.3m
0. 1026
0.1917
0.2209
Koopman
108k
0.0001
0.0962
2.0315
Learnable parameters for the embedding/transfommer model.
Learnable oulput parameters/fixed input and reservoir parameters.
Table 2: Test set relative mean- mrean-squared-eee -eroor (MSE) for surrogate modeling the Lorenz system at several time-step intervals with noisy data.
Relative MSE 1 % Noise
Relative MSE 5%
Noise
Model
(0-64)
(64-128)
(128-192)
(0-64)
(64-128)
(128-192)
Transformer
0.0021
0.0216
0.0429
0.0210
0.0759
0.1292
LSTM
0.0045
0.0218
0.0437
0.0212
0.0758
0.1324
Autoregressive
0.0114
0.0417
0.0901
0.0760
0.2060
0.2065
Echo State
0.0859
0.1686
0.2102
0.1000
0.1581
0.2051
Koopman
0.0047
0.1192
0.1597
0.0200
0.0787
0.1906
The proposed transformer and alternative models' relative mean squared errors for the test set are plotted in Fig. a5las function of time and listed in segmeeee into several intervals based the transformer's context length. We can see that the transformer model is able to achieve the best performance using the least number parameters with the exception of the Koopman only model for early times. However, the Koopman model is unstable for long time predictions rendering useless for surrogate predictions. To quantify accuracy of the chaotic dynamics of each model, the Lorenz map is plotted in Fig. Fig.5bjwa which is well-defined relation between successive local maximas despite the Lorenz's 's chaotic nature. Calculated using 25k time-step predictions from each model, again we can see that the transformer model agrees the best with the numerical solver indicating that has learned the best physical dynamics of all the tested models.
Additionally we test the transformer's sensitivity to contaminated data by adding white noise to the training observations scaled by the magnitude the state variables. Each model tested with clean data is retrained with data perturbed by 1% and 5% noise. The effects of these two different noise levels is qualitatively illustrated in Appendix B The errors are listed in Table 2 In general, we indeed that the transformer performing model for 1 % noise and still being competitive, particularly at later time-steps, with 5% noise.
still perform adequately with noisy data by being the best

---

Under review as a conference paper at ICLR 2021
4 2D FLUID DYNAMICS
The next dynamical system we will test is transient 2D fluid flow governed by the Savier-Stokes equations:
ou: du, 1 0p 01
+ du: + Uj Ozj p 8x; dx
in which u, and p are the velocity and pressure, respectively. v is the viscosity of the fluid. We consider modeling the classical problem of flow around a cylinder at various Reynolds number defined by Re u,,d/v in which Uin 1 and d are the inlet velocity and cylinder diameter, respectively. In this work, we choose to develop surrogate model to predict the solution at any Reynolds number between Re V (100, 750). This problem a fairly classical flow investigate in scientific machine learning with various levels of difficulty (Lee & You |
Morton al. 2018 Lusch et al., 2018 Hann et al., 2019; Xu & Duraisamy 2020: Geneva & Zabaras 2020b). Here we choose of the
difficult forms: model the flow starting at steady state flow field f 0. Meaning the model is provided zero information on the structure of the cylinder wake during testing.
Training, validation and testing data is obtained using the OpenFOAM simulator (Jasak al. (007), from which rectangular structured sub-domain is sampled centered around the cylinder's wake. Given that the flow is two-dimensional, our model will predict the x-velocity, y-velocity and pressure fields, [ug Myy P E R331288611 The training, validation and test data sets consist of 27, and fluid flow simulations respectively with 400 time-steps each physical time-step size of At 0.5. As base line model, we train a convolutional encoder-decodee model with a stack of three convolutional LSTMs (Xingiian al., 2013) in the center. We note that convolutional LSTMs have been used extensively in recent scientific machine learning literature for modeling various physical systems (Han et al.) 2019, Wiewel et al., 2019 Tang et al. 2020; Geneva & Zabaras, (020b), thus can be considered a state-of-the-art approach. The convolutional LSTM model is trained for 500
epochs.
Additionally, three different embedding methods are implemented: the first is the proposed Koop- man observable embedding using a convolutional auto-encoder. This model encodes the fluid x-velocity, y-velocity, pressure and viscosity fields to embedded dimension of 128, F R4x 128x18864 + R 1222 g 1 - 3 128x64 The second embedding method using the same convolutional auto encodrr model but without the enforcement of Koopman dynamics on the em- bedded variables. The last embedding method tested was principal component analysis (PCA), a classical dimensionality reduction benchmark method for all fields of science. For each embedding method an identical transformer model with context length of 128 with 4 transformer decoder lay- ers is trained. Similar the previous example, the embedding models are trained for 300 epochs when applicable and the transformer is trained for 200.
H4
9000
9225
Figure 6: Vorticity, u. V Uz of two test case predictions using the proposed transformer with Koopman embeddings at Reynolds numbers 233 (top) and 633 (bottom).
Each trained model is tested on the test set by providing the initial laminar state at t = 0 with the fluid viscosity and allowing the model predict 400 time-steps into the future. Two test predictions using the proposed transformer model with Koopman embeddings are plotted in Fig, G which the predicted vorticity fields in good agreement with the true solution. The test set relative mean

---

Under review as a conference paper at ICLR 2021
square error for each output field for each model is plotted in Fig. 7 The errors of each field over the entire time-series are listed in T Additional results are provided in Appendix C
For all alternative models, rapid error increase can be seen between the transition from the laminar flow into vortex shedding. This error then plateaus since each model is able to produce stable vortex shedding, the instantaneous states predicted from these models deviate from the true numerical solution. is clear that only the transformer model with Koopman embeddings can correctly predict these dynam- ies demonstrating the benefits of enforcing Koopman dynamics on the embedded representations. Compared to the widely used ConvLSTM model, the proposed transformer offers far more reliable predictions with less learnable parameters.
[50, 100] which is due to
illustrated in Figs. [5 & T6 in Appendix C however
V-ldecity
Cian
Nee
041-
Baleaxe MRE
010
0.00
8:01
Figure 7: Test set relative mean-squared-error (MSE) of the base-line convolutional LSTM model (ConvLSTM) and transformers with Koopman (KM), auto-encoder (AE) and PCA embedding meth- ods over time.
Table 3: Test set relative leean--quaredeeree -rroor (MSE) of each output field for surrogate modeling 2D fluid flow past cylinder. Models listed include the convolutional LSTM model (ConvLSTM) and transformers with Koopman (KM), auto-encoder (AE) and PCA embedding methods.
Relative MSE 00
400)
Model
Parameters
U
U
p
Transformer-KNn KM
224k/628kk
0.0036
0.0175
0.0018
Transformer-AF
224k/628kf
0.0226
0.09606
0.0088
Transformer-PCH
m./m2k8
0.0251
0.1062
0.0108
ConvLSTM
934k
0.0220
0.0862
0.0086
Leeamable parameters for the embedding/ transformer model.
3D REACTION-DIFFUSION DYNAMICS
The final numerical example demonstrate the proposed transformer model is 3D reaction- diffusion system governed by the Gray-Scott model:
Ou
uv² + f (), du Tu dx?
Ov
do + vn² ( k) To ax?
(8)
Øt
Dt.
in which u and v are the concentration of two species, Fa and ra are their respective diffusion rates, & is the kill rate and f is the feed rate. This is a classical system of particular application
chemical processes as it models the following chemical reaction: U + 2V 3V: V + P. For a set region of feed and kill rates, this seemingly simple system can yield a wide range of complex dynamics (Pearson 1993 Lee al.. 1993). Hence, under the right settings, this system an excellent study to push the proposed methodology to its predictive limits. In this work, will use the parameters: r. = 0.2, r. 0.1, A = 0.055 and f = 0.025 which results in a complex dynamical reaction seen in Fig. 8 Akin the first numerical example, the initial condition of this system is stochastic such that the system is seeded with 3 randomly placed perturbations within the periodic domain.
Training, validation and testing data are obtained from a Runge- Kutte-Kutta finite difference simulation on structured grid, {u, v) E RR2x32x32x32, The training, validation and test data sets consist
8

---

Under review as a conference paper at ICLR 2021
of 512, 16 and 56 time-series, respectively, with 200 time-steps each at a physical time-step size of = 5. A 3D convolutional encoder- used to embed the two species into a 512 embedding dimension, F 32233xx32333 + R2 9 R + 222332x32x32 Transformer models with varying depth are trained all with a context length of 128. All other model and training parameters for the transformer models consistent. A test prediction using the transformer model is shown in Fig. [ and the errors for each trained transformer are listed in Table 4 Despite this system having complex dynamics in 3D space, the transformer is able produce acceptable predictions with very similar structures as the numerical solver. To increase the model's 's predictive accuracy we believe the limitation here is not in the transformer, but rather the number of training data and the inaccuracies of the embedding model due to the dimensionality reduction needed. This is supported by the fact that increasing the transformer's depth does not yield considerable improvements for the test errors in Table Additional, results are provided in Appendix D
Figure 8: u volume plots of the target (top) and transformer prediction (bottom) for a test case of the Gray-Scott system. Isosurfaces displayed span the range [0.3, 0.5] show the inner structure.
Table 4: Test set relative -ean-squared-error (MSE) for surrogate modeling 3D Gray- Scott system.
Relative MSE [0- 200
Model
Layers
Parameters
u
Transformer
6. 6.2m//66mm
0.0159
0.0120
Transformer
6.2m/ 1.2m/11.900 12.9m
0.0154
0.0130
Transformer
6.2m/25.5m
0.0125
0.0101
Learnable parameters for the embedding/ transformer model.
6 CONCLUSION
While transformers and self-attention models have been established as a powerful framework for NLP tasks, the adoption of such methodologies has yet to fully permute other fields. In this work, we have demonstrated the potential transformers have for modeling dynamics of physical phenomena. The transformer architecture with self-attention allows the model to learn longer and more com- plex temporal dependencies than alternative machine learning methods, which is benign for many physical systems that exhibit multi-scale dynamics. This can be directly attributed to the transform- ability to draw information from many past time-steps efficiently through self-attention rather than just the most recent time-step which occurs in auto-regressive, Fuler-based and standard RNN models.
The key challenge of using the transformer model is identifying appropriate embeddings to repre- sent the physical state the system, for which we propose the use of Koopman dynamics to enforce dynamical context. Using the proposed methods, our transformer model can outperform alternative models widely seen recent scientific machine learning literature. The investigation of unsuper- vised pre-training of such transformer models as well as gaining better understanding of what attention mechanisms imply for physical systems will be the subjects of works the near future.
ACKNOWLEDGMENTS
The authors acknowledge computing resources provided by the AFOSR Office of Scientific Re- search through the DURIP program.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Ashesh Chattopadhyay, Pedram Hassanzadeh, Devika Subramanian, and Krishna Palem. Data- driven prediction of multi-scale Lorenz 96 chaotic system using a hierarchy of deep learning methods: Reservoir computing, ANN, and RNN-LSTM. 2019.
Mark Chen, Alec Radford, Rewon Child, Jeff Wu, Heewoo Jun, Prafulla Dhariwal, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In Proceedings of the 37th International Conference on Machine Learning, 2020.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdi- nov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:/810.044: 2018.
Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3146-30 3154, 2019.
Philip Gage. A new algorithm for data compression. Users Journal, 12(2):23-38, 1994.
Nicholas Geneva and Nicholas Zabaras. Modeling the dynamics of PDE systems with physics- constrained deep auto-regressive networks. Journal of Computational Physies, 403:109056, 2020a.
Nicholas Geneva and Nicholas Zabaras. Multi-fidelity generative deep learning turbulent flows.
arXiv preprint wrXiv:2006.04731, 2020b.
R. Goreález-Garefa, R. Rico-Martínez, and I.G. Kevrekidis. Identification of distributed parameter
systems: A neural net based approach. Computers & Chemical Engineering, 22:5965 S968, 1998. ISSN 0098-1354. doi: http://iennn/aatcceeeeeeee/
100.0000009981115599000199444 URL http:
com/ ropean Symposium on Computer Aided Process Engineering-8.
www. www.scceecedirece.o http://///..............////////
Eu-
Renkun Han, Yixing Wang, Yang Zhang, and Gang Chen. A novel spatial-temporal prediction method for unsteady wake flows based on hybrid deep neural network. Physics of Fluids, (12); 127101, 2019.
Hrvoje Jasak, Aleksandar Jemcov, Zeljko Tukovic, et al. Openfoam: A c++ library for complex physics simulations. In International workshop on coupled methods numerical dynamics, vol- ume 1000, pp. 1-20. IUC Dubrovnik Croatia, 2007.
Bernard O Koopman. Hamiltonian systems and transformation in hilbert space. Proceedings of the national academy of sciences of the united states of america, 17(5):315, 1931.
Milan Korda and Igor Mezić. Linear predictors for nonlinear dynamical systems: Koopman operator meets model predictive control. Automatica, 93:149 160, 2018. ISSN 0005-1098. doi: https: (oooooorr//...111///aaoomoooioo
URL http:/ www. sciencedirect. com/
sci
00000098888111111 830133X
Milan Korda, Mihai Putinar, and Igor Mezić. Data-driven spectral analysis of the koopman opera-
tor. Applied and Computationnll Harmonic Analysis, 48(2):599 629, 2020. ISSN 1063-5203. doi: http://ccoooo//rrreeeee///////1s com/
002. URL http: htt.:://www.wcoe.
Kyoung WD McCormick, Qi Ouyang, and Harry L Swinney. Pattern formation by interacting chemical fronts. Science, 261 1993.
Sangseung Lee and Donghyun You. Prediction laminar vortex shedding over a cylinder using deep learning. arXiv preprint arXiv: 2017.
10

---

Under review as a conference paper at ICLR 2021
Qianxiao Li, Felix Dietrich, Erik M Bollt, and loannis G Kevrekidis. Extended dynamic mode decomposition with dictionary learning: A data-driven adaptive spectral decomposition of the koopman operator. Chaos: An Interdisciplinary Journal of Nonlinear Science, 27(10):103111, 2017.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprini arXiv: 1692, 2019.
Mantas Lukoševičius. A practical guide to applying echo state networks. In Neural networks: Tricks of the trade, pp. 686. Springer, 2012.
Bethany Lusch, Nathan Kutz, and Steven I. Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature communications, 911):1-10, 2018.
Romit Maulik, Romain Egele, Bethany Lusch, and Prasanna Balaprakash. Recurrent neural network architecture search for geophysical emulation. arXiv preprint arXiv:2004. 10928, 2020.
Oren Melamud, Jacob Goldberger, and Ido Dagan. context2vec: Learning generic context embed- ding with bidirectional Istm. In Proceedings of the 20th SIGNLL conference on computational natural language learning, pp. 51-61, 2016.
Igor Mezic. On numerical approximations of the Koopman operator. arXiv:2009.05883, 2020.
arXiv preprint
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. arXiv preprint arXiv: 1301.3/301 20.3a.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations words and phrases and their compositionality. In Advances in neural information pro- cessing systems, pp. 311 111111119 19, 2013b.
Shaoxing Mo, Yinhao Zhu, Nicholas Zabaras, Xiaoqing Shi, and Jichun Wu. Deep convolutional encoder-decoder networks for uncertainty quantification of dynamic multiphase flow in heteroge- neous media. Water Resources Research, 55(1):703-728, 2019.
Jeremy Morton, Antony Jameson, Mykel J Kochenderfer, and Freddie Witherden. Deep dynamical modeling and control of unsteady fluid flows. In Advances in Neural Information Processing Systems, pp. 9258-9268, 2018.
Samuel E Otto and Clarence W Rowley. Linearly recurrent autoencoder networks for learning dynamics. SIAM Journal on Applied Dynamical Systems, 18(1):558-593, 2019.
John E Pearson. Complex patterns in a simple system. Science, 261 1993.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word
representation. In Proceedings the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations, arXiv preprint arXiv: /00.:0v3000.333533 2018.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. OpenAl Blog, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8)9, 2019.
Alvaro Sanchez-Gonzaler Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter W. Battaglia. Learning simulate complex physics with graph networks. In International Confer- ence on Machine Learning, 2020.
Anna Shalova and Ivan Oseledets. Tensorized transformer for dynamical systems modeling. arXiv preprint ar arxiv:2006. 2tl.20006.03445, 2020.
11

---

Under review as a conference paper at ICLR 2021
Naoya Takeishi, Yoshinobu Kawahara, and Takehisa Yairi. Learning koopman invariant subspaces for dynamic mode decomposition. In Advances in Neural Information Processing Systems, pp- 1130-1140, 2017.
2017.
Meng Tang, Yimin Liu, and Louis J Durlofsky, A deep-learning-basee surrogate model for data similation in dynamic subsurface flow problems. Journal of Compulational Physics, pp. 109456, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv: 2017.
Steffen Wiewel, Moritz Becher, and Nils Thuerey. Latent space physics: Towards learning the temporal evolution of fluid flow. In Computer Graphics Forum, volume 38, 71-82. Wiley Online Library, 2019.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jemnite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's 's transformers: State- of-the-art natural language processing. ArXiv, abs/ 1b00/191,., 2019.
SHI Xingjian, Zhourong Chen, Hao Wang, Dit- Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional Istm network: A machine learning approach for precipitation nowcasting. In Ad- vances in neural information processing systems, pp. 802-810, 2015.
Jiayang Xu and Karthik Duraisamy. Multi-level convolutional autoencoder networks for parametric prediction of spatio-temporal dynamics. Computer Methods in Applied Mechanics and Engineer- ing. 372: 13379, 2020.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. International Conference on Machine Learning, pp. 7354-7363. PMLR, 2019.
12

---

Under review as a conference paper at ICLR 2021
A KOOPMAN EMBEDDING MODELS
The architectures for the Koopman embedding models are provided. In general, each is classical auto-encoder architecture using standard machine learning methodologies. For complete details we refer the reader to the code.
500
500
K
Gas
Layer Norn
Figure 9: Fully-connected embedding network with ReLU activation functions for the Lorenz sys- tem.
4x
x2xx 16
64x8x
E
50
K
K
12x 4x8 4x4x8
&
{+11
32 l6x 32
16x x2.66
3 64 128
6xxx 16
4x4x8 128 128x4x8
Figure 10: 2D convolutional embedding network with ReLU activation functions for the flow around cylinder system consisting of 5 convolutional encoding/decoding layers. Each convolutional oper- ator has a kernel size of (3. 3). In the decoder, the feature maps are up-sampled before applying a standard convolution. Additionally, two auxiliary fully-connected networks used predict the diagonal and off-diagonal elements of the Koopman operator for each viscosity v.
Ei11
5/2
Layer Norm
12stesia
Figure 11: 3D convolutional embedding network with leaky ReLU activation functions for the Gray- Scott system. noraaaaormalizttionn used between each the convolutional layers. In the decoder, the feature maps up-sampled before applying standard 3D convolution.
13

---

Under review as a conference paper at ICLR 2021
B LORENZ SUPPLEMENTARY RESULTS
Predictions for all of the tested models for three test cases are plotted in Fig. 12 in which we can see the transformer model is consistently accurate within the plotted time frame. Additional test predictions for the Lorenz system using the transformer model are shown in Fig. [3 Lastly, in Fig. 14 we illustrate the two different noise levels used to contaminate the training data for the results listed in Table
Figure 12: Three Lorenz test case predictions using each tested model for 128 time-steps.
Figure 13: Lorenz test predictions using the transformer model for 320 time-steps. 19
Figure 14: Comparison between the clean and contaminated (noisy) training data.
14

---

Under review as a conference paper at ICLR 2021
C CYLINDER SUPPLEMENTARY RESULTS
Prediction fields of all the tested models for a single test case are plotted in Figs. 15 & 16 While all models are able to perform adequately with qualitatively good results, the transtormer model with Koopman embeddings TTaasformer-KM, outperforms alternatives.
The evolution of the flow field projected onto the dominate eigenvectors of the learned Koopman operator for two Reynolds numbers are plotted in Fig. [7] This reflects the dynamical modes that were learned by the embedding model to impose physical "context" on the embeddings. Given that complex, we plot both the magnitude, | and angle, Ltb. For both Reynolds numbers it is easy to see the initial transition region, < 100, before the system enters the periodic vortex shedding. Once in the periodic region, we can see that the higher Reynolds number has higher frequency which reflects the increased vortex shedding speed.
the eigenvectors
2-9
1-22
Figure 15: Velocity magnitude predictions of test case at Re 633 using each tested model.
14180
1-20
1-333
Figure 16: Pressure predictions of a test case at Re 633 using each tested model.
(a) Re 233
(b) Re 633
Figure 17: The dynamics of the fluid flow around a cylinder projected onto the 8 most dominate eigenvectors of the learned Koopman operator, K. in the embedding model.
15

---

Under review as a conference paper at ICLR 2021
D GRAY-SCOTT SUPPLEMENTARY RESULTS
Additionally, to better visualize the accuracy of the trained transformer model, contour plots for three test cases are also provided in Fig. 20 Both the volume and contour plots illustrate the complex and unique structures that arise in this system for each initial condition.
Volume plots two test cases for both species are provided in Figs, 18 & 19
(a) target (top) and transfommer prediction (bottom)
(b) target (top) and transformer prediction (hottom)
Figure 18: Test case volume plots for the Gray- Scott system. Isosurfaces displayed span the range
u, |0.3, 0.5) to show the inner structure.
(a) target (top) and transformer prediction (bottom)
(b) target (top) and transformer prediction (bottom)
Figure 19: Test case volume plots for the Gray-Scott system. Isosurfaces displayed span the range u. 10.3, 0.5) to show the inner structure.
16

---

Under review as a conference paper at ICLR 2021
Ind
1405
I+33
bw10
twite
Inlit
20
2
Figure 20: y plane contour plots of three Gray-Scott test cases sliced at 16.
17


Reviews:
[{"review_id": "mYUIMKI32uA", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**:\nThe paper proposes applying transformer models to modeling physical systems. The state at each time step is embedded into a continuous vector using a pretrained encoder-decoder model based on Koopman’s theory. The experiments are performed on three physical systems and generally show that (1) a transformer model outperforms alternative machine learning methods, (2) a transformer model with the proposed embedding outperforms transformer models with alternative embeddings based on autoencoders or PCA, and (3) more transformer layers help (but only slightly).\n\n**General comments**:\nThis is, to my knowledge, a new application area for transformer models, and the fact that they outperform alternatives could be interesting to the ML community.\nFurthermore, the application of transformer models is not entirely standard, in that the paper proposes learning embeddings that are specialized to the application area.\nThe proposed approach is compared both with alternative (non-transformer) techniques and with ablations of the transformer model.\n\n**Questions**:\nThe proposed embeddings are based on a neural network-based encoder and decoder pair and these are trained either with an objective based on Koopman’s theory or a standard AE objective. Have you considered training the embeddings jointly with the transformer parameters? I think this baseline is needed to motivate the need for pretrained embeddings and it could alleviate the concern that the model limitations are due to the \"inaccuracies of the embedding model\".", "extra_scores": {}, "extra_text": {}}, {"review_id": "V_OY67Q-8Qf", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary:\nThe paper proposes to use transformers for modelling dynamical systems. The transformer is combined with a linear dynamical system to enforce Koopman features and is trained using the reconstruction and prediction loss. Finally, the proposed algorithm is applied to the different tasks with 1, 2 & 3 dimensions. On each simulated task the proposed algorithm marginally outperforms sufficient baselines.\n\n### Clarity & Style:\nThe paper is clearly written and understandable. The figures could be drastically improved by optimizing the white space and axes spacing, e.g. figure 5. Furthermore, having videos of the solver and network predictions would be preferable. It's a bit surprising that the authors don't cite https://arxiv.org/pdf/2002.09405.pdf, which would also be an interesting but not necessarily required baseline.\n\n\n### Quality, Originality & Significance:\nWhile the paper is clearly written and well executed in terms of experiments and baselines, the paper is missing originality and significance. The paper does not introduce new perspective or demonstrate a previously unaccomplished task. It simply applies a straight-forward method and demonstrates that it works marginally better compared to existing methods. Especially the usefulness of the model is questionable. The Lorenz system is a **three** parameter dynamical system and is approximated with a transformer with **54,000** parameters! What is the advantage of the transformer network compared to the analytical solvers? Is it faster and if yes how much more energy does it consume compared to the solver? Why is it better compared to a look-up table with 54,000 values, which does not require tuning network hyper-parameters? Does an analytic version of the Koopman operator exist for the Lorenz system? There might be a good chance as the Lorenz system is a polynomial dynamic system. How many parameters does this have? If so how do the learned features compare to the analytic features?\n\nFurthermore, the paper lacks to address the real challenges of learning dynamical systems (*in my opinion*). For clean data (or data with clean Gaussian noise) and a single coherent timescale most methods perform reasonably well. IMO the main problems are:\n\n* Messy data from real systems including, non-Gaussian noise, discretization errors, time delays. Can such an over-parameterized transformer work for such data and recover the underlying structure or does it overfit?  \n* Dynamical systems with multiple space & time-scales. Is the transformer model able to learn a good dynamical system with multiple time-scales?\n\nTo increase the significance of the paper the authors would need to address one of these two questions.\n\n\n###  Conclusion:\nAll in all, I think the paper is well written and the performed experiments are well executed. However, right now the paper misses significance to be published at ICLR. To improve the papers the authors would either have to address messy real-world data or dynamical systems with multiple time-scales. I am open for discussion, why this model is a significant advancement over prior methods, but right now I don't see that.\n\n### **Post Discussion Comments:** \nThe authors provided videos of the task. While I still find that this paper misses clear benefits/use-case for the machine-learning community. I still don't clearly understand why one would need 54,000 parameters for a  3 parameter system. One could store a really big table of this one 1d system for this amount of parameters. However, the paper is well executed and clearly written in regards of technical aspects. The motivation remains doubtful for me. Due to the execution and clear writing, i increased my score to weak accept.  \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "MZGiUX9adty", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Quality\n\n\n- The experiments are clear and the results are easily understandable.\n- In all three experiments, the paper achieves compelling results with low errors compared to baselines, especially for high number of time steps.\n\n\n\nClarity\n\n\n- I find the mathematical notations in Section 2.2 could be more clear. I'm a bit confused about the infinite dimensionality of the state observables and how the Koopman operator handles it, whereas in the Figure 2 shown the Koopman operator K looks like a banded diagonal linear operator. It is not apparent why equation (3) leads to the construction of the encoder-decoder model shown. More explanations on the Koopman operator would be appreciated to make the paper more self-contained.\n\n- The plots make the experiment results quite clear in terms of the model's capability to estimate the dynamics.\n\n- Some minor typos such as \"on going\" which should be \"ongoing\"\n\n\n\n\nOriginality\n\n- This work uses a Transformer model instead of recurrent networks to perform dynamical system simulation. The paper also proposes using an embedding model. Overall, the novelty is moderate. \n\n\nSignificance\n\n\n\nHigh-level pros and cons\n\nPros\n\n- Modeling dynamical systems is an interesting deep learning application.\n- Good results compared to baselines in all experiments shown.\n\nCons\n\nThe clarity can be improved in my opinion.", "extra_scores": {}, "extra_text": {}}, {"review_id": "3ink3-dD-w_", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes the adaptation of the recent and SOTA framework\nused in NLP to dynamical systems. The idea is nice and well\nmotivated. Two main steps are described: Transformer model for\nmarkovian prediction of time series, along with a Koopman inspired\nmethod to learn embeddings of the state space.  \n\nThe interaction between the main training steps (embedding and\ntransformer parts) is not clear in the paper: pre-training of the\nembeddings, followed by a fine tuning step (or not ?). This is an\nissue of the paper since this interaction is important for NLP\napplications. The reader should be able to fully understand what is\ngoing on here, only by reading the paper.\n\nMy second concern is about the experimental part. The dynamical\nsystems are also limited in complexity, but this is not crucial here.\nHowever, the difference in the results is not well documented and for\nexample, with the fluid flow behind a cylinder, the low results\nobtained with CNN is surprinsing (too bad ?).\n\nThese concerns are not individually prohibitive for such a article,\nand maybe the paper could be improved before\npublication. Nevertheless, the ideas are really interesting.\n\n\nIn the introduction, authors claim that : \"Standard deep neural\nnetwork architectures such as auto-regressive, recurrent and LSTM\nbased models have been largely demonstrated to be effective at\nmodeling various physical dynamics (...) However, the current literature is focused on systems of limited complexity and domain size.\"\n\nIt is maybe important here to better characterize the term complexity,\nwhy it is limited ? and why it is important ?  Moreover, the\ncorrelation between this complexity and the originality of using of\ntransformers is not straightforward. Complex architecture based on\nLSTM for instance could be also effective or should better explain why\ntransformers are different when faced with larger complexity and\ndomain size.\n\nThe notations in the section 2 are a bit heavy and maybe it could be\neasier to follow one of the experimental example, without loss of\ngenerality, to introduce the whole picture.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a framework for learning dynamical system models from observations consisting of discrete spatio-temporal series. It is composed of two components trained sequentially.  A first one learns embedding from observations using a seq2seq approach, where the embeddings are constrained to follow linear dynamics. This is inspired by approximation schemes for Koopman operators. These embeddings are then used as the spatio-temporal series representions and are fed to a transformer trained as an autoregressive predictor. Experiments are performed on different problems using data generated from PDEs through numerical schemes. Comparisons are performed with different ML baselines.

The paper is well written with experiments on problems with different complexities. The original contributions of the paper are 1) the combination of pretrained embeddings with a transformer auto-regressor, 2) a seq2seq architecture for learning time series representations constrained by linear dynamics.

On the cons side, the paper original contribution and significance are over claimed. Closely related ideas for learning approximate Koopman operators and observables have already been developed and used in similar contexts. Besides there is no discussion here on the properties or physical interpretability (which is often an argument for Koopman) of the learned representations. Then the baselines are mainly composed of simple regressors (LSTM, conv-LSTM, etc.) and this is not a surprise that they cannot learn dynamics over long term horizons. There is no comparison with dynamical models incorporating numerical integration schemes that could model the temporal dynamics of the system. There is now a large literature on this topic exploiting discrete (ResNet like) or continuous formulations (as started with Neural ODE).
Decision: Reject