SEA: State-Exchange Attention for High-Fidelity Physics Based Transformers
Authors: Parsa Esmati
Amirhossein Dadashzadeh
Vahid Goodarzi Ardakani
Nicolas Larrosa
Nicolò Grilli
Venue: NeurIPS 2024 poster (2024)
Keywords: Generative
Spatio-temporal
AI4SCIENCE
Physics informed
Transformers in physics
Computational fluid dynamics
Primary area: machine_learning_for_physical_sciences

Abstract: Current approaches using sequential networks have shown promise in estimating field variables for dynamical systems, but they are often limited by high rollout errors. The unresolved issue of rollout error accumulation results in unreliable estimations as the network predicts further into the future, with each step's error compounding and leading to an increase in inaccuracy. Here, we introduce the State-Exchange Attention (SEA) module, a novel transformer-based module enabling information exchange between encoded fields through multi-head cross-attention. The cross-field multidirectional information exchange design enables all state variables in the system to exchange information with one another, capturing physical relationships and symmetries between fields. Additionally, we introduce an efficient ViT-like mesh autoencoder to generate spatially coherent mesh embeddings for a large number of meshing cells. The SEA integrated transformer demonstrates the state-of-the-art rollout error compared to other competitive baselines. Specifically, we outperform PbGMR-GMUS Transformer-RealNVP and GMR-GMUS Transformer, with a reduction in error of 88% and 91%, respectively. Furthermore, we demonstrate that the SEA module alone can reduce errors by 97% for state variables that are highly dependent on other states of the system. The repository for this work is available at: https://github.com/ParsaEsmati/SEA

SEA: State-Exchange Attention for High-Fidelity Physics Based Transformers
Parsa Esmati
Amirhossein Dadashzadeh University of Bristol
University of Bristol
parsa. esmati@bristol.ac. uk
a. dadashraddeherristol...
Vahid Goodarzi Ardakani
Nicolas Larrosa University of Bristol
Sabe Technology Limited
vahid. goodazzieeda.e tech
nicolas wawrosa@bristol..cc
Nicolò Grilli
University of Bristol
nicolo.
nicolo.grillf@brrr...... wwiggrrrriiisooo...... ac. uk
Abstract
Current approaches using sequential networks have shown promise in estimating field variables for dynamical systems, but they are often limited by high rollout errors. The unresolved issue of rollout error accumulation results in unreliable estimations the network predicts further into the future, with each step's 's er- ror compounding and leading to an increase in inaccuracy. Here, we introduce the State-Fxchange Attention (SEA) module, novel transformer-based module enabling information exchange between encoded fields through multi-head cross- attention. The cross-field multidirectional information exchange design enables all state variables in the system to exchange information with one another, capturing physical relationships and symmetries between fields. Additionally, we introduce an efficient ViT-like mesh autoencoder to generate spatially coherent mesh em- beddings for large number of meshing cells. The SEA integrated transformer demonstrates the state-of-the-a rollout error compared to other competitive base- lines. Specifically, we outperform PbGMR-GMUS Transformer- Realer and GMR-GMUS Transformer, with reduction in error of 88% and 91 %, respectively. Furthermore, demonstrate that the SEA module alone can reduce errors by 97% for state variables that are highly dependent on other states of the system. The repository for this work available at: https://githubb.com www.ttuub.ooo///eeeeeeeeeeeeeeee www.ecom//aaeeeee commParsaEEmattiSE...
Introduction
Solving partial differential equations (PDE) has been primary concern of many fields in science and engineering, including physics (Salvini et al. 2024], chemistry [Grilli et al.) 22020). and material sciences (Grilli et al. 2018 Esmati al. 2024 In many cases where direct analytical solution of the PDE is impossible to obtain, numerical simulations are used. To solve these equations numerically, the domain is discretized into smaller cells using discretization method such finite element finite volume methods. In some cases, the domain of interest is divided into millions smaller elements forming large matrices representing the equation [Moukalled al. 2016 Solving these discretized equations typically follows an iterative technique, which can take days and sometimes weeks of runtime converge to the full solution the defined temporal domain [Liu al. 2024
Jiang et al. 2023 In some cases to resolve some specific features of the system a fine mesh is
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

required. The graphical representation of small droplet [Um et al. 2018 cracking and brittle behaviour in thin components [Pfaif al. 2014], multiphase and turbulent flows computational fluid dynamics instances of such scenario.
et al. 2021 Thuerey et al., 2020 Heyse et al. 202 b are all
Finer mesh however comes at the price of computational cost, which may not be feasible in industry settings. Consequently, there is critical need for frameworks that can either bypass these detailed simulations or accelerate the solvers.
Recent advances in sequential networks have shown promising results in estimating the state variables of dynamical systems over time. However, they have not yet been effectively utilized to bypass or accelerate numerical models (Yousif et al.] 2022] Key challenges include lengthy training times and steep gradients in rollout errors [Sun et al. 2023 Han et al. 2022]. Additionally, many of these networks task-specific, necessitating retraining for each unique test case [Li et al. 2020a]. If a model is to be integrated into a solver, steep error accumulation necessitates frequent retraining of networks maintain efficacy when bypassing numerical solvers. This leads to unnecessary computational costs, making it challenging to integrate these models with solvers. Ideally, a model should be capable of learning the underlying physics and constitutive laws minimize error gradients. Such a model could reduce retraining frequency and enable reuse through transfer learning approaches Yosinski al. 2014].
The current trend towards probabilistic models, such as diffusion models for science and physics, tends overcome the rollout errors (Han et al., 2024 Valencia al., 2024]. However, regardless of their domain (science, video, etc.), these models can produce unrealistic motions, especially for long sequence generations, and result in temporal limitation (Weng et al. 20244 Hence, the autoregressive generation seems unavoidable for longer sequences.
Transformer- architectures, in particular, are at the forefront of the autoregressive generations [Sun et al., 2023| Zhao et al., 2023]. By drawing parallel between the video and spatio-temporal PDE simulations, we intend improve the unavoidable autoregressive rollout error for long sequence generation using transformers as the building block. Hence, inspired by the matio-temppral cross- attention mechanisms used in vision models [Lin et al., 2022 Chen al. 2021], propose Exchangx Attention (SEA) module for physics-based transformers, designed to mimic PDE by explicitly capturing variable coupling. The effective multidirectional information exchange between fields enables the correction some fields by others, allowing the model to learn the complex features of the physical system. This work employs Vision Transformer (ViT) like (Dosovitskiy al. 2020 encoder for mesh embedding. The embedded mesh at each instance is then used in the temporal State Baaha-Exchange Attention (SEA) integrated Transformer to predict the system's 's future states.
In summary, the contributions of this work include:
1. Design and integration of novel SEA module for physies-domain Transformer models. This module enables learning the underlying physics through a multidirectional information exchange process between the state variables.
2. Assembly of full ViT-SEA integrated framework that demonstrates state-of-the-an results in generating the complete sequence of the physical system given the initial sequence and specific time-invariant parameters representing the model.
3. Comprehensive evaluation of SEA module, and VIT-SEA integrated transformer across different computational fluid dynamics cases, showing over 60% reduction in error in all cases compared to state-of-the- -frt---rt- models.
Related work
In recent years, deep learning has led to major advancements in modeling physical systems, with contributions ranging from applying computer vision techniques to enhance the resolution of coarse meshes to incorporating physical symmetries and constraints through innovative modifications to learning objectives.
The work of (Raissi et al. 2019 demonstrated the feasibility of directly incorporating physical
information into the objective function, including initial conditions and necessary physical residuals.
Other examples of such approaches include [Jeon et al. 2024 Haghighat et al. 2021 Yu et al. 2022

---

While these techniques embed physical knowledge into the objective function, they lack generality, as the obtained parameters tend to be case- aase-specific. The broader underlying physics is not fully captured. and only the physical symmetries specific to a particular case are addressed.
Another class of methods is neural operators [L et al. 2020b] These methods generalize well across the PDEs they are trained on and are not case-specific. However, these models require data to be represented in higher dimensions to capture complex relationships. The application of an integral kernel in a high-dimensional representation of complex geometries with dense data points poses a significant computational challenge (Li al., 2020a]. While neural operators can employ different architectures to reduce rollout error, they lack explicit mechanisms for integrating temporal data, unlike transformers, which handle sequence dependencies robustly through their attention mechanisms. This limitation can affect their effectiveness in applications that require sequential data processing.
notable trend in recent years involves the use of encoder-decoder pairs to process dynamical states in latent space. The work of [Wiewel et al., 2019] an early example of this approach. In general, the input must first be encoded into the latent space while preserving the context. A sequential network, such LSTM, GRU, other variants, is then trained on the encoded data. Thus, these approaches typically require two components: an encoder-decoder pair and a temporal model. Another example of this approach is Mesh Graph Networks (MGN) [Pfaff al. 2020] and Graph-Network-biit Simulators (GNS) [Sanchez- Gonzzlz et al. 2020]. The encoder modules of these models are based on graph networks. After processing nodes, edges, and features, they use a simple multi-layer perceptron (MLP) to compute the derivatives of the features over time and update the state using forward-Fule scheme. While the encoder-decoder pair is powerful tool, especially for unstructured mesh spaces, the lack of a robust time-stepping model significantly limits performance, with rollout error becoming a dominant issue.
Current state-of-the-art models that demonstrate optimal performance on baseline datasets typically combine a graph network encoder with more advanced time-stepping algorithms. Notably, the study by (Han et al. [0222] employs Graph Mesh Reducer (GMR) for encoding, along with sequential time-stepping network and a Graph Mesh Up- Samplin (GMUS) decoder. This research explored various sequential models, including LSTM, GRU, and Transformers, with the latter achieving the lowest rollout error. Building on these principles, [Sun et al. 2023) introduced a modified version of GMR-GMUS adding RealNVP normalizing flow model to the timesttepiig transformer. While adding a normalizing flow model does not yield a fully tractable model, allows for the direct maximization of log- likeehooo over the final data point, improving the overall objective and resulting in the most competitive baseline model reported thus far. However, these models still cannot fully address the rollout error in systematic way that incorporates our physical understanding of the system.
As result, improving rollout errors and reducing training time remain sigmificant challenges in this field. In this work, we demonstrate that both objectives can be addressed by separating the fields and allowing them to learn the inherent physical relationships and symmetries. To this end, we introduce the SEA module, implemented on top of a Vision Transformer (ViT) based mesh autoencoder.
3 Methodology
3.1 Problem statement
Temporal generation of the states in a dynamical system from an initial condition is analogous to video generation tasks, where the process is conditioned on both the initial state and external input. Similarly, the evolution of a dynamical system can be conditioned on known parameters, such as the Reynolds number in fluid flow and the system's initial condition. Given this similarity to autoregressive generative models like ART-V [Weng et al. 2024), we formulate the temporal evolution as autoregressive generation of states, conditioned on both the initial state and time-invariant parameter.
However, autoregression on the mesh space is challenging due to the large size of the elements and the number of variables stored on each element. Hence the mesh must be embedded into manageable embedding, and our formulation becomes autorcgressive sequence generation in latent space.
3

---

If we denote the fields with index i, and time with index t, then the stored data on field at time t is presented by X Given the proposed framework with the SEA module, we intend to encode the field groups separately. The field group refers to fields with the same dimensions (e.g., velocity in the x- and y-directions belong to the same group, while pressure lies the partition of fields into groups based on their dimensions (e.g.. velocity, pressure). Each group is then encoded separately with the mesh encoder &, resulting in an encoded representation z at time The final encoded representation time is then given by:
separate group). Let 9 represent
z, (Concat (ooncat((X()ara ((X))eG))), VG& E g
We now denote z as the encoded representation of the group GR. where j indexes the group embedding space. This embedding is then used to formulate the autoregressive sequence generation in time where the initial condition Zi and the time invariant parameter O (e.g.. Reynolds number) are always available and used to condition the generation. We can formally express this problem as arg maxz; P(Z Zy θ) where we aim to find the sequence of latent variables Z that
maximizes the conditional probability given the initial condition Z and the time-invariant parameter O. This is achieved through a pointwise estimation of the conditional probabilities in the continuous embedding space, with optimization performed by minimizing the L2 loss. Further detail objectives and training is provided in Appendix
3.2 VIT mesh autoencoder
The backbones commonly used to create embedding spaces in image and video models, such as Latent Diffusion Models (LDM) [Rombach et al., [022] and Video Diffusion Models (VDM) (Ho) et al. 2022]. cannot be directly applied to mesh data due to their inherent structured pixel inductive biases. Therefore, our proposed autoencoder must specifically overcome these biases for unstructured mesh space. Given that the temporal model employs transformer backbone, the embedding must generate tokens compatible with the transformer's 's input requirements. These tokens are generated similarly to that of ViT (Dosovitskiy al. 2020 Following the approach used in ViT, the space is divided into multiple patches, with each patch containing a number of cells. Let Ω denote the domain in which our study is conducted, with dimension d. Assume that the domain is discretized into set of nodes {æ; N where each node z; represents a point in Rd. To construct patches, we define a partitioning function fP Rd > f1, MM. which assigns each node v; to one of M patches based on its coordinates.
Assuming the boundaries between patches are showing with B {bi bm then the function fr is defined as follows:
1
if x4 < b1, if by I <b for 2 2 j S m,
Sp(xz)
j
m + 1
if x4 2 bm
To address the challenge of irregular and unstructured meshes, which lead to varying numbers of nodes per patch, each patch is padded to align with the length of the largest patch. A padding value of 0 is used throughout the framework. Moreover, bias terms are excluded in the embedding process, and the Gaussian Error Linear Unit (GELU) activation function [Hendrycks and Gimpel, 2016) is applied to ensure that the padded elements do not influence the spatial encoding. To achieve spatially aware embeddings and coherent reconstructions. we apply a multi-head self-attention mechanism (MHSA). This padding and embedding strategy is illustrated in Figure []]
The output generated by the Vision Transformer (ViT) embedding module is subsequently flattened and utilized as tokens within the temporal model. The complete process is thoroughly explained in appendix[A]
3.3 Temporal and State-Exchange Attention model
The Stahe-Exchange Attention module integrated into the temporal model in this work to enhance the autoregressive generation of sequences in time. The temporal model utilizes transformer

---

Mesh Embedding
MHSA
Layer-Norm
MLP (On patch)
Figure I: The ViT-based mesh autoencoder divides the domain into patches and pads them to ensure equal sizes. An MLP is then applied to reduce the dimensionalityy and the MHSA mechanism provides global awareness create spatially coherent reconstructions with minimal patch artifacts. These patches are subsequently flattened and adapted as tokens for the temporal model.
architecture to capture the temporal dependencies of the state variables. This transformer includes the adaptive layer-norm by [Peebles and Xie 2023) and the rotational positional embedding (RoPE) as developed in [Su et al. [0224 and adapted by state of the art autoregressive image generation models [Lu et al. 2024 Sun al. 2024). The adaptive layer-norm employed in this work is modified to take the continuous time invariant parameters as input and act as a secondary conditioning mechanism on the temporal model. Full detail of the temporal model is provided in appendix B
This work initializes decoder block for each state variable in the given PDE For instance, the Navier-Stokes equations governing the fluid dynamics, requires the resolution of variables such as velocity and pressure each of these are assigned an expert decoder. The SEA module is designed to allow the exchange of information amongst these experts with cross attention. We further investigate other modes information exchange in appendix D
The flow of information through the expert layers and the SEA module can be formulated by building on the well-known attention mechanism. For clarity, the terms regarding the positional embedding are omitted here. We start from the encoded groups of variables presented in section 3.1 denoted by
Z/_ To simplify the notation and remove the explicit dependence on i, we represent the sequence of encoded variables across all time steps as a single matrix, Zz, which stacks the encodings all time steps. The attention mechanism then reads:
Q(Z') Q(Z))K(ZA)) V(Z))
Attention(Z') softmax
(I)
dde
Where KZZ)), Q(Z²), and (ZZ) represent the key, query, and value matrices, and are obtained from the linear transformation of the input Z by set of learnable weights WK. Wo. and Wy.
Additionally de represents the model dimension.
Furthermore the adaptive layer-norm is demonstrated by 'AdaLNN' and hence following a pre-norm convention the multihead self-attention (MHSA) becomes:
Z SA
Z + Attention (AdaLN(ZZ)). Vj E
(2)
Given the autoregressive task at hand all the attention mechanisms including the presented MHSA have causal mask to improve autoregressive generation.
The field information flows through the information exchange module after the temporal self attention. This information exchange module is represented by the state-exchange attention in Figure 2 where we allow the state variables to exchange relevant information with a causal cross attention mechanism. This is formulated as:
5

---

Concallansle Flelds
Adel
AdwlN
GELU
MLP
MLP
project
AdaLN
AdaLN
<D+
TIP1
(b) Time invariant parameter injection
(TiPI)
SEA
PD
Up projection
Up projection
Masked
Multi
Head
Balf-Altention
Sett-Altertion
Fiwid to Fleid Attention
Pleid to Field Attention
k KAAAA
T.AAA
AdaLN
AdaLN H
Jown projection
Down projection
Flow j-1
Flow of 2-2
AdaLN
AdaLN
(a) architecture schematic
(c) Eaile-xchange Attention
Figure 2: (a) State- Eaath-Exchange Attention (SEA) integrated Transformer model architecture, incorpo- rating the additional modules of SEA and Time Invariant Parameter Injection (TIPI). Dashed lines represent the inclusion of additional fields. (b) Representation of the TIPI, designed to incorporate time-invariant information after the SEA module. (c) Schematic of the SEA module, illustrating how fields communicate through this module.
(Z' (Z')SFA- )SEa (Z')
fSEA (AdaLN )SN), Js))) VjEg
kEO
Here, (.)) represents our information exchange mechanism SEA. This module takes in two arguments: the first the adaptive layer norm of the expert for which the attention is taking place, and the second is the adaptive layer norm from the expert to which the module is attending. These are represented by AdaLN(ZZ) and AdaLN(Z*)) )), where and j are non-equal embedding field indices.
To enhance efficiency during information exchange, we adopt bottleneck mechanism inspired by expert-based architectures such as [Lee et al. 2024]. Introducing a bottleneck this stage helps keep the model scalable by enabling selective information exchange in lower-dimensions space. Another instance of such strategy is the Perceiver architecture [Jaegle al.) [021]) where cross-attention creates a bottleneck for high-dimensional data from different modalities. In our case, we employ simpler method, using droj-projection with learnable parameters, following the approach [Lee] et al. 2024]. If the down- and up-projection matrices for mapping to the bottleneck are denoted by W and W respectively, the State- ExchanEx Attention mechanism is fully described by Equation
4 The arguments to this function are the AdaLN of (Z') )Sa and (Z*) SsA shown in Equation 3
However, for clarity in illustrating the equation, we use A and A to represent the input arguments here.
QIWAA)) Vd
SSEA (A = W softmax
v(WSA*)
(4)
In the presented equation KWW ((W) and V(W) represent the key, query and value matrices obtained form the down projection of the inputs.

---

The conditioning of the generations on external parameters such as the Reynolds number is done using an indirect method of adaptive layer norm and direct method of time invariant parameter injection module (TIPI). These components replace the more computationally intensive attention mechanism commonly used in physics domain autoregressive models. The TIPI component processes the time- invariant parameters using a multilayer perceptron (MLP) with GELU activation function. The MLP maps the time-invariant parameters @ to the model's embedding dimension through learnable parameters. As shown in Equation 5 the information injector, represented by TIPI(, -] injects these processed parameters into the current state by summing the model's embedded information with upscaled time-invariant parameters.
( Ourput (Z. )SEA MLP (AdaLN (TIPI [ZZ SEAA e)))
(5)
Figure 2 illustrates the architecture's detailed schematic.
4 Experiments
4.1 Procedure
In this section, we evaluate the proposed model on two benchmark datasets and compare its perfor- mance with other frameworks. First, the complete model is tested on the cylinder flow benchmark, a widely used dataset in computational fluid dynamics. This evaluation includes the comparison of the full model with recent physics domain autoregressive models.
We then explore multiphase case, where the model is tasked to resolve the interface by taking into account the fluxes caused by the velocities using the SEA module. this section, only the temporal aspects of the model are varied (SEA module), while the ViT mesh autoencoder is fixed to isolate and eliminate the impact of encoding method on model's performance. To this end, separate decoders are assigned, one for velocity and another for volume fraction, similar structure depicted 2 The inclusion of the volume fraction allows us to study the extent of the model's 's capability to resolve interfaces and the effect of State-Exchange Attention on capturing the multiphase scenarios.
For consistent comparison with state-of-the-ann models [Sun et al. 2023 Han et al., 2022]. relative mean squared eтror is used to quantify the errors. The model trained an A100 GPU for approximately 2 hours for both datasets. Furthermore, a consistent Transformer architecture was adopted in both cases, utilizing 1 layer and 8 attention heads. The embedding dimension of the model for the cylinder flow case was set to 1024, in line with the literature [Sun et al.] 2023), while dimension of 2048 used for the multiphase flow case effectively capture the interface. Full details of the configurations and datasets are provided in Appendix G and respectively.
4.2 Evaluation of VIT mesh autoencoder
The autoencoder used in the following experiments trained exclusively with a reconstruction objective. The complete training procedure for this model is detailed in Appendix A Reconstruction errors, compared recent graph-bar based autoencoders, are presented in Table I
Table I: Relative reconstruction error for cylinder flow and multiphase flow reported in the unit of 1 x 10-3
Dataset
Ours (ViT based autoencoder)
GMR-GMUS
PGGMR-GMUS
Cylinder flow Multiphase flow
1.7
14.3
1.9
6.3
4.3 Evaluation of temporal model on general case
We assess complete architecture using the 2D cylinder flow, a benchmark dataset employed by other leading baseline models. In this case, the Navier-Stokes equation governs the motion of the fluid throughout the domain. Consequently, the trajectory to track is the velocity and pressure. The
7

---

mesh was initially tokenized at all time steps based on the explained ViT mesh encoder. These tokens were then fed through the SEA integrated model illustrated Figure During the training. the entire sequence was fed to the network for each batch. During inference, the model was set to estimate the trajectory autoregressively, and hence, the test errors correspond to the total rollout error. For the cylinder flow dataset, the error was evaluated over the case with Reynold's number of 400. This case was chosen to keep consistent with the other competitive models.
The recorded rollout error, and its comparison with other baseline models is illustrated in Figure
SEA
10P
buSk-C tranafarraT Crandtarmer
10'-
MON
MCN-M
Taroformes Roo
50
100
1.50
200
250
300
352
Tane Stop
Stop
(a) Cylinder flow rollout error comparison
(b) Field-wise error compared to basic model
Figure Comparison of the rollout error for the cylinder flow dataset.
In Figure 3a show that model outperforms the established state-of-the-art models across the board. Specifically, our results indicate an average improvement of 88% and 91% in reducing the error in autoregressive sequence generation compared to the PbGMR-GMUS Transformer- Meefoorner-RalNVVP and GMR-GMUS Transformer architectures, respectively. Furthermore, our model outperforms both variants of MGN architecture with 99% and 98% improvement over the base MGN model and MGN-NI, respectively. Full detail of the errors are presented Table
Table 2: Time average rollout eтrоr of the presented model compared to other competitive models. The presented error after decoding and corresponds the real field error. The reported values are in the unit of x 10
Models
p
Avg
MGN (Pfaff et 2020
98
2036
673
935.6
MGN-NI Plaff et al. 2020
25
778
136
313
EMR-GMUS Transfommer Han
2022 BOGMR-GMUS Transformer-Reall Sun et al. 2023
4.9
89
38
43.96
3.8
74
20
32.6
Ours (Full ViT-SEA Transformer)
0.35
10.7
0.3
3.7
The evaluated test cases were post- post-processed to generate contour map for further observation of the learned patterns and potential areas of error. For consistency with the work in literature (Han) al. 2022 Sun et al.) 2023 the contour map of the case with Reynolds number of 400 is presented here. fully explore our model's ability to capture complex flow features, such as the downstream vortex, we present visualizations the 250th timestep, point at which the Von Karman vortex street fully developed, as shown in Figure 4 This timestep was deliberately chosen to visualize complex dynamics that are often challenging for traditional models to capture.
4.4 Evaluation of temporal model on multi-phase
Evaluation the presented model is extended include a multiphase flow scenario, which comple- ments the analysis presented in Section 4.3 The evaluated test case in this experiment corresponds to an immiscible collapse of two blocks of liquid due density differential. flows, critical aspect is the precise identification of fluid interfaces. This is accomplished by using a volume fraction state variable, denoted by a, which indicates the region occupied by the fluid. The value of ranges from 0 in one fluid to I in the other, effectively distinguishing between the
studies of multiphase

---

SEPER 04200 DETEC
Prediction,
Prediction,
Prediction, P
Ground truth,
Ground truth,
Ground truth,
Contour maps the generated fields at Re=400, and time step of 250 where Von Karman vortex street is formed.
Figure 4:
two phases. To capture the phase, an additional block is assigned to the volume fraction which can communicate with other fields (Velocity in this case). Given the minor variations of pressure, we discard this variable here and only focus on the importance of the field communication between velocities and volume fraction through SEA module.
We investigate three possible variations of the Transformers to achieve this. First, we estimate the rollout егrоr of the model with the SEA module. Second, we evaluate a basic model that encodes the fields into different latent spaces with mode of information exchange. Finally, a model that encodes all fields together into a single latent space, referred to as the Field Fusion Encoder (FFE) Transformer. This latter model corresponds the Transformer architecture used and EMR-GMUS Transformer, as indicated in the
assess
in the P6GMR-GMUL TransformerRReall provided results. The rollout error of these models are presented in Figure
10"
10
10-11
22-1
Rasic-Iransformer
ViT Basic Transformer
FFE-Trasformmr
FRE-Transformer
100
378
320
100
115
Time
(a) Average over fields
(b)
12a
Besic-Transformer
Taass--Trnnfrrmmrr
100
125
175
200
100
L00
150
175
200
(c)
(d)
Figure 5: Comparison between the Transformer with SEA module and other variations of Transform- used in the literature over the multi-phase dataset.
From the comparative results presented in Figure it evident that the models with SEA module outperform the other variations. Most of the error observed in the average error plot in Figure corresponds to the error in the volume fraction. The mean volume fraction errors over all time steps are 0.1 0.25, and 4.61 for the SEA-integrated Transformer, basic Transformer, and FFE Transformer,

---

Prediction, ts=5
Prediction, ts=10
Prediction, ts=20
Ground truth, ts=5
Ground truth, ts=10
Ground truth, ts=20
Figure 6: Comparison of the contour maps of the volume fraction (a), between the predictions and ground truth in the case of p 850
respectively. This represents approximately 52% and 97% reduction in error with the integration of the SEA module, compared to the basic and FFE Transformers. Additionally, improvements of 48.5% and 40% are observed in the averaged velocity components.
Further demonstration of the contour maps is provided in F where the actual interface tracking capability of the SEA enhanced transformer be observed. Further visual results on the velocities are provided in Appendix H
Discussion
The presented ViT-based mesh autoencoder and State- Exchangx Attention (SEA) integrated Trans- former module evaluated through two different experiments computational fluid dynamics (CFD) problems. A significant improvement relative mean squared error was observed when the SEA module deployed. During the cylinder flow evaluation, the full ViT-SEA integrated transformer framework achieved over 80% improvement compared to all competitive baselines. This improvement was accompanied by lower gradient in the error, demonstrating a form of self- correction through information exchange between the velocity and pressure fields. The isolated SEA module was then tested multiphase case, resulting in 97% improvement the volume fraction compared to the field fusion encoder transformer, where the entire field is encoded into the same latent space. In the multiphase case, it was evident that the velocities exhibited marginal error difference; approximately 40-50%; however, the volume fraction dominated the overall improvement. This is due to the significance velocity in the displacement the interface, whereas the volume fraction did not provide any valuable information to the velocity field. The error reduction observed with the deployment of the SEA module suggests that SEA module provides the necessary tools for the underlying physics of the governing to be captured.
The presented module, however, may face challenges when scaling equations involving large number of state variables. For instance, in multiphase flows with more than two phases or in grain growth within materials where each grain is represented by state variable, the model would require a corresponding number of transformers to operate in parallel. This could lead to inefficiencies as the number of variables increases significantly.
6 Conclusion
The presented work introduces SEA, novel module that enables the state variables of a physical system to exchange information within the transformer architecture. Transformers integrated with SEA demonstrated state-of- other transformer-basee of-the--rr aht performance, surpassing previously established benchmarks by models with improved autoregressive rollout error. The improved rollout emror an indication that the SEA module is enabling the model learn the underlying physics. In future works, will explore the scaling complexity of SEA to handle more state variables and its application other domains. Furthermore, the integration SEA with probabilistic models, such as diffusion models, will be investigated.
10

---

References
Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vi- sion transformer for image classification. In Proceedings the IEEE/CVF international conference on computer vision, pages 357-366, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1199 11929, 2020. doi: https//oo..ay///4455555xxx...1..
Parsa Esmati, Thomas Flint, Fatma Akyel, Simon Olschok, Uwe Reisgen, Philip Cardiff, Nicolas O. Larrosa, and Nicolò Grilli. Simulating chemical mixing and molten pool shape in dissimilar welds using thermal fluid dynamics. International Journal of Heat and Mass Transfer, 226:12544,, 125449, 125444 URL
2024. ISSN 0017-9310.
doi:
N. Grilli, K.G.F. Janssens, J. Nellessen, S. Sandlöbes, and D. Raabe. Multiple slip dislocation
patterning in a dislocation- based crystal plasticity finite element method. International Journal of Plasticity, 100:104-12) 121, 2018. ISSN 0749-6419. doi: https://oooarg///00000////iiippp 15......
URL https: /wtw://wwww..o www.seceeeeeiccieeeoeocceeeeeeee com/ wttteetccooooocceeeeeeeeeeeeeeee http:///i/////ttt//////////////0
Nicolò Grilli, Philip Earp, Alan C.F. Cocks, James Marrow, and Edmund Tarleton. Characterisation of slip and twin activity using digital image correlation and crystal plasticity finite element simulation: Application to orthorhombic a-uranium. Journal of the Mechanics and Physics of Solids, 135:103800, 2020. ISSN 0022-5096. doi: https://doi..or/110.10000 h000 URL https:://///. //www. www.escieencce encedirect..........c.....cce..e com/
Ehsan Haghighat, Maziar Raissi, Adrian Moure, Hector Gomez, and Ruben Juanes. A physics- informed deep learning framework for inversion and surrogate modeling in solid mechanics.
Computer Methods in Applied Mechanics and Engineering, 379:113741, 2021. ISSN 0045-
7825. doi: http::///doi.org10.1110000. 201.:/6667......211111111.44114 URL URLhttps::/ http:///wwww.iinnciiire....... com/ httpa///aaaiiiieee///////5//////
Jiaqi Han, Minkai Xu, Aaron Lou, Haotian Ye, and Stefano Ermon. Geometric trajectory diffusion models. arXiv preprint arXiv:2410.13007,, 2024.
Xu Han, Han Gao. Tobias Pfaff, Jian-Xun Wang, and Li- Li-Ping Liu. Predicting physics in mesh-reduced space with temporal attention. arXiv preprint arXiv:2201.091 2022.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint
arXiv:1606.084155 011660600841,,.. 2016.
Jan Felix Heyse, Aashwin A Mishra, and Gianluca Iaccarino. Estimating rans model uncertainty using machine learning. Journal of the Global Power and Propulsion Society, 2021 (021(May::1114 2021a.
Jan Felix Heyse, Aashwin Ananda Mishra, and Gianluca laccarino. Data driven physics constrained perturbations for turbulence model uncertainty estimation. In AAAI Spring Symposium: MLPS, 2021b.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633-8666 8646, 2022.
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. International conference on machine learning, pages 4651-4664 PMLR, 2021.
Joongoo Jeon, Juhyeong Lee, Ricardo Vinuesa, and Sung Joong Kim. Residual-basee physics- informed transfer learning: A hybrid method for accelerating long-term cfd simulations via deep learning. Internutional Journal of Heal and Mass Trunsfer, 220:124900, 2024. ISSN 0017-9310. doi: http:////oooooggl....////aaaaaaa
124900. URIttttps::/ //ww:

---

Jundou Jiang. Guanxiong Li, Yi Jiang, Laiping Zhang, and Xiaogang Deng. Transefd: A transformer- based decoder for flow field prediction. Engineering Applications of Artificcal Intelligence, 123: 106340, 2023. ISSN 0952-1976. doi: https:/doo.oogglloo000fiiiggaapp 106340. URL
Dmitrii Kochkov, Jamie A Smith, Ayya Alieva, Qing Wang. Michael P Brenner, and Stephan Hoyer. Machine learninn-acceleraann computational fluid dynamics. Proceedings of the National Academy of Sciences, 118(2))=e101784411 18, 2021.
Dongho Lee, Jongseo Lee, and Jinwoo Choi. Cast: Cross-attention in space and time for video action recognition. Advances in Neural Information Processing Systems, 36, 2024.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895 2020a. doi: https://doi.org: 10///10...0 wwwpo//....5//an/..i.i. (0.444050/2xi..110...5555
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Andrew Stuart, Kaushik Bhattacharya, and Anima Anandkumar. Multipole graph neural operator for parametric partial differential equations. Advances in Neural Information Processing Systems, 33:6755-6766, 2020b.
Hezheng Lin, Xing Cheng, Xiangyu Wu, and Dong Shen. Cat: Cross attention in vision transformer. In 2022 IEEE international conference on mullimedia and expo (ICME), pages 1-6. IEEE, 2022.
Xin- Yang Liu, Min Zhu, Lu Lu, Hao Sun, and Jian-Xun Wang. Multi-resolution partial differential equations preserved learning framework for spatiotemporal dynamics. Communications Physics, 7 (1):31, 2024.
Jiasen Lu. Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages (66439226433 June 2024.
F. Moukalled, L. Mangani, and M. Darwish. The Finite Volume Method, pages 103-135. Springer In- ternational Publishing, Cham, 2016. ISBN 678-4-619-16874 doi: 10.1007/978-3-319-166444 5. URL https: www.//...//........ 10. 1007 000111007777111 978 9117------1---174--665-
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages $195-4205, 2023.
Tobias Pfaff, Rahul Narain, Juan Miguel De Joya, and James F O'Brien. Adaptive tearing and cracking of thin sheets. ACM Transactions Graphics (TOG), 33(4):1-9, 2014.
Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonnalezz, and Peter W Battaglia. Learning mesh- based simulation with graph networks. arXiv preprint arXiv:20/0.03-00) 2020.
M. Raissi, P. Perdikaris, and G.E. Karniadakis. Physics-informed neural networks: A deep learning
framework for solving forward and inverse problems involving nonlinear partial differential
equations. Journal of Computational Physics, 378:686-707, 2019. ISSN 0021-9991. doi:
http:://tto.or///1111166//../.p/ 10.045. (ttt.sttts::///ee http://www.iiineeereetttt. wwt.ccccccccceeeee com/sciience. Brticcle/pi1/S00219999333 att.ciee/pi//500119911188111
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pages 10684-10695, 2022.
Michael Salvini, Nicolò Grilli, Eralp Demir, Siqi He, Tomas Martin, Peter Flewitt, Mahmoud Mostafavi, Christopher Truman, and David Knowles. Effect of grain boundary misorientation and carbide precipitation on damage initiation: A coupled crystal plasticity and phase field damage study. International Journal of Plasticity, 103854,,,,, 2024. ISSN 0779-6419. doi: https:
//tb..or//10.0016...ipps..... 103854. URI. httpt::t/////wtt http///www.ciinceeeret..t. www.s/eeettttccccocoeeeeeeeee com/science, artziclee/pi///77444441222200000
12

---

Alvaro Sanzhez-Gonzalez. Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning simulate complex physics with graph networks. In International conference on machine learning, pages 8459 8468. PMLR, 2020.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Luning Sun, Xu Han, Han Gao, Jian-Xun Wang, and Liping Liu. Unifying predictions of deter- ministic and stochastic physics in mesh-reduced space with sequential flow generative model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 60636-60661 Curran Associates, Inc., 2023. URL htttp:://proceediini www. hewriip.cccppperr paper.pppp...... wtllle//paperr222////aa5 httpaaaaaaaacaal00220aaaaaaaeaaa
pdf
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525, 2024.
Nils Thuerey, Konstantin Weißenow, Lukas Prantl, and Xiangyu Hu. Deep learning methods for reynolds-averaged navier-stokes simulations of airfoil flows. AIAA Journal, 58(1):25-36, 2020. doi: https:/dcoiorgg10.55441....15331
Kiwon Um, Xiangyu Hu, and Nils Thuerey. Liquid splash modeling with neural networks. In Computer Graphics Forum, volume 37, pages 171-182. Wiley Online Library, 2018.
Mario Lino Valencia, Nils Thuerey, and Tobias Pfaff. Se (3)-cquivariant diffusion graph nets: Synthesizing flow fields by denoising invariant latents on graphs. In ICML 2024 for Science Workshop, 2024.
Wenming Weng, Ruoyu Feng, Yanhui Wang, Qi Dai, Chunyu Wang, Dacheng Yin, Zhiyuan Zhao, Kai Qiu, Jianmin Bao, Yuhui Yuan, et al. Art-v: Auto-regressive text-to-video generation with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7395-7405, 2024.
Steffen Wiewel, Moritz Becher, and Nils Thuerey. Latent space physics: Towards learning the temporal evolution of fluid flow. In Computer graphics forum, volume 38, pages 71-82. Wiley Online Library, 2019. doi: htps://doi.org/10..1 11/cgf.13620
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in neural information processing systems, 27, 2014. doi: https://doi.orr/11...o 4ttp//0000555///iii.ia 14.11 141..
Mustafa Z Yousif, Linqi Yu, and HeeChang Lim. Physics-guided deep learning for generating turbulent inflow conditions. Journal of Fluid Mechanics, 936:A21, 2022.
Jeremy Yu, Lu Lu, Xuhui Meng, and George Em Karniadakis. informed neural networks for forward and inverse pde problems. Computer Methods in Ap- plied Mechanics and Engineering, 393:114823, 2022. ISSN 0045-7825. doi: https://doi.org//
Gradient-enhanced physics-
1016/j.cma.2022.1112 114233 URL httpst//:://aet wwwp://wwwwwwwwwwwwctctccmm wwtpe//eetttccccciiieeeee http://reer..ooo/ccccc/////tttta hwtp//iiiiineaaice//ppp..... $0045782522001435
Leo Zhiyuan Zhao, Xueying Ding, and B Aditya Prakash. Pinnsformer: A transformer-based framework for physics-informed neural networks. arXiv preprint arXiv:2307.1188 2023.
13

---

Appendix
A
VIT autoencoder for Mesh detail
Embedding mesh data presents a greater challenge compared to images, where a structured pixel grid allows for the natural application of convolutional kernels. The unstructured nature of meshes requires novel approaches for embedding, such as graph neural networks [Pfaff et af., [020], J or systematic methods for padding within the kernels. However, determining the appropriate padding and their position in the convolution is not a simple task.
VIT models, originally developed for image classification, partition images into patches and apply linear projection each patch. Inspired by this structure, we developed ViT-like mesh autoencoder, where patches are generated on the mesh, and each patch is encoded individually. In this approach, the number of cells per patch becomes readily available, allowing all patches be padded to the length of the patch with the maximum number of cells. To enhance the processing capability during the embedding stage, we replace the original patch projection in the ViT with an MLP layer. The self-attention block applied after this step further refines the embedding, creating spatial coherence by allowing each patch to capture the global context in addition to the local context encoded by the MLP.
D
Padded Patches
Patch
Patcña
Patchas
Stack
MHSA
P
Potched
Emid
Figure 7: Presentation of the ViT mesh autoencoder, where a number of patches are created on the mesh and an embedding space created for each patch using the encoder E. The multihead self attention is then applied to create global awareness. The decoder D simply maps the embedded patches to the original space.
The following list outlines the evolution of the data structure as it passes through the ViT-based mesh autoencoder. Here, B denotes the batch size, T represents the number of time steps, C is the total number of mesh cells, F refers to the number of fields (analogous to channels in image or video contexts), P is the number of patches, Cp is the number of cells per patch, and D is the embedding dimension. In this study, batches are derived from the generated trajectories, as described in Section
Input: F]
Patchify (Patchedy: [B.TCC.FI
Permute: (B,T,P,C (B.T,,,CCCC
Collapse batch with time and field with cell: [B=T,P,F,C, [p] [B Cp) P.D. Global awareness with MHSA (Z): TT + [B
Encode with MLP (Embd): [B T, P. F Cp]
Decode with MLP (P): [B + T.P. C₁]
The presented procedure is done for fields within each field group separately. The field groups refer the groups of fields with the same physical dimension as formalised in Section 3.1
For completeness, we also provide variational version of the encoder the repository, incorporating weighted KL divergence penalty, following the approach in [Rombach et al. 2022]. However, in our
14

---

projection 2)
Up projection (factor
Up projection (fector
Up projection (factor
Field to Attention
Attention
AgAAA
k
KAGAvA
Down projection (factor 2)
Down projection (factor
Down projection (fachor 2)
Down projection (factor 2)
(a) Attention information exchange
(b) Addition information exchange
Figure 8: The information exchange mechanisms evaluated for this work, these include the informa- tion exchange with (a) cross-attention and (b) addition.
experiments, the variational encoding does lead to significant improvements, and the embedded space error follows the real error (Refer to Appendix D). Therefore, we choose to proceed with pointwise encoding, consistently results in better reconstruction accuracy in the studied cases.
The hyperparameters associated with this autoencoder include the number of patches, the MLP dimensions and architecture, the number of heads in the attention mechanism, the number of attention layers, and the embedding dimension. The attention block is configured similarly to the original VIT, with exact specifications provided Appendix G The number of patches is primarily determined by the number of cells and the mesh dimensions. In this work, we use a constant number of 64 patches, arranged as 8 patches along the x-direction and patches along the y-direction. Consequently, the number of cells per patch depends on both the total number of cells and the mesh density. Table summarizes this information.
Table 3: Patch and cell information after padding for the cylinder flow and the multiphase flow experiments.
Case
patch in x patch in
Cp
Cylinder flow
8 8
232
16
Multiphase flow
156
32
B Temporal transformer detail
The transformer used to capture temporal dependencies is designed with a decoder-only architecture, incorporating the SEA and TIPI modules. In this work, develop and evaluate two variations of information exchange methods, namely exchange by addition and exchange via cross-attention, shown in Figure Both methods operate within information bottleneck, where the data is nown-projected to retain only the relevant information required for exchange. To ensure improved gradient flow and prevent information loss due to this bottleneck, we introduce residual connection from the output of the multi-head self-attention (MHSA) to the flow outside the SEA module. The down-projection within the bottleneck is
factor of 2, following the approach in [Lee al.) 2024 The architecture referred to as the basic transformer does not incorporate any connections or information exchange between the decoders assigned to different variables.
Additionally, the TIPI module is incorporated as a direct conditioning mechanism, while AdaLN serves an indirect conditioning technique. In the final design, the primary TIPI module utilizes the addition-basee approach, as shown in Figure 9 However, we also explored injection via attention, which is commonly used conditioning technique in physics-based based models.
The adaptive layer normalization (AdaLN) technique, used as the indirect conditioning method, is adopted from [Peebles and Xiel 2023 In this approach, conditioning is applied by replacing the standard layer normalization with scale and shift parameters derived from the time-invariant parameter e. To remain consistent with [Peebles and Xie 2023
MLP to compute the
15

---

Information
Linear (up)
Linee (up)
(A) TIPI with addition
(b) TIPI with attention
Figure 9: The information exchange mechanisms evaluated for this work, these include the informa- tion exchange with cross-altention and (b) addition.
scale and shift values. First, is up-projected to the model dimension, followed by the application of the SiLU activation function. A subsequent layer is then used to obtain the final scale and shift parameters.
An schematic of the AdaLN block is presented in Figure 10
7
Linear
SILU
Scale and Shift
Linear (up)
Layer Norm
Information
Figure 10: The block of adaptive layer norm adopted in the temporal model.
For the temporal model, the data is first structured similarly to the section A After the encoding process, we obtain a data structure of [B*T. D] for each field group. To process different trajectories separately and avoid mixing, we separate the time from the batch and collapse the patch in the encodings. Hence. the final shape processed by the temporal model [B. T. P*D], where B represents different trajectories, which we batch from and learn the temporal evolution.
C Training procedures
Autoencoder:
The autoencoder trained wtih the objective of the reconstruction error with L2
loss:
- C=
X x
(6)
Where X represents the model's output, and the expectation is done with respect to the data drawn from the batch.
For consistent comparison with the literature we provide the errors using relative mean squared error where we refer to this as relative reconstruction error, and relative rollout error for the autocncoder
16

---

and temporal models respectively. The relative rollout MSE is formulated as:
X
X,
ReIMSE = EXxD
Here the summation done on the cell dimension.
Temporal model: The temporal model is trained using L2 loss in teacher forcing manner, where the model generates the encoded field for the subsequent time step after observing the full trajectory. To accurately assess the autoregressive performance of the model. evaluation every 250 epochs, allowing us to track the learning history for the required autoregressive task. This evaluation includes monitoring both the encoded and decoded errors. We observe that, although the magnitude of variations in error is not perfectly aligned, the trends in the embedding space and decoded space errors follow similar trajectory, peaking simultaneously. The following figure provides an example of the autoregressive learning curves for both the embedded space and the decoded space, using the VIT mesh autoencoder described in earlier sections.
conduct an autoregressive
Encoded
Encoded
Decodes
Decoded
10"
101
350
1000
1500
2000
2500
3003
300
1000
1350
2000
2530
3000
3000
(a) Cylinder flow validation.
(b) Multiphase flow validation.
Figure 11: Comparison of the autoregressive evaluation in embedding space and after decoding where we observe they follow the same trend.
Provided curves in Figure 11 demonstrate the dynamic of the joint ViT-autoencoder and the temporal model. Informed by these curves, non-variational encoding approach is adopted, and we assign weights the KL term used for the regularization of the latent space.
D Ablation study
Ablation of the information exchange method is performed using vanilla architecture. The ar- chitecture incorporates layer normalization, a common practice in transformer models within the physics domain. We also initialize with absolute positional embeddings. Following exploration of the vanilla transformer with the information exchange module, we extend the ablation studies to include the components of the main architecture. Specifically, show that adding TIPI and AdaLN as conditioning mechanisms further reduces error and eliminates the need for an explicit attention mechanism for conditioning.
Table 4: Effect of different information exchange modes and no explicit exchange mode on the ReIMSE. Reported errors are multiplied by 1
mode
CF (u)
CF (v)
CF (p)
MP (u)
MP (v)
MP (a)
Avg
No exchange
626
15.3
1043
1352
136.8
530.5
Mixture- 16
8.4
550.8
10.1
Mixture-32 32
8.6
579.2
10.7
984.3
1235.8
258.9
512.9
Mixture-66 64
938.5
1285.5
127
Addition
4.9
265.1
6.2
874.3
1168
164.5
413.83
SEA
4.1
206.5
5.4
904.1
1213.1
116.9
408.35
17

---

Table Effect of our primary conditioning mechanism TIPI on the relative MSE. Reported errors are multiplied by 10².
TIPI model
CF
MP
No conditioning
158.63
780
Attention
80
755.5
Early-TIPI
102.5
692.9
Late-TIPI
110
744.6
To demonstrate the effect of the TIPI module, we investigate how different types of conditioning impact the error. This information is summarized in Table where "no conditioning" refers to the absence of TIPI, and "attention" indicates conditioning through attention mechanisms. Additionally, we evaluate whether the optimal placement of the TIPI module is before or after the SEA module.
It is evident that all three types of conditioning significantly reduce the error across both datasets. Furthermore, our secondary conditioning mechanism, AdaLN, reduces the errors even further, resulting in the lowest recorded values. We compare the effect of using AdaLN versus not using NN along with the best errors obtained from Table 5 as shown in Table 6 The addition of the second conditioning mechanism yields similar results regardless of TIPI placement, effectively removing TIPI placement as a critical model parameter.
The effect of RoPE embedding has been widely investigated in the literature, and thus, do not explore this any further. However, our experiments, the incorporation of RoPE significantly improved convergence within this specific domain.
Table 6: Effect of our secondary conditioning mechanism with inclusion of the standard TIPI module presented in this work.
AdaLN
CF
MP
X
80
692.9
38.7
685.9
E Theoretical error
To gain deeper insight into the problem and the sources of error, explore the errors addressed by our model and framework. this analysis, we disregard the spatial and temporal discretization errors originating from the numerical solver and instead focus on the potential errors introduced by the autoregressive models grounded in physical simulations.
The coupling of variables in PDEs is a common occurrence. Many physical systems governed by sets of linear or nonlinear coupled PDEs. Examples include Maxwell's equations (describing electromagnetism), the Ginzburg- Landau equation (used in superconductivity and phase transitions), the Einstein field equations (in general relativity), and the Navier-Stokes equations (governing fluid dynamics).
common approach to modeling these equations involves embedding the variables into a shared latent space, where the system is resolved temporally in an autoregressive manner. For long trajec- tories, an autoregressive approach becomes essential, even for temporally-awand diffusion models. Consequently, it is critical to ensure that the embedded space closely resembles the real physical fields maintain accuracy throughout the prediction process.
Assume v. and ? are two non-equal coupled field variables governed by the following advection equation, where f and g are any functions enforcing the coupling:
1
+
f(()2)
0t
dr
(8)
262
0x62
+ o2
g(b
Øt
dr
18

---

Say v is variable that includes both uo! and 2 then the entropy of y follows the following condition:
H(V) max(H(()),
For the rate-distortion trade-off, this implies that, at the same rate, we can achieve lower field-wise distortion when the embedding is performed separately. the case of two fields, achieving the same distortion requires doubling the rate to embed them together. This results in higher-dimensional em- bedding space, which introduces two significant challenges for the temporal model. First, transformers scale quadratically with the embedding dimension, leading greater computational complexity when the dimension increases, as opposed to using separate decoders for each field. Second, in higher dimensions, our temporal model begins to suffer from sparsity, resulting in poor generalization. Let the distortion error be denoted as ep. Morcover, embedding the physical fields together may result in an uneven distribution of information from the fields, potentially weakening the coupling described in Equation g
We can now demonstrate how any form of information exchange theoretically leads to lower error when separate embedding spaces are created. Based on the definition of mutual information, we have:
I( + = (1'))(())))) HIb! 2
(10)
where I [wb"; 2) represents the mutual information between vo and 2b² and H(bb) is the entropy of v/2 (())) is non-zero, then from the PDE, we know that I(s];; xf2) is non-zero as well. This implies that the entropy of v/t, conditioned on v/², is lower than the entropy of g/d alone. A lack of such conditioning
the model introduces an additional error term due to increased uncertainty, which we refer to the coupling error EC. Our SEA module addresses this error by establishing channel for information exchange between the fields.
To summarise the errors addressed in our work:
Distortion error introduced the from joint embedding of the variables cD-
Coupling error introduced by lack of information exchange or unjust compression of infor- mation due to joint embedding EC.
F
Dataset information
The dataset used in this work consists of two fluid mechanics simulations motivated by physical phenomena: flow around cylinder and the mixing of immiscible fluids. The governing equations for these problems are the conservation of mass (continuity equation) and the conservation of linear momentum (Navier-Stokes equation), which were numerically solved using the finite volume method implemented in the open-source software OpenFOAM. For the multiphase flow simulation, an additional iiivectionddiffusiio equation was solved to track the volume fraction of the phases. The results these simulations were labeled and used as the ground truth in our experiments. The continuity and momentum equations for incompressible fluid flow given by:
V V.u u 0 u
(11)
Ou - (u V
Vp + + +
(12)
Øt
where is the kinematic viscosity, P the density, p is the pressure, is the velocity vector, and 9.81 m/s² is the acceleration due to gravity.
The simulation of flow around 2D cylinder was conducted using icoFoam, transient incompressible solver for laminar flows in OpeMFOAM. To ensure comparability with the literature, we generated 70 trajectories at different Reynolds numbers, with 60% used for training and 20% for validation. The input flow variables for this test case are the velocity components the z- and y-directions (u, v) and pressure (p).
In the second test, the mixing of two fluids in the liquid phase with different densities was simulated using the tvoLiquidMixingFFam solver. The flow modeled transient, incompressible, and
19

---

isothermal, with the dynamics of the mixture being the primary focus of this test case. The model input parameters included the velocity components and the volume fraction, which separates the fluids and ranges between 0 and I. We generated 40 trajectories for this case, with a similar ratio for training, validation, and testing as in the cylinder flow case.
G Final configuration and hyperparameters
Table details the configurations for the ViT-based mesh autoencoder and the temporal model, including key hyperparameters such as the number of patches and TIPI-specific settings. These include scaling mechanisms (MLP, Gaussian Fourier projection, linear) and injection methods (addition, attention). The bottleneck down-projection, adapted from Lee et al. 2024]. is introduced information exchange-specific hyperparameter.
Table 7: The configurations and parameters for the ViT-based mesh autoencoder, temporal transformer architecture, and training parameters. represents the volume of fluid for multiphase flow (Refer to Appendix
ViT-based Mesh Autoencoder
Model parameters
Cylinder flow
Multiphase flow
Number of layers
12
Number of heads
MLP scale ratio
Embedding
16
32
Dropout
Variational
Patches in
Patches in y
Field groups
[u, v). (p)
{a, (a)
Temporal Transformer Architecture
Model parameters
Cylinder flow
Multiphase flow
Number of layers
Number of heads
MLP scale ratio
Model size Dropout
1024
2048
0.3
0
Down projection
TIPI scale mode
MLP
MLP
TIPI injection mode
add
add
Training Parameters
Autoencoder
Optimizer
AdamW
AdamW
I lernnng rate
le-4
le-4
Weight decay
lc-5
le-5
Optimizer momentum (5,, (2)
(0.9, 0.999)
(0.9, 0.999)
Batch size (suapshot)
128
128
Epochs
600
1000
Temporal Model
Optimizer
AdamW le-4
AdamW
Learning rate
8e-5
Weight decay
le-5
le-5
Optimizer momentum (5,, (2)
(0.9. 0.999)
(0.9, 0.999)
Batch size (trajcctory)
Epochs
3500
2500
H Additional results
Figures 12 and 13 preset contour of velocity components and pressure for the flow around the cylinder. Figures 16 show the contour of velocity components and volume fraction for the mixing two fluids test case.
20

---

Prediction,
Prediction, v
Prediction,
Ground truth,
Ground truth,
Ground truth. P
Figure 12: Contour maps of the generated fields at Re=800, and time step of 250 where Von Karman vortex street is formed.
Prediction,
Prediction, v
Prediction,
Ground truth,
Ground truth,
Ground truth, p
Figure 13: Contour maps of the generated fields at Rc=850, and time step of 250 where Von Karman vortex street is formed.
Prediction, 5
Prediction, ts=10
Prediction, ts=20
Ground truth, ts=5
Ground truth, ts=10
Ground truth, ts=20
Figure 14: Comparison of the contour maps of the velocity component u, between the predictions and ground truth in the 850.
Prediction, 5
Prediction, ts=10
Prediction, ts=20
Ground truth, ts=5
Ground truth, ts=10
Ground truth, ts=20
Figure 15: Comparison of the contour maps of the velocity component between the predictions and ground truth in the case of 850.
Prediction, ts=30
Prediction, ts=30
Prediction, ts=30
Ground truth, ts=30
Ground truth, ts=30
Ground truth, ts=30
Figure 16:
Comparison of the contour maps of the volume fraction (a), and velocity components between the predictions and ground truth in the case of 850.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Sections 4.3 4.4
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA].
22

---

Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Sections
3
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer:
[No]
Justification:
The code is being processed and will be uploaded on github soon.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
criiiinnocccccy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //ippe.cc/pullic/guuii wwwtc//ggui////eeeeceoocc CoodeSubeiiie ssi ssionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In sections 3 and appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars were not relevant
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http:///eeiiss..eeee https///cpppp...ttccccccooeeeee
Answer: [Yes]
Justification:
The authors read and respect the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
25

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee www.ree.wthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
26

---

Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "h5T5PhGShJ", "reviewer": "Reviewer_Nand", "summary": "The study proposes an approach for autoregressive spatiotemporal estimation of a dynamical system state, e.g. solution of time-dependent PDE. In particular, a Vision Transformer (VIT) model is adapted to solving PDEs where each state variable is tokenized similarly to tokenizing an image and a novel State-Exchange Attention (SEA) module allows interaction and fusion of these tokens through cross-filed attention. This enables state variables in the system represented by tokens to exchange information and capture the physical relationships and symmetries between fields. The approach is evaluated on the problem of flow past a cylinder benchmark showing improvement of the error against current neural networks approaches.", "questions": "Q1. What is the computational complexity (time, storage) of SEA and how does it compare to other approaches and PDE simulation?\n\nQ2. Does SEA apply to high dimensional state variables, e.g. 3D flows or ND state variables? If yes, it would be important to include in the manuscript (appendix) practical steps of extending the model to higher dimensions.", "limitations": "The technical contributions of this work are subtle since ViT are well known as powerful models and the SEA component seems to be sensible and inherits structure from prior approaches. This could be considered as a limitation, however, on the other hand, the impact of this work lies in the application of solving dynamical equations and simulating dynamical processes and appears to be significant advancement. The ability to implement existing components in such a way that they will be effective is not a straightforward endeavor. Hence, ablation results, discussion of computational complexity and more explanation of the error would be vital in this work. Also, addressing Weaknesses and Questions above would improve clarity, generalization and evaluation of this promising work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "S1. The work presents a novel system based on vision transformer to solve PDEs in autoregressive way. The system is constructed in logical way and the SEA component developed for interaction of field variable is sensible and appears to be applicable for capturing such interactions.\n\nS2. Evaluation of the model on 2D flow past a cylinder shows advantage of the model vs. existing approaches.\n\nS3. The model seems to be generalizable to other PDEs and even to simulations of physical processes that do not have formulated equations.", "weaknesses": "W1. It is claimed that the rollout errors are effectively controlled by the SEA model. There's very little explanation/derivations of the source of these errors. It would be helpful to define these errors rigorously and also if possible to show (graphically or rigorously) how SEA model allows for better control of the error.\n\nW2. Ablations of SEA model components and how their contribution to the overall accuracy of solution are missing. These could inform the critical parts of SEA and better characterize SEA.\n\nW3. While generalization seems plausible, evaluation was performed on only one basic dataset from which introduces uncertainty in how the approach generalizes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***\n**W1. It is claimed that the rollout errors are effectively controlled by the SEA model. There's very little explanation/derivations of the source of these errors. It would be helpful to define these errors rigorously and also if possible to show (graphically or rigorously) how SEA model allows for better control of the error.**\n\nThe overall errors of this model with regards to the numerical results (Neglecting the discretization errors) consist of two main components, namely reconstruction error (Relevant to the encoder-decoder) and temporal error (Relevant to the temporal transformer). The SEA module explicitly addresses the temporal error accumulating during inference (Autoregressive generation) to form the total rollout error. Given a set of coupled partial differential equations with M-state variables, the optimal temporal model would estimate the exact solution as the discretized numerical model. It should be noted that a discretized equation for each variable contains all or a set of other state variables since these equations are coupled. Hence, embedding all the variables in the same embedding space with the sole objective of reconstruction loss may result in unjust information from each field, which would result in an additional error term regarding the coupling of different variables. Our work eliminates this source of error by creating different embedding spaces for different variables, and the SEA module mimics the coupling between different variables. A theoretical analysis has been prepared and will be attached to the supplementary materials. \n***\n**W2. Ablations of SEA model components and how their contribution to the overall accuracy of solution are missing. These could inform the critical parts of SEA and better characterize SEA.**\n\nThe ablation study is summarized in table 1 provided in the attached PDF. This table will also be added to the supplementary material for better understanding of the model. The included table contains the following information:\n1. performance of the model with SEA module for information exchange\n2. Addition of the states to other states for information exchange\n3. No information exchange (variables solved with different transformers)\n4. A single transformer architecture with a shared latent space for all variables (Common in literature).\nThis table demonstrates that models utilizing modes of information exchange outperform those without information exchange or with a shared latent space.\n***\n**W3. While generalization seems plausible, evaluation was performed on only one basic dataset from which introduces uncertainty in how the approach generalizes.**\n\nCylinder flow and the multiphase cases here were chosen to demonstrate the model's performance in cases with different governing PDEs. In the multiphase case, in addition to the Navier stoke's, the volume of fluid PDE is solved coupled with the momentum. In this case, we focus on demonstrating how the SEA module creates this coupling between volume fraction and velocity. Given the clear decreasing trend of error on every variable, as shown in the ablation table in the provided PDF, and the identified sources of error (Explained in W1), we believe this model generalizes well to cases with different governing PDEs, initial conditions, or boundary conditions.\n***\n**Q1. What is the computational complexity (time, storage) of SEA and how does it compare to other approaches and PDE simulation?**\n\nThe time complexity of the encoder-decoder is similar to the transformer and is unaffected by the number of state variables. Similarly the storage is the same as the ViT base model. However, the temporal model with the SEA module has an additional quadratic term regarding the number of state variables. If we denote the number of state variables as M, the time complexity becomes O(M^2 * D * N^2) where D, and N are the model dimension and the input block size respectively.\nThe current state-of-the-art studies resolving full trajectories use techniques with a transformer backbone, which is dominated by the self-attention term O(D*N^2) [1,2].\nAlthough the inference time remains orders of magnitude below the numerical simulations, the time complexity increases quadratically with respect to the state variables which is the main limitation of this module. More information on computational aspects are provided in the global section.\n\n**Q2. Does SEA apply to high dimensional state variables, e.g. 3D flows or ND state variables? If yes, it would be important to include in the manuscript (appendix) practical steps of extending the model to higher dimensions.**\n\nThe model is capable of 3D flows, and in general n dimensional and multi-variate problems. The procedure for this will be added to the appendix and the the code will be released with the paper which includes a complete implementation for any specified number of variables. \n***\n# References\n[1] Luning Sun, Xu Han, Han Gao, Jian-Xun Wang, and Liping Liu. Unifying predictions of deter-ministic and stochastic physics in mesh-reduced space with sequential flow generative model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 60636–60660. Curran Associates, Inc., 2023.\n\n[2] Xu Han, Han Gao, Tobias Pfaff, Jian-Xun Wang, and Li-Ping Liu. Predicting physics in mesh-reduced space with temporal attention. arXiv preprint arXiv:2201.09113, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NdaWzhkDqv", "reviewer": "Reviewer_NXYL", "summary": "The authors present a novel and interesting approach to physics-domain transformer models that exchanges information between state variables and demonstrates strong performance improvements relative to SOTA models on hard fluid dynamics problems. The paper is well written and the results compelling.", "questions": "- How does the padding work in the case of graphs with highly variable node density (for example on an adaptive mesh with much denser representation near the cylinder as is common in practice)?", "limitations": "The authors have touched on potential limitations related to number of state variables but could discuss this in more detail, as many problems of interest contain more than two state variables.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I find the multidimensional information exchange architecture employed by the authors to be interesting, the improvement over existing models to be significant on challenging problems, and the paper and figures to be quite well done. I would like to see more thorough evaluation of the models, perhaps including application to a non-fluid domain or to a problem with a few more underlying state variables, and it is highly desirable that code be made available with the paper.", "weaknesses": "#### Major concerns:\n\n1. Perhaps I missed this somewhere, but will code be made available with this paper? It seems too much to require readers to re-implement this architecture to use it.\n\n2. The supplement is helpful but could contain further experimental details, further details on the layers (and their trainability), how projections are performed, and more complete explanation of results and metrics.\n\n3. If training the models takes just 15-20 minutes, it would be nice to see the effects of the embedding dimensionality and simulation parameters explored more thoroughly.\n\n4. The authors point out that a limitation might be scaling to a large number of state variables. Could they present an example on a problem with more than two state variables so that the \"multidirectional\" component is more than bidirectional, and perhaps add some thoughts on how this might scale? \n\n#### Minor concerns:\nThere are a number of places the notation could be clarified and some typos, for example:\nBy line:\n- 126: argmax optimization variable is not specified (presumably z_1:T)\n- 159: the X variable is never defined, and may be superfluous here... maybe just use z_i^k? \n- 166: T should be italic?\n- A number of variables throughout seem to be used without introduction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***\n**W1 and W2: Perhaps I missed this somewhere, but will code be made available with this paper? It seems too much to require readers to re-implement this architecture to use it.**\n\nThe code is prepared and will be released with the paper. We will also add more information in the implementation of supplementary materials for better reproducibility. \n***\n**W3: If training the models takes just 15-20 minutes, it would be nice to see the effects of the embedding dimensionality and simulation parameters explored more thoroughly.**\n\nThe optimal embedding dimensions were determined for both cases by experimenting. It was found that lower embedding dimensions made reconstruction difficult and amplified errors, while higher dimensions required more computation and led to model overfitting. A detailed study, included in the provided PDF (Figure 1) and will be added to the supplementary material, to further clarify the chosen dimensions.\n\n***\n**W4: The authors point out that a limitation might be scaling to a large number of state variables. Could they present an example on a problem with more than two state variables so that the \"multidirectional\" component is more than bidirectional, and perhaps add some thoughts on how this might scale?**\n\nOne potential field in the physics and material sciences with a large number of state variables would be the phase field simulations to capture the mesoscale dynamics of the grains. These 3D problems tend to have one state variable per grain, resulting in M-coupled equations where M represents the number of variables. Another example would be the multiphase flows with more than two phases. Given the history of each variable is resolved with its corresponding transformer, this work becomes limited in these cases. Even though the number of variables does not theoretically limit the model itself the required computation increases quadratic with the number of variables. However, this is not different from other works in the literature where all the variables are in the same latent space, and one transformer resolves the dynamics of all variables. In those cases, the latent dimension must be increased when more variables are stored on the mesh, resulting in higher computation. Scalability is further discussed in the global response. We also intend to address the scalability in the future works and include a wider range of test cases in different domains to evaluate the performance of the model. \n\n***\n**minor problems:**\nWe appreciate the corrections. These problems were all corrected. \n\n***\n**Q1. How does the padding work in the case of graphs with highly variable node density (for example on an adaptive mesh with much denser representation near the cylinder as is common in practice)?**\n\nIn this study, all patches are uniformly sized, similar to the Vision Transformer (ViT). To ensure consistency, each patch is padded with zeros up to the maximum number of elements found in any patch within the domain. Prior to the mentioned padding, we introduce a small noise in the fields stored on the mesh such that there is no element with an exact value of 0 (In order to keep these elements active in the learning and separate them from the pads). Further, we have adopted a GeLU activation function during the downsampling of each patch, resulting in no contribution from the pads in the encoder-decoder pair. This treatment, by default, captures more information in the patches with high mesh density and lower information in those with a more sparse structure (Highly padded patches).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jeFAbWZVCN", "reviewer": "Reviewer_M5vu", "summary": "The paper introduces the State-Exchange Attention (SEA) module, a new cross-attention module that enables information exchange between state variables in physics-domain transformer modules. The authors evaluate the performance of this module in both single-phase and multi-phase 2D fluid settings.", "questions": "1. Are there ablation studies available for single-phase flow (as is sort of provided for multi-phase flow). In other words, how does a basic transformer fair in this setting? \n2. How resource-intensive is training this model vs. traditional transformer architectures? Some indication of this would be very helpful.", "limitations": "See weaknesses.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Novelty: The State-Exchange Attention architecture represents an interesting and new contribution in architecture for physics-domain transformer models. This module enables multi-directional information exchange between state variables, and as far as I know is the first of its kind for physics-domain transformer architectures.\n2. Performance: The SEA module significantly reduces rollout error accumulation, which is a major issue in current sequential models. The paper reports substantial improvements over competitive baseline models in both a single-phase and multi-phase context. \n3. Strong Evaluation: The SEA-integrated model is well evaluated across different computational fluid dynamics (CFD) cases, showing consistent performance improvements. This includes detailed experiments on both single-phase and multiphase flows.", "weaknesses": "1. Scalability Concerns: The current architecture is relatively small, using a transformer with only 1 layer and 8 attention heads. I am curious how the model will scale at larger architecture sizes. Moreover, as the authors themselves note, there are also concerns about how the model will scale with larger number of state variables. \n2. Diversity of experiments: The authors evaluate performance on one instance of single-phase and one instance of multi-phase flow, both in two dimensions. How does the model fair in 1D and 3D settings? How does it fair in a wider variety of fluid mechanics problems (like those presented in PDEBench/PDEArena)? \n\nMinor comments:\n- It seems like the general gist of paragraphs on lines 179-186 and lines 187-192 are almost identical. Was this paragraph repeated by accident?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***\n**W1. Scalability Concerns: The current architecture is relatively small, using a transformer with only 1 layer and 8 attention heads. I am curious how the model will scale at larger architecture sizes. Moreover, as the authors themselves note, there are also concerns about how the model will scale with larger number of state variables.**\n\nAs dictated by the available data and complexity trade-off, increasing the complexity of the model would not be ideal in either case. We carefully studied the architectural aspects and converged on the same architecture as the transformer based works we compared our results to [1,2]. We also provide the our model dimension study in the provided PDF..\n***\n**W2. Diversity of experiments: The authors evaluate performance on one instance of single-phase and one instance of multi-phase flow, both in two dimensions. How does the model fair in 1D and 3D settings? How does it fair in a wider variety of fluid mechanics problems (like those presented in PDEBench/PDEArena)?**\n\nOur model is specifically designed to address the coupling between state variables governed by different PDEs. In a 1D scenario, reducing the model will yield the same results as those found in the literature, as there is only one state variable and information exchange becomes redundant (here, we focus on the temporal model and not the ViT encoder-decoder). The 2D problems studied, such as the 2D Navier-Stokes and 2D Navier-Stokes with volume of fluid PDE, were chosen due to the strong coupling between the variables. We avoided 3D cases in this study for clearer demonstrations, however the model is by no means limited to 2D.\n***\n**W3. It seems like the general gist of paragraphs on lines 179-186 and lines 187-192 are almost identical. Was this paragraph repeated by accident?**\n\nThis is indeed an editting problem, we appreciate your detailed revision and pointing this out. \n***\n**Q1. Are there ablation studies available for single-phase flow (as is sort of provided for multi-phase flow). In other words, how does a basic transformer fair in this setting?**\n\nThe ablation studies are presented in an additional table within the provided PDF and the supplementary materials. This table demonstrates the model's performance under different information exchange modes: addition of information for exchange, SEA attention mechanism for exchange, and no information exchange. Additionally, this table includes the case where state variables share a mutual latent space, and no explicit information exchange is defined.\n ***\n**Q2. How resource-intensive is training this model vs. traditional transformer architectures? Some indication of this would be very helpful.**\n\nWe have added the information regarding the training time to the global rebuttal, and further demonstrated the inference time for the full trajectories of studied cases with respect to the model dimension in the provided PDF. \n\n***\n# References\n[1] Luning Sun, Xu Han, Han Gao, Jian-Xun Wang, and Liping Liu. Unifying predictions of deter-ministic and stochastic physics in mesh-reduced space with sequential flow generative model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 60636–60660. Curran Associates, Inc., 2023.\n\n[2] Xu Han, Han Gao, Tobias Pfaff, Jian-Xun Wang, and Li-Ping Liu. Predicting physics in mesh-reduced space with temporal attention. arXiv preprint arXiv:2201.09113, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "po05eORbg1", "reviewer": "Reviewer_Yocw", "summary": "This paper presents a novel neural network architecture (SEA) for solving partial differential equations (PDEs) in physical problems. This architecture effectively utilizes information exchange between multiple fields and the conservation quantities of physical systems to correct model predictions, achieving smaller errors and improved generalization. This has significant value for industrial applications, as it can greatly reduce the time cost for solving specific problems.", "questions": "See cooments", "limitations": "yes", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "**Originality:** The authors propose a novel neural network architecture (SEA) that effectively utilizes information exchange between multiple fields while cleverly incorporating physical conservation quantities. This maximizes the use of the physical system's symmetry to reduce errors.\n\n**Clarity:** The paper clearly defines concepts and provides a clear and explicit explanation of the motivation for proposing this architecture. The presentation of experimental results is detailed, and the overall model structure is presented in an intuitive and clear manner.\n\n**Significance:** The model in this paper offers valuable practical insights into integrating neural networks into PDE solvers, providing a fresh perspective in this field. This has important implications for enhancing productivity in the industrial sector.\n\n**Quality:** The language of the paper is rigorous and clear, with appropriate citations of previous work. The overall approach is straightforward and easy to understand. The experimental results include comparisons with multiple similar models for reasonable and effective evaluation.", "weaknesses": "Further exploration of the model's generalization ability is needed, such as evaluating model errors under more complex boundary conditions. Additionally, presenting the training and inference costs of the model would provide a better assessment of its potential for industrial applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***\n**W1. Further exploration of the model's generalization ability is needed, such as evaluating model errors under more complex boundary conditions. Additionally, presenting the training and inference costs of the model would provide a better assessment of its potential for industrial applications.**\n\nOur future work will focus on addressing the scalability of the architecture and evaluating the model on a wider range of scenarios. We plan to conduct further evaluations using the PDEBench dataset and other datasets from different domains. Details on costs and time complexity are discussed in the global rebuttal and provided in the attached PDF.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
***
We would like to appreciate all referees for spending time giving great feedback, and instructions on how to improve the paper. 

Here we would like to address two main points that were questioned by all the referees, first the limitation of the work and then the efficiency of the model.

***
# Limitation
We first briefly review the limitation and then suggest a potential way forward for the future works. 
The SEA module effectively acting as a cross attention amongst the state variables is quadratic in time with respect to the number of state variables which limits the module extending to large number of variables. The time complexity of this model can be expressed as O(M^2 * D * N^2) , where M is the number of variables, D is the dimension of the model, and N is the sequence length. However, various strategies can potentially be studied in future work to mitigate this complexity. A key technique involves leveraging prior knowledge of the equations or the physical system to inform the cross-attention process. This prior knowledge can be utilized to adjust the cross-attention between variables, ensuring the method is applied only to strongly coupled variables (e.g., velocity and pressure). Another approach involves embedding variables of the same type together. For example, in the 3D Navier-Stokes equations, the three components of velocity would be embedded together, while source terms like pressure and other external or internal forces would be embedded together. In our future work, we will conduct a comprehensive investigation of these factors. Additionally, we will explore the implementation of this model within a probabilistic framework (Guiding the output based on the coupling between variables), leveraging its ability to effectively resolve the history of variables and their coupling and address uncertainty during inference.

***
# Computational aspects
Another point raised by all the reviewers pertains to the computational aspects of the model. We have recorded the training and inference times for both cases. In the paper, the reported training was conducted using teacher forcing on both the training and validation sets. Additionally, we present the training time for the scenario where validation is tracked in an autoregressive manner. While both approaches yield meaningful training, we include the autoregressive validation case because the model's ultimate task is to estimate full trajectories.
The training for all cases is done with a single A100 GPU. In cylinder flow we resolve 5600 time points, with 7697 meshing points, and 3 variables of u, v, and p stored on each point.
For the multiphase case we have a total of 7400 time steps and 8241 meshing points. Similar to the cylinder flow we have three variables here namely u, v, and alpha where alpha is the volume fraction capturing the phases and their interface. The following computational detail will enter the supplementary materials. 

The training cost of each case is listed below:
1) Training with validation using teacher forcing
* Cylinder flow: 20-30 Minutes
* Multiphase flow: 60-70 Minutes*

2) Training with validation using autoregressive approach
* Cylinder flow: 45-60 Minutes
* Multiphase flow: 2 Hours

*A typo in the main manuscript reported the multiphase flow training time as 15 minutes. The correct training time is provided here and will be corrected in the paper.

Inference of full trajectory for all cases remains below 30 seconds a detailed plot of which with respect to model dimension is provided in the attached PDF.

***
# Provided PDF
In the provided PDF we have included a detailed ablation study which also contains results of the case with shared latent space between all variables (Table 1). We also evaluate the performance of the time invariant parameter injection module in table 2. Another critical parameter in training this model is the batch size. Increasing the batch size was observed to raise the error in autoregressive generation and potentially cause divergence. Relevant information on the batch size is provided in Table 2. To fully understand the impact of dimension choice, we included several plots illustrating the reconstruction error (MSE) of the ViT encoder-decoder, temporal error (relative MSE), and inference time. These plots, along with the provided training cost, informed our dimension selection.

The provided results in the PDF will be appended to the supplementary materials along with a theoretical analysis of errors to demonstrate which component of error is being addressed by our model.

Meta-review: Reviewers appreciate how the paper adapts vision transformers to PDEs with a "state exchange attention (SEA)" mechanism and a "time invariant parameter injection (TIPI)" component. I personally find the paper to be frustratingly lacking in key details, and I strongly encourage the authors to address these concerns in the camera ready version. 

1. The authors do a decent job of motivating what they want to accomplish with SEA, but the mathematical description in eq. (3) doesn't make sense. Index $j$ appears on the right-hand side but not the left, and there's no specification of how results are reduced over $j$.
2. The so-called "time invariant parameter injection (TIPI)" component is not well motivated, and again the mathematical description in eq. (5) doesn't make sense: a mysterious "information injector" $f_{IJ}(\mu)$ is introduced without explanation. The variable $\mu$ is never even defined!
3. The captions for Figure 1 and Figure 2 are insufficient to describe what is being shown. The text sometimes refers to Figure 5 when it is meant to refer to Figure 3. The tick labels are illegible in Figures 4-6.
4. Table 1 presents the authors' method in bold even when it does not achieve the lowest error (specifically for $u$).
5. The authors use \citet when they should use \citep.
6. The paper needs to be edited for spelling mistakes, grammatical errors, and redundant phrases. For example, "We can formally formulate this problem..." (line 125) and "To process and prepare the spatial information..." (line 130).
Decision: Accept (poster)