A composable machine-learning approach for steady-state simulations on high-resolution grids
Authors: Rishikesh Ranade
Derek Christopher Hill
Lalit Ghule
Jay Pathak
Venue: NeurIPS 2022 Accept (2022)
Keywords: machine learning
partial differential equations
computational science

Abstract: In this paper we show that our Machine Learning (ML) approach, CoMLSim (Composable Machine Learning Simulator), can  simulate PDEs on highly-resolved grids with higher accuracy and generalization to out-of-distribution source terms and geometries than traditional ML baselines. Our unique approach combines key principles of traditional PDE solvers with local-learning and low-dimensional manifold techniques to iteratively simulate PDEs on large computational domains. The proposed approach is validated on more than 5 steady-state PDEs across different PDE conditions on highly-resolved grids and comparisons are made with the commercial solver, Ansys Fluent as well as 4 other state-of-the-art ML methods. The numerical experiments show that our approach outperforms ML baselines in terms of 1) accuracy across quantitative metrics and 2) generalization to out-of-distribution conditions as well as domain sizes. Additionally, we provide results for a large number of ablations experiments conducted to highlight components of our approach that strongly influence the results. We conclude that our local-learning and iterative-inferencing approach reduces the challenge of generalization that most ML models face.

A composable machine-learning approach for steady-state simulations on high-resolution grids
Rishikesh Ranade Office of CTO
Chris Hill
Fluids Business Unit
Ansys Inc. Canonsburg, PA 15317
Ansys Inc. Lebanon, NH. 03766
rishikesh, ran.eeessss.aaadeaaa.aaaa com
com
Lalit Ghule
Jay Pathak
Office of CTO Ansys Inc. Canonsburg, PA, 15317 lalit. www.tttggueeenssssoooooo com
Office of CTO Ansys Inc. San Jose, CA, 95134 jay. www.akeansys.cooommmmmmmoom com
Abstract
In this paper we show that our Machine Learning (ML) approach, CoMLSim (Composable Machine Learning Simulator), can simulate PDEs on highly-resolved grids with higher accuracy and generalization to out-of-distribution source terms and geometries than traditional ML baselines. Our unique approach combines key principles of traditional PDE solvers with local local-learnig and Now-dimensional manifold techniques to iteratively simulate PDEs on large computational domains. The proposed approach is validated on more than 5 steady-state PDEs across different PDE conditions on highly-resolved grids and comparisons are made with the commercial solver, Ansys Fluent as well as 4 other state-of of-the-art MI methods. The numerical experiments show that our approach outperforms ML baselines terms of I) accuracy across quantitative metrics and 2) generalization to out-of- distribution conditions as well as domain sizes. Additionally, we provide results for large number of ablations experiments conducted to highlight components of our approach that strongly influence the results. We conclude that our local-leariing and iterative- iettrreeeiferennngggg. approach reduces the challenge of generalization that most ML models face.
Introduction
Engineering simulations utilize solutions of partial differential equations (PDEs) model various complex physical processes like weather prediction, combustion in car engine, thermal cooling on electronic chips etc., where the complexity in physics driven by several factors such as Reynolds number, heat source, geometry, etc. Traditional PDE solvers use discretization techniques to ap- proximate PDEs on discrete computational domains and combine them with linear and non-linear equation solvers to compute PDE solutions. The PDE solutions are dependent on several conditions imposed on the PDE, such as geometry and boundary conditions of the computational domain as well source terms such as heat generation, buoyancy etc. Moreover, many applications in engineering require highly-resolved computational domains to accurately approximate the PDEs solutions as well as to capture the intricacies of conditions such geometry and source terms imposed on the PDEs. As result, traditional PDE solvers, although accurate and generalizable across various PDE conditions, be computationally slow, especially in the presence of complicated physies and
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Ansys
CoMLSim
Figure I: CoMLSim performance vs Ansys Fluent for a high Reynolds number, steady-state turbulent flow over a car (never seen during training) at 90 mph. The vectors and contours predictions match reasonably with MAE of 0.0145. High flow gradients that affect the wake pattern are resolved accurately and comparable to Fluent.
large computational domains. In our work, we specifically aim to increase the speed of engineering simulations using ML techniques complex PDE conditions used for enterprise simulations.
The idea of using Machine Learning (ML. with PDEs is not unique and has been explored for several decades [T] 22. ML approaches are computationally fast, but they fall short in terms of accuracy, generalization wide range and out-of-distribution PDE conditions and in their ability scale highly-resolved computational grids, when compared to traditional PDE solvers. A few shortcomings of the current methods are outlined below and subsequently are motivations for this work.
1. ML approaches static inferencing to predict PDE solutions function of the PDE conditions. In many cases, solutions and conditions have high-dimensional and sparse repre- sentations, which are challenging to generalize with such block-boxed static inferencing.
2. Most engineering applications require highly-resolved computational grids to capture de- tailed solution features, such as hot spots on an electronic chip surface. This adversely impacts training in terms of GPU memory, computational time and data requirements.
3. Most ML approaches fail to use powerful information from traditional PDE solvers related to solver methods, numerical discretization etc.
In this work, introduce novel ML approach, Composable Machine Learning Simulator (CoML- Sim) to work with high-resolution grids without compromising accuracy. To achieve that, our method decomposes highly-resolved computational grids into smaller local subdomains with Cartesian grids. PDE solutions and conditions on each subdomain are represented by lower-dimensional latent vectors determined using autoencoders. The latent vectors of solutions corresponding conditions are predicted in each subdomain and stitched together to maintain local consistency, analogous to flux conservation traditional solvers. this paper, we describe our method (shown in fig. 2 for building CoMLSim for different PDEs, showcase different experiments to compare accuracy and generalizability with different ML methods and provide an extensive ablation study to understand the impact of CoMLSim's 's different components on the results.
Significant contributions of this work:
1. The CoMLSim approach combines traditional PDE solver strategies such as domain dis- cretization, flux conservation, solution methodologies etc. with ML techniques accurately model numerical simulations on high-resolution grids.
2. Our approach operates on local subdomains and solves PDEs in a low-dimensional space. This enables generalizing to out-of-distributio PDE conditions and scaling to bigger do- mains with large mesh sizes.
3. The iterative inferencing algorithm self-supervised and allows for coupling with traditional PDE solvers
Related Works: The use of MI. for solving PDEs has gained tremendous traction the past few years. Much of the research has focused on improving neural network architectures and optimization

---

techniques to enable generalizable and accurate learning of PDE solutions. More recently, there has been lot of focus on learning PDE operators with neural networks (NNs) B 5
8 9
The neural operators are trained on high-fidelity solutions generated by traditional PDE solvers computational grid of specific resolution and do not require any knowledge of the PDE. However, accuracy and diitr--dittiibution generalizability deteriorates as the resolution of computational grid increases (for example, 2048² in 2-D or 256 3-D). Furthermore, these methods are limited by an upper cap GPU memory.
different research direction focuses on training neural networks with physics constrained optimiza- tion [10 IT. These method use Automatic Differentiation (AD) [12] to compute PDE derivatives. The physics- based approaches have been extended to solve complicated PDEs representing complex IS
29 30 More recently. alternate approaches that use discretization techniques using higher order derivatives and specialize numerical schemes compute derivatives have shown to provide better regularization for faster convergence However, the use of optimization techniques to solve PDEs, although accurate, has proved to be extremely slow compared to traditional solvers and hence, non-scalable high resolution meshes.
physies [3 14
[6] [7] [8] [9,20 21 22 23 24
[25] 26 27
28
31
32
34
Domain decomposition techniques have been successfully used in the context of ML applied to PDEs 35]. These techniques enable local learning from smaller restricted domains and have proved to accelerate learning of neural networks. In [3 35), the authors proposed a hybrid method combining ML with domain decomposition reduce the computational cost of finite element solvers. Similarly, domain decomposition has been extensively used in the context of physics informed neural networks
to reduce training costs by enabling distributed training on multiple GPUs (37 38
and learn on localized domains but infer on larger computational domains using stitching process. 41
39
and 442 learn coefficients of numerical discretization schemes from high fidelity data, which is sub-sampled on coarse grids. 43 learn surrogate models for smaller components to allow for cheaper simulations. 44 learn local prolongation operators from discretization matrices, to improve the rate of convergence of the multigrid linear solvers. Alternatively, other methods such as 45 46 47 make us graph-basee based models to improve generalization by learning on local elements. In this work, we Cartesian grid discretize each subdomain. However, GNNs 48 49 50 51 52 as well as FEM use other discretizations such as triangle or polygonal meshes. In future, these ideas can be used to handle unstructured meshes in subdomains. Additionall, other MI. methods improve the learning of ML models by compressing PDE solutions on to lower-dimensional manifolds. This has shown to improve accuracy and generalization capability of neural networks
[53 54 55
56
58 Although,
here we propose an iterative inferencing approach in the context of solving PDEs, is inspired from
the implicit deep learning approach used variety of machine learning tasks 59 60 6] 62] In
this work, domain decomposition is combined with latent space learning is employed to accurately represent solutions on local subdomains and allow scalability bigger computational domains.
2
CoMLSim approach
2.1 Similarities with traditional PDE solvers
Consider a of coupled steady-state PDEs with n solution variables (n for example). The coupled PDEs are defined as follows:
L (u, v) F = 0: L2(u, F2
(1)
u( y, 2) and v(z, y, 2) are defined on a computational domain with boundary conditions specified on the boundary of the computational domain, Ωz, Here, L1, L2 denote PDE operators, Fi, F2
User Specs.
selution. geametry, and ssurce icrms
demain/PDD candition
Computational
Fized point derrations
Final PDE sulutiun
Break
Encade
Flax subdommii
Decads solation
Tir
lig
04 solution encoding escoding
to
boundary encoling, iu eccoding
Figure CoMLSim solution algorithm 3

---

represent PDE source terms and = v for 52 S2. The PDE operators can vary for different PDEs. For example, in non-linear PDE such as the unsteady, incompressible Navier- Stokes equation the operator, L = V. A.V Traditional FVM/FDM based PDE solvers solve PDEs in Eq. T by computing solutions variables, u, and their linear/non- linear derivatives discretized computational domain. Iterative solution algorithms are used to conserve fluxes between neighboring computational elements and determine consistent PDE solutions over the entire domain at convergence. The CoMLSim approach is designed to perform similar operations but at the level of subdomains (n elements) using MI. techniques. Additional details are provided in supplementary materials sections A.4.
2.2 Solution algorithm for steady-state PDEs
The solution algorithm CoMLSim approach is shown Fig. 2 and described in detail in Alg. [ Similar to traditional solvers, the CoMLSim approach discretizes the computational domain into smaller subdomains contain n" computational elements, where p refers to spatial dimensionality and n is predetermined. Each subdomain has constant physical size and represents both PDE solutions and conditions. For example, a subdomain cutting across the cylinder in Fig. represents the geometry and boundary conditions of the part-cylinder well the corresponding solution. In a steady-state problem, the solutions on the domain are initialized either with uniform solutions or they are generated from coarse-grid PDE solvers. The initial solution F (u(z), v(T) is shown with randomly oriented flow vectors in Fig. 2 Pretrained encoders Ej, Eg. eb, ea are used to encode the initial solutions well user-specified PDE conditions into liver-dimensional latent vectors, Mg corresponding to PDE solutions and Ig: The corresponding to geometry. boundary conditions and source terms, respectively. Solution and condition latent vectors on groups of neighboring subdomains are concatenated together evaluated using pre-trained flux conservation autoencoder (O) get new set of solution latent vectors (1/g) on each subdomain, which are more locally consistent than the original vectors. The new solution latent vectors are iteratively passed through to improve their local consistency and the iteration is stopped when the L2 norm of change in solution latent vectors meets a specified tolerance, otherwise the iteration continues with the updated latent vectors. The latent vectors of the PDE conditions are not updated and help in steering the solution latent vectors to an equilibrium state that is decoded PDE solutions using pretrained decoders (gp) on the computational domain. The converged solution in Fig. 2 is represented with flow vectors that are locally consistent with neighboring subdomains. The iterative procedure used in the CoMLSim approach can be implemented using several linear or non-linear equation solvers, such as Fixed point iterations [59], Gauss Seidel, Newton's 's method etc., that are used in commercial PDE solvers. Out of those we have explored Point Jacobi and Gauss Seidel, which are described below. The use of autoencoders for compressed solution and condition representation tandem with iterative inferencing solution algorithm is inspired from [58] 31 but the main difference in this work that the solution procedure is carried out on local subdomains as opposed to entire computational domains to align with principles used in traditional PDE solvers. Supplementary material section G explains our algorithm in more details using an example.
Algorithm 1: Solution methodology of CoML approach
Domain Decomposition: Computational domain Ω + Subdomains Ω. 2 Initialize solution on all Ω2: p(7)= = 0.0 for all j g Ωr
Encode solutions on Ω2: ny e. (f.)))
Encode conditions on Ω2.:
Th = eg (g())) 7h- = es (b(S))))) 1/s es (s(O.)
Set convergence tolerance: e, -8
7 while e > & do
for Qe E Ω do
Gather neighbors of Qoi Pnd = [ac Siefe Prighe....]
ab nb 17/2)) - O mb ,g
10
Compute La norm: e || 1/all2
12
Update: 77 + 7/3 for all Q2 E 5
13 Decode PDE solution on all Ω2: p = 9 g)(()))))

---

refers to the latent vectors and Why flux conservation networks work? The flux conservation network (0) is simply an autoencoder which takes encoding of solutions and conditions on group of neighboring subdomains as both inputs and outputs. Similar to traditional solvers that use approximations to represent relationships between neighboring solutions on elements, the flux conservation autoencoder does the same for encoded subdomain solutions and corresponding conditions. The training of this network is carried out on converged, locally consistent solutions generated for a specific PDE for arbitrary conditions. Since this network has only learnt locally consistent solutions, if one were initialize group of neighboring subdomains with random noise and iteratively pass it through this network the output of such a procedure would be an equilibrium solution corresponding to some locally consistent PDE solution. However, the correct PDE solution it converges to depends on the fixed condition specified in this procedure. Additional results and details provided in the supplementary materials sections A. A.4 and C.I.5.
Stability of flux conservation autoencoder: The iterative inferencing approach proposed in this work is similar to traditional approaches used in solving system of linear equations. Our algorithm uses fixed point iterations to solve Eq. 2
1/p (n mb
110 "i m)
where, θ corresponds to the weights of flux conservation network, Tho nb refers to neighboring subdomains defined below in Eq.
7p =
np 77 .36
left right top bottom
;yp 17
(3)
Similar linear systems. the stability of our iterative inferencing approach is governed by the condition number [63] defined in Eq.
00 (1p The Fg: ns)) || < 1
(4)
Ong
Solution algorithms: Point Jacobi vs Gauss Seidel: Since the algorithm loops over all the sub- domains in a specified order, while updating the solution encoding of a subdomain there subdomains in the neighborhood that already have an updated solution encoding. The Gauss Seidel method uses these updated solution encodings (175)) on neighboring subdomains update the solution encoding on subdomain, and hence can be easily vectorized for significantly faster computation. The implementation has a remarkable similarity with how traditional solvers use these methods (64] and leaves the door open to use other non-lineen optimizers with physics- physics-based constraints.
On the other hand, Point Jacobi method does not make use of this
2.3 Neural network components in CoMLSim
Our algorithm employs two types of autoencoders, CNN autoencoders to establish lower- dimensional representation of PDE solutions and conditions from local subdomains grids and FCNN Autoencoders for flux conservation, where the goal is to learn reduced representation of solution and condition latent vectors in a local neighborhood. Let us consider an example of solving the Laplace equation, V = 0 in 2D for arbitrarily shaped computational domains, Pg. In this example, the CoML Sim algorithm will require: 3 autoencoders, described in Eq. 5 to learn lower-dimensional representations local subdomains.
n' S(u, nh))
g' G(g, ng)
(5)
The O ()
where, u. Tu, S refers to the solution, its latent vector and the weights of the autoencoder, respectively. g, gg G refers to representation of geometry, such as Signed Distance Fields (SDF) [65], its latent vector and the weights of the autoencoder, θ refers to the weights of flux conservation autoencoder and Vnb represents a set of concatenated latent vectors, 7/h and 7/ on group of neighboring subdomains. All the autoencoders are trained with samples of PDE solutions generated for the same use case. In the case of coupled PDEs, a single autoencoder is trained for solution variables. Additional details related to the autoencoder networks are provided in the supplementary materials section A.
5

---

Why Autoencoders?: Solutions to classical PDEs such as the Laplace equation can be represented by homogeneous solutions as follows:
(x, y) = do a1 a2y + az(2² y²) + 0s (4
+
(6)
where, A = dg. d1, 02. ... an are constant coefficients that can be used to reconstruct the PDE solution on any local subdomain. A can be considered as compressed encoding of the Laplace solutions. Since, it is not possible to explicitly derive such compressed encodings for other high dimensional and non-linear PDEs, the CoMLSSim approach relies autoencoders compute them. It is known that non-linear autoencoders with good compression ratios can learn powerful non- non-linear generalizations 666 67 [68. Autoencoders also have great denoising abilities, which improve robustness and stability, when used in iterative settings [58]. In their paper, Park and Lee [69] demonstrate that the latent manifold established by a trained autoencoder is stable for varying intensities of Gaussian noise. This quality of autoencoders is useful for the convergence of the iterative inferencing approach.
3 Experiments
In this section well as in the supplementary materials section C, we consider number of use cases with varying degrees of difficulty resulting from the PDE formulation as well as source terms, geometry and boundary conditions. The PDEs have applications in fluid mechanics, structural mechanics and semiconductor simulations.
Details of experiments: Here we provide some details about the experiments. Additional details and experiments on Laplace and Darcy equation may be found in the supplementary materials section D.
1. 2D Poisson equation: The Poisson's 's equation, shown in Eq. is very popular in engineer- ing simulations, for example, chip temperature prediction, pressure equation in fluids etc.
where, t is the solution variable and f is the source term. This PDE is solved on 1024x1024 grid resolve the high-frequency features of the source term. The source, S, sampled from a Gaussian mixture model shown in Eq. g
1024 1024
10211024
(- (-(-² Hz,k
ΣΣ
My.k
A exp
(8)
Oz,k
yy.k
km
where, I, y correspond to the grid coordinates. As randomly assumes either 0 or 1 to vary the number of active Gaussians in the model. Pz By and az, Fy the mean and standard deviations Gaussians in x and directions, respectively. The means and standard deviations vary randomly between O to and 0.001 to 0.01, respectively. The smaller magnitude of standard deviation results hot spots that require highly-resolved grids. In this case, 256 training and 100 testing solutions are generated using Ansys Fluent.
2. 2D non-linear coupled Poisson equation: The coupled non-linear Poisson's equation is shown below in Eq. 9 u, are the solution variables and f is the source term, similar the description in Eq. This PDE has applications in reactive flow simulations.
V = u²;
V
22 +
The data generation is similar to Experiment 2 with slight difference that the source term is less stiff with a standard deviation that varies between 0.005 to 0.05.
3. 3D Reynolds-Averagee Navier-Stokes external flow: This use case consists of 3-D channel flow with resolution 304x64x64 at high Reynolds number over arbitrarily shaped objects. The corresponding PDEs are presented in supplementary materials section C.3 or may be referred from [70]. The characteristies of flow generated on the downstream has important applications in the design of automobiles and airplanes. In this case, we use primitive 3-D geometries namely, cylinder, cuboid, trapezoid, airfoil wing and wedge and their random combinations and rotational augmentations to create 150 training geometries and 50 testing geometries, which are solved with Ansys Fluent to generate the data. Out-of- distribution testing carried out on simplistic automobile geometries as shown in Figure

---

(MAE
(A)
LMAE)
(B)
MAB
(C)
(D)
0.0
3.25
500
0.5
Depple::
0.10
400
3.15
110
110
POW
115
3330
1.05
0.05
300
2.0
1.00
Grid resolution
Bottleneck layer
Figure 3: (A): Error vs No. of Gaussians for linear Poisson's; (B): Error vs No. of Gaussians for non-linear Poisson's: s; (C): Error grid resolution for linear Poisson's; (D): MAE and convergence iterations vs flux conservation bottleneck size
4. 3D chip cooling with Natural convection: In this case, we extend the complexity of Experiments 2 and 3 industrial 3-D case of chip cooling with natural convection. In this problem, the computational grid (128 resolution) consists of a chip that is subjected to the powermap specified by Eq. The heating the chip results generation of velocity in the fluid and at steady- state, balance is achieved. We use Ansys Fluent to generate 300 training and 100 testing solutions for velocity, pressure, temperature for arbitrary sources.
4 Results and Ablation studies
In this section, we provide variety of results for the experiments outlined in Section Additionally, we conduct thorough ablation studies to understand the different components of the CoML.Sim approach in more detail. We use the 2-D Linear Poisson's 's equation experiment for these studies unless otherwise specified. Details regarding the CoMLSim set up for each experiment, baseline network architectures well additional results corresponding contour and line plots and comparisons across other metrics are provided in supplementary materials section C.
4.1 Comparison with Ansys Fluent and other ML baselines
The CoMLSim is compared with other ML baselines namely, UNet (71), FNO S and DeepONet for the experiments outlined in Section We compare the mean absolute error with respect to Ansys Fluent and averaged over the unseen testing cases and all solution variables. In the case 3-D suited this industrial application. It may be observed from Table that the CoMLLSim approach performs better than other ML basclines. All the ML methods perform better for the non-linear Poisson's 's case as compared to the linear Poisson's 's because the source term is less stiff but the CoMLSim approach does a better job in modeling the non-linearity arising due solution coupling. It must be noted that CoMLSim well as the baselines are trained with reasonably training samples. The baseline performance can have different outcomes with increasing the size of the training data.
chip cooling report the Too norm of temperature this metric is
Table Comparison with baselines Metric
Experiment
CoMLSim
UNet
FNO
DeepONet
FCNN
2-D Linear Poisson's
L
0.011
0.132
0.031
0.061
0.267
2-D non linear Poisson's
L1
0,0053
0.0877
0.0278
0.527
0.172
3-D NS external flow
L1
0.012
0.0625
0.038
0.81
0.125
3-D chip cooling
Loo
15.2
95.21
60.836
45.27
192.7
4.2 Assessment of generalizability
In this section, we assess the generalizability of our method specifically with respect to out-of- distribution PDE conditions and scaling to higher resolution grids. The results are compared with Ansys Fluent as well as other ML baselines discussed section 4.1
Out-of-distribation souree term: The source term for linear and non-linear Poisson equa- tions are sampled from a Gaussian mixture model described in Eq. g where the num- ber of Gaussian is randomly chosen between 0 and 30. In this experiment, we evaluate

---

the generalizability of our approach for source terms with exactly 30, 40, 50 and 60 num- ber of Gaussians, corresponding to out-of- -iitti--iittiittiinn for the training data distributions. For each case, the results averaged over 10 testing samples are compared with Ansys Fluent. may be observed from AA and B, that the accuracy of all MI. approaches decreases as you move further away from the source term distribution. However, the CoMLSim approach performs signif- icantly better than other ML baselines and the accuracy is reasonable even for the case with 60 Gaussian mixture model, which is substan- tially different from the training distribution.
Ansys (u)
CoMLSim (w)
Ansys (V)
CoMLSim (V)
Increasing mesh size by increasing domain size: Next, we evaluate the performance of CoMLSim for bigger domains with larger grid size and compare it with other MI. baselines. In this experiment, we compute solutions for the 100 source term conditions in the test set but on different domains with mesh sizes, 1024², 20482, 40962, 81922, respectively. The mean and standard deviation of Gaussians in the testing set are proportionally increased with the mesh size to ensure that the problem definition does not change. It is important to note that none of the networks are retrained for larger sized grids. It maybe observed from table 3 that the CoMLSim continues to scale to larger mesh sizes with similar accuracy as the original mesh size. The slight drop in accuracy due to increase in mesh size is attributed to accumulation of error caused by the flux conservation networks resulting from the increase in number of subdomains. Other MI baselines such as UNet and FNO scale up to grid size of 2048² but cannot evaluate beyond that due to GPU memory constraints. DeepONet has point-wise inference and can also scale to larger-sized grids but its accuracy is lower than CoMLSim.
02
00
Figure 4: Example of out-of- distribution source (60 Gaus- sians) comparison for non- linear Poisson's
diitt--iiitiiittion geometries: Next, we evaluate the performance of CoMLSim on 2 unseen car models presented at different 3-D angles of rotation to the external flow. It should be noted that these geometries are more complicated than the primitive objects considered for training our approach 3] The mean absolute errors with respect to Ansys Fluent across all solution variables are 0.0145, 0.06924, 0.04175, 0.91 and 0.12525 for CoMLSim, UNet. FNO, DeepONet and FCNN, respectively, Additional results and analysis are provided in supplementary materials C.3.3 and C.3.4.
4.3 Analysis of Subdomain size
In this experiment, we analyze the effect of subdomain resolution on accuracy and computational cost of the CoML.Sim approach. We train instances at resolutions of 322. 642 and 128?. The compression ratio in autoencoders is kept the same for the different subdomain resolutions. The mean absolute errors on testing set are 0.015, 0.011 and 0.029, respectively. Additionally, the iterations required to convergence are 420, 162 and 89, respectively. The convergence history is shown in Fig. 5A. The 1282 subdomain resolution has a lower accuracy because it is challenging to train accurate autoencoders as bigger subdomains capture a large amount of information. On the other hand, the computational cost is the lowest for the 1282 resolution because the number of subdomains in the entire computational domain are significantly less and hence, the solution algorithm converges faster.
4.4 Flux conservation bottleneck layer size
The flux conservation autoencoder as it the primary workhorse of the CoMLSim solution algorithm. As result, we evaluate the effect of the bottleneck layer size on the accuracy and computational speed. Each instance of CoMLSim with different bottleneck size is tested on 100 unseen test cases. We verify that all models satisfy the stability criterion specified in Eq. 3] It may be observed from FigureB) that as the compression ratio of the flux conservation autoencoder decreases, it begins to overfit and the testing error as well as the number of convergence iterations and computational time significantly increase. On the other hand, if the compression ratio is too large the testing error increases because the autoencoders underfit. In alignment with the collective intuition about autoencoders, there exists an optimum bottleneck size compression ratio where the best testing error is obtained for small computational times.
8

---

4.5 Robustness and stability
A long standing challenge in the field of numerical simulation is to guarantee the sta- bility and convergence of non-linear PDE solvers. However, we believe that the de-
noising capability of autoencoders [72] [6] [73 74 [7] used in our iterative solu- tion algorithm presents unique benefit, irrespective of the choice of initial condi- tions. Here, we empirically demonstrate the robustness and stability of our apDroach. In scenario A. we randomly sample 25 initial solutions from uniform distribution and in scenario B we sample from 6 differ- ent distributions, namely Uniform, Gumbel, Laplace, Gamma, Normal, Logistic etc. The mean convergence trajectory is plotted in Fig. [C and 55 show that the log of L2 norm of the conver- gence error falls below an acceptable tolerance for all cases and demonstrates the stability of our approach.
Log
(A)
-2
-3
-5
-7
4.6 CoML.Sim solution methods: Point Jacobi vs Gauss Seidel
1.0
1.5
2.0
(B)
Next, we compare the performance of the Point Jacobi and Gauss Seidel implementations in the CoMLSim approach across 3 met- rics, mean absolute error, computational cost and iterations to converge. Both methods have a similar prediction accuracy with a mean absolute error of 0.013 and 0.017, respectively but the Point Jacobi method is computationally cheaper requiring 4.3 seconds to converge as compared to 72 seconds of Gauss Seidel. It may be observed from Fig. EE that the Gauss Seidel method takes significantly fewer convergence iterations, but the Point Jacobi method faster because it can be efficiently vectorized on a GPU.
-3
10%
44
-5
-6-
-8
0.5
1.5
(C)
-2
-3
-5
4.7 Coupling
with traditional PDE solvers and super-resolution
-8
0.5
1.0
1.5
2.0
In this experiment, analyze the potential of coupling the CoML.Sim approach with a traditional PDE solver such as Ansys Fluent and subsequently demonstrate how approach inher-
(D)
-1
-2
ently possesses the capability to perform super-resolution. We mainly carry out experiments, where we start from a coarse grid solution for the 100 test samples of linear Poisson's 's equation solved by Ansys Fluent on a 64x64 grid (256x smaller than the fine resolution). add random noise to the coarse initial solution with varying amplitudes (10%, 25% and 50%), and use that as an initial solution in the CoMLSim approach. The convergence trajectory in Fig. 5 shows that the case with coarse grid ini- tialization converges the fastest, followed by initialization with noise and zero initialization. All solutions converge to the same accuracy.
-8
0.5
1.0
2.0
(E)
Now jact
4.8 Computational speed
-8
We observe that the CoMLSim approach is about 40-50x faster as compared to commercial steady-state PDE solvers such as Ansys Fluent for the same mesh resolution and physical domain size in all the experiments presented in this work. In comparison to the ML baselines, our approach is expected to be slightly slower because it adopts an iterative inferencing approach. But, this is compensated by solution accuracy, generalizability and robust- ness on high-resolution grids. A detailed analysis is provided in supplementary materials section E.
0.0
0.5
1.0
1.5
2.0
(ogat(itrattion))
Figure 5: Convergence history
for (A): varying subdomain sizes; (B): coupling with traditional PDE solvers; (C, D): Robustness and stability; (E): Point Jacobi vs Gauss Seidel

---

4.9 Importance of local learning and latent-space representation
To understand the effect of local learning, we set up the CoMLSim approach for single subdomain (size equal to the computational domain). From this experiment, conclude that the solution emros with single subdomain is significantly higher than having multiple subdomains (See supplementary material section C.1.5.1). Additionally, study the impact of latent space representations experiment with the latent vector sizes as well as case where CoMLSSm solves the solution space. We conclude that latent space representations have larger impact the computational cost than accuracy (See supplementary material section C.1.5.2).
4.10 Solution evolution during inference
In Figure. 6 we present the evolution of the PDE solution for the Poisson's s equation experiment at different iterations of the iterative inferencing. The results are plotted for 3 unseen test cases with different source term distribution. In each case, the solution the domain is initialized with random noise sampled from a uniform distribution between -1 and The contour plot at iteration O shows an attenuated noisy solution because the initial solution is passed through the solution encoder and decoder. It may be observed that at iteration I, the PDE solution starts evolving from the regions of high source term. As iterations progress, the solution begins to diffuse through the solution domain due to the repeated operation of the flux conservation network, until converges. The diffusion process is dominant in the case of Poisson's equations and is effectively captured by the flux conservation autoencoder in CoMLSim. The error plot shows the solution accuracy at different iterations during the inference procedure. The log error is the largest initially and drops linearly as the iterations progress.
Souree tert
Amsys Floent
IeraaionnO
Iteration-1
Iteration-60
Itcration-120
Log(mae) vs iter
Source tenn
Ansys Floent
Iteration=0
Iteration=1
Iteration=90
Iteration=180 180
Log(mne) vs iter
Source term
Ansys Fluent
Itenation00
Isenation
Itoration 100
Iteration-200 2000
Log(mae) vs ieer
Figure 6: Evolution of PDE solution during inference
Conclusion
In this work, we introduce the CoMLSSm approach, which is self-supervised, low-dimensional and local machine learning approach for solving PDE on highly-resolved grids and generalizing across wide range PDE conditions. Our approach is inspired from strategies employed in traditional PDE solvers and adopts iterative inferencing. The proposed approach is demonstrated to predict accurate solutions for a range of PDEs, generalize reasonably across geometries, source terms and BCs. Moreover, it scales to bigger domains with larger mesh size.
Broader impact, future work & limitations Although the proposed ML-model can generalize to out-of- cittof-distribution geometries, source terms and BCs, but like other ML approaches, extrapolation to any and all PDE conditions that significantly different from the original distribution still remains challenge. However, this work takes big step towards laying down the framework on how truly generalizable ML-based solvers can be developed. these challenges of generalizability and scalability by training autoencoders on random, application agnostic PDE solutions and enforeing PDE-based constraints the iterative inferencing procedure. Future work will also investigate the potential for hybrid solvers and extensions to transient PDEs and inverse problems. Finally, we will also address extension of the current approach unstructured meshes, which is current limitation.
future, we would like address
10

---

References
[1] James P Crutchfield and BS McNamara. Equations of motion from a data series. Complex systems, 1(417-452):21, 1987.
[2] Ioannis Kevrekidis, William Gear, James M Hyman, Panagiotis G Kevrekidid, Olof Runborg, Constantinos Theodoropoulos, et al. Equation-free free. coarse-grained multiscale compu- tation: Enabling mocroscopic simulators to perform system-level analysis. Communications in Mathematical Sciences, 1(4):715-762, 2003.
[3] Kaushik Bhattacharya, Bamdad Hosseini, Nikola Kovachki, and Andrew M Stuart. Model reduction and neural networks for parametric pdes. arXiv preprint arXiv:2005.00 180, 2020.
[4] Anima Anandkumar, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Nikola Kovachki, Zongyi Li, Burigede Liu, and Andrew Stuart. Neural operator: Graph kemne network for partial differential equations. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations, 2020.
[5] Zongyi LI, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Multipole graph neural operator for parametric partial differential equations. arXiv preprint arXivv2006.0955: 2020.
[6] Ravi G Patel, Nathaniel Trask, Mitchell Wood, and Eric Cyr. physics-informed operator regression framework for extracting data-driven continuum models. Computer Methods in Applied Mechanics and Engineering, 13500, 2021.
[7] Lu Lu, Pengzhan Jin, Guofei Pang. Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3):218-22)) 2021.
[8] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differen- tial equations. arXiv preprint arXiv:2010.08895, 2020.
[9] Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzadenesheli, and Anima Anandkumar. Physics-informer neural operator for learning partial differential equations. arXiv preprint arXiv:2111.03700 2021.
[10] Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
[11] Maziar Raissi and George Em Karniadakis. Hidden physics models: Machine learning of nonlinear partial differential equations. Journal of Computational Physics, 357:125-141, 2018.
[12] Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation machine learning: a survey. Journal of machine learning research, 18, 2018
[13] Xiaowei Jin, Shengze Cai, Hui Li, and George Em Karniadakis. Nsfnets (navier-stokes flow nets): Physics-ifformed neural networks for the incompressible navier-stokes equations. Journal of Computational Physics, 426:10995
2021.
[14] Zhiping Mao, Ameya D Jagtap, and George Em Karniadakis. Physics-informen neural networks for high-speed flows. Computer Methods in Applied Mechanics and Engineering, 360: 12789, 2020.
[15] Chengping Rao, Hao Sun, and Yang Liu. Physics-informan deep learning for incompressible laminar flows. Theoretical and Applied Mechanics Letters, 10(3):207-212, 2020.
[16] Jin-Long Wu, Heng Xiao, and Eric Paterson. Physics-informed machine learning approach for augmenting turbulence models: A comprehensive framework, Physical Review Fluids, 3(7): 074602, 2018
11

---

[17] Elizabeth Qian, Boris Kramer, Benjamin Peherstorfer, and Karen Willcox. Lift & learn: Physics- informed machine learning for large-scale nonlinear dynamical systems. Physica D: Nonlinear Phenomena, 406:132400 132401, 2020.
[18] Vikas Dwivedi, Nishant Parashar, and Balaji Srinivasan. Distributed learning machines for solving forward and inverse problems in partial differential equations. Neurocomputing, 420: 299-316, 2021.
[19] Ehsan Haghighat, Maziar Raissi, Adrian Moure, Hector Gomez, and Ruben Juanes. physics- informed deep learning framework for inversion and surrogate modeling in solid mechanics. Computer Methods in Applied Mechanics and Engineering, 379: 113741, 2021.
[20] Ehsan Haghighat and Ruben Juanes. Sciann: A keras/tensorflow wrapper for scientific compu- tations and physics-informed deep learning using artificial neural networks. Computer Methods in Applied Mechanics and Engineering, 373:1 13552, 2021.
[21] Mohammad Amin Nabian, Rini Jasmine Gladstone, and Hadi Meidani. Efficient training of physies-informed neural networks via importance sampling. Computer-Aided Civil and Infrastructure Engineering, 2021.
[22] Ehsan Kharazmi, Zhongqiang Zhang, and George Em Karniadakis. hp-vpinns: Variational physics-informed neural networks with domain decomposition. Computer Methods in Applied Mechanics and Engineering, 13347, 2021.
[23] Shengze Cai, Zhicheng Wang, Frederik Fuest, Young Jin Jeon, Callum Gray, and George Em Karniadakis. Flow over an espresso cup: inferring 3-d velocity and pressure fields from tomographic background oriented schlieren via physics-informed neural networks. Journal of Fluid Mechanics, 915, 2021.
[24] Shengze Cai, Zhicheng Wang, Sifan Wang, Paris Perdikaris, and George Karniadakis. Physics- informed neural networks (pinns) for heat transfer problems. Journal of Heat Transfer, 2021.
[25] Mathis Bode, Michael Gauding, Zeyu Lian, Dominik Denker, Marco Davidovic, Konstantin Kleinheinz, Jenia Jitsev, and Heinz Pitsch. Using physics-informed enhanced super-resolution generative adversarial networks for subfilter modeling in turbulent reactive flows. Proceedings of the Combustion Institute, 38(2):2617-25 2625, 2021.
[26] Ehsan Taghizadeh, Helen M Byrne, and Brian D Wood. Explicit physics- inffore- neural networks for non-linear upscaling closure: the case of transport in tissues. arXiv preprint arXiv:2104.01476, 2021.
[27] Lu Lu, Xuhui Meng, Zhiping Mao, and George Em Karniadakis. Deepxde: A deep learning library for solving differential equations. SIAM Review, 63(1):208 228, 2021.
[28] Khemraj Shukla, Ameya D Jagtap, and George Em Karniadakis. Parallel physics-informed neural networks via domain decomposition. arXiv preprint arXiv:2104. 10013, 2021.
[29] Oliver Hennigh, Susheela Narasimhan, Mohammad Amin Nabian, Akshay Subramaniam, Kaustubh Tangsali, Max Rietmann, Jose del Aguila Ferrandis, Wonmin Byeon, Zhiwei Fang. and Sanjay Choudhry. Nvidia simnet" (TM): an ai-accelerated multi-physics simulation framework. arXiv preprint arXiv:2012.079.88 2020.
[30] Li Li, Stephan Hoyer, Ryan Pederson, Ruoxi Sun, Ekin Cubuk, Patrick Riley, Kieron Burke, et al. Kohn-sham equations as regularizer: Building prior knowledge into machine-learned physics. Physical review letters, 126(3):036401, 2021.
[31] Rishikesh Ranade, Chris Hill, and Jay Pathak. Discretizationnet: A machine-learning based solver for navier-stokes equations using finite volume discretization. Computer Methods in Applied Mechanics and Engineering, 378:1 13722, 2021.
[32] Han Gao, Luning Sun, and Xun Wang. Phygeonet: physics-ifformed geometry-adaptiv convolutional neural networks for solving parameterized steady-state pdes on irregular domain. Journal of Computational Physics, 428:110079, 2021.
12

---

[33] Nils Wandel, Michael Weinmann, and Reinhard Klein. Learning incompressible fluid dy- namics from scratch towards fast, differentiable fluid models that generalize. arXiv preprint arXiv:2006.08762, 2020.
[34] Haiyang He and Jay Pathak. An unsupervised learning approach to solving heat equations on chip based on auto encoder and image gradient. arXiv preprint arXiv:2007 09684, 2020.
[35] Alexander Heinlein, Axel Klawonn, Martin Lanser, and Janine Weber. Combining machine learning and domain decomposition methods for the solution of partial differential equations- review. MaMMi-nnteillungen, 44(1) 44(1):=202 100001, 2021.
[36] Alexander Heinlein, Axel Klawonn, Martin Lanser, and Janine Weber. Machine learning in adaptive domain decomposition methods praddccis-eeiiiiiing the geometric location of constraints. SIAM Journal on Scientific Computing, 41(6):A3887-43391 2019.
[37] Ke Li, Kejun Tang, Tianfan Wu, and Qifeng Liao. D3m: A deep domain decomposition method for partial differential equations. IEEE Access, 8:5283-5294, 2019.
[38] Ameya D Jagtap, Ehsan Kharazmi, and George Em Karniadakis. Conservative physics-informed neural networks on discrete domains for conservation laws: Applications forward and inverse problems. Computer Methods in Applied Mechanics and Engineering, 365:11 13028, 2020.
[39] Lu Lu, Haiyang He, Priya Kasimbeg, Rishikesh Ranade, and Jay Pathak. One-shot learning for solution operators of partial differential equations. arXiv preprint arXiv:2104.05512, 2021.
[40] Hengjie Wang, Robert Planas. Aparna Chandramowlishwwran, and Ramin Bostanabad. Train once and use forever: Solving boundary value problems in unseen domains with pre-trained deep learning models. arXiv preprint S00Iv::20041087... 2021.
[41] Yohai Bar-Sinai, Stephan Hoyer, Jason Hickey, and Michael P Brenner. Learning data-driven discretizations for partial differential equations. Proceedings of the National Academy of 2019.
Sciences,
[42] Dmitrii Kochkov Jamie A Smith, Ayya Alieva, Qing Wang, Michael P Brenner, and Stephan Hoyer. Machine learning accelerated computational fluid dynamics. arXiv preprint arXiv:2102.01010, 2021.
[43] Alex Beatson, Jordan Ash, Geoffrey Roeder, Tianju Xue, and Ryan P Adams. Learning com- posable energy surrogates for pde order reduction. Advances in Neural Information Processing Systems, 33, 2020.
[44] Daniel Greenfeld, Meirav Galun, Ronen Basri, Irad Yavneh, and Ron Kimmel. Learning to optimize multigrid pde solvers. In International Conference on Machine Learning, pages 2415-2423. PMLR, 2019.
[45] Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter W Battaglia. Learning mesh-based simulation with graph networks. arXiv preprint arXiv:2010.03409, 2020.
[46] Jaayang Xu, Aniruddhe Pradhan, and Karthikeyan Duraisamy. Conditionally parameterized, discretization-awaal neural networks for nesh- modeling of physical systems. Advances in Neural Information Processing Systems, 34:1634-1645, 2021.
[47] Lukas Harsch and Stefan Riedelbauch. Direct prediction of steady-state flow fields in meshed domain with graph networks. arXiv preprint arXiv:2105.0255: 2021.
[48] Ferran Alet, Adarsh Keshav Jeewajee, Maria Bauza Villalonga, Alberto Rodriguez, Tomas Pozano-Perre. and Leslie Kaelbling. Graph element networks: adaptive, structured computation and memory. In International Conference on Machine Learning, pages 212-222. PMLR, 2019.
[49] Valerii Iakovlev, Markus Heinonen, and Harri Lähdesmäki. Learning continuous-time pdes from sparse data with graph neural networks. arXiv preprint arXiv:2006.08955 2020.
[50] Wenzhuo Liu, Mouadh Yagoubi, and Mare Schoenauer. Multi-resolution graph neural networks for pde approximation. In International Conference on Artificial Neural Networks, pages 151-163. Springer, 2021.
13

---

[51] Zijie Li and Amir Barati Farimani. Graph neural Computers Graphics, 103:201-211, 2022.
lagrangian fluid simulation.
[52] J Chen, E Hachem, and J Viquerat. Graph neural networks for laminar flow prediction around random two-dimensional shapes. Physics of Fluids, 33(12):123607, 2021.
[53] Steffen Wiewel, Byungsoo Kim, Vinicius C Azevedo, Barbara Solenthaler, and Nils Thuerey. Latent space subdivision: stable and controllable time predictions for fluid flow. In Computer Graphics Forum, volume 39, pages 15-25, Wiley Online Library, 2020.
[54] Romit Maulik, Bethany Lusch, and Prasanna Balaprakash. Reduced-order modeling of advection-dominated systems with recurrent neural networks and convolutional autoencoders. arXiv preprint arXiv:2002.004400 2020.
[55] Byungsoo Kim, Vinicius C Azevedo, Nils Thuerey, Theodore Kim, Markus Gross, and Barbara Solenthaler. Deep fluids: generative network for parameterized fluid simulations. In Computer Graphics Forum, volume 38, pages 59-70. Wiley Online Library, 2019.
[56] Takaaki Murata, Kai Fukami, and Koji Fukagata. Nonlinear mode decomposition with convolu- tional neural networks for fluid dynamics. Journal of Fluid Mechanics, 882, 2020.
[57] Kai Fukami, Taichi Nakamura, and Koji Fukagata. Convolutional neural network based hierarchical autoencoder for nonlinear mode decomposition of fluid field data. Physics of Fluids, 3 32(9):095110, 10, 2020.
[58] Rishikesh Ranade, Chris Hill, Haiyang He, Amir Maleki, and Jay Pathak. A latent space solver for pde generalization. arXiv preprint arXiv:2104. 020402. 2021.
[59] Shaojie Bai, Zico Kolter, and Vladlen Koltun. Deep equilibrium models. arXiv preprint arXiv: 1909.09997777 2019.
[60] Yulia Rubanova, Alvaro Sanchez-Gonzalez, Tobias Pfaff, and Peter Battaglia. Constraint-based graph network simulator. arXiv preprint arXiv:2112.091611 2021.
[61] Edoardo Remelli, Artem Lukoianov, Stephan Richter, Benoît Guillard, Timur Bagautdinov, Pierre Baque, and Pascal Fua. Meshsdf: Differentiable iso-surface extraction. Advances in Neural Information Processing Systems, 33:22468 -24778, 2020.
[62] Yilun Du, Shuang Li, Joshua Tenenbaum, and Igor Mordatch. Learning iterative reasoning through energy minimization. In International Conference on Machine Learning, pages 5570- 5582. PMLR, 2022.
[63] Lloyd N Trefethen and David Bau III. Numerical linear algebra, volume 50, Siam, 1997.
[64] Yousef Saad. Iterative methods for sparse linear systems. SIAM, 2003.
[65] Amir Maleki, Jan Heyse, Rishikesh Ranade, Haiyang He, Priya Kasimbeg, and Jay Pathak. Geometry encoding for numerical simulations. arXiv preprint wxXiv:2104.0777 2021.
[66] Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT press Cambridge, 2016.
[67] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning internal represen- tations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
[68] Dor Bank, Noam Koenigstein, and Raja Giryes. Autoencoders. arXiv preprint arXiv:2003.05991, 2020.
[69] Saerom Park and Jaewook Lee. Stability analysis of denoising autoencoders based on dynamical projection system. IEEE Transactions on Knowledge and Data Engineering, 33(8):3155-3155 2020.
[70] Alexandre Joel Chorin. Numerical solution of the navier-stokes equations. Mathematics of computation, 22(104):745-766 1968.
14

---

[71] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: biomedical image segmentation. In International Conference on Medical image computing and computer-ussisted intervention, pages 234-241. 234iinr, 2015.
Convolutional networks for
[72] Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of machine learning research, 11(12), 2010.
[73] Bo Du, Wei Xiong, Jia Wu, Lefei Zhang, Liangpei Zhang, and Dacheng Tao. Stacked convolu- tional denoising auto-encoders for feature representation. IEEE transactions on cybernetics, 47 (4):1017-1027, 2016.
[74] Yoshua Bengio, Li Yao, Guillaume Alain, and Pascal Vincent. Generalized denoising auto- encoders as generative models. arXiv preprint (rXivv1305.66666 2013.
[75] Mare Ranzato, Christopher Poultney, Sumit Chopra, Yann LeCun, et al. Efficient learning of sparse representations with an energy-based model. Advances in neural information processing systems, 19:1137, 2007.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] You are strongly encouraged to include a justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license to the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/]]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] Yes
(b) Did you describe the limitations of your work? [Yes] Yes. See Section
(c) Did you discuss any potential negative societal impacts of your work? [N/A] N/A. This not relevant to our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] Yes, our paper conforms the ethical guidelines.
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A] N/A
(b) Did you include complete proofs of all theoretical results? [N/A] N/A
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL)? [No] No. The source code is proprietary. But, we will provide the data and code for the main experiment once the paper is accepted. Instructions for generalization provided in the supplementary materials section F.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] Yes, the details are provided in the main paper as well as supplementary materials.
15

---

(c) Did you report enror bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] We test the robustness and stability of our solver for different initial conditions and provide the mean and standard deviation of convergence error for these experiments in Section 4.5
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] Yes, all the details are provided in the main paper as well as supplementary materials.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
16


Reviews:
[{"review_id": "njWSQRm88oI", "reviewer": "Reviewer_3nTn", "summary": "The paper presents a method to efficiently perform neural simulation of a high-resolution PDE, by dividing the simulation domain into patches, encoding per-patch latents, and learning the simulation on a stencil of neighboring latents.", "questions": "I think with a bit of work on clarifying method and results this could be a much stronger paper. If those are addressed sufficiently I will raise my score.\n\nMethod:\n- The paper talks a lot about using insights from classical PDE solvers, but I couldn't find how those factor into the method. Can you make this clearer? E.g. is there any reason to believe the flux-conservation autoencoder actually preserves flux better than a standard method trained on overall MSE? \n- How exactly is the flux-conservation module trained? The simulation output is already at the fix point, so if trained on this data wouldn't it just learn to pass through eta_c and ignore the neighbors?\n- Similarly, isn't the distribution shift between train (consistent patches) vs. the first iteration on test (random values with possible discontinuities) a problem?\n\nExperiments/Results:\n- How many iterations are needed? How accurate is the model when just run for 1 step?\n- Are the baselines also iterated or run one-step?\n- How is resolution scaled? CoMLSim could use more patches at test time, but e.g. UNet resolution can't easily be changed without re-training\n- Why are the baselines so bad? I certainly have seen much better prediction with UNet/ResNet-like architectures on very similar datasets.\n- In general, I can see how this method is more memory efficient and can tackle bigger domains compared to basic UNet, and perhaps better at generalization. But it's very surprising to me why those would have worse accuracy in the in-domain setting; if anything, I would have suspected the reverse, due to UNet's scale hierarchy which is generally very helpful for steady-state PDEs. If it isn't just an undertuned baseline, the reason for this should be properly investigated.\n", "limitations": "There isn't really any discussion on limitations beyond that OOD generalization is hard.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper differentiates itself from the common way of learning PDEs by (1) patch-based encoding and (2) iterative application of a local kernel to solve a steady-state problem.\nI think the paper's biggest strength is using (1) to solve systems which are otherwise too big to ingest into a NN model; while there are lots of related methods (e.g. GNN-based physics models like MeshGraphNets also operate on local elements to improve generalization), I haven't really seen demonstrations on really big systems like 256^3.\nThere's quite a few methods which use iterative updates to solve PDEs (e.g. the class of methods surrounding Bai et al.'s Deep Equilibrium Models, or Rubanova et al. Constraint-based graph network simulator), although in a slight different setting. This aspect of the paper however feels very understudied to me (see questions below). \nMore broadly, while the paper shows plenty of experiments and ablations, I found it very hard to distill concrete findings out of it. It's e.g. unclear what part of the model's performance are due to (1) or (2), and due to vague writing it's unclear how exactly experiments and comparisons were performed. The same is true for the method; it introduces lots of similar-looking symbols which are not properly defined (e.g. how are boundaries and source terms encoded and processed), and the training procedure was very unclear (see questions).\nFinally, I'm a bit suspicious of the comparison results. UNets are generally a solid baseline for this kind of problem, but UNet results looked unreasonably bad; this may be a case of a misimplemented or undertuned baseline.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "lVeiV_hUK9S", "reviewer": "Reviewer_1Y4R", "summary": "This paper proposes an algorithm to solve PDEs via an iterative procedure that uses a learned solver for a subdomain. Iterations on multiple of these inferred solutions are performed to obtain a global solution. As such the method represents a hybrid approach between more traditional iterations and learned solvers, with the main goal of achieving solvers for high resolutions without having to re-train models each time. The paper focuses on steady state problems, which limits the scope but at the same time represents a meaningful starting point.\n", "questions": "One aspect that I didnt understand was highlighting that the training is \"unsupervised\". As explained in the appendix, the training inherently needs many precomputed solutions. This seems misleading to me, unsupervised implies that solutions are not precomputed, or did I overlook something here?\n", "limitations": "I think these are covered quite well via the ablations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The method is presented in terms of domain subdivisions, with a flux-based coupling for the interfaces between the rectangular subdomains. The iterative solve is performed with an encoded latent space representation for the local solutions. The resulting algorithm is quite straight forward, but the \"convergence\" condition could be explained a bit more, I believe. This simply seems to be a delta of the latent space vectors, i.e., not a convergence to a true or reference solution, but only one in terms of satisfying the states between neighbors (there's no \"residual\" or other metric for the overall quality of the solution here).\n\nThe first example shown is, unfortunately, directly one of the less convincing ones. The wake behind the \"car\" and the boundary layer show some clear differences, and the learned solutions seems to have stronger axis aligned artifacts which I guess stem from the piece-wise solutions of the NN. It's good to see a 3D case, though.\n\nThe other ablations are interesting, and nicely illustrate the properties of the algorithm. I didn't find the de-noising too convincing, though - this is a classic task for autoencoders, but not a typical issue for numerical solvers.\n\nOne aspect that I didnt understand was highlighting that the training is \"unsupervised\". As explained in the appendix, the training inherently needs many precomputed solutions. This is misleading: unsupervised implies that solutions are not precomputed. Please clarify in the main paper, that the autoencoders are trained with collections of pre-computed sample solutions.\n\nOverall, I think the paper presents an interesting combination of learned and traditional solvers, and combinations of the form proposed here have not been studied a lot. It's good to see how well these piece-wise autoencoders work, and I'm sure the authors can iron out the smaller issues above in a final revision.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_iNwb94YUu7", "reviewer": "Reviewer_hAL3", "summary": "This paper presents a neural network model for solving steady-state PDEs on regular grids. The proposed approach is rooted in the principles of Domain Decomposition, which consists of two major steps: solving PDEs on subdomains in parallel and synchronizing the solutions across the subdomain boundaries, i.e., geometric interfaces partitioning the whole domain. The proposed method makes heavy use of encoders and decoders to solve PDEs in each subdomain. On top of that, the method uses fixed-point iterations to stitch these solutions together.\n\nRoughly speaking, the proposed method can be seen as a deep-learning version of Domain Decomposition solved in a learned latent space. \n", "questions": "I actually like the general idea in this paper, but I think many critical technical contents are described quite vaguely and are difficult to evaluate.\n\n**Problem definition**\n1. The proposed approach is tied to a specific spatial discretization scheme (Cartesian grids). While using grids is definitely OK, it should be clearly stated earlier in the paper, e.g., acknowledging it as an assumption in your problem definition.\n2. The related work section should also briefly mention papers like GNNs or finite-element methods using triangle or hex meshes so that readers are aware of other spatial discretization schemes.\n3. Eqn. (1) is an incomplete definition of PDEs without stating their boundary conditions.\n4. The discretization scheme lacks details and misses some subtle points, e.g., how is the smooth geometric surface in the car example handled using your regular grid?\n5. Similarly, the concept of “PDE conditions” is also vague and refers to various things in different problems (Sec. A.1.1: “source” in Poisson equation, “geometry” in RANS, physical parameters in other problems, etc.). It is probably better to rename “PDE conditions” so that readers like me don’t mistakenly think it refers to boundary conditions.\n\n**Technical method**\n1. Lines 112-114: “Iterative solution algorithms…at convergence.” This statement is specifically for Domain Decomposition, but the context seems to imply all PDE solvers fall into this pattern. Perhaps better to clarify it?\n2. Some background knowledge about flux conservation is probably necessary.\n3. Overall, I feel Sec. 2.2 and 2.3 lack enough details for people to reproduce the method. The Laplacian example (Eqn. 5) does not help to clarify the method (Eqn. 5 isn’t more informative than Fig 2 or Alg 1). A concrete, step-by-step description of running your approach on a sample problem (Poisson or Laplacian) will be much appreciated.\n\n**Experiments**\nWhile I appreciate the tremendous efforts this paper puts into experiments, I feel many of them are repetitive. Experiments that can tell us what makes combining Domain Decomposition and neural networks a good idea are rare in the paper. One experiment that I really like is Fig. 3C because\n- Domain Decomposition is designed for solving large PDE problems due to its superiority in time/spatial complexity;\n- Therefore, we have a good reason to hypothesize the proposed method should scale up to resolutions unseen by previous neural network methods;\n- Fig. 3C confirms this point with convincing results.\n\nHowever, other experiments are not very informative to me, although it definitely does not hurt to have them.\n- Just like I don’t expect Domain Decomposition to solve PDEs more *accurately* than other traditional methods (CG/multigrid without splitting the domain), I don’t see a strong reason to believe the proposed approach could solve PDEs more accurately than other network models, yet Fig. 3A-3C shows your method achieves smaller MAE than all other baselines across the board. What component in your method makes this happen? Are baselines carefully tuned? I would prefer to see an experiment analyzing it more deeply, e.g., by looking into the differences between your network architecture and that in the baselines.\n- Another experiment that I would like to see is to study the influence of latent vector dimensions. As a thought experiment, if we set the latent vector dim = input dim and enforce encoder = identity, does it reduce your method to the standard Domain Decomposition method? By varying the latent vector dim, I would like to understand how it pushes your approach away from the standard Domain Decomposition approach. \n", "limitations": "**Limitations**: Sort of. I think the paper should elaborate more on its limitation (See my comments above), but I can understand it may be difficult due to the page limits and won’t hold it against the paper.\n\nI really appreciate that the authors put “steady-state” in the title to reveal the scope of the PDEs they consider in the first place.\n\n**Potential negative societal impact**: N/A.\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper shows a respectful attempt to combine a classic numerical method with modern deep-learning techniques.\n2. The paper presents lots of experiments to evaluate the proposed approach.\n\nWeaknesses:\n1. The description of the technical method is vague without rigorous mathematical details. Part of the approach is still mysterious to me.\n2. The method and experiments did not reveal much insight from drawing the analogy between Domain Decomposition and neural networks.\nI will elaborate on them in the next section.\n\nSome small comments about the figures:\n1. It is probably better to place Fig. 2 on page 4.\n2. I have found Fig. 19 in the supplemental material helpful. I suggest putting a simplified version of it in the main text if space permits.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept